mirror of
https://github.com/tiennm99/litellm.git
synced 2026-06-23 17:36:32 +00:00
Alexsander Hamir
6e70c279f8
* fix(router): use cacheable prefix for prompt caching cache keys Fix issue where requests with same cacheable prefix but different user messages were routing to different deployments, preventing cached token reuse. The cache key now correctly includes only the cacheable prefix (up to and including the last cache_control block) instead of the entire messages array. ## New Functions ### extract_cacheable_prefix() Static method that extracts the cacheable prefix from messages for prompt caching. The cacheable prefix is defined as everything UP TO AND INCLUDING the LAST content block (across all messages) that has cache_control with type "ephemeral". This includes ALL blocks before the last cacheable block (even if they don't have cache_control themselves). - Finds the last content block with cache_control across all messages - Returns all messages and content blocks up to and including that last cacheable block - Excludes everything after the last cacheable block (including user messages that come after) - Returns empty list if no cacheable blocks are found ## Changed Functions ### get_prompt_caching_cache_key() Modified to use the cacheable prefix instead of the full messages array when generating cache keys. This ensures that requests with the same cacheable prefix but different user messages generate the same cache key, enabling proper routing to the same deployment. - Now calls extract_cacheable_prefix() to get only cacheable content - Returns None if no cacheable prefix is found (can't generate key) - Cache key is now based on cacheable prefix only, not full messages ### async_get_model_id() Completely refactored to use the cacheable prefix directly instead of the previous workaround that checked progressively shorter message slices. The previous implementation was inefficient and unreliable. - Removed progressive message slicing logic (messages[:-1], messages[:-2], etc.) - Now uses single direct cache lookup with cacheable prefix-based key - More efficient (1 lookup instead of up to 4) - More reliable (uses correct cache key based on cacheable prefix) - Returns None if no cacheable prefix found ### add_model_id() Added None check for cache_key to prevent caching when no cacheable prefix is found. This ensures we don't attempt to cache when there's no meaningful cache key to use. - Added guard: returns early if cache_key is None - Prevents attempting to cache when no cacheable prefix exists ### async_add_model_id() Added None check for cache_key to prevent caching when no cacheable prefix is found. Matches the behavior of add_model_id() for consistency. - Added guard: returns early if cache_key is None - Prevents attempting to cache when no cacheable prefix exists ### get_model_id() Added None check for cache_key to handle cases where no cacheable prefix is found. Ensures consistent behavior across all cache methods. - Added guard: returns None if cache_key is None - Prevents calling get_cache() with None key ## Test ### test_router_prompt_caching_same_cacheable_prefix_routes_to_same_deployment() New end-to-end test that validates the fix. Tests that requests with the same cacheable prefix (system blocks with cache_control) but different user messages: 1. Generate the same cache key 2. Successfully perform cache lookup 3. Route to the same deployment This test reproduces the exact scenario from the user's bug report where three requests with different user messages should route to the same deployment but were previously routing to different ones. Fixes issue where cached tokens couldn't be reused because requests were routed to different providers due to different cache keys. * fix(router): use cast() for proper type handling in extract_cacheable_prefix Replace type annotation with type: ignore comment with proper cast() from typing module, matching the pattern used throughout the codebase for creating modified AllMessageValues dictionaries.
Router component unit tests.
Please name all files with the word 'router' in them.
This is used to ensure all functions in the router are tested.