* fix(router): use cacheable prefix for prompt caching cache keys
Fix issue where requests with same cacheable prefix but different user
messages were routing to different deployments, preventing cached token
reuse. The cache key now correctly includes only the cacheable prefix
(up to and including the last cache_control block) instead of the
entire messages array.
## New Functions
### extract_cacheable_prefix()
Static method that extracts the cacheable prefix from messages for
prompt caching. The cacheable prefix is defined as everything UP TO
AND INCLUDING the LAST content block (across all messages) that has
cache_control with type "ephemeral". This includes ALL blocks
before the last cacheable block (even if they don't have cache_control
themselves).
- Finds the last content block with cache_control across all messages
- Returns all messages and content blocks up to and including that
last cacheable block
- Excludes everything after the last cacheable block (including user
messages that come after)
- Returns empty list if no cacheable blocks are found
## Changed Functions
### get_prompt_caching_cache_key()
Modified to use the cacheable prefix instead of the full messages array
when generating cache keys. This ensures that requests with the same
cacheable prefix but different user messages generate the same cache
key, enabling proper routing to the same deployment.
- Now calls extract_cacheable_prefix() to get only cacheable content
- Returns None if no cacheable prefix is found (can't generate key)
- Cache key is now based on cacheable prefix only, not full messages
### async_get_model_id()
Completely refactored to use the cacheable prefix directly instead of
the previous workaround that checked progressively shorter message
slices. The previous implementation was inefficient and unreliable.
- Removed progressive message slicing logic (messages[:-1], messages[:-2], etc.)
- Now uses single direct cache lookup with cacheable prefix-based key
- More efficient (1 lookup instead of up to 4)
- More reliable (uses correct cache key based on cacheable prefix)
- Returns None if no cacheable prefix found
### add_model_id()
Added None check for cache_key to prevent caching when no cacheable
prefix is found. This ensures we don't attempt to cache when there's
no meaningful cache key to use.
- Added guard: returns early if cache_key is None
- Prevents attempting to cache when no cacheable prefix exists
### async_add_model_id()
Added None check for cache_key to prevent caching when no cacheable
prefix is found. Matches the behavior of add_model_id() for consistency.
- Added guard: returns early if cache_key is None
- Prevents attempting to cache when no cacheable prefix exists
### get_model_id()
Added None check for cache_key to handle cases where no cacheable
prefix is found. Ensures consistent behavior across all cache methods.
- Added guard: returns None if cache_key is None
- Prevents calling get_cache() with None key
## Test
### test_router_prompt_caching_same_cacheable_prefix_routes_to_same_deployment()
New end-to-end test that validates the fix. Tests that requests with
the same cacheable prefix (system blocks with cache_control) but
different user messages:
1. Generate the same cache key
2. Successfully perform cache lookup
3. Route to the same deployment
This test reproduces the exact scenario from the user's bug report
where three requests with different user messages should route to the
same deployment but were previously routing to different ones.
Fixes issue where cached tokens couldn't be reused because requests
were routed to different providers due to different cache keys.
* fix(router): use cast() for proper type handling in extract_cacheable_prefix
Replace type annotation with type: ignore comment with proper cast()
from typing module, matching the pattern used throughout the
codebase for creating modified AllMessageValues dictionaries.
* Attempt CI/CD Fix
* Adding test for coverage
* Adding max depth to copilot and vertex
* Fixing mypy lint and docker database
* Fixing UI build issues
* Update playwright test
* perf(router): Optimize prompt management model check with early exit
Add early return for models without '/' to avoid expensive get_model_list()
calls for 99% of standard model requests (gpt-4, claude-3, etc).
- Refactor _is_prompt_management_model() with "/" check before model lookup
- Add unit tests to verify optimization doesn't break detection
* perf(caching): optimize Redis batch cache operations and reduce unnecessary queries
This commit introduces several performance optimizations to the Redis caching layer:
**DualCache Improvements (dual_cache.py):**
1. Increase batch cache size limit from 100 to 1000
- Allows for larger batch operations, reducing Redis round-trips
2. Throttle repeated Redis queries for cache misses
- Update last_redis_batch_access_time for ALL queried keys, including those
with None values
- Prevents excessive Redis queries for frequently-accessed non-existent keys
3. Add early exit optimization
- Short-circuit when redis_result is None or contains only None values
- Avoids unnecessary processing when no cache hits are found
4. Optimize key lookup performance
- Replace O(n) keys.index() calls with O(1) dict lookup via key_to_index mapping
- Reduces algorithmic complexity in batch operations
5. Streamline cache updates
- Combine result updates and in-memory cache updates in single loop
- Only cache non-None values to avoid polluting in-memory cache
**CooldownCache Improvements (cooldown_cache.py):**
1. Enhanced early return logic
- Check if all values in results are None, not just if results is None
- Prevents unnecessary iteration when no valid cooldown data exists
These changes significantly improve Redis caching performance, especially for:
- High-throughput batch operations
- Scenarios with frequent cache misses
- Large-scale deployments with many concurrent requests
* fix: remove unnecessary test
* refactor: move default_max_redis_batch_cache_size to constants
- Add DEFAULT_MAX_REDIS_BATCH_CACHE_SIZE constant (default: 1000)
- Update DualCache to use constant from constants.py
- Document new environment variable in config_settings.md
* fix: only use in memory cache when set
* fix(router): improve prompt management model detection with smart early return
The previous early return optimization in _is_prompt_management_model() was
checking if the model name parameter contained '/' and returning False if it
didn't. This broke detection for model aliases (e.g., 'chatbot_actions') that
don't have '/' in their name but map to prompt management models
(e.g., 'langfuse/openai-gpt-3.5-turbo').
Changed the early return logic to only exit early when:
- Model name contains '/' AND
- The prefix is NOT a known prompt management provider
This maintains the performance optimization for 99% of direct model calls
(avoiding expensive get_model_list lookups) while correctly handling:
- Direct prompt management calls (e.g., 'langfuse/model')
- Model aliases without '/' (e.g., 'chatbot_actions')
- Regular models with/without '/' (e.g., 'gpt-3.5-turbo', 'openai/gpt-4')
Fixes test: test_router_prompt_management_factory
* perf(router): optimize _pre_call_checks with shallow copy (1400x faster)
Replace deepcopy with list() in _pre_call_checks - runs on every request.
Only pops from list, never modifies deployment dicts, so shallow copy is safe.
Performance: 1400x faster on hot path
Impact: 2-5x overall throughput improvement for routing workloads
Tests: Added regression test to ensure no mutation + filtering works
* perf(router): replace deepcopy with shallow copy for default deployment
Replace expensive copy.deepcopy() with shallow copy for default_deployment
in _common_checks_available_deployment() hot path.
Changes:
- Use dict.copy() for top-level deployment dict
- Use dict.copy() for nested litellm_params dict
- Only the 'model' field is modified, so deep recursion is unnecessary
Impact:
- 100x+ faster for default deployment path (every request when used)
- deepcopy recursively traverses entire object tree
- Shallow copy only copies two dict levels (exactly what's needed)
Test coverage:
- Added regression test to verify deployment isolation
- Ensures returned deployments don't mutate original default_deployment
- Validates multiple concurrent requests get independent copies
* perf(router): remove unnecessary dict copy in completion hot paths
Remove unnecessary deployment['litellm_params'].copy() in _completion
and _acompletion functions. The dict is only read and spread into a new
dict, never modified, making the defensive copy wasteful.
Changes:
- Remove .copy() in _completion (sync hot path)
- Remove .copy() in _acompletion (async hot path)
Impact:
- Every completion request (highest traffic endpoints)
- Eliminates unnecessary dict allocation and copy on every call
- Dict spreading already creates new dict, so no mutation possible
Test coverage:
- Added tests verifying deployment params unchanged after calls
- Tests both sync and async completion paths
- Validates optimization doesn't introduce mutations
* perf(router): optimize deployment filtering in pre-call checks
Replace O(n²) list pop pattern with O(n) set-based filtering in
_pre_call_checks() to improve routing performance under high load.
Changes:
- Use set() instead of list for invalid_model_indices tracking
- Replace reversed list.pop() loop with single-pass list comprehension
- Eliminate redundant list→set conversion overhead
Impact:
- Hot path optimization: runs on every request through the router
- ~2-5x faster filtering when many deployments fail validation
- Most beneficial with 50+ deployments per model group or high
invalidation rates (rate limits, context window exceeded)
Technical details:
Old: O(k²) where k = invalid deployments (pop shifts remaining elements)
New: O(n) single pass with O(1) set membership checks
* add: memory profiler
feat(proxy): Add configurable GC thresholds and enhance memory debugging endpoints
- Add PYTHON_GC_THRESHOLD env var to configure garbage collection thresholds
- Add POST /debug/memory/gc/configure endpoint for runtime GC tuning
- Enhance memory debugging endpoints with better structure and explanations
- Add comprehensive router and cache memory tracking
- Include worker PID in all debug responses for multi-worker debugging
* refactor: reduce complexity in get_memory_details endpoint
Extract 6 helper functions from get_memory_details to fix linter
error PLR0915 (too many statements). Improves maintainability
while preserving functionality.
* fix(router): remove incorrect early exit in _is_prompt_management_model
Removes early exit optimization that checked model_name prefix instead
of the actual litellm_params model. This incorrectly returned False for
custom model aliases that map to prompt management providers.
Example: "my-langfuse-prompt/test_id" -> "langfuse_prompt/actual_id"
The method now correctly checks the underlying model's prefix.
Fixes test_is_prompt_management_model_optimization
* fix(proxy): add explicit type annotations to debug_utils dictionaries
Resolved 6 mypy type errors in proxy/common_utils/debug_utils.py by adding
explicit Dict[str, Any] annotations to dictionary variables where mypy was
incorrectly inferring narrow types. This allows the dictionaries to accept
different value types (strings, nested dicts) for error handling and various
return structures.
Fixed:
- Line 246: caches dictionary in get_memory_summary()
- Line 371: cache_stats dictionary in _get_cache_memory_stats()
- Line 439: litellm_router_memory dictionary in _get_router_memory_stats()
* fix(proxy): fix Python 3.8 compatibility in debug_utils type annotations
- Replace tuple[...], list[...] with Tuple[...], List[...] from typing
- Replace Dict | None with Optional[Dict] for Python 3.8 compatibility
- Add missing imports: List, Optional, Tuple to typing imports
Fixes TypeError: 'type' object is not subscriptable in Python 3.8
---------
Co-authored-by: AlexsanderHamir <alexsanderhamirgomesbaptista@gmail.com>
* Implement fix for thinking_blocks and converse API calls
This fixes Claude's models via the Converse API, which should also fix
Claude Code.
* Add thinking literal
* Fix mypy issues
* Type fix for redacted thinking
* Add voyage model integration in sagemaker
* Add config file logic
* Use already exiting voyage transformation
* refactor code as per comments
* fix merge error
* refactor code as per comments
* refactor code as per comments
* UI new build
* [Fix] router - regression when adding/removing models (#15451)
* fix(router): update model_name_to_deployment_indices on deployment removal
When a deployment is deleted, the model_name_to_deployment_indices map
was not being updated, causing stale index references. This could lead
to incorrect routing behavior when deployments with the same model_name
were dynamically removed.
Changes:
- Update _update_deployment_indices_after_removal to maintain
model_name_to_deployment_indices mapping
- Remove deleted indices and decrement indices greater than removed index
- Clean up empty entries when no deployments remain for a model name
- Update test to verify proper index shifting and cleanup behavior
* fix(router): remove redundant index building during initialization
Remove duplicate index building operations that were causing unnecessary
work during router initialization:
1. Removed redundant `_build_model_id_to_deployment_index_map` call in
__init__ - `set_model_list` already builds all indices from scratch
2. Removed redundant `_build_model_name_index` call at end of
`set_model_list` - the index is already built incrementally via
`_create_deployment` -> `_add_model_to_list_and_index_map`
Both indices (model_id_to_deployment_index_map and
model_name_to_deployment_indices) are properly maintained as lookup
indexes through existing helper methods. This change eliminates O(N)
duplicate work during initialization without any behavioral changes.
The indices continue to be correctly synchronized with model_list on
all operations (add/remove/upsert).
* fix(prometheus): Fix Prometheus metric collection in a multi-workers environment (#14929)
Co-authored-by: sotazhang <sotazhang@tencent.com>
* Add tiered pricing and cost calculation for xai
* Use generic cost calculator
* Resolve conflicts in generated HTML files
* Remove penalty params as supported params for gemini preview model (#15503)
* fix conversion of thinking block
* add application level encryption in SQS (#15512)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* build: bump version
* bump: version 1.78.0 → 1.78.1
* add application level encryption in SQS
* add application level encryption in SQS
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: deepanshu <deepanshu.lulla@hq.bill.com>
* [Feat] Bedrock Knowledgebase - return search_response when using /chat/completions API with LiteLLM (#15509)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* add AnthropicCitation
* fix async_post_call_success_deployment_hook
* fix add vector_store_custom_logger to global callbacks
* test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call
* async_post_call_success_deployment_hook
* add async_post_call_streaming_deployment_hook
* async def test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call_streaming(setup_vector_store_registry):
* fix _call_post_streaming_deployment_hook
* fix async_post_call_streaming_deployment_hook
* test update
* docs: Accessing Search Results
* docs KB
* fix chatUI
* fix searchResults
* fix onSearchResults
* fix kb
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
* [Feat] Add dynamic rate limits on LiteLLM Gateway (#15518)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* build: bump version
* bump: version 1.78.0 → 1.78.1
* fix: KeyRequestBase
* fix rpm_limit_type
* fix dynamic rate limits
* fix use dynamic limits here
* fix _should_enforce_rate_limit
* fix _should_enforce_rate_limit
* fix counter
* test_dynamic_rate_limiting_v3
* use _create_rate_limit_descriptors
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
* Add google rerank endpoint
* Add docs
* fix mypy error
* fix mypy and lint errors
* Add haiku 4.5 integration
* Add haiku 4.5 integration for other regions as well
* Handle citation field correctly
* Fix filtering headers for signature calcs
* Add haiku 4.5 integration (#15650)
---------
Co-authored-by: Leslie Cheng <leslie.cheng5@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com>
Co-authored-by: Lucas <10226902+LoadingZhang@users.noreply.github.com>
Co-authored-by: sotazhang <sotazhang@tencent.com>
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: deepanshu <deepanshu.lulla@hq.bill.com>
Update test_generate_model_id_with_deployment_model_name to accept the new
error message format that results from the list+join optimization.
The function still correctly rejects None values with a TypeError, but the
error message changed from 'unsupported operand type(s) for +=' to
'expected str instance, NoneType found' due to the implementation change
from string concatenation to list joining.
Add AST-based test to detect 'for ... in self.model_list' anti-pattern.
Enforces use of index maps (model_id_to_deployment_index_map and
model_name_to_deployment_indices) for O(1) lookups instead of O(n) iteration.
Add AST-based test to detect 'for ... in self.model_list' anti-pattern.
Enforces use of index maps (model_id_to_deployment_index_map and
model_name_to_deployment_indices) for O(1) lookups instead of O(n) iteration.
Update test_generate_model_id_with_deployment_model_name to accept the new
error message format that results from the list+join optimization.
The function still correctly rejects None values with a TypeError, but the
error message changed from 'unsupported operand type(s) for +=' to
'expected str instance, NoneType found' due to the implementation change
from string concatenation to list joining.
* perf(router): add model_name index for O(1) deployment lookups
Add model_name_to_deployment_indices mapping to optimize _get_all_deployments()
from O(n) to O(1) + O(k) lookups.
- Add model_name_to_deployment_indices: Dict[str, List[int]]
- Add _build_model_name_index() to build/maintain the index
- Update _add_model_to_list_and_index_map() to maintain both indices
- Refactor to use idx = len(self.model_list) before append (cleaner)
- Optimize _get_all_deployments() to use index instead of linear scan
* test(router): add test coverage for _build_model_name_index
Add single comprehensive test for _build_model_name_index() function to fix
code coverage CI failure.
The test verifies:
- Index correctly maps model_name to deployment indices
- Handles multiple deployments per model_name
- Clears and rebuilds index correctly
Fixes: CI code coverage error for _build_model_name_index
* fix: remove redundant deep copy
set_model_list already does the deep copy at the beginning of the call.
* fix: remove unused model_list arguments
The `model_list` parameter was being passed to classes that did not use it.
* fix: reduce per-request memory and time from O(N×M) to O(N)
No need to create a whole array for a simple look up.
* add: missing test
* fix: remove unused parameter
* fix(access group): allow access group on mcp tool retrieval
* fix(test): fix broken tests and add test case for access group
* fix(mypy): fix typing issues
* fix(memory file): add content type to in memory file
* fix intent params
* Add responses
* fix unrelated test
* test fix - fireworks API endpoint is down
* test fix fireworks ai is having an active outage
* test_completion_cost_databricks
* dbrx fix test API currently not responding
* Update OpenAI Realtime handler to use the correct endpoint and include all query parameters. Adjusted error messages for missing API base and key. Updated health check URL construction to pass model as a query parameter.
* Enhance OpenAI Realtime handler tests to ensure model parameter inclusion in WebSocket URL. Added new tests to verify correct URL construction with model and additional parameters, preventing 'missing_model' errors. Updated existing tests for consistency.
* Remove debug print statements for API base and key in OpenAIRealtime handler to clean up the code.
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* fix unsupported operand type(s) for +=: 'NoneType' and 'str' on clientside auth creds for responses
* fix the client side auth to use correct metadata
* add more tests
* fix tests
* EditAutoRouterTabProps
* Revert "EditAutoRouterTabProps"
This reverts commit 2835d3a3743e6411b9914a0b01381050e2273ad7.
* add EditAutoRouterTab
* delete edit
* fixes for edit auto-router
* fix accessing model edit
* working edit auto router
* fix - edit remove custom model name
* fixes for edit auto router settings
* qa for adding a model router
* test fix
* feat(router.py): translate the model in jsonl for create file deployment to use the deployment model name
* test: add unit test for replace model in jsonl
* test(test_router.py): add unit tests
* test: add unit tests
* fix(router.py): write file to all deployments
allows unified file id to work across multiple deployments
* fix(view_logs/index.tsx): show call type in request logs
* fix(router.py): pass a deep copy of kwargs to avoid conflict across multiple runs
* fix(batch_utils.py): broaden check
* fix(router_utils.py): handle null type for function name
* fix(proxy_track_cost_callback.py): fix ruff check error
* fix(router.py): handle healthy_deployments as a dict
* feat(managed_files.py): support encoding / decoding unified batch id … (#10711)
* feat(managed_files.py): support encoding / decoding unified batch id when using managed files
allows routing retrieve batch to the right model id
* fix: fix linting error
* test: add unit tests
* fix: fix ruff check
* fix(user_api_key_auth.py): add 'headers' to constructed request for websocket
Fix issue on some datastructure versions which require a headers field in scope
* test(test_user_api_key_auth.py): add unit testing for headers in scope change
* fix(router.py): migrate `_arealtime` to generic router endpoint
Fix infinite loop on model name missing for realtime api calls
* test(test_router_helper_utils.py): cleanup test post refactor
Ishaan Jaff