* perf(router): add model_name index for O(1) deployment lookups
Add model_name_to_deployment_indices mapping to optimize _get_all_deployments()
from O(n) to O(1) + O(k) lookups.
- Add model_name_to_deployment_indices: Dict[str, List[int]]
- Add _build_model_name_index() to build/maintain the index
- Update _add_model_to_list_and_index_map() to maintain both indices
- Refactor to use idx = len(self.model_list) before append (cleaner)
- Optimize _get_all_deployments() to use index instead of linear scan
* test(router): add test coverage for _build_model_name_index
Add single comprehensive test for _build_model_name_index() function to fix
code coverage CI failure.
The test verifies:
- Index correctly maps model_name to deployment indices
- Handles multiple deployments per model_name
- Clears and rebuilds index correctly
Fixes: CI code coverage error for _build_model_name_index
* fix: remove redundant deep copy
set_model_list already does the deep copy at the beginning of the call.
* fix: remove unused model_list arguments
The `model_list` parameter was being passed to classes that did not use it.
* fix: reduce per-request memory and time from O(N×M) to O(N)
No need to create a whole array for a simple look up.
* add: missing test
* fix: remove unused parameter
* fix(access group): allow access group on mcp tool retrieval
* fix(test): fix broken tests and add test case for access group
* fix(mypy): fix typing issues
* fix(memory file): add content type to in memory file
* fix intent params
* Add responses
* fix unrelated test
* test fix - fireworks API endpoint is down
* test fix fireworks ai is having an active outage
* test_completion_cost_databricks
* dbrx fix test API currently not responding
* Update OpenAI Realtime handler to use the correct endpoint and include all query parameters. Adjusted error messages for missing API base and key. Updated health check URL construction to pass model as a query parameter.
* Enhance OpenAI Realtime handler tests to ensure model parameter inclusion in WebSocket URL. Added new tests to verify correct URL construction with model and additional parameters, preventing 'missing_model' errors. Updated existing tests for consistency.
* Remove debug print statements for API base and key in OpenAIRealtime handler to clean up the code.
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* fix unsupported operand type(s) for +=: 'NoneType' and 'str' on clientside auth creds for responses
* fix the client side auth to use correct metadata
* add more tests
* fix tests
* EditAutoRouterTabProps
* Revert "EditAutoRouterTabProps"
This reverts commit 2835d3a3743e6411b9914a0b01381050e2273ad7.
* add EditAutoRouterTab
* delete edit
* fixes for edit auto-router
* fix accessing model edit
* working edit auto router
* fix - edit remove custom model name
* fixes for edit auto router settings
* qa for adding a model router
* test fix
* feat(router.py): translate the model in jsonl for create file deployment to use the deployment model name
* test: add unit test for replace model in jsonl
* test(test_router.py): add unit tests
* test: add unit tests
* fix(router.py): write file to all deployments
allows unified file id to work across multiple deployments
* fix(view_logs/index.tsx): show call type in request logs
* fix(router.py): pass a deep copy of kwargs to avoid conflict across multiple runs
* fix(batch_utils.py): broaden check
* fix(router_utils.py): handle null type for function name
* fix(proxy_track_cost_callback.py): fix ruff check error
* fix(router.py): handle healthy_deployments as a dict
* feat(managed_files.py): support encoding / decoding unified batch id … (#10711)
* feat(managed_files.py): support encoding / decoding unified batch id when using managed files
allows routing retrieve batch to the right model id
* fix: fix linting error
* test: add unit tests
* fix: fix ruff check
* fix(user_api_key_auth.py): add 'headers' to constructed request for websocket
Fix issue on some datastructure versions which require a headers field in scope
* test(test_user_api_key_auth.py): add unit testing for headers in scope change
* fix(router.py): migrate `_arealtime` to generic router endpoint
Fix infinite loop on model name missing for realtime api calls
* test(test_router_helper_utils.py): cleanup test post refactor
* add GET responses endpoints on router
* add GET responses endpoints on router
* add GET responses endpoints on router
* add DELETE responses endpoints on proxy
* fixes for testing GET, DELETE endpoints
* test_basic_responses api e2e
* fix(router.py): support reusable credentials via passthrough router
enables reusable vertex credentials to be used in passthrough
* test: fix test
* test(test_router_adding_deployments.py): add unit testing
* Fixed issue #8246 (#8250)
* Fixed issue #8246
* Added unit tests for discard() and for remove_callback_from_list_by_object()
* fix(openai.py): support dynamic passing of organization param to openai
handles scenario where client-side org id is passed to openai
---------
Co-authored-by: Erez Hadad <erezh@il.ibm.com>
* add initial test for assembly ai
* start using PassthroughEndpointRouter
* migrate to lllm passthrough endpoints
* add assembly ai as a known provider
* fix PassthroughEndpointRouter
* fix set_pass_through_credentials
* working EU request to assembly ai pass through endpoint
* add e2e test assembly
* test_assemblyai_routes_with_bad_api_key
* clean up pass through endpoint router
* e2e testing for assembly ai pass through
* test assembly ai e2e testing
* delete assembly ai models
* fix code quality
* ui working assembly ai api base flow
* fix install assembly ai
* update model call details with kwargs for pass through logging
* fix tracking assembly ai model in response
* _handle_assemblyai_passthrough_logging
* fix test_initialize_deployment_for_pass_through_unsupported_provider
* TestPassthroughEndpointRouter
* _get_assembly_transcript
* fix assembly ai pt logging tests
* fix assemblyai_proxy_route
* fix _get_assembly_region_from_url
* feat(router.py): support request prioritization for text completion calls
* fix(internal_user_endpoints.py): fix sql query to return all keys, including null team id keys on `/user/info`
Fixes https://github.com/BerriAI/litellm/issues/7485
* fix: fix linting errors
* fix: fix linting error
* test(test_router_helper_utils.py): add direct test for '_schedule_factory'
Fixes code qa test
* fix use _update_kwargs_before_fallbacks
* test assert standard_logging_object includes model_group
* test_datadog_non_serializable_messages
* update test
* fix(router.py): fix reading + using deployment-specific num retries on router
Fixes https://github.com/BerriAI/litellm/issues/7001
* fix(router.py): ensure 'timeout' in litellm_params overrides any value in router settings
Refactors all routes to use common '_update_kwargs_with_deployment' which has the timeout handling
* fix(router.py): fix timeout check
* fix(main.py): support passing max retries to azure/openai embedding integrations
Fixes https://github.com/BerriAI/litellm/issues/7003
* feat(team_endpoints.py): allow updating team model aliases
Closes https://github.com/BerriAI/litellm/issues/6956
* feat(router.py): allow specifying model id as fallback - skips any cooldown check
Allows a default model to be checked if all models in cooldown
s/o @micahjsmith
* docs(reliability.md): add fallback to specific model to docs
* fix(utils.py): new 'is_prompt_caching_valid_prompt' helper util
Allows user to identify if messages/tools have prompt caching
Related issue: https://github.com/BerriAI/litellm/issues/6784
* feat(router.py): store model id for prompt caching valid prompt
Allows routing to that model id on subsequent requests
* fix(router.py): only cache if prompt is valid prompt caching prompt
prevents storing unnecessary items in cache
* feat(router.py): support routing prompt caching enabled models to previous deployments
Closes https://github.com/BerriAI/litellm/issues/6784
* test: fix linting errors
* feat(databricks/): convert basemodel to dict and exclude none values
allow passing pydantic message to databricks
* fix(utils.py): ensure all chat completion messages are dict
* (feat) Track `custom_llm_provider` in LiteLLMSpendLogs (#7081)
* add custom_llm_provider to SpendLogsPayload
* add custom_llm_provider to SpendLogs
* add custom llm provider to SpendLogs payload
* test_spend_logs_payload
* Add MLflow to the side bar (#7031)
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com>
* (bug fix) SpendLogs update DB catch all possible DB errors for retrying (#7082)
* catch DB_CONNECTION_ERROR_TYPES
* fix DB retry mechanism for SpendLog updates
* use DB_CONNECTION_ERROR_TYPES in auth checks
* fix exp back off for writing SpendLogs
* use _raise_failed_update_spend_exception to ensure errors print as NON blocking
* test_update_spend_logs_multiple_batches_with_failure
* (Feat) Add StructuredOutputs support for Fireworks.AI (#7085)
* fix model cost map fireworks ai "supports_response_schema": true,
* fix supports_response_schema
* fix map openai params fireworks ai
* test_map_response_format
* test_map_response_format
* added deepinfra/Meta-Llama-3.1-405B-Instruct (#7084)
* bump: version 1.53.9 → 1.54.0
* fix deepinfra
* litellm db fixes LiteLLM_UserTable (#7089)
* ci/cd queue new release
* fix llama-3.3-70b-versatile
* refactor - use consistent file naming convention `AI21/` -> `ai21` (#7090)
* fix refactor - use consistent file naming convention
* ci/cd run again
* fix naming structure
* fix use consistent naming (#7092)
---------
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Yuki Watanabe <31463517+B-Step62@users.noreply.github.com>
Co-authored-by: ali sayyah <ali.sayyah2@gmail.com>
