- Add ttl parameter to cache_control for Gemini models
- Support Google's TTL format (e.g., '3600s', '7200s')
- Implement robust TTL extraction and validation
- Extract TTL before system message transformation to handle all cases
- Add comprehensive test suite with 17 test cases in tests/test_litellm/
- Update documentation with TTL usage examples
- Maintain backward compatibility with existing cache_control usage
Fixes#9810
* feat(route_checks.py): allow admin to disable proxy management endpoints on instance
useful for preventing multiple instances from doing admin actions
* docs(scaling_multiple_instances.md): add architecture doc on scaling multiple litellm instances
provide guidance on scaling proxy
* docs(scaling_multiple_instances.md): add doc on scaling across multiple regions for litellm
* fix(route_checks.py): allow disabling llm api endpoints on an instance
allows pure admin instance to exist
* refactor(enterprise/route_checks.py): refactor env var checks
* refactor: finish refactoring
* docs(control_plane_and_data_plane.md): refactor docs
* test: update tests
* [Feat] New LLM API Integration - Add Moonshot API (Kimi) (#12551)
* Add Moonshot AI provider support to LiteLLM
Co-authored-by: ishaan <ishaan@berri.ai>
* Refactor Moonshot provider params handling and transformation logic
Co-authored-by: ishaan <ishaan@berri.ai>
* fix constants
* add Moonshot AI
* fix get_supported_openai_params
* handle kimi temp
* add tool choice handling
* test moonshot unit tests
* fix kimi
* fix linting
* Add pricing information for Moonshot AI's kimi-k2 model (#12566)
* Add pricing information for Moonshot AI's kimi-k2 model
* Update model name to kimi-k2-0711-preview
- Changed model name from moonshot/kimi-k2 to moonshot/kimi-k2-0711-preview
- This reflects the specific model version as requested
* Update moonshot_models list to match model_context JSON
---------
Co-authored-by: openhands <openhands@all-hands.dev>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
* update docs
* docs moonshot
* fixes model cost map
* fix map_openai_params
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
* Get the basics of the integration working.
* Cleanup bytez integration.
* Update user agent for Bytez integration.
* Use the config class directly. Create the start of the docs.
* Finish up bytez documentation. Include a provider integration guide.
* Fix typing bug in custom_logger_utils. Add tests for bytez integration.
* Add token tracking for model usage for Bytez integration.
* Create a units test for the Bytez config.
* Make changes to Bytez transformation code per PR feedback.
* Cleanup coment in Bytez transformation test.
* Remove LRU usage for bytez integration.
* Consolidate Bytez tests into a single file. Conform to project structure for tests.
* Fix linting error with Bytez impl.
* Add Bytez to the list of providers in the docs.
* just use 1 param for mcp groups
* fix just use 1 param for access groups
* test_get_tools_from_mcp_servers
* docs access groups
* group MCPs
* test fix
* fix screenshots on docs
* TestMCPAccessGroupsE2E
* update img
* fix MCP connect
* Get the basics of the integration working.
* Cleanup bytez integration.
* Update user agent for Bytez integration.
* Use the config class directly. Create the start of the docs.
* Finish up bytez documentation. Include a provider integration guide.
* Fix typing bug in custom_logger_utils. Add tests for bytez integration.
* Add token tracking for model usage for Bytez integration.
* Create a units test for the Bytez config.
* Make changes to Bytez transformation code per PR feedback.
* Cleanup coment in Bytez transformation test.
* Remove LRU usage for bytez integration.
* Consolidate Bytez tests into a single file. Conform to project structure for tests.
* Fix linting error with Bytez impl.
* Update Vertex Model Garden documentation to deploy + use OpenAI Chat Completion tutorial.
* update anchor description
* update to LiteLLM specific tutorial
* update the right doc
* Reverted docs/my-website/docs/providers/vertex.md to match main
* revert to main vertex tutorial
* fix(internal_user_endpoints.py): initial commit removing logic to create new budget for new user if default max budget in team set
* feat(proxy_setting_endpoints.py): update team member budget when set via default internal user endpoint
removes need to create a unique budget per user
* feat(proxy_server.py): set team max member budget on startup, if set on config.yaml
* fix(prometheus.py): support custom tags for tracking on prometheus
Allows tracking user agent values on prometheus metrics
* test(test_internal_user_endpoints.py): fix test
* update types
* add STDIO to client
* allow using STDIO with MCP manager
* add Stdio-specific fields to schema.prisma
* fixes for MCP mgmt
* fix for adding stdio MCP to DB
* ui - allow adding stdio MCPs
* fix MCP server manager
* docs stdio MCP
* add_stdio_mcp.png
* new stdio tests
* allow adding MCPs through config.yaml
* fix tool test panel
* use TestMCPClient
* ui fixes for testing circle ci mcp
* feat(custom_guardrail.py): allow admin to run guardrails on specific modes based on request tags
Enables pre-call checks for openwebui but logging for claude code guardrails
* refactor(custom_guardrail.py): move tag based mode to enterprise
* test(test_custom_guardrail.py): update test
* feat(quick_start.md): add doc on tag-based guardrail mode
* docs(quick_start.md): cleanup doc
* docs: cleanup
* docs(openweb_ui.md): add tutorial to doc
* fix(custom_guardrail.py): ensure user is premium user
* test: update test
* Added dashscope as a provider
* Fix some leftover references on nebius
* Porting the dashscope api endpoit international version
* explicit tool_choice = True in config
* add support of bearer token for bedrock integration
* fix linting issue
* fix type checking issue
* reoder arguments to address type checking issue
* switch to use get_secret_str to fetch env variable
Co-authored-by: 0x-fang <fanggong@amazon.com>
* fix(auth_checks.py): resolve a model group alias when key has access to underlying model
Fixes LIT-293
* feat(anthropic/): add mock_response to anthropic /v1/messages
makes it easy to test fallback logic
* fix(router.py): support fallbacks on /v1/messages
adds working fallbacks on generic api route
* refactor(router.py): point _ageneric_api_call_with_fallbacks to updated function
* test: add unit test for new helper on router
* fix(router.py): use correct metadata variable name
* fix(router.py): use correct metadata field
* docs(config_settings.md): document new param
* fix(db_spend_update_writer.py): fix db query
* fix(litellm_pre_call_utils.py): support passing anthropic-beta headers when 'forward_client_headers_to_llm_api' is True
allows user to pass along extra headers to vertex ai anthropic models
* docs(config_settings.md): update docs
* add _get_mcp_auth_header_from_headers
* test_process_mcp_request_with_custom_auth_header
* Using a different Authentication Header
* fix customize MCP Auth header name
Fix the header to highlight the website guardrailsai.com which is the real name/website of this provider.
Guardrails.ai leads to some different website, which is confusing.
* fix(key_management_endpoints.py): add new `/key/service_account/generate` endpoint
enables controlling team member permissions for creating service account keys
* refactor(ui/): use new /key/service-account/generate endpoint when creating keys on UI
enables team member permissions for service accounts
* fix(permission_definitions.tsx): add permission for creating service accounts on UI
* fix(key_management_endpoints.py): ignore user id for service account keys
* fix(key_management_endpoints.py): improve docstring
* docs(service_accounts.md): update docs
* refactor(aim.py): refactor to support adding aim guardrails on UI
* fix(base.py): add ui_friendly_name to config model
* feat(ui/): support loading new guardrails from backend api call
removes need to onboard each guardrail to ui
* fix: don't show optional params if not set and don't show ui_friendly_name (internal param0
* fix(ui/add_guardrail_form.tsx): ensure dynamic provider value is used
* fix(ui/): just one-time update the provider map dictionary
* fix(ui/): show masked api base / api key on guardrail update
* refactor(aporia_ai/): refactor to show on UI
* feat(aporia_ai/): add aporia ai guardrail to UI
* refactor(guardrails_ai/): refactor to add via UI
* refactor(lasso.py): refactor to enable adding lasso guardrails via UI
* feat(pangea.py): add pangea guardrail on UI
* feat(panw): add panw prisma airs through UI
* test: update tests
* fix: fix ruff linting error
* test: update tests
* fix: add missing docs
* fix: fix guardrail init
* fix: suppress linting errors
* fix(proxy_server.py): fix linting error
Marcelo Díaz