- Add ttl parameter to cache_control for Gemini models
- Support Google's TTL format (e.g., '3600s', '7200s')
- Implement robust TTL extraction and validation
- Extract TTL before system message transformation to handle all cases
- Add comprehensive test suite with 17 test cases in tests/test_litellm/
- Update documentation with TTL usage examples
- Maintain backward compatibility with existing cache_control usage
Fixes#9810
- Add support for envVars (simple key-value pairs) in migrations job
- Add support for extraEnvVars (complex environment variable configurations)
- Include comprehensive test coverage for both envVars and extraEnvVars
- Ensure backward compatibility with existing configurations
- Tests verify proper rendering of environment variables in container spec
* refactor(prisma_migration.py): refactor to support use_prisma_migrate for helm hook
* fix(prisma_migration.py): don't use subprocess
* fix(prisma_migration.py): fix cli commands
* fix(prisma_migration.py): still run prisma generate
* feat(route_checks.py): allow admin to disable proxy management endpoints on instance
useful for preventing multiple instances from doing admin actions
* docs(scaling_multiple_instances.md): add architecture doc on scaling multiple litellm instances
provide guidance on scaling proxy
* docs(scaling_multiple_instances.md): add doc on scaling across multiple regions for litellm
* fix(route_checks.py): allow disabling llm api endpoints on an instance
allows pure admin instance to exist
* refactor(enterprise/route_checks.py): refactor env var checks
* refactor: finish refactoring
* docs(control_plane_and_data_plane.md): refactor docs
* test: update tests
Added cache_read_input_token_cost (25% of regular input cost) to all 39 Gemini 2.x models
to properly support implicit context caching cost calculations. Previously, cached tokens
were being charged at full price instead of the discounted rate.
Fixes#11156
* [Feat] New LLM API Integration - Add Moonshot API (Kimi) (#12551)
* Add Moonshot AI provider support to LiteLLM
Co-authored-by: ishaan <ishaan@berri.ai>
* Refactor Moonshot provider params handling and transformation logic
Co-authored-by: ishaan <ishaan@berri.ai>
* fix constants
* add Moonshot AI
* fix get_supported_openai_params
* handle kimi temp
* add tool choice handling
* test moonshot unit tests
* fix kimi
* fix linting
* Add pricing information for Moonshot AI's kimi-k2 model (#12566)
* Add pricing information for Moonshot AI's kimi-k2 model
* Update model name to kimi-k2-0711-preview
- Changed model name from moonshot/kimi-k2 to moonshot/kimi-k2-0711-preview
- This reflects the specific model version as requested
* Update moonshot_models list to match model_context JSON
---------
Co-authored-by: openhands <openhands@all-hands.dev>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
* update docs
* docs moonshot
* fixes model cost map
* fix map_openai_params
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
* Get the basics of the integration working.
* Cleanup bytez integration.
* Update user agent for Bytez integration.
* Use the config class directly. Create the start of the docs.
* Finish up bytez documentation. Include a provider integration guide.
* Fix typing bug in custom_logger_utils. Add tests for bytez integration.
* Add token tracking for model usage for Bytez integration.
* Create a units test for the Bytez config.
* Make changes to Bytez transformation code per PR feedback.
* Cleanup coment in Bytez transformation test.
* Remove LRU usage for bytez integration.
* Consolidate Bytez tests into a single file. Conform to project structure for tests.
* Fix linting error with Bytez impl.
* Add Bytez to the list of providers in the docs.
* just use 1 param for mcp groups
* fix just use 1 param for access groups
* test_get_tools_from_mcp_servers
* docs access groups
* group MCPs
* test fix
* fix screenshots on docs
* TestMCPAccessGroupsE2E
* update img
* fix MCP connect
* fix chat ui
* fixes for types
* ui - fix back button on MCP
* mcp - show existing groups when creating new mcps
* fix design
* fix color scheme
* fix linting
* fixes for mapped tests
- Use getattr() with default value when accessing mcp_server.env
- Prevents AttributeError when Prisma doesn't include the field
- Fixes startup error: 'LiteLLM_MCPServerTable' object has no attribute 'env'
The issue occurs when Prisma fetches records and doesn't include
null/undefined fields in the returned object.
* fix(model_hub_table.tsx): fix null link
* fix(model_hub_table.tsx): add tooltip on disabled make public
tell user how to enable button
* feat(model_hub_table.tsx): make Model Hub OSS
allow more teams to share available models via litellm
* fix(model_hub_table_columns.tsx): make table view only model hub view, rename 'model' column to 'public model name', make model hub url copyable
* fix(page.tsx): fix logo re-rendering
* fix(public_model_hub.tsx): fix theme
* style(public_model_hub.tsx): move filters into same card as model table
filters are for table
* fix(page.tsx): fix ui linting error
* refactor(model_hub_table.tsx): refactor public model selection to inside the 'make public' button - modal
similar flow to adding guardrails
consistent flow for doing stuff on the proxy
* style(ui/): make 'make public' modal wider, to have all filters on same horizontal axis
* style(model_hub_table.tsx): have already public models be pre-selected
* fix(model_hub_table.tsx): ensure only admin can make models public via UI
* fix: fix linting errors
* fix(model_hub_table.tsx): make model hub purpose clearer
* fix(model_hub_table_columns.tsx): fix linting error
* fix(model_hub_table.tsx): fix null link
* fix(model_hub_table.tsx): add tooltip on disabled make public
tell user how to enable button
* feat(model_hub_table.tsx): make Model Hub OSS
allow more teams to share available models via litellm
* fix(model_hub_table_columns.tsx): make table view only model hub view, rename 'model' column to 'public model name', make model hub url copyable
* fix(page.tsx): fix logo re-rendering
* fix(public_model_hub.tsx): fix theme
* style(public_model_hub.tsx): move filters into same card as model table
filters are for table
* fix(page.tsx): fix ui linting error
* Get the basics of the integration working.
* Cleanup bytez integration.
* Update user agent for Bytez integration.
* Use the config class directly. Create the start of the docs.
* Finish up bytez documentation. Include a provider integration guide.
* Fix typing bug in custom_logger_utils. Add tests for bytez integration.
* Add token tracking for model usage for Bytez integration.
* Create a units test for the Bytez config.
* Make changes to Bytez transformation code per PR feedback.
* Cleanup coment in Bytez transformation test.
* Remove LRU usage for bytez integration.
* Consolidate Bytez tests into a single file. Conform to project structure for tests.
* Fix linting error with Bytez impl.
* added mcp tools on internal user and divide it by teams
* add support for server api call
* Added frontend for test key
* added tools used output
* fix ui for servers
* All servers to personal
* change columns format
* revert ui logic
* Added vertical align
* fix mapped tests
* fix lint
* fix lint
* remove extra file
* fix ui test
* comments fixes
* change query type
* change query type
* mcp acces group init
* add ability to change server display on ui through access groups
* Mcp access group names UI (#12486)
* Added ui changes to reflect mcp_access_groups
* fix edit mcp page
* change to string array (#12491)
* change to string array
* Remove print
* add ability to change server display on ui through access groups
* Litellm mcp access groups accesses (#12498)
* added mcp access groups for keys and teams
* added access groups above servers
* fixed ruff
* fixed mypy
* revert couple changes
* fix test
* fixed double asterisks
* Litellm mcp groups UI (#12522)
* add ui for teams
* fix object permissions
* fix mcp servers test object permission
* remove print
* add helper method
* add tests + remove logs
* add mcp access group servers to test key
* add mcp access group support for headers
* lint fix
* add tests and helper function
* fixed test
* change list -> List
* tests
* added ui for selecting servers
* added mcp tools on internal user and divide it by teams
* add support for server api call
* Added frontend for test key
* added tools used output
* fix ui for servers
* All servers to personal
* change columns format
* revert ui logic
* Added vertical align
* fix mapped tests
* fix lint
* fix lint
* remove extra file
* fix ui test
* comments fixes
* change query type
* change query type
* mcp acces group init
* add ability to change server display on ui through access groups
* Mcp access group names UI (#12486)
* Added ui changes to reflect mcp_access_groups
* fix edit mcp page
* change to string array (#12491)
* change to string array
* Remove print
* add ability to change server display on ui through access groups
* Litellm mcp access groups accesses (#12498)
* added mcp access groups for keys and teams
* added access groups above servers
* fixed ruff
* fixed mypy
* revert couple changes
* fix test
* fixed double asterisks
* Litellm mcp groups UI (#12522)
* add ui for teams
* fix object permissions
* fix mcp servers test object permission
* remove print
* add helper method
* add tests + remove logs
* add mcp access group servers to test key
* add mcp access group support for headers
* lint fix
* add tests and helper function
* fixed test
* change list -> List
* tests
* fix(proxy_setting_endpoints.py): require store model in db is enabled for setting user default settings
* test(test_proxy_server.py): update test
* fix(reset_budget_job.py): initial commit adding reset budget logic for team members
* test: update unit testing
* test(test_proxy_budget_reset.py): validate team member budget was reset
* test(test_reset_budget_job.py): update unit tests
* test: update tests
* Update Vertex Model Garden documentation to deploy + use OpenAI Chat Completion tutorial.
* update anchor description
* update to LiteLLM specific tutorial
* update the right doc
* Reverted docs/my-website/docs/providers/vertex.md to match main
* revert to main vertex tutorial
* create new model button
* cleanup + prettier
* prettier
* create new model button added for model and model info+ back button added
* prettier
* team and team info has create team button + back to teams button added in team_info
* prettier
* better prettier
* always show create new organization button and add back to organizations button
* minor, prettier
* prettier
* remove user header and prettier
* move top buttons to the left
* prettier
* back to all logs button style fixed
* prettier
* modify button text
* create new guardrail button style
* prettier
* button modifications
* guardrails page done
* prettier
* add new guardrail text button + prettier on mcp
* button related changes
* prettier
* mcp servers page done
* fix(internal_user_endpoints.py): initial commit removing logic to create new budget for new user if default max budget in team set
* feat(proxy_setting_endpoints.py): update team member budget when set via default internal user endpoint
removes need to create a unique budget per user
* feat(proxy_server.py): set team max member budget on startup, if set on config.yaml
* fix(prometheus.py): support custom tags for tracking on prometheus
Allows tracking user agent values on prometheus metrics
* test(test_internal_user_endpoints.py): fix test
