* feat(bulk_user_update/): support updating all users on proxy
* fix(bulk_edit_user.tsx): persist user settings when 'add to team' clicked
* fix(team_endpoints.py): bulk add all proxy users to team
supports flow from UI to add all existing users to a team
* fix: minor fixes
* feat(user_edit_view.tsx): support setting no default model on user edit
allows preventing users from calling models outside team scope
* fix(user_edit_view.tsx): prevent triggering submit when 'cancel' is clicked
* refactor(internal_user_endpoints.py): refactor to reduce function size
* fix(ui_sso.py): fix form action on login when sso is enabled
* fix: multiple fixes - fix resetting env var in proxy config + add key to exception message on key decryption
fixes issue where env vars would be reset
* refactor(proxy_server.py): cleanup redundant decryption line
* fix(proxy_setting_endpoints.py): show saved ui access mode
allows admin to know what they'd previously stored in db
* feat(proxy_server.py): working guardrails on streaming output
ensures guardrail actually raises an error if flagged during streaming output
* test: add unit tests
* feat(advanced_settings.tsx): add guardrails option as ui component on model add
enables setting guardrails on model add
* feat(add_model_tab.tsx): fix add model form
* feat(model_info_view.tsx): support adding guardrails on model update
* fix(add_model_tab.tsx/): working health check when guardrails selected
* fix(proxy_server.py): fix yield
* feat(key_management_endpoints.py): Support new 'key_type' field
allow user to specify if key should be 'management' or 'llm api' key
Security fix
* test(test_route_checks.py): add unit tests
* fix(create_key_button.tsx): add ui component to select key type
allows specifying if key can call llm api vs. management routes
* feat(create_key_button.tsx): add specifying key type to ui
* fix(route_checks.py): add sensitive data masker for user id on not allowed error message
prevent leaking sensitive information
* feat(litellm_pre_call_utils.py): add num_retries to litellm data for backend call
allow user to pass in num retries via request headers
* test(test_litellm_pre_call_utils.py): add unit test
* docs(request_headers.md): document new request header
* fix(common_daily_activity.py): show spend breakdown by model group
Partial fix for https://github.com/BerriAI/litellm/issues/12887
* feat(new_usage.tsx): new tab switcher for viewing usage by model group vs. received model
Closes https://github.com/BerriAI/litellm/issues/12887
* fix(team_endpoints.py): always remove team member budget from updated_kv
this is not a field for the litellm team table
Prevents startup issue
* test(test_team_endpoints.py): add unit test to ensure 'team_member_budget' is never in update to table - separate logic
* refactor: cleanup
When using Model Armor guardrail with explicit project_id in config,
the project_id was being overwritten to None due to incorrect
initialization order between ModelArmorGuardrail and VertexBase parent class.
This fix ensures that user-provided project_id is preserved by initializing
parent classes before setting instance attributes.
Fixes#12757
* feat: integrate Google Cloud Model Armor guardrails (LIT-298)
- Add ModelArmorGuardrail class that extends CustomGuardrail and VertexBase
- Support for both pre-call (sanitizeUserPrompt) and post-call (sanitizeModelResponse) sanitization
- Integrate with existing Vertex AI authentication using VertexBase
- Add configuration model for Model Armor in guardrail types
- Register Model Armor in guardrail initializers and registry
- Include comprehensive test suite for Model Armor functionality
- Support content masking for both requests and responses
- Handle streaming responses with content sanitization
This integration allows LiteLLM to use Google Cloud Model Armor API for
content moderation and sanitization, providing similar functionality to
Bedrock Guards but using Google Cloud's security infrastructure.
* fix: remove unused imports flagged by ruff linter
- Remove unused asyncio import at top level (moved to local import where needed)
- Remove unused TextCompletionResponse import
* fix: remove additional unused imports
- Remove unused TextChoices import from line 34
- Remove duplicate asyncio import from line 384
- Replace asyncio.iscoroutine() with hasattr check for __await__
* fix: remove final unused imports
- Remove unused 'import sys' from line 9
- Remove unused 'StreamingChoices' from imports
* fix: remove unused import from model_armor.py
- Remove unused 'import os' from the top of the file
* fix: remove commented-out header from model_armor.py
- Eliminate unnecessary comments at the top of the file to improve code clarity.
* feat(guardrails): Add Model Armor UI support
- Convert model_armor.py to a directory structure with __init__.py for dynamic discovery
- Add get_config_model() method to ModelArmorGuardrail class for UI integration
- Add ui_friendly_name() to ModelArmorConfigModel returning "Google Cloud Model Armor"
- Remove manual registration from guardrail_registry.py to use dynamic discovery
- Model Armor now appears in the UI guardrails dropdown with proper configuration fields
This enables users to configure Model Armor guardrails through the LiteLLM UI interface.
* fix(guardrails): Fix undefined name 'GuardrailConfigModel' in Model Armor
- Import TYPE_CHECKING and GuardrailConfigModel from base module
- Fixes F821 linting error for undefined name in type annotation
- Follows same pattern as other guardrail implementations
* fix(guardrails): Fix Model Armor type errors and config model inheritance
- Create ModelArmorGuardrailConfigModel that properly inherits from GuardrailConfigModel base class
- Move config model to litellm/types/proxy/guardrails/guardrail_hooks/model_armor.py following convention
- Update get_config_model() to return the properly typed config model
- Remove ModelArmorConfigModel from LitellmParams inheritance chain
- Add template_id field to BaseLitellmParams instead
This fixes the mypy type errors and follows the same pattern as other guardrail implementations.
* fix(guardrails): Add missing Model Armor fields to BaseLitellmParams
- Add location, credentials, api_endpoint, and fail_on_error fields
- Fixes mypy errors about missing attributes in LitellmParams
- All Model Armor configuration parameters are now properly defined
* fix: Apply PR review feedback for Model Armor guardrail
- Move test file to tests/test_litellm/ for GitHub Actions
- Extract only last consecutive user messages to avoid context limits
- Use get_content_from_model_response helper for response extraction
- Handle non-ModelResponse types (e.g., TTS) gracefully
- Maintain newline separation for multi-part content
* refactor: Simplify message content extraction in ModelArmorGuardrail
- Removed the custom _extract_content_from_messages method.
- Integrated get_last_user_message helper for improved content extraction.
- Updated test to reflect changes in content formatting.
* refactor: Remove unused import in model_armor.py
- Deleted the unused import of AllMessageValues to clean up the codebase.
* Add unit tests for Model Armor guardrail functionality
- Implement tests for pre-call and post-call hooks, including content sanitization and blocking behavior.
- Validate error handling for API responses and credential management.
- Test streaming responses and handling of list content in user messages.
- Ensure proper assertions for API interactions and response sanitization.
* Add comprehensive test coverage for Model Armor guardrail
- Add test for requests with no messages field
- Add test for empty message content handling
- Add test for system/assistant-only messages
- Add test for fail_on_error=False behavior
- Add test for custom API endpoint configuration
- Add test for dictionary credentials (non-file path)
- Add test for action=NONE response handling
- Add test for missing sanitized_text field fallback
- Add test for non-text response types (TTS/image)
- Add test for auth token refresh behavior
All tests ensure robust edge case handling and proper error management.
* feat: improve Model Armor handling of non-ModelResponse types
- Add debug logging when skipping non-text responses (TTS, images, etc.)
- Improve docstring to clarify behavior for non-text responses
- Add test coverage for non-ModelResponse handling
- Ensure guardrail gracefully skips processing for response types it cannot handle
* build: move build_and_test to use prisma migrate
* fix(guardrails_ai.py): default to guardrail accepting 'llmOutput' as the input param
enables same guardrail to work for pre call and post call
* fix(__init__.py): set default value
* fix(guardrails_ai.py): updates
* fix: fix linting error
Replace MagicMock with AsyncMock for litellm_teamtable.update to fix:
TypeError: object MagicMock can't be used in 'await' expression
The test was failing because it tried to await a MagicMock object.
Added AsyncMock for the update method to properly handle async operations.
* fix(team_endpoints.py): ensure user id correctly added when new team created with user email as member
Fixes issue where user not correctly added to team on /team/new
* fix(internal_user_endpoints.py): make user email validation check case insensitive
Fixes issue where uppercase email was added even when lowercase email existed
* test: update test
* build: move build_and_test to use prisma migrate
* feat(proxy_setting_endpoints.py): encrypt env var before storing in db
Ensures env var can be read when loaded in from DB
Fixes issue when trying to add SSO from admin UI
* test: update tests
* fix: Handle circular references in spend tracking metadata JSON serialization
- Fixes issue #12634 where circular references in metadata caused
ValueError: Circular reference detected when logging spend data
- Adds _safe_json_dumps() function that detects and handles circular
references by replacing them with placeholder strings
- Maintains full functionality for normal objects while preventing
crashes from circular references
- Adds comprehensive tests for circular reference handling
- Critical fix for v1.74.3 stable release
* fix: Replace bare except clauses with specific Exception handling
- Fixes E722 linting errors in _safe_json_dumps function
- Maintains same error handling behavior while following best practices
- All tests continue to pass
* refactor: Use existing safe_dumps utility instead of custom implementation
- Replace custom _safe_json_dumps() with existing safe_dumps() from litellm_core_utils
- Remove duplicate code and leverage existing circular reference handling
- Update tests to use safe_dumps function
- Maintains same functionality while reducing code duplication
- All tests continue to pass
* feat(route_checks.py): allow admin to disable proxy management endpoints on instance
useful for preventing multiple instances from doing admin actions
* docs(scaling_multiple_instances.md): add architecture doc on scaling multiple litellm instances
provide guidance on scaling proxy
* docs(scaling_multiple_instances.md): add doc on scaling across multiple regions for litellm
* fix(route_checks.py): allow disabling llm api endpoints on an instance
allows pure admin instance to exist
* refactor(enterprise/route_checks.py): refactor env var checks
* refactor: finish refactoring
* docs(control_plane_and_data_plane.md): refactor docs
* test: update tests
* just use 1 param for mcp groups
* fix just use 1 param for access groups
* test_get_tools_from_mcp_servers
* docs access groups
* group MCPs
* test fix
* fix screenshots on docs
* TestMCPAccessGroupsE2E
* update img
* fix MCP connect
* added mcp tools on internal user and divide it by teams
* add support for server api call
* Added frontend for test key
* added tools used output
* fix ui for servers
* All servers to personal
* change columns format
* revert ui logic
* Added vertical align
* fix mapped tests
* fix lint
* fix lint
* remove extra file
* fix ui test
* comments fixes
* change query type
* change query type
* mcp acces group init
* add ability to change server display on ui through access groups
* Mcp access group names UI (#12486)
* Added ui changes to reflect mcp_access_groups
* fix edit mcp page
* change to string array (#12491)
* change to string array
* Remove print
* add ability to change server display on ui through access groups
* Litellm mcp access groups accesses (#12498)
* added mcp access groups for keys and teams
* added access groups above servers
* fixed ruff
* fixed mypy
* revert couple changes
* fix test
* fixed double asterisks
* Litellm mcp groups UI (#12522)
* add ui for teams
* fix object permissions
* fix mcp servers test object permission
* remove print
* add helper method
* add tests + remove logs
* add mcp access group servers to test key
* add mcp access group support for headers
* lint fix
* add tests and helper function
* fixed test
* change list -> List
* tests
* fix(proxy_setting_endpoints.py): require store model in db is enabled for setting user default settings
* test(test_proxy_server.py): update test
* fix(reset_budget_job.py): initial commit adding reset budget logic for team members
* test: update unit testing
* test(test_proxy_budget_reset.py): validate team member budget was reset
* test(test_reset_budget_job.py): update unit tests
* test: update tests
* fix(internal_user_endpoints.py): initial commit removing logic to create new budget for new user if default max budget in team set
* feat(proxy_setting_endpoints.py): update team member budget when set via default internal user endpoint
removes need to create a unique budget per user
* feat(proxy_server.py): set team max member budget on startup, if set on config.yaml
* fix(prometheus.py): support custom tags for tracking on prometheus
Allows tracking user agent values on prometheus metrics
* test(test_internal_user_endpoints.py): fix test
* update types
* add STDIO to client
* allow using STDIO with MCP manager
* add Stdio-specific fields to schema.prisma
* fixes for MCP mgmt
* fix for adding stdio MCP to DB
* ui - allow adding stdio MCPs
* fix MCP server manager
* docs stdio MCP
* add_stdio_mcp.png
* new stdio tests
* allow adding MCPs through config.yaml
* fix tool test panel
* use TestMCPClient
* ui fixes for testing circle ci mcp
* added mcp tools on internal user and divide it by teams
* add support for server api call
* Added frontend for test key
* added tools used output
* fix ui for servers
* All servers to personal
* change columns format
* revert ui logic
* Added vertical align
* fix mapped tests
* fix lint
* fix lint
* remove extra file
* fix ui test
* comments fixes
* change query type
* change query type
* mcp acces group init
* add ability to change server display on ui through access groups
* Mcp access group names UI (#12486)
* Added ui changes to reflect mcp_access_groups
* fix edit mcp page
* change to string array (#12491)
* change to string array
* Remove print
* add ability to change server display on ui through access groups
* Litellm mcp access groups accesses (#12498)
* added mcp access groups for keys and teams
* added access groups above servers
* fixed ruff
* fixed mypy
* revert couple changes
* fix test
* fixed double asterisks
* Litellm mcp groups UI (#12522)
* add ui for teams
* fix object permissions
* fix mcp servers test object permission
* remove print
* add helper method
* add tests + remove logs
* added mcp tools on internal user and divide it by teams
* add support for server api call
* Added frontend for test key
* added tools used output
* fix ui for servers
* All servers to personal
* change columns format
* revert ui logic
* Added vertical align
* fix mapped tests
* fix lint
* fix lint
* remove extra file
* fix ui test
* comments fixes
* change query type
* change query type
* mcp acces group init
* add ability to change server display on ui through access groups
* Mcp access group names UI (#12486)
* Added ui changes to reflect mcp_access_groups
* fix edit mcp page
* change to string array (#12491)
* change to string array
* Remove print
* add ability to change server display on ui through access groups
* Litellm mcp access groups accesses (#12498)
* added mcp access groups for keys and teams
* added access groups above servers
* fixed ruff
* fixed mypy
* revert couple changes
* fix test
* fixed double asterisks
* fix(guardrails_ai.py): initial commit adding pre-call hook support for guardrails ai
enables running user input through guardrails ai - if set
* feat(guardrails_ai.py): working pre call guardrail
enables pii detection to work via guardrails ai
* feat(guardrails_ai.py): support logging hook
enables masking input via guardrails ai on logging integrations
* test(test_guardrails_ai.py): add unit test for new input processing function
* fix(auth_checks.py): resolve a model group alias when key has access to underlying model
Fixes LIT-293
* feat(anthropic/): add mock_response to anthropic /v1/messages
makes it easy to test fallback logic
* fix(router.py): support fallbacks on /v1/messages
adds working fallbacks on generic api route
* refactor(router.py): point _ageneric_api_call_with_fallbacks to updated function
* test: add unit test for new helper on router
* fix(router.py): use correct metadata variable name
* fix(router.py): use correct metadata field
* docs(config_settings.md): document new param
* added mcp tools on internal user and divide it by teams
* add support for server api call
* Added frontend for test key
* added tools used output
* fix ui for servers
* All servers to personal
* change columns format
* revert ui logic
* Added vertical align
* fix mapped tests
* fix lint
* fix lint
* remove extra file
* fix ui test
* comments fixes
* change query type
* change query type
* change query type
* Update model_checks.py
* add _get_mcp_auth_header_from_headers
* test_process_mcp_request_with_custom_auth_header
* Using a different Authentication Header
* fix customize MCP Auth header name
* fix(common_daily_activity.py): initial commit with working mock BE endpoint for mcp usage
* feat(ui/): show mcp server activity on UI
allows admin to know which mcp's are being used
* feat(common_daily_activity.py): return activity by key
* feat(ui/): show top api keys for a given model / mcp server
allow user to know which key is driving spend
* fix(common_daily_activity.py): use known mcp server names
* feat(server.py): log the namespaced tool name (includes server prefix)
allow accurate cost tracking
* feat(db_spend_update_writer.py): log by mcp_namespaced_tool_name
store aggregate daily activity by mcp_namespaced_tool_name
Enables cost / usage tracking by mcp tool name
* fix(server.py): add key/user metadata to mcp calls
* refactor(common_daily_activity.py): update to return mcp activity in API
* fix(common_daily_activity.py): handle empty key
* fix(common_daily_activity.py): track when api key is empty
* test(test_spend_management_endpoints.py): update tests
* fix: fix ui linting error
* fix: fix linting errors
* test: add missing key
* build(schema.prisma): add mcp tool tracking
* fix(migration.sql): add schema migration file
* feat(server.py): add request logging for mcp calls
enables storing the mcp calls
* fix(new_usage.tsx): fix linting errors
* fix: fix code qa errors
* fix(activity_metrics.tsx): fix ui linting errors post-merge
* fix(types/utils.py): fix linting error
* fix(server.py): always have name
Krrish Dholakia