* added mcp guardrails doc in mcp.md
* add button to reload models
* Added button changes
* added button for scheduling reload
* add multi pod support to reloading the model price json
* fix ruff
* feat(proxy/utils.py): track pre-call hooks in OTEL
some pre call hooks can cause latency in high traffic - make sure this is tracked
* fix(router.py): move redis call on deployment_callback_on_success to pipeline operation
reduces p99 latency by half when redis is enabled
* fix(parallel_request_limiter_v3.py): only run check if any item has rate limits set
Prevents unnecessary latency added by rate limit checks
* test: add unit tests
* Latency Improvements: only track tpm/rpm usage when set on deployment+ LLM Caching - use an in-memory cache to reduce redis calls + OTEL - track time spent on LLM caching (#13472)
* fix(router.py): only track usage for deployments with tpm/rpm set
ensures additional latency avoided for non-tpm/rpm models
* fix(caching_handler.py): log time spent on request get cache to OTEL
enables easy debugging of call latency
* fix(caching_handler.py): use dual cache object for in-memory caching + trace redis call within caching handler
* fix(caching_handler.py): working in-memory cache for redis calls
ensures dual cache works when redis cache setup for llm calls
makes calls quicker by only checking redis when in-memory cache missed for llm api call
* test: remove redundant test
* test: add unit tests
* (#13284) add avector_store_create to route_type which doesn't require model
* (#13284) exclude hidden params in metadata when create vector store
* (#13284) fix lint error
* (#13284) keep metadata None if metadata is None(not empty dict)
* (#13284) add test code
* (#13284) change test code name
* (#13284) add avector_store_search to route_type which doesn't require model
* fix(guardrails): Fix PANW Prisma AIRS post-call hook method name
- Changed async_post_call_hook to async_post_call_success_hook to match proxy calling convention
- Added event_hook parameter to initialization to ensure proper hook registration
- Fixes post-call response scanning for PANW Prisma AIRS guardrails
Resolves issue where post-call hooks were not being invoked due to method name mismatch.
* Update PANW Prisma AIRS tests to use correct method name
* ensure original client is disconnected when re-creating
* test_recreate_prisma_client_successful_disconnect
* test_recreate_prisma_client_successful_disconnect
* feat: Add dot notation support for all JWT fields
- Updated all JWT field access methods to use get_nested_value for dot notation support
- Enhanced get_team_id to properly handle team_id_default fallback with nested fields
- Added comprehensive unit tests for nested JWT field access and edge cases
- Updated documentation to reflect dot notation support across all JWT fields
- Maintains full backward compatibility with existing flat field configurations
Supported fields with dot notation:
- team_id_jwt_field, team_ids_jwt_field, user_id_jwt_field
- user_email_jwt_field, org_id_jwt_field, object_id_jwt_field
- end_user_id_jwt_field (roles_jwt_field was already supported)
Example: user_id_jwt_field: 'user.sub' accesses token['user']['sub']
* fix: Add type annotations to resolve mypy errors
- Add explicit type annotation for team_ids variable in get_team_ids_from_jwt
- Add type ignore comment for sentinel object return in get_team_id
- Resolves mypy errors while maintaining functionality
* fix: Resolve mypy type error in get_team_ids_from_jwt
- Remove explicit List[str] type annotation that conflicts with get_nested_value return type
- Simplify return logic to use 'team_ids or []' ensuring always returns List[str]
- Fixes: Incompatible types in assignment (expression has type 'list[str] | None', variable has type 'list[str]')
* fix: Add proper type annotation for team_ids variable
- Use Optional[List[str]] type annotation to satisfy mypy requirements
- Resolves: Need type annotation for 'team_ids' [var-annotated]
- Maintains functionality while ensuring type safety
* refactor: remove outdated JWT unit tests and consolidate JWT-related functionality
- Deleted the test_jwt.py file as it contained outdated and redundant tests.
- Consolidated JWT-related tests into test_handle_jwt.py for better organization and maintainability.
- Updated tests to ensure proper functionality of JWT handling, including token validation and role mapping.
- Enhanced test coverage for JWT field access and nested claims handling.
* test: add comprehensive unit tests for JWT authentication
- Introduced a new test file `test_jwt.py` containing unit tests for JWT authentication.
- Implemented tests for loading configuration with custom role names, validating tokens, and handling team tokens.
- Enhanced coverage for JWT field access, nested claims, and role-based access control.
- Added fixtures for Prisma client and public JWT key generation to support testing.
- Ensured proper handling of valid and invalid tokens, including user and team scenarios.
* revert test_handle_jwt.py
* rename file
* test: remove outdated JWT nesting tests and add new nested field access tests
- Deleted the `test_jwt_nesting.py` file as it contained outdated tests.
- Introduced new tests in `test_handle_jwt.py` to verify nested JWT field access.
- Enhanced coverage for accessing nested values using dot notation and ensured backward compatibility with flat field names.
- Added tests for handling missing nested paths and appropriate default values.
- Improved handling of metadata prefixes in nested field access.
* restore file
* Fix: Add support for GOOGLE_API_KEY environment variables for Gemini API authentication
* added test cases
* incoperated feedback to make it more maintainable
* fix failed linting CI
* Fix security vulnerability in list_team_v2 endpoint
- Add missing allowed_route_check_inside_route security check to list_team_v2
- Add @management_endpoint_wrapper decorator for consistency with list_team
- Add comprehensive tests to verify security checks work correctly
- Ensure non-admin users can only query their own teams
- Ensure admin users can query all teams
This fixes a security bug where non-admin users could potentially access
team information they shouldn't have access to through the list_team_v2
endpoint, which was missing the authorization check present in list_team.
* Fix test
* Test fixes
* Fixed test
* Restored invalid delete
* Revert
---------
Co-authored-by: openhands <openhands@all-hands.dev>
* fix(azure/chat/gpt_transformation.py): support api_version="preview"
Fixes https://github.com/BerriAI/litellm/issues/12945
* Fix anthropic passthrough logging handler model fallback for streaming requests (#13022)
* fix: anthropic passthrough logging handler model fallback for streaming requests
- Add fallback logic to retrieve model from logging_obj.model_call_details when request_body.model is empty
- Fixes issue #12933 where streaming requests to anthropic passthrough endpoints would crash due to missing model field
- Ensures downstream logging and cost calculation work correctly for all streaming scenarios
- Maintains backwards compatibility with existing non-streaming requests
* test: add minimal tests for anthropic passthrough logging handler model fallback
- Add unit tests for the model fallback logic in _handle_logging_anthropic_collected_chunks
- Test existing behavior when request_body.model is present
- Test fallback logic when request_body.model is empty but logging_obj.model_call_details has model
- Test edge cases where both sources are empty or missing
- Ensure backwards compatibility and graceful degradation
* fix(anthropic_passthrough_logging_handler.py): add provider to model name (accurate cost tracking)
* fix(anthropic_passthrough_logging_handler.py): don't reset custom llm provider, if already set
* fix: fix check
---------
Co-authored-by: Haggai Shachar <haggai.shachar@backline.ai>
* feat(bulk_user_update/): support updating all users on proxy
* fix(bulk_edit_user.tsx): persist user settings when 'add to team' clicked
* fix(team_endpoints.py): bulk add all proxy users to team
supports flow from UI to add all existing users to a team
* fix: minor fixes
* feat(user_edit_view.tsx): support setting no default model on user edit
allows preventing users from calling models outside team scope
* fix(user_edit_view.tsx): prevent triggering submit when 'cancel' is clicked
* refactor(internal_user_endpoints.py): refactor to reduce function size
* fix(ui_sso.py): fix form action on login when sso is enabled
* fix: multiple fixes - fix resetting env var in proxy config + add key to exception message on key decryption
fixes issue where env vars would be reset
* refactor(proxy_server.py): cleanup redundant decryption line
* fix(proxy_setting_endpoints.py): show saved ui access mode
allows admin to know what they'd previously stored in db
* feat(proxy_server.py): working guardrails on streaming output
ensures guardrail actually raises an error if flagged during streaming output
* test: add unit tests
* feat(advanced_settings.tsx): add guardrails option as ui component on model add
enables setting guardrails on model add
* feat(add_model_tab.tsx): fix add model form
* feat(model_info_view.tsx): support adding guardrails on model update
* fix(add_model_tab.tsx/): working health check when guardrails selected
* fix(proxy_server.py): fix yield
* feat(key_management_endpoints.py): Support new 'key_type' field
allow user to specify if key should be 'management' or 'llm api' key
Security fix
* test(test_route_checks.py): add unit tests
* fix(create_key_button.tsx): add ui component to select key type
allows specifying if key can call llm api vs. management routes
* feat(create_key_button.tsx): add specifying key type to ui
* fix(route_checks.py): add sensitive data masker for user id on not allowed error message
prevent leaking sensitive information
* feat(litellm_pre_call_utils.py): add num_retries to litellm data for backend call
allow user to pass in num retries via request headers
* test(test_litellm_pre_call_utils.py): add unit test
* docs(request_headers.md): document new request header
* fix(common_daily_activity.py): show spend breakdown by model group
Partial fix for https://github.com/BerriAI/litellm/issues/12887
* feat(new_usage.tsx): new tab switcher for viewing usage by model group vs. received model
Closes https://github.com/BerriAI/litellm/issues/12887
* fix(team_endpoints.py): always remove team member budget from updated_kv
this is not a field for the litellm team table
Prevents startup issue
* test(test_team_endpoints.py): add unit test to ensure 'team_member_budget' is never in update to table - separate logic
* refactor: cleanup
