Two code paths in key_management_endpoints.py call hash_token()
unconditionally when invalidating the user_api_key_cache after a key
update. When the caller passes a pre-hashed token ID (not an sk-
prefixed key), hash_token() double-hashes it, producing a cache key
that does not match the actual cached entry. Cache invalidation
silently fails.
This is compounded by update_cache() which writes the stale cached key
object back with a fresh 60s TTL after every successful request,
preventing natural TTL expiry. The stale entry (with outdated fields
like max_budget=None) persists indefinitely under load.
PR #24969 fixed this in update_key_fn but missed two other call sites:
- _process_single_key_update (bulk update path)
- _execute_virtual_key_regeneration (key rotation path)
Fix: replace hash_token() with _hash_token_if_needed() in both
locations, matching the pattern already used elsewhere in the file.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add _verify_org_access to deprecated POST /organization/info endpoint
- Move get_user_object to module-level import in organization_endpoints.py
- Add tests for _verify_team_access 403 denial path
- Introduced a new method in `FileContentStreamingHandler` to resolve streaming request parameters, enhancing the routing logic based on credentials.
- Updated the `should_stream_file_content` method to check against supported providers.
- Cleaned up type hints and imports across multiple files for better organization and clarity.
- Added comprehensive tests to validate the new routing behavior and ensure original data integrity during streaming requests.
* feat(guardrails): optional skip system message in unified guardrail inputs
Made-with: Cursor
* feat(dashboard): skip_system_message_in_guardrail in guardrail UI
Add a tri-state control (inherit / yes / no) when creating or editing
guardrails so admins can set litellm_params.skip_system_message_in_guardrail
without YAML. Table edit merges existing litellm_params before PUT to avoid
wiping content-filter and other provider fields.
Document the dashboard flow in the guardrails quick start with a screenshot.
Made-with: Cursor
* fix(guardrails): type structured_messages as AllMessageValues for mypy
Use AllMessageValues in openai_messages_without_system and cast adapter
request messages so GenericGuardrailAPIInputs matches TypedDict.
Made-with: Cursor
- Changed the import path for `upload_file_to_storage_backend` in test files to reflect the new module structure.
- Ensured consistency in mocking for storage backend service tests.
The raw SQL queries in spend_management_endpoints.py, spend_tracking_utils.py,
and analytics_endpoints.py cast date params to ::timestamptz while comparing
against the plain-timestamp "startTime" column. Postgres resolves the type
mismatch by promoting the column using the DB session timezone, which drifts
the filter window and date_trunc buckets whenever session TZ is not UTC —
silently dropping rows at UTC day boundaries and, for narrow windows, losing
rows entirely.
Wraps every such comparison with `AT TIME ZONE 'UTC'` so the param side
resolves to a plain timestamp matching the column type. Both sides end up
as plain timestamp, Postgres does no implicit conversion, and session TZ
plays no role in the query. The fix is constant-foldable so the existing
startTime index (PR #17504) remains usable.
Also marks strptime-produced datetimes as tz-aware UTC at the call sites
for intent clarity and consistency with parse_date.
Fixes#22529 (Logs page missing recent rows under non-UTC session TZ) and
the Global Usage single-day-returns-two-days symptom reported internally.
- Added validation to ensure credentials include a custom LLM provider before routing.
- Cleaned up type casting for better readability.
- Introduced a new test to verify behavior when a non-OpenAI provider is used, ensuring proper handling of streaming responses.
- Updated imports to include necessary modules for testing.
- Updated `FileContentStreamingHandler` to utilize `custom_llm_provider` from credentials for routing.
- Added error handling for missing `custom_llm_provider` in credentials.
- Introduced new tests to validate streaming behavior with routed providers and non-OpenAI providers.
- Cleaned up imports and ensured proper type casting for improved clarity.
- Static Methods for Streaming Handler Function
- Remove the afile_content_streaming wrapper function. Enabled with a stream boolean in afile_content
- Cleaned up test cases after refactor
- Removed unused imports and streamlined type hints in `litellm/utils.py` and `litellm/files/main.py`.
- Moved `FileContentStreamingResult` to a new `litellm/files/types.py` for better organization.
- Updated `FileContentStreamingResponse` in `litellm/files/streaming.py` to include asynchronous close methods and improved logging capabilities.
- Enhanced tests to ensure proper closure of streaming iterators in `tests/test_litellm/llms/openai/test_openai_file_content_streaming.py` and `tests/test_litellm/proxy/openai_files_endpoint/test_files_endpoint.py`.
- Route multipart forwarding on forward_multipart instead of empty _parsed_body
so litellm_logging_obj no longer forces json= for file uploads.
- Remove custom_body from pass-through endpoint signatures; FastAPI treated it
as a JSON body and rejected multipart before the handler ran. Bedrock passes
JSON via request.state (LITELLM_PASS_THROUGH_CUSTOM_BODY_STATE_KEY).
- Use build_request + send(stream=True) for streaming multipart; httpx 0.28
AsyncClient.request does not accept stream=.
- Add regression test for non-empty _parsed_body multipart path; update Bedrock
custom-body test and query-params test for forward_multipart.
Made-with: Cursor
Non-admins previously skipped RBAC when prisma_client was None but could
still read payloads from custom loggers. Return 403 unless admin view.
Add test_ui_view_request_response_forbids_non_admin_without_db.
Made-with: Cursor
- Drop module-level common_utils import; import team helpers inside callers.
- Inline admin-view role check in _is_admin_view_safe to break import cycle.
- Require non-null row.user before treating spend log as owned by the key
(fixes None==None bypass for service keys).
- Document deferred proxy_server imports in _get_permitted_team_ids_for_spend_logs.
- Update tests (common_utils patches, regression test, ruff cleanups).
Made-with: Cursor
- Introduced `afile_content_streaming` and `file_content_streaming` functions in `litellm/files/main.py` to handle asynchronous and synchronous file content streaming.
- Added `FileContentStreamingResponse` class in `litellm/files/streaming.py` to manage streaming responses with logging capabilities.
- Updated OpenAI API integration in `litellm/llms/openai/openai.py` to support new streaming methods.
- Enhanced file content retrieval in `litellm/proxy/openai_files_endpoints/files_endpoints.py` to route requests for streaming.
- Added unit tests for the new streaming functionality in `tests/test_litellm/llms/openai/test_openai_file_content_streaming.py` and `tests/test_litellm/proxy/openai_files_endpoint/test_files_endpoint.py`.
- Refactored type hints and imports for better clarity and organization across modified files.
- Defense-in-depth: warn instead of hard-fail for legacy servers
- Move os import to module level in _types.py
- Document args residual risk in allowlist comment
- Add UpdateMCPServerRequest allowlist test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add command allowlist for MCP stdio transport to prevent RCE via
/mcp-rest/test/* endpoints. Restrict test endpoints to PROXY_ADMIN
role. Fix docker/README.md MASTER_KEY -> LITELLM_MASTER_KEY.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allow JWT tokens matching routing_overrides to use OAuth2 introspection without enabling global OAuth2 while keeping OAuth2 routing limited to LLM/info routes. Add regression coverage for management-route boundary and tighten opaque-token assertions; update docs to reflect selective-mode route scope.
Made-with: Cursor
Adds a new endpoint to bulk-update team_member_permissions across
teams. Supports apply_to_all_teams (with cursor-based pagination)
or a specific list of team_ids. Merges new permissions into each
team's existing set rather than overwriting.
Also fixes test isolation bug in test_get_prompt_info_by_base_id
where leaked prisma_client state from other tests caused a
TypeError on await.
* added support for metadata (#24261)
* added support for metadata
* fix: PR review - meta truthiness, BlobResourceContents mimeType, add Blob+empty meta tests
Made-with: Cursor
* pyproject to .25
* feat(teams): resolve access group models/MCPs/agents in team endpoints
Add access_group_models, access_group_mcp_server_ids, and
access_group_agent_ids to /team/info and /v2/team/list responses.
These fields contain resources inherited from access groups, kept
separate from direct assignments so the UI can distinguish the source.
Backend: _resolve_access_group_resources() helper resolves access
group resources via existing _get_*_from_access_groups() functions.
UI: Teams table and detail view show direct models as blue badges
and access-group-sourced models as green badges.
* perf(teams): single-pass access group resolution + asyncio.gather in list endpoint
- Fetch each access group object once and extract all 3 resource fields
in a single pass instead of 3 separate calls (3N → N lookups)
- Use asyncio.gather to resolve access groups across teams concurrently
in list_team_v2 instead of sequential awaits
- Add 5 unit tests for _resolve_access_group_resources
* docs: add default_team_params to config reference and update examples
- Add default_team_params to litellm_settings reference table in
config_settings.md with all sub-fields documented
- Update self_serve.md and msft_sso.md examples to include
team_member_permissions, tpm_limit, and rpm_limit
- Fix misleading comment that implied default_team_params only applies
to SSO auto-created teams — it applies to all /team/new calls
* docs: clarify that models sub-field only applies to SSO auto-created teams
* fix: lazy import get_access_object to break cyclic import + short-circuit all-proxy-models display
- Remove get_access_object from module-level import in team_endpoints.py
and use a lazy _get_access_object wrapper to avoid cyclic dependency
- Add _prisma_client is None early-exit guard in _resolve_access_group_resources
- Short-circuit UI to show "All Proxy Models" when team.models is empty
or contains "all-proxy-models", skipping access group model resolution
* add: making organizations a select instead of read only badges
* fix(ui): only send organization_id when changed and use raw initial value
* fix(ui): add paginated team search to usage page filter
Replace the static team dropdown on the usage page with a new
TeamMultiSelect component that uses the paginated v2/team/list
endpoint with debounced server-side search and infinite scroll.
* fix(ui): fix imports and update placeholder for team multi select
* fix(ui): wire team_id filter to key alias dropdown on Virtual Keys tab
The Key Alias dropdown on the Virtual Keys page was showing aliases from
all teams regardless of which team was selected. The team_id was never
passed through the frontend chain to the backend /key/aliases endpoint.
- Backend: add optional team_id query param to /key/aliases endpoint
- networking.tsx: add team_id param to keyAliasesCall
- useKeyAliases: accept and forward team_id to API call and query key
- filter.tsx: pass allFilters context to custom filter components
- PaginatedKeyAliasSelect: read Team ID from allFilters and pass to hook
* fix(tests): correct mock targets in TestResolveAccessGroupResources
Three tests were patching the non-existent `get_access_object` instead
of `_get_access_object` (the lazy-import wrapper), causing AttributeError.
Also added missing `prisma_client` mock so tests get past the early-exit
guard and actually exercise the resolution logic.
* fix: use direct attribute access with or [] fallback in _resolve_access_group_resources
Replace getattr(ag, "field", []) with ag.field or [] for cleaner
access and safe handling if a field is None.
* fix(ui): remove model source legend from team detail view
The blue/green color distinction is self-explanatory; the legend added
visual clutter without providing enough value.
* fix(ui): add missing access_group fields to TeamData.team_info type
The TeamData interface was missing access_group_models,
access_group_mcp_server_ids, and access_group_agent_ids fields,
causing a TypeScript build failure.
* perf(teams): batch-fetch access groups in single DB query
Replace per-ID _resolve_access_group_resources loop with a single
find_many call that deduplicates IDs across all teams. Removes the
N+1 query pattern on cold cache for the team list endpoint.
* refactor(proxy): extract helpers to fix PLR0915 violations
Extract `_apply_non_admin_alias_scope` from `key_aliases`,
`_resolve_team_access_group_resources` from `team_info`, and
`_enforce_list_team_v2_access` from `list_team_v2` to bring each
function under ruff's 50-statement limit. No behavior changes.
* test(ui): update tests to match new team_id / access-group signatures
- useKeyAliases, PaginatedKeyAliasSelect: add trailing `undefined` to
spy matchers for the new `team_id` param on `useInfiniteKeyAliases`
and `keyAliasesCall`.
- EntityUsage: mock new `TeamMultiSelect` child so QueryClientProvider
is not required for team-entity tests.
- ModelsCell: replace the overflow-accordion test with one that
verifies the new collapse-on-`all-proxy-models` behavior (no
accordion, single badge).
* fix(ui): send null (not '') for cleared organization_id on team update
AntD <Select allowClear> returns undefined when the user clears the
selection. Coalescing to "" caused the team-update payload to carry
organization_id: "" instead of null, relying on the backend to coerce
it. Send null directly so the intent is explicit at the source.
* poetry
* chore: regen poetry.lock for litellm-proxy-extras 0.4.64 bump
* chore: update Next.js build artifacts (2026-04-04 17:55 UTC, node v22.16.0)
---------
Co-authored-by: shivam <shivam@uni.minerva.edu>
Co-authored-by: Ryan Crabbe <ryan@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* Tag query fix (#25094)
* feat(tag-spend): implement separate scheduler job for daily tag spend updates
* fix(docker): add g++ to build dependencies in Dockerfile
* initial test cases. TODO: check scheduler init and test cases in proxy_server related to it
* resolved QPS issue when redis transaction buffer is enabled
* resolving circular import error flagged by greptile
* fix(mypy): use Optional[str] for api_base in PydanticAI provider to match superclass signature
---------
Co-authored-by: Shivam Rawat <shivam@berri.ai>
Co-authored-by: shivam <shivam@uni.minerva.edu>
Co-authored-by: Ryan Crabbe <ryan@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Harish <harishgokul01@gmail.com>
Co-authored-by: Ishaan Jaffer <ishaan@berri.ai>
* Litellm ishaan march23 - MCP Toolsets + GCP Caching fix (#25146)
* feat(mcp): MCP Toolsets — curated tool subsets from one or more MCP servers (#24335)
* feat(mcp): add LiteLLM_MCPToolsetTable and mcp_toolsets to ObjectPermissionTable
* feat(mcp): add prisma migration for MCPToolset table
* feat(mcp): add MCPToolset Python types
* feat(mcp): add toolset_db.py with CRUD helpers for MCPToolset
* feat(mcp): add toolset CRUD endpoints to mcp_management_endpoints
* fix(mcp): skip allow_all_keys servers when explicit mcp_servers permission is set (toolset scope fix)
* feat(mcp): add _apply_toolset_scope and toolset route handling in server.py
* fix(mcp): resolve toolset names in responses API before fetching tools
* feat(mcp): add mcp_toolsets field to LiteLLM_ObjectPermissionTable type
* feat(mcp): register LiteLLM_MCPToolsetTable in prisma client initialization
* feat(mcp): validate mcp_toolsets in key-vs-team permission check
* feat(mcp): register toolset routes in proxy_server.py
* feat(mcp): add MCPToolset and MCPToolsetTool TypeScript types
* feat(mcp): add fetchMCPToolsets, createMCPToolset, updateMCPToolset, deleteMCPToolset API functions
* feat(mcp): add useMCPToolsets React Query hook
* feat(mcp): add toolsets (purple) as third option type in MCPServerSelector
* feat(mcp): extract toolsets from combined MCP field in key form
* feat(mcp): extract toolsets from combined MCP field in team form
* feat(mcp): show toolsets section in MCPServerPermissions read view
* feat(mcp): pass mcp_toolsets through object_permissions_view
* feat(mcp): add MCPToolsetsTab component for creating and managing toolsets
* feat(mcp): add Toolsets tab to mcp_servers.tsx
* feat(mcp): pass mcpToolsets to playground chat and responses API calls
* feat(mcp): generate correct server_url for toolsets in playground API calls
* docs(mcp): add MCP Toolsets documentation
* docs(mcp): add mcp_toolsets to sidebar
* fix(mcp): replace x-mcp-toolset-id header with ContextVar to prevent client forgery
* fix(mcp): use ContextVar + StreamingResponse for toolset MCP routes (fixes SSE streaming)
* fix(mcp): cache toolset permission lookups to avoid per-request DB calls
* test(mcp): add tests for toolset scope enforcement, ContextVar isolation, and access control
* fix(mcp): cache toolset name lookups in MCPServerManager to avoid per-request DB calls
* fix(mcp): prevent body_iter deadlock + use cached toolset lookup in responses API
- _stream_mcp_asgi_response: add done callback to handler_task that puts
the EOF sentinel on body_queue when the task exits, preventing body_iter
from hanging forever if the handler raises after headers are sent.
- litellm_proxy_mcp_handler: replace raw get_mcp_toolset_by_name() DB call
with global_mcp_server_manager.get_toolset_by_name_cached() so toolset
resolution uses the 60s TTL cache added for this purpose instead of
hitting the DB on every responses-API request.
* fix(mcp): toolset access control, asyncio fix, and real unit tests
- server.py: _apply_toolset_scope now enforces that non-admin keys must
have the requested toolset_id in their mcp_toolsets grant list;
admin keys always bypass the check.
- mcp_management_endpoints.py: three access-control fixes:
* fetch_mcp_toolsets: non-admin keys with mcp_toolsets=None now
return [] instead of all toolsets (only admins get 'all' when
the field is absent)
* fetch_mcp_toolset: non-admin keys that haven't been granted the
requested toolset_id now get 403 instead of the full result
* add_mcp_toolset: duplicate toolset_name now returns 409 Conflict
instead of an opaque 500
- proxy_server.py: use asyncio.get_running_loop() instead of
get_event_loop() inside an already-running coroutine (Python 3.10+).
- test_mcp_toolset_scope.py: replace four hollow tests that only
asserted local variable properties with real tests that call the
production fetch_mcp_toolsets() and handle_streamable_http_mcp()
functions with mocked dependencies.
* fix(mcp): add mcp_toolsets to ObjectPermissionBase, fix multi-toolset overwrite, fix delete 404, allow standalone key toolsets
* fix(mcp): add auth check on toolset resolution in responses API; union mcp_servers in _merge_toolset_permissions
* fix(mcp): handle RecordNotFoundError in update_mcp_toolset; union direct servers with toolset servers
* fix(mcp): use _user_has_admin_view; deny None mcp_toolsets for non-admin; use direct RecordNotFoundError import; fix docstring
* fix(mcp): add @default(now()) to MCPToolsetTable.updated_at; fix test for non-admin toolset access
* fix: use UniqueViolationError import; guard _ensure_eof for error/cancel only
* fix(mcp): preserve mcp_access_groups in toolset scope, use shared Redis cache for toolset perms
- Remove mcp_access_groups=[] from _apply_toolset_scope (server.py) and the
responses API toolset path (litellm_proxy_mcp_handler.py). A key's access-group
grants remain valid even when the request is scoped to a single toolset; clearing
them silently revoked legitimate entitlements.
- Switch resolve_toolset_tool_permissions and get_toolset_by_name_cached to use
user_api_key_cache (Redis-backed DualCache in production) instead of per-instance
in-memory dicts. Cache entries are now shared across workers, eliminating the
per-worker stale-toolset-permission window flagged as a P1 by Greptile.
- Use union merge (set union of tool names per server) when applying toolset
permissions in the responses API path so direct-server tool restrictions are not
overwritten by toolset permissions.
* fix(mcp): return 404 when edit_mcp_toolset target does not exist
* fix(mcp): align mcp_toolsets default to None in LiteLLM_ObjectPermissionTable
* fix(mcp): admin toolset visibility, in-place tool name mutation, test helper coercion
* fix(mcp): treat None/[] team mcp_toolsets as no restriction in key validation
* fix(mcp): allow_all_keys backward compat, blocked_tools API write-path, efficient startup query
* fix(mcp): use _mcp_active_toolset_id ContextVar to detect toolset scope, avoiding DB-default false-positive
* fix(mcp): remove dead toolset cache stubs, log invalidation failures, align schema updated_at defaults
* fix(mcp): deserialise MCPToolset from Redis cache hit, replace fastapi import in test
* fix(mcp): evict name-cache on toolset mutation, 409 on rename conflict, warning-level list errors
* fix(redis): regenerate GCP IAM token per connection for async cluster (#24426)
* fix(redis): regenerate GCP IAM token per connection for async cluster clients
Async RedisCluster was generating the IAM token once at startup and
storing it as a static password. After the 1-hour GCP token TTL, any
new connection (including to newly-discovered cluster nodes) would fail
to authenticate.
Fix: introduce GCPIAMCredentialProvider that implements redis-py's
CredentialProvider protocol. It calls _generate_gcp_iam_access_token()
on every new connection, matching what the sync redis_connect_func
already does. async_redis.RedisCluster accepts a credential_provider
kwarg which is invoked per-connection.
* refactor(redis): move GCPIAMCredentialProvider to its own file
Extract GCPIAMCredentialProvider and _generate_gcp_iam_access_token
into litellm/_redis_credential_provider.py. _redis.py imports them
from there, keeping the public API unchanged.
* fix: address Greptile review issues
- GCPIAMCredentialProvider now inherits from redis.credentials.CredentialProvider
so redis-py's async path calls get_credentials_async() properly
- move _redis_credential_provider import to top of _redis.py (PEP 8)
- remove dead else-branch that silently no-oped (gcp_service_account from
redis_kwargs.get() was always None since it's popped by _get_redis_client_logic)
- remove mid-function 'from litellm import get_secret_str' inline import
- remove unused 'call' import from test_redis.py
* chore: retrigger CI/review
* chore: sync schema.prisma copies from root
* chore: sync schema.prisma copies from root
* fix(proxy_server): use bounded asyncio.Queue with maxsize to prevent unbounded growth
* fix(a2a/pydantic_ai): make api_base Optional to match base class signature
* fix(a2a/pydantic_ai): make api_base Optional in handler and guard against None
* fix(mcp): remove unused get_all_mcp_servers import
* fix(mcp): remove unused MCPToolset import
* refactor(mcp): extract toolset permission logic to reduce statement count below PLR0915 limit
* fix(tests): update reload_servers_from_database tests to mock prisma directly
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(toolset_db): lazy-import prisma to avoid ImportError when prisma not installed
* fix(tests): update UI tests for toolset tab and updated empty state text
* fix(tests): add get_mcp_server_by_name to fake_manager stub
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Bug 1: internal users hit route-level 403 on /guardrails/submissions.
The route wasn't in self_managed_routes, so the route allowlist rejected
non-admin callers before our endpoint's team-scoping ran. Added
/guardrails/submissions and /guardrails/submissions/{guardrail_id} to
self_managed_routes.
Bug 2: register_guardrail 403'd non-admins registering for teams in
their user.teams list. It used get_team_membership() which reads the
litellm_teammembership join table, but that row is only created when
the team has a budget (management_helpers/utils.py:225). Switched to
the _get_user_team_ids helper (reads user_obj.teams), making it
consistent with list_guardrail_submissions.
UI: moved the Test Playground tab inside the isAdmin conditional in
guardrails.tsx. Internal users now see only the Submitted Guardrails
tab; admins still see all four.
Tests: added coverage for non-admin register paths (cross-team allowed
and cross-team forbidden).
* feat(router): integrate allowed_fails_policy into health check failures (#24988)
* feat(router): integrate allowed_fails_policy into health check failures
Health check failures now increment the same per-deployment failure
counters used by allowed_fails_policy, so users can control how many
health check failures of each error type are required before a
deployment enters cooldown.
- ahealth_check() preserves the original exception in its return dict
- run_with_timeout() returns a litellm.Timeout on health check timeout
- _perform_health_check() propagates exceptions to unhealthy endpoints
- _write_health_state_to_router_cache() calls _set_cooldown_deployments
for each unhealthy endpoint that has an exception
- When allowed_fails_policy is set, the binary health check filter is
bypassed so cooldown is the sole routing exclusion mechanism
- Safety net: if all deployments are in cooldown with
enable_health_check_routing=True, the cooldown filter is bypassed
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(router): add health_check_ignore_transient_errors flag
When enabled, health check failures with 429 (rate limit) or 408 (timeout)
status codes are skipped from the cooldown pipeline. These are transient
load issues, not broken deployments. Auth errors (401), 404, and 5xx errors
still increment counters and trigger cooldown as before.
Config (general_settings):
health_check_ignore_transient_errors: true
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(router): also exclude 429/408 from health state cache when ignore_transient_errors set
The previous fix only skipped cooldown counter increments. The health state
cache was still marking 429/408 endpoints as is_healthy=False, causing the
binary health check filter to exclude them from routing.
Now, when health_check_ignore_transient_errors=True, 429/408 endpoints are
also excluded from the unhealthy list passed to build_deployment_health_states(),
so the binary filter treats them as unaffected (not unhealthy).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(router): add health check driven routing guide
New standalone page covering the full health check routing feature:
allowed_fails_policy integration, health_check_ignore_transient_errors,
architecture SVG, step-by-step setup, and gotchas (TTL, AllowedFails semantics).
Replaces the inline section in health.md with a link to the new page.
Added to the Routing & Load Balancing sidebar.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(health-check-routing): fix three CI failures
- Add "exception" to ILLEGAL_DISPLAY_PARAMS in health_check.py so the
exception object is stripped before the health endpoint serializes
results to JSON (fixes TypeError: 'URL' object is not iterable)
- Add allowed_fails_policy = None to FakeRouter stubs in
test_router_health_check_routing.py (fixes AttributeError)
- Add health_check_ignore_transient_errors to config_settings.md router
settings reference table (fixes documentation test)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix litellm/tests/proxy_unit_tests/test_proxy_server.py
* fix(router): address greptile review comments
- Narrow cooldown safety-net bypass: only fires when allowed_fails_policy
is set (cooldown is health-check driven). Without a policy, cooldowns
are from real request failures and must not be bypassed.
- Restore cooldown deployments DEBUG log that was accidentally removed.
- Fix test_health TypeError: move exception extraction to a separate
exceptions_by_model_id dict returned alongside endpoints, so exception
objects never appear in the endpoint dicts that get JSON-serialized
by the /health response.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(health-check-routing): properly isolate exceptions from health response
Return exceptions_by_model_id as a separate third value from
_perform_health_check / perform_health_check so exception objects
(which contain non-JSON-serializable httpx URL types) never appear
in the endpoint dicts that get serialized by the /health response.
Callers updated: _health_endpoints.py, shared_health_check_manager.py,
proxy_server.py background loop. All use the exceptions dict only for
cooldown integration, not for display.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(shared-health-check): fix remaining 2-value return sites and update type annotation
* fix(health-check-routing): fix P0 cooldown integration never firing
The cooldown loop was reading endpoint.get("exception") which is always
None because exceptions are now returned via exceptions_by_model_id, not
stored in endpoint dicts. Fixed to use _exceptions.get(model_id).
Also fixes the transient-error filter to use _exceptions instead of
endpoint.get("exception"), and fixes all remaining 2-value return sites
in shared_health_check_manager.py. Tests updated to pass exceptions via
exceptions_by_model_id parameter instead of endpoint dicts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(health-check-routing): fix P1 transient-error filter broken on cache hits
When SharedHealthCheckManager returns cached results, exceptions_by_model_id
is always {} so the transient-error filter defaulted to status 500 for all
endpoints, incorrectly marking 429/408 endpoints as unhealthy.
Fix: store integer exception_status on each unhealthy endpoint dict in
_perform_health_check. _get_endpoint_exception_status() uses the live
exception object when available (direct path) and falls back to the stored
integer (cache-hit path). The integer is JSON-serializable and survives
the shared cache round-trip.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(health-check-routing): gate cooldown loop behind allowed_fails_policy
Without the policy, cooldown is not the routing exclusion mechanism.
Firing _set_cooldown_deployments for all enable_health_check_routing users
was a backwards-incompatible change — 401s would immediately cooldown
deployments that the binary filter would have recovered on the next cycle.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* revert: undo allowed_fails_policy gate on cooldown loop
Cooldown integration via health checks is intentional for all
enable_health_check_routing users, not just those with allowed_fails_policy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(docs+tests): fix health_check_ignore_transient_errors doc section and test coverage
- Move health_check_ignore_transient_errors from router_settings to
general_settings in config_settings.md (code reads it from general_settings)
- Remove duplicate enable_health_check_routing / health_check_staleness_threshold
entries that were incorrectly listed under router_settings
- Replace TestHealthCheckEndpointExceptionPropagation tests with ones that
exercise the real _perform_health_check code path via mocked ahealth_check,
verifying exceptions appear in exceptions_by_model_id and NOT in endpoint dicts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tests+docs): fix tuple unpacking and docs test failures
- Update test mocks that return (healthy, unhealthy) to return
(healthy, unhealthy, {}) to match the new 3-value signature
- Update test unpackings of perform_shared_health_check to use
healthy, unhealthy, _ = ...
- Add health_check_ignore_transient_errors to router_settings section
in config_settings.md (it is a Router constructor param, so the doc
test requires it there; it also lives in general_settings for proxy use)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix CodeQL errors
* fix(tests): fix 2-value unpackings of _perform_health_check in test_health_check.py
* fix(tests): fix mock _perform_health_check returning 2-tuple instead of 3
* fix team routing
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add distributed lock for key rotation job (#23364)
* fix: add distributed lock for key rotation job
* fix: address Greptile review feedback on key rotation lock (#23834)
* fix: address Greptile review feedback on key rotation lock
* fix req changes greptile
* feat(proxy): Optional on_error for guardrail pipeline (API / technical failures) (#24831)
* guardrails fallback
* docs
* docs: add LITELLM_KEY_ROTATION_LOCK_TTL_SECONDS to environment variables reference
* fix(mypy): accept Union[Dict, Any] in _get_deployment_order and use typed list to fix min() type error
* fix(mypy): use Optional[str] for api_base in PydanticAI provider to match superclass signature
---------
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com>
Co-authored-by: Shivam Rawat <shivam@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Backend:
- list_guardrail_submissions no longer 403s non-admins; it returns only
submissions whose team_id matches one of the caller's teams (via
get_user_object.teams). Admins still see all.
- Filtering by a team the caller is not in returns 403.
- Users with no team memberships get an empty list (no DB query).
- get_guardrail_submission applies the same scoping to single-item GETs.
Frontend:
- Remove admin-only bail-out in TeamGuardrailsTab.fetchSubmissions so
internal users actually load their team's submissions.
- Finish antd migration in guardrails.tsx: drop the last Tremor Button.
- Remove guardrailsList.length === 0 gate on the Test Playground tab;
the playground already renders a "No guardrails available" inline
empty state, which is more discoverable than a disabled tab.
Tests:
- Cover non-admin scoped access, empty teams, cross-team filter 403,
and per-submission GET scoping.
* fix(pricing): add unversioned vertex_ai/claude-haiku-4-5 entry
Missing unversioned entry causes cost tracking to return $0.00 for
all requests using vertex_ai/claude-haiku-4-5. All other Vertex AI
Claude models have both versioned and unversioned entries.
* fix(router): skip misleading tags error when no candidates (e.g. cooldown)
Return early from get_deployments_for_tag when healthy_deployments is empty so
tag-based routing does not raise no_deployments_with_tag_routing after cooldown
filters all deployments. Adds regression test.
Made-with: Cursor
* feat(oci): add embedding support and update model catalog
- Add OCIEmbeddingConfig for OCI GenAI embedding models
- Add 16 new chat models (Cohere, Meta Llama, xAI Grok, Google Gemini)
- Add 8 embedding models (Cohere embed v3.0, v4.0)
- Update documentation with embedding examples
- Update pricing for all new models
* test(oci): add unit tests for OCI embedding support
- 17 unit tests covering OCIEmbeddingConfig
- Tests for URL generation, param mapping, request/response transform
- Tests for model pricing JSON completeness
* style(oci): format with black and ruff
* fix(oci): correct embedding request body format
OCI embedText API expects inputs, truncate, and inputType at the
top level of the request body, not nested under embedTextDetails.
Fixed transformation and updated tests accordingly.
Verified with real OCI API: 3/3 embedding models working.
* docs: clarify tag routing early return and test intent
Made-with: Cursor
* fix(oci): address code review findings from Greptile
- P1: Fix signing URL mismatch with custom api_base by accepting
api_base parameter in transform_embedding_request
- P2: Remove encoding_format from supported params (OCI does not
support it, was silently dropped)
- P2: Raise ValueError for token-array inputs instead of silently
converting to string representation
- Add test for token-list rejection
* fix(mcp): add STS AssumeRole support for MCP SigV4 authentication
MCPSigV4Auth only supported static AWS credentials or the boto3 default
credential chain. Production Kubernetes environments typically authenticate
via IAM role assumption (sts:AssumeRole), which was not possible.
Add aws_role_name and aws_session_name parameters to the MCP SigV4 auth
stack. When aws_role_name is provided, MCPSigV4Auth calls sts:AssumeRole
to obtain temporary credentials before signing requests. Explicit keys,
if also provided, are used as the source identity for the STS call;
otherwise ambient credentials (pod role, instance profile) are used.
* fix: stop logging credential values and add missing redaction patterns
Replaces raw credential values in debug/error log messages with
boolean presence checks or type names. Adds PEM block, GCP token,
JWT, SAS token, and service-account blob patterns to the redaction
filter. Fixes private_key pattern to capture full PEM blocks instead
of stopping at the first whitespace.
Addresses: Vertex AI credential JSON (including RSA private key)
being logged to stderr on health check failures.
* fix: log only field names for UserAPIKeyAuth, not full object
* style: apply black formatting to experimental_mcp_client/client.py
* style: fix black/isort formatting and mypy error in proxy_server.py
- Fix black formatting in experimental_mcp_client/client.py (done in prev commit)
- Fix black/isort formatting in key_management_endpoints.py, proxy_server.py, transformation.py
- Fix mypy: iterate over optional list safely (access_group_ids or []) in proxy_server.py
* fix(test): patch check_migration.verbose_logger directly to fix xdist ordering issue
When test_proxy_cli.py tests run before test_check_migration.py in the same
xdist worker, litellm.proxy.db.check_migration is already in sys.modules.
Patching litellm._logging.verbose_logger has no effect on the already-bound
reference. Patch the correct target (check_migration.verbose_logger) and
import the module before patching so the order doesn't matter.
* fix(mypy): make api_base Optional in PydanticAIProviderConfig to match base class signature
---------
Co-authored-by: Ihsan Soydemir <soydemir.ihsan@gmail.com>
Co-authored-by: Milan <milan@berri.ai>
Co-authored-by: Daniel Gandolfi <danielgandolfi@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
Tests cover: matching deployments, wrong team_id filtering, string-encoded
model_info, empty results, invalid model_info, and mixed deployment filtering.
Also updates MockPrismaClient.find_many to support the new startswith query.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add brave/search to model_prices_and_context_window.json (#25042)
Brave Search is supported by litellm as a search provider (documented at
docs.litellm.ai/docs/search/brave and listed in provider_endpoints_support.json)
but was missing from model_prices_and_context_window.json, making it invisible
to any code that discovers search providers from litellm.model_cost.
Cost: $0.005/query ($5 per 1,000 requests) per https://brave.com/search/api/
* feat(models): add NVIDIA Nemotron 3 Super 120B on Bedrock (#24588)
* feat(models): add NVIDIA Nemotron 3 Super 120B on Bedrock
Add model definition for nvidia.nemotron-3-super-120b-a12b-v1 via
Bedrock Converse API with pricing, context window (256k/32k), and
capability flags (function calling, tool choice, system messages).
* fix model ID to nvidia.nemotron-super-3-120b + add tests
Correct the Bedrock model ID from nvidia.nemotron-3-super-120b-a12b-v1
(NVIDIA's internal name) to nvidia.nemotron-super-3-120b (the actual
AWS Bedrock programmatic model ID). Add unit tests verifying model
resolution, pricing, and context window.
* fix(proxy): allow JWT auth for /v1/mcp/server sub-paths (#24698)
mcp_routes only contained "/v1/mcp/server" (exact match). Starlette's
compile_path produces an end-anchored regex, so sub-paths like
/register, /health, /submissions, /oauth/* all failed the JWT
allowed_routes_check. Add a {path:path} wildcard entry so all
sub-paths are covered.
---------
Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Adds tenant_id, client_id, and client_secret to the Azure provider entry
in provider_create_fields.json so the credential add/edit modals and the
add-model form surface Service Principal auth as an alternative to api_key.
The Azure handler already reads these fields from litellm_params at request
time via get_azure_ad_token(); this change makes them inputtable from the
UI without code changes to the React components (the form is driven by
GET /public/providers/fields).
