Renames the new per-guardrail opt-out field from `disabled_global_guardrails`
to `opted_out_global_guardrails` to eliminate the one-character collision with
the legacy `disable_global_guardrails` boolean kill switch. Adds a type guard
on the new gate so a misnamed bool can't crash the guardrail check. Filters
duplicates out of the team-edit guardrail display for legacy teams that have a
global name persisted in `metadata.guardrails` from before this PR. Drops the
unused `isGuardrailsLoading` and `guardrailsError` destructures left in
AddModelForm after the hook refactor.
Adds Python tests for the new gate behavior (root, litellm_metadata, metadata,
non-matching name, empty list, malformed bool value, opt-in coexistence) and
extends useGuardrails.test.ts to exercise the global / optional partition
logic that the rebuilt hook performs in its `select` transform.
Wires the legacy kill switch and the new opt-out list together in the team
edit form so they can never fall out of sync:
- Toggling the kill switch reactively updates the Guardrails Select via
`onValuesChange` — switch on strips all globals from the selection, switch
off re-adds them. Existing opt-in extras are preserved either way.
- When the switch is on, global options in the Select are individually
disabled (greyed out) so the user can still manage opt-in guardrails but
cannot accidentally re-enable a global the kill switch is bypassing.
- The save handler writes both fields together: `disable_global_guardrails`
reflects the switch, and `opted_out_global_guardrails` is set to either
every global (when the switch is on) or the user's explicit opt-outs.
- `effectiveGuardrails` for the form's initialValues honors the kill switch
on legacy teams so the form opens in a state that matches what the runtime
gate is actually doing — fixes the visual lie where chips appeared active
while the switch was bypassing them.
The backend gate already reads the list as the primary path with the bool
as a fallback, so untouched legacy teams keep working until they get edited,
at which point they migrate naturally.
Rename disable_global_guardrail → disable_global_guardrails to match
the key name used by litellm_pre_call_utils.py, the API endpoints,
and the UI when propagating key/team metadata.
The singular form was introduced in PR #16983 and has never matched
the plural form written by the rest of the codebase, so the feature
silently did nothing.
Re-applies fix originally from #25488. Original commit could not be
merged due to missing signature.
Co-Authored-By: Remi Mabon <remi.mabon@redcare-pharmacy.com>
* fix(s3): add retry with exponential backoff for transient S3 503/500 errors
S3 occasionally returns 503 "Slow Down" during PUT operations when
request rates spike above partition limits. The current code makes a
single upload attempt via httpx — unlike boto3, httpx has no built-in
retry for transient S3 errors. Failed uploads permanently lose the
request's audit/logging data.
Add exponential backoff retry (3 attempts, 1s/2s delays) for S3
500/503 responses in both async_upload_data_to_s3 and
upload_data_to_s3. Logs a warning on each retry with the S3 object
key for observability.
In production we observed ~18 permanent S3 upload failures per day
(124 over 7 days) — all transient 503s that would have succeeded on
a single retry.
* test(s3): add unit tests for S3 upload retry logic
Tests cover:
- Async retry on 503 (succeeds on second attempt)
- Async retry on 500
- Exhausted retries on persistent 503 (calls handle_callback_failure)
- No retry on 4xx errors (403)
- Sync retry on 503
* style(s3): move time import to module level
Address review feedback: move `import time` from inside
upload_data_to_s3 to the top-level imports per project style guide.
* build: migrate packaging metadata to uv
* ci: move automation and local tooling to uv
* docker: migrate image builds and runtime setup to uv
* docs: update install and deployment guidance for uv
* chore: align auxiliary scripts and tests with uv
* test: harden test_litellm isolation
* fix: keep release and health check images self-contained
* build: pin uv tooling and health check deps
* test: isolate bedrock image request formatting from suite state
* test: cover sandbox executor requirements flow
* ci: fix circleci no-op command steps
* ci: fix circleci publish workflow parsing
* fix: stabilize remaining uv migration CI checks
* ci: increase matrix test timeout headroom
* fix: restore published docker and license coverage
* fix: restore proxy runtime build parity
* fix: restore proxy extras parity and venv migrations
* ci: persist uv path across circleci steps
* fix: keep psycopg binary in default test env
* docker: preserve prisma cache across stages
* test: run local proxy checks through uv python
* build: restore runtime deps moved into ci
* build: refresh uv lock after upstream merge
* fix: restore module import in test_check_migration after merge
The conflict resolution imported only the function but the test body
references check_migration as a module throughout.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: revert dependency promotions, remove nodejs-wheel-binaries, fix Docker layer caching
- Move google-generativeai, Pillow, tenacity back to ci group (they are
lazily imported and bloat the base SDK install needlessly)
- Remove nodejs-wheel-binaries from extra_proxy and proxy-dev (redundant
in Docker where system Node.js is already installed via apk)
- Remove all nodejs-wheel node replacement and venv npm patching blocks
from Dockerfiles since the wheel is no longer installed
- Add --no-default-groups to CodSpeed benchmark workflow so the benchmark
environment matches the old minimal pip install footprint
- Apply standard uv two-phase Docker pattern: copy metadata first, install
deps (cached layer), then copy source and install project
- Replace CircleCI enterprise no-op with proper uv sync command
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: regenerate uv.lock after removing nodejs-wheel-binaries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): use cache/restore instead of cache to prevent cache poisoning
The old workflow used actions/cache/restore (read-only). The uv migration
changed it to actions/cache (read-write), which zizmor flags as a cache
poisoning risk. Restore the safer read-only variant.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): disable setup-uv built-in cache to silence cache-poisoning alert
The setup-uv action enables caching by default, which zizmor flags as a
cache poisoning risk. Disable it since we already use a read-only
cache/restore step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): disable setup-uv cache in publish workflow
Silences zizmor cache-poisoning alert. Publishing workflow runs
infrequently on protected branches so caching adds no real benefit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(test): remove duplicate verbose_logger mock in test_check_migration
The logger was patched twice — first via mocker.patch() then via
mocker.patch.object(autospec=True). The second call fails because
autospec cannot inspect an already-mocked attribute. Remove the
redundant first patch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): free disk space before Docker build in test-server-root-path
The Dockerfile.non_root build ran out of disk on the CI runner. Remove
Android SDK, .NET, Boost, and GHC toolchains (~12GB) to free space.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The deployment hook now converts stream=True→False in wrapper_async's
scope so the streaming early-return path is skipped and logging executes.
logging_obj.stream is synced after the hook, and the original stream
intent is recovered for the short-circuit path.
Made-with: Cursor
* fix(vertex_ai): support pluggable (executable) credential_source for WIF auth (#24700)
The WIF credential dispatch in load_auth() only handled identity_pool and
aws credential types. When credential_source.executable was present (used
for Azure Managed Identity via Workload Identity Federation), it fell
through to identity_pool.Credentials which rejected it with MalformedError.
Add dispatch to google.auth.pluggable.Credentials for executable-type
credential sources, following the same pattern as the existing identity_pool
and aws helpers.
Fixes authentication for Azure Container Apps → GCP Vertex AI via WIF
with executable credential sources.
* feat(logging): add component and logger fields to JSON logs for 3rd p… (#24447)
* feat(logging): add component and logger fields to JSON logs for 3rd party filtering
* Let user-supplied extra fields win over auto-generated component/logger, tighten test assertions
* Feat - Add organization into the metrics metadata for org_id & org_alias (#24440)
* Add org_id and org_alias label names to Prometheus metric definitions
* Add user_api_key_org_alias to StandardLoggingUserAPIKeyMetadata
* Populate user_api_key_org_alias in pre-call metadata
* Pass org_id and org_alias into per-request Prometheus metric labels
* Add test for org labels on per-request Prometheus metrics
* chore: resolve test mockdata
* Address review: populate org_alias from DB view, add feature flag, use .get() for org metadata
* Add org labels to failure path and verify flag behavior in test
* Fix test: build flag-off enum_values without org fields
* Gate org labels behind feature flag in get_labels() instead of static metric lists
* Scope org label injection to metrics that carry team context, remove orphaned budget label defs, add test teardown
* Use explicit metric allowlist for org label injection instead of team heuristic
* Fix duplicate org label guard, move _org_label_metrics to class constant
* Reset custom_prometheus_metadata_labels after duplicate label assertion
* fix: emit org labels by default, remove flag, fix missing org_alias in all metadata paths
* fix: emit org labels by default, no opt-in flag required
* fix: write org_alias to metadata unconditionally in proxy_server.py
* fix: 429s from batch creation being converted to 500 (#24703)
* add us gov models (#24660)
* add us gov models
* added max tokens
* Litellm dev 04 02 2026 p1 (#25052)
* fix: replace hardcoded url
* fix: Anthropic web search cost not tracked for Chat Completions
The ModelResponse branch in response_object_includes_web_search_call()
only checked url_citation annotations and prompt_tokens_details, missing
Anthropic's server_tool_use.web_search_requests field. This caused
_handle_web_search_cost() to never fire for Anthropic Claude models.
Also routes vertex_ai/claude-* models to the Anthropic cost calculator
instead of the Gemini one, since Claude on Vertex uses the same
server_tool_use billing structure as the direct Anthropic API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(anthropic): pass logging_obj to client.post for litellm_overhead_time_ms (#24071)
When LITELLM_DETAILED_TIMING=true, litellm_overhead_time_ms was null for
Anthropic because the handler did not pass logging_obj to client.post(),
so track_llm_api_timing could not set llm_api_duration_ms. Pass
logging_obj=logging_obj at all four post() call sites (make_call,
make_sync_call, acompletion, completion). Add test to ensure make_call
passes logging_obj to client.post.
Made-with: Cursor
* sap - add additional parameters for grounding
- additional parameter for grounding added for the sap provider
* sap - fix models
* (sap) add filtering, masking, translation SAP GEN AI Hub modules
* (sap) add tests and docs for new SAP modules
* (sap) add support of multiple modules config
* (sap) code refactoring
* (sap) rename file
* test(): add safeguard tests
* (sap) update tests
* (sap) update docs, solve merge conflict in transformation.py
* (sap) linter fix
* (sap) Align embedding request transformation with current API
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) mock commit
* (sap) run black formater
* (sap) add literals to models, add negative tests, fix test for tool transformation
* (sap) fix formating
* (sap) fix models
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) commit for rerun bot review
* (sap) minor improve
* (sap) fix after bot review
* (sap) lint fix
* docs(sap): update documentation
* fix(sap): change creds priority
* fix(sap): change creds priority
* fix(sap): fix sap creds unit test
* fix(sap): linter fix
* fix(sap): linter fix
* linter fix
* (sap) update logic of fetching creds, add additional tests
* (sap) clean up code
* (sap) fix after review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) add a possibility to put the service key by both variants
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) update test
* (sap) update service key resolve function
* (sap) run black formater
* (sap) fix validate credentials, add negative tests for credential fetching
* (sap) fix validate credentials, add negative tests for credential fetching
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) fix after bot review
* (sap) lint fix
* (sap) lint fix
* feat: support service_tier in gemini
* chore: add a service_tier field mapping from openai to gemini
* fix: use x-gemini-service-tier header in response
* docs: add service_tier to gemini docs
* chore: add defaut/standard mapping, and some tests
* chore: tidying up some case insensitivity
* chore: remove unnecessary guard
* fix: remove redundant test file
* fix: handle 'auto' case-insensitively
* fix: return service_tier on final steamed chunk
* chore: black
* feat: enable supports_service_tier to gemini models
* Fix get_standard_logging_metadata tests
* Fix test_get_model_info_bedrock_models
* Fix test_get_model_info_bedrock_models
* Fix remaining tests
* Fix mypy issues
* Fix tests
* Fix merge conflicts
* Fix code qa
* Fix code qa
* Fix code qa
* Fix greptile review
---------
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: Josh <36064836+J-Byron@users.noreply.github.com>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Alperen Kömürcü <alperen.koemuercue@sap.com>
Co-authored-by: Vasilisa Parshikova <vasilisa.parshikova@sap.com>
Co-authored-by: Lin Xu <lin.xu03@sap.com>
Co-authored-by: Mark McDonald <macd@google.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* update bedrock models in tests
* updated more tests and model_prices_and_context_window
* fix model id and pricing
* replace more sonnet models
* update tests
* git push
* update pricing
* flaky total cost
* monkey patch
* relax the cost change
* fix and revert some changes
* revert the pricing
* chore: move cost/pricing changes to bedrock-cost-fixes branch
* chore: split Bedrock file-api beta stripping to separate branch
Removes strip_unsupported_file_api_betas_for_bedrock_invoke from this branch;
see litellm_bedrock_invoke_strip_file_api_betas for that fix.
Made-with: Cursor
Verify that spend_logs_metadata is correctly merged into combined_metadata
and flows through to Prometheus custom labels. Tests cover: basic extraction,
precedence when keys overlap, all three metadata sources combined, and None
handling.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Skip short-circuit for providers that have a BaseAnthropicMessagesConfig
(bedrock, vertex_ai, azure_ai, anthropic) — they use the agentic loop
which includes a follow-up LLM synthesis step. Short-circuiting would
return raw search text instead of an LLM-synthesized answer.
- Add fallback to litellm.get_llm_provider() for custom_llm_provider
derivation when litellm_params is overwritten by kwargs.
- Add test for bedrock guard.
Addresses Greptile review comments #3 and #4.
- Replace hand-rolled _extract_search_query with existing
get_last_user_message from common_utils
- Use full UUID (str(uuid.uuid4())) to match codebase convention
- Move uuid import to module level per CLAUDE.md
Addresses Greptile review feedback:
- Save original stream flag before pre-request hooks convert it, so
streaming callers get SSE events instead of a plain dict
- Propagate custom_llm_provider derived inside _execute_pre_request_hooks
when it was not explicitly passed by the caller
- Add tests covering both scenarios
For providers like github_copilot that don't natively support web search,
Claude Code's search sub-conversations were falling through to the adapter
path which strips the web_search tool and has no stream reconversion.
Instead of routing search requests through the full LLM pipeline, detect
web-search-only requests early (all tools are web_search, simple prompt)
and execute the search directly via Tavily/Perplexity, returning a
synthetic Anthropic response. No adapter, no backend LLM call needed.
Fixes#21733
* add DD Tracing (#24033)
* feat(models): add Azure GPT-5.4 mini and nano variants (#24045)
Add `azure/gpt-5.4-mini` and `azure/gpt-5.4-nano` to the model
database with official pricing from Azure OpenAI:
- GPT-5.4 mini: $0.75/M input, $0.075/M cached, $4.5/M output
- GPT-5.4 nano: $0.20/M input, $0.02/M cached, $1.25/M output
Both models support:
- 1.05M input / 128K output context window
- Chat, batch, and responses endpoints
- Function calling, tools, vision, reasoning
- Prompt caching with automatic tiered pricing
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add new model pricing details for volcengine Doubao-Seed-2.0 series (#23871)
Add entries for volcengine Doubao-Seed-2.0 series
* fix(mcp): support refresh_token grant type in OAuth token endpoint (#23701)
* fix(mcp): support refresh_token grant type in OAuth token endpoint (#23700)
The .well-known/oauth-authorization-server metadata advertises
refresh_token as a supported grant type, but the token endpoint
rejected it with HTTP 400. This adds refresh_token grant support
so MCP clients can refresh expired tokens without re-authenticating.
* test(mcp): add tests for refresh_token grant type in OAuth token endpoint
* fix(mcp): move code_verifier guard into authorization_code branch
code_verifier is only relevant for authorization_code grants (PKCE).
Move it inside the else branch so it doesn't apply to refresh_token.
* fix(mcp): guard None client_secret and forward scope in token exchange
- Conditionally include client_secret in form data to prevent httpx
from sending the literal string "None" (applies to both
authorization_code and refresh_token branches)
- Forward optional scope parameter per RFC 6749 §6, allowing clients
to request a subset of originally-granted scopes on refresh
* fix(mcp): validate code param in authorization_code grant
Guard against None code being form-encoded as literal string "None"
by httpx, symmetric with the existing refresh_token guard.
* docs: add incident report for guardrail logging secret exposure (#24059)
Add blog post documenting the guardrail logging path exposing internal
request data (e.g. Authorization headers) in spend logs and OTEL traces.
Fix available in LiteLLM 1.82.3+.
Made-with: Cursor
* [Fix] Datadog LLM Observability tags format (env, service, version missing) (#23673)
* tag fix
* greptile comment
* fix(ci): stabilize 6 failing CI jobs
1. mypy: remove duplicate type annotation for token_data in discoverable_endpoints.py
2. integrations tests: add parameterized to CI test deps
3. doc quality: document OTEL_IGNORE_CONTEXT_PROPAGATION env key
4. security: allowlist CVE-2026-2673, CVE-2026-3644, CVE-2026-4224 (no fix available)
5. proxy_store_model_in_db: fix missing x-litellm-call-id header on error responses
6. google tests: add --retries 3 for transient Vertex AI rate limits
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(streaming): handle RuntimeError during model_copy in streaming handler
The race condition occurs when model_copy(deep=True) tries to deepcopy
_hidden_params dict while it's being concurrently modified by logging
callbacks. Fall back to shallow copy if the deep copy fails.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(cost): handle non-string traffic_type in cost calculator + add retries
1. Fix AttributeError in _map_traffic_type_to_service_tier when traffic_type
is an integer (cast to str before calling .upper()). This was causing
pass-through vertex spend logging to fail silently.
2. Add --retries to llm_translation_testing for flaky external API calls.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: ExMatics HydrogenC <33123710+HydrogenC@users.noreply.github.com>
Co-authored-by: Jack Venberg <jack.venberg@rover.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
LangSmith reads the Cost column from outputs.usage_metadata.total_cost,
but LangsmithLogger._prepare_log_data never wrote to that key. The
response_cost was already computed in StandardLoggingPayload but was
not forwarded to the outputs dict.
Inject usage_metadata with input_tokens, output_tokens, total_tokens,
and total_cost into the outputs dict so LangSmith can display cost.
Fixes#24001
Made-with: Cursor
Replace per-request `httpx.AsyncClient` with `get_async_httpx_client`
to avoid the +500ms latency penalty from creating new clients per
request. Updates tests to mock the cached client factory.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix genuine regression in responses_api_bridge_check where the second
call assigned to `model_info` instead of `responses_api_model_info`,
preventing gpt-5.4 + tools + reasoning_effort from routing to the
Responses API bridge.
Also update outdated tests:
- Vantage tests: match "csv" file key and use supported column names
- Anthropic caching test: add "type": "custom" to expected tool payload
- Claude Agent SDK test: remove non-deterministic LLM content assertion
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests were outdated after #23472 added pagination (take/order) to find_many
and stale-row cleanup via update_many. Updated assertions to match new call
signatures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- DELETE /vantage/delete now removes the in-memory VantageLogger from
litellm.callbacks via remove_callbacks_by_type, preventing the
scheduler from continuing to fire exports with stale credentials
- Add test_should_cast_decimal_columns_to_float covering the
Decimal→Float64 cast in FocusCsvSerializer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use timedelta(hours=1) instead of replace(hour=hour+1) in _window()
to avoid ValueError when hour=23
- Add test_should_batch_by_row_count covering the >10K rows batching
path (previously only the 2 MB size-limit path was tested)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a pluggable Vantage destination to the existing FOCUS export pipeline,
enabling LiteLLM to export spend data in FOCUS format directly to Vantage's
cost-import API. Supports automatic hourly exports via scheduled background job,
with admin API endpoints for manual control and configuration. Includes CSV
serializer, batching for 10K row / 2MB API limits, and enriched Tags JSON with
team/user/key metadata for Vantage Token Allocation feature.
- Add CSV serializer (FocusCsvSerializer) for FOCUS data
- Add Vantage API destination with automatic batching
- Add VantageLogger that wraps FocusLogger with Vantage defaults
- Add proxy endpoints: /vantage/{init,settings,export,dry-run,delete}
- Register "vantage" callback in logger registry and literal type
- Wire up background job in proxy_server.py startup
- Populate Tags column with JSON metadata (team_id, user_id, user_email, etc.)
- Add 14 unit tests covering serializer, destination, and factory
All tests pass (23 focus tests total, no regressions).
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When guardrails return the full data dict (e.g. guardrails_ai), the
guardrail response logged to spend logs and OTEL traces could contain
data["secret_fields"].raw_headers with plaintext Authorization headers.
This adds a pop("secret_fields") in the guardrail logging path,
matching the existing pattern used by Langfuse and Arize integrations.
Tested: Verified fix removes secret_fields/raw_headers/authorization
from both /spend/logs/ui responses and OTEL trace span attributes.
