mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-07 04:24:12 +00:00
2b00ea9ee486ee86300a41a2d85df214ab1e86d1
9701
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Mateo Wang
|
2b00ea9ee4 |
test(ci): skip Fireworks tests on 404 + Gemini image-size test on 429
Four pre-existing flakes on main that gate this branch's workflow even though they're unrelated to the reasoning_effort_grid suite: 1. tests/local_testing/test_completion.py::test_completion_fireworks_ai 2. tests/local_testing/test_completion_cost.py::test_completion_cost_fireworks_ai[fireworks_ai/llama-v3p3-70b-instruct] 3. tests/llm_translation/test_fireworks_ai_translation.py::test_document_inlining_example[False] The Fireworks-hosted `llama-v3p3-70b-instruct` deployment is currently returning 404 "Model not found, inaccessible, and/or not deployed". These tests pass when the model is deployed; the issue is upstream capacity, not our code path. Wrap the live call in a try/except that pytest.skip's on litellm.NotFoundError so a Fireworks deployment hiccup no longer fails CI for unrelated PRs. 4. tests/llm_translation/test_gemini.py::test_gemini_image_size_limit_exceeded The test fetches the 32MB "Blue Marble 2002" image from Wikimedia to exercise the 50MB image-size cap. CI runners share an IP pool with noisy traffic, so Wikimedia routinely returns HTTP 429. The size-limit check never gets a chance to fire. Catch the 429 BadRequestError and pytest.skip in that case. None of these belong on this PR conceptually, but they're included per request to unblock the workflow before morning. |
||
|
Mateo Wang
|
e29ea53c31 |
fix(reasoning_effort_grid): classify status by exception status_code, not class
The anthropic_messages route wraps client-side BadRequestError as AnthropicError (a BaseLLMException subclass) with status_code=400, so "except BadRequestError" missed those cells and they fell through to the generic Exception arm, returning 500 instead of the expected 400. Replace the isinstance-on-BadRequestError check with a tiny classifier that prefers BadRequestError membership, then falls back to the exception's status_code attribute (set by every BaseLLMException subclass), then 500. Apply to both _call_chat and _call_messages for consistency. Fixes the 13 CircleCI llm_translation_testing failures on bedrock_invoke_messages cells where the effort was disabled / invalid / empty / xhigh-on-unsupported / max-on-unsupported. |
||
|
Mateo Wang
|
90cdbb92d7 |
fix(tests): use litellm.anthropic_messages entrypoint + drop unstable openapi field
Two CI failures, both pre-existing in different ways:
1. reasoning_effort_grid: all 33 bedrock_invoke_messages cells failed with
AttributeError("module 'litellm' has no attribute 'messages'"). litellm
exposes the async Anthropic Messages entrypoint as litellm.anthropic_messages
(via "from .llms.anthropic.experimental_pass_through.messages.handler
import *" in litellm/__init__.py), not litellm.messages.acreate. Swap
the call.
2. tests/test_litellm/interactions/test_openapi_compliance.py::TestResponseCompliance::test_interaction_response_fields
asserts the live Google spec contains "steps". Google's spec has churned
through "outputs" -> "steps" -> neither, and presently carries neither.
The test broke on main as soon as upstream dropped "steps"; pulling the
key off the assert list realigns the test with the live schema. Re-add
the per-turn output field once upstream stabilizes on a name.
The openapi-compliance fix doesn't belong to this PR conceptually but is
included here per request to unblock CI before the morning.
|
||
|
Mateo Wang
|
d084782661 | Merge branch 'litellm_grid-v4-e2e-tests-cZRwz' of http://127.0.0.1:41997/git/BerriAI/litellm into litellm_grid-v4-e2e-tests-cZRwz | ||
|
Mateo Wang
|
f77324d766 |
refactor(tests): move reasoning_effort grid suite under llm_translation, drop v4 naming
- Drop the "v4" suffix throughout: it referred to the QA sweep iteration,
not this test suite. There's only one regression suite, so just call it
reasoning_effort_grid.
- Move tests/test_litellm/reasoning_effort_grid_v4/ -> tests/llm_translation/
reasoning_effort_grid/. Two reasons:
1. The parent tests/test_litellm/conftest.py installs an autouse fixture
(isolate_host_aws_config) that clears every AWS_* env var before each
test, which would silently skip every Bedrock cell.
2. tests/llm_translation/conftest.py already wires up the Redis-backed
VCR persister and auto-applies @pytest.mark.vcr to every collected
item via apply_vcr_auto_marker_to_items. Living under that conftest
means the suite gets cassette replay for free -- first CI run with
provider creds records 231 cassettes, every subsequent run replays
them with no live spend.
- Trim the suite's own conftest down to just the wire_capture fixture; the
inherited llm_translation conftest covers the VCR plumbing.
- Drop the dedicated reasoning_effort_grid_v4_e2e CircleCI job. The existing
llm_translation_testing job globs tests/llm_translation/**/test_*.py, so
the suite is gated by an existing job with no new wiring.
|
||
|
Cursor AgentandYassin Kortam
|
51dff1ff79 |
fix(reasoning_effort_grid_v4): grant sonnet-4-6 entries the max-effort cap
The runtime _validate_effort_for_model allows effort='max' for any Claude 4.6 model (opus or sonnet), and model_prices_and_context_window sets supports_max_reasoning_effort: true for claude-sonnet-4-6. The grid spec previously gave sonnet-4-6 entries _CAPS_NONE, so expected() returned status=400 for effort='max', which mismatched the runtime's status=200 and caused 6 cells (one per route) to fail. Rename _CAPS_OPUS_4_6 to _CAPS_4_6 (since the cap set is shared by opus and sonnet 4.6) and assign it to all sonnet-4-6 entries. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Cursor AgentandYassin Kortam
|
432742778a |
fix(reasoning_effort_grid_v4): cleanup unused fixture, parse converse body, guard budget tokens
- Remove unused vertex_credentials_path fixture (and now-unused os import) from conftest.py. - Parse Bedrock Converse complete_input_dict (logged as a JSON string by converse_handler.py) before passing to _assert_cell, so dict accessors work uniformly across routes. - Extend _BUDGET_TOKENS with xhigh and max entries so the budget-mode branch in expected() cannot KeyError if a future budget model gains the matching cap. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Mateo Wang
|
fec4ae69e0 |
test(ci): add reasoning_effort grid v4 e2e regression suite
Encode the 231-cell QA sweep (21 provider x model combos x 11 effort values) from #27039 / #27074 as an automated CircleCI-gated regression suite. Each cell hits the real provider endpoint, captures the outgoing wire body via a pre-call CustomLogger, and asserts: - thinking.type, output_config.effort, thinking.budget_tokens, max_tokens in the captured request body (regression signal for silent drops/strips in any provider transformation) - HTTP status (200 vs BadRequestError -> 400) returned by litellm (regression signal for clean-error vs leaked-500 mappings) The matrix is encoded as a small rule set keyed by (model_mode, effort) plus per-model xhigh/max capability overrides, then expanded across the five chat-completion routes (Anthropic direct, Azure AI Foundry, Vertex AI, Bedrock Converse, Bedrock Invoke /chat) and the Bedrock Invoke /v1/messages route. Cells skip at runtime when the route's provider env vars are absent, so PR builds without credentials no-op gracefully. Wired into CircleCI as the reasoning_effort_grid_v4_e2e job behind the existing main / litellm_* branch filter. |
||
|
|
c459646162 |
feat: add OTEL GenAI latest-experimental semantic convention support (#27418)
- Introduce `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` opt-in that switches OTEL traces to conform with the OpenTelemetry GenAI semantic conventions specification
- Extract all semconv behavior into a new `OTELGenAISemconvMixin` class in `gen_ai_semconv.py`, mixed into `OpenTelemetry` to keep concerns separated
- In semconv mode, span name follows `{operation} {model}` pattern (e.g. `chat gpt-4`) and span kind is set to `CLIENT` instead of legacy `litellm_request`
- Replace `gen_ai.system` with `gen_ai.provider.name` and drop `llm.is_streaming` in semconv mode; add `gen_ai.request.{frequency_penalty,presence_penalty,top_k,seed,stop_sequences,stream,choice.count}` and `gen_ai.usage.cache_{creation,read}.input_tokens` attributes
- Replace per-message `gen_ai.content.prompt` / per-choice `gen_ai.content.completion` log events with a single consolidated `gen_ai.client.inference.operation.details` event; omit `gen_ai.input/output.messages` when content capture is disabled
- Suppress the non-standard `raw_gen_ai_request` child span entirely in semconv mode
- Support both programmatic (`OpenTelemetryConfig.semconv_stability_opt_in` field) and environment variable activation; the two sources are unioned so either or both can enable the opt-in
- Extract OTEL SDK `LogRecord` / `SeverityNumber` version-compatibility shim into a reusable `_otel_log_types()` static method to deduplicate the `< 1.39.0` / `>= 1.39.0` import branching
- Add 30+ unit tests covering opt-in gating, span naming, attribute emission/omission rules, stop sequence normalization, cache token attributes, and the consolidated event lifecycle
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
|
||
|
yuneng-jiangandGitHub
|
361a84ccb0 |
Merge pull request #28022 from BerriAI/litellm_/hardcore-albattani-e592b4
fix(proxy): make /config/update env-var encryption idempotent |
||
|
Mateo WangandGitHub
|
2c733c00f5 |
chore(ci): modernize model references in tests and configs (#27856)
* test: modernize models used in CircleCI e2e test suites
Replaces obsolete models (gpt-4o, gpt-4o-mini, gpt-3.5-turbo,
claude-3-5-sonnet-20240620, claude-sonnet-4-20250514) with current
equivalents across the e2e_openai_endpoints and
proxy_e2e_anthropic_messages_tests CircleCI jobs.
- gpt-4o -> gpt-5.5 (responses API e2e tests)
- gpt-4o-mini -> gpt-5-mini (websocket responses, oai_misc_config)
- gpt-4o-mini-2024-07-18 -> gpt-4.1-mini-2025-04-14 (fine-tuning,
still actively fine-tunable)
- gpt-4 / gpt-3.5-turbo target_model_names example -> gpt-5.5 /
gpt-5-mini
- bedrock claude-3-5-sonnet-20240620 batch entry -> haiku-4-5-20251001
(also aligning oai_misc_config model_name with what
test_bedrock_batches_api.py actually requests)
- bedrock claude-sonnet-4-20250514 (deprecated, retires 2026-06-15)
-> claude-sonnet-4-5-20250929
* test: point bedrock-claude-sonnet-4 alias at Sonnet 4.6, not 4.5
Greptile/Cursor flagged that after the previous commit, the
bedrock-claude-sonnet-4 alias collided with bedrock-claude-sonnet-4.5
(both pointed to claude-sonnet-4-5-20250929). Rename to
bedrock-claude-sonnet-4.6 and point it at the Sonnet 4.6 Bedrock ID
(us.anthropic.claude-sonnet-4-6, already in the litellm model
registry) so the alias name matches the underlying model version.
* test: modernize models across remaining CI-mounted configs & tests
Expands the modernization sweep to all CircleCI-mounted proxy configs
and to test directories where the model literal is a fixture/route key
(not the test's subject).
Config changes:
- proxy_server_config.yaml: bump gpt-3.5-turbo / gpt-3.5-turbo-1106 /
gpt-4o / gemini-1.5-flash / dall-e-3 underlying models; rename
gpt-3.5-turbo-end-user-test alias to gpt-5-mini-end-user-test; bump
text-embedding-ada-002 underlying to text-embedding-3-small. User-
facing aliases (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, etc.)
preserved for backward compatibility with tests.
- simple_config.yaml, otel_test_config.yaml, spend_tracking_config.yaml:
bump gpt-3.5-turbo underlying to gpt-5-mini.
- pass_through_config.yaml: claude-3-5-sonnet / claude-3-7-sonnet /
claude-3-haiku entries replaced with claude-sonnet-4-5 / claude-
haiku-4-5 / claude-opus-4-7.
- oai_misc_config.yaml: align alias name with the gpt-5-mini rename.
Test changes (proactive: claude-sonnet-4-20250514 / claude-opus-4-
20250514 retire 2026-06-15):
- tests/llm_translation/test_anthropic_completion.py: bump 3 references
+ paired Vertex AI ID to claude-sonnet-4-5.
- tests/llm_translation/test_optional_params.py: bump 2 references.
- tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py
and test_bedrock_anthropic_messages_test.py: bump router fixtures
using the deprecated model IDs.
- tests/pass_through_unit_tests/base_anthropic_messages_tool_search_test.py:
modernize docstring examples.
- tests/test_end_users.py: update references to renamed alias.
* test: modernize placeholder model literals in router_unit_tests
Mass replace_all on fixture/placeholder model literals across the
router_unit_tests/ suite (model name is a routing key / label, not the
test subject). Sub-agent sweep so far — additional commits will follow
for logging_callback_tests/, enterprise/, top-level tests/test_*.py,
and other CI-mounted dirs.
Mappings applied:
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 / claude-3-opus-20240229 /
claude-3-haiku-20240307 / claude-3-5-sonnet-20240620 ->
claude-sonnet-4-5-20250929 / claude-opus-4-7 /
claude-haiku-4-5-20251001 as appropriate
Explicitly preserved:
- gpt-4o-mini-* variants (transcribe, tts, etc.) where they're current
- gpt-4-turbo / gpt-4-vision-preview / gpt-4-0613 (subject literals)
- JSONL batch body literals
- Mock LLM response model fields (must match upstream)
- Fake/mock identifiers
* test: modernize placeholder model literals across remaining CI suites
Sub-agent sweep across logging_callback_tests/, guardrails_tests/,
enterprise/, pass_through_unit_tests/, otel_tests/,
llm_responses_api_testing/, batches_tests/, spend_tracking_tests/,
litellm_utils_tests/, unified_google_tests/, and a few top-level
tests/test_*.py files where the model literal is a fixture or
placeholder (router model_list, mock standard logging payload, mock
callback data) rather than the test's subject.
Mappings applied (see scope notes below):
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5.5 (corrected from initial gpt-5 — bare gpt-5
is not a valid OpenAI alias; only gpt-5.5 / gpt-5.4 / gpt-5.2-codex
/ gpt-5-mini exist)
- gpt-4o-mini (bare) -> gpt-5-mini
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 -> claude-sonnet-4-5-20250929
- claude-3-opus-20240229 -> claude-opus-4-7
- claude-3-haiku-20240307 -> claude-haiku-4-5-20251001
- claude-3-5-sonnet-20240620/20241022 -> claude-sonnet-4-5-20250929
- claude-3-7-sonnet-20250219 -> claude-sonnet-4-6
- gemini-1.5-flash -> gemini-2.5-flash
- gemini-1.5-pro -> gemini-2.5-pro
Explicitly preserved (not modernized):
- llm_translation/ tests where model is the SUBJECT (provider-specific
translation/transformation logic). Only the deprecated 20250514
references were already bumped in a prior commit.
- Cost-calc / tokenizer subject tests in test_utils.py (skip-ranges
documented by the sub-agent).
- Bedrock model IDs in test_health_check.py path-stripping tests.
- JSONL batch request bodies and mock LLM response bodies (must match
upstream literal).
- Langfuse expected-request-body JSON fixtures (cost values are exact-
match-asserted; changing the model would shift response_cost).
- gpt-3.5-turbo-instruct (text-completion endpoint; no modern OpenAI
equivalent).
- Top-level tests calling the proxy through user-facing aliases
(gpt-3.5-turbo, gpt-4, text-embedding-ada-002, dall-e-3) — aliases
in proxy_server_config.yaml stay; only the underlying model was
bumped.
- tests/test_gpt5_azure_temperature_support.py (the test's whole point
is model-name handling).
- Fake / mock / openai/fake identifiers.
Notable side fixes:
- test_spend_accuracy_tests.py: UPSTREAM_MODEL now matches what
spend_tracking_config.yaml's proxy actually routes to (gpt-5-mini),
resolving a latent inconsistency.
- proxy_server_config.yaml: bare `gpt-5` alias renamed to `gpt-5.5`
(bare gpt-5 is not a valid OpenAI alias).
- test_batches_logging_unit_tests.py: explicit_models list entries
kept distinct (gpt-5-mini + gpt-5.5) after bulk rename.
* test: fix CI failures from model modernization sweep
CI surfaced 4 categories of regression from the bulk modernization:
1. Azure deployment names are customer-specific. Reverted:
- tests/litellm_utils_tests/test_health_check.py: azure/text-
embedding-3-small -> azure/text-embedding-ada-002 (the CI Azure
account does not have a text-embedding-3-small deployment).
- tests/logging_callback_tests/test_custom_callback_router.py:
same revert for two router fixtures driving aembedding.
2. gpt-5 family does not accept temperature != 1. Tests that pass a
custom temperature swapped from gpt-5-mini to gpt-4.1-mini (modern
non-reasoning OpenAI mini that still accepts temperature/logprobs):
- tests/logging_callback_tests/test_datadog.py
- tests/logging_callback_tests/test_langsmith_unit_test.py
- tests/logging_callback_tests/test_otel_logging.py
3. proxy_server_config.yaml's gpt-3.5-turbo-large alias was routing to
gpt-5.5 (a reasoning model that rejects logprobs). The proxy test
tests/test_openai_endpoints.py::test_chat_completion_streaming
exercises logprobs/top_logprobs through that alias. Bumped the
underlying model to gpt-4.1 (non-reasoning, still modern).
4. tests/logging_callback_tests/test_gcs_pub_sub.py asserts against a
pinned JSON fixture (gcs_pub_sub_body/spend_logs_payload.json) with
hardcoded model="gpt-4o" and a model-specific spend value. Reverted
the litellm.acompletion calls in the test to model="gpt-4o" so the
fixture's exact-match assertions still hold.
5. tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py:
anthropic.messages.create routing to openai/gpt-5-mini returned an
empty content[0] with max_tokens=100 (reasoning-token consumption).
Swapped to openai/gpt-4.1-mini.
* test: fix Assistants API model + 2 cursor[bot] review nits
1. pass_through_unit_tests/test_custom_logger_passthrough.py: gpt-5.5
isn't accepted by the /v1/assistants endpoint
("unsupported_model"). Switch to gpt-4.1-mini (modern, Assistants-
API-supported, non-reasoning).
2. example_config_yaml/pass_through_config.yaml: the previous sweep
bumped the claude-3-7-sonnet alias to claude-opus-4-7, which is a
tier change (Sonnet -> Opus). Map to claude-sonnet-4-6 to keep the
Sonnet tier intact. (Cursor bugbot review.)
3. example_config_yaml/simple_config.yaml: model_name was left as
gpt-3.5-turbo while the underlying was bumped to gpt-5-mini, which
muddles the "simple" example. Make both sides gpt-5-mini so the
most basic example is a straight 1:1 mapping again. (Cursor bugbot
review.)
* fix: revert gpt-4/gpt-3.5-turbo alias underlying to non-reasoning models
tests/test_openai_endpoints.py::test_completion calls the proxy alias
"gpt-4" with temperature=0, and other tests call gpt-3.5-turbo with
custom temperature / logprobs / the legacy /v1/completions endpoint.
The earlier modernization mapped both aliases to gpt-5.5 / gpt-5-mini,
which are reasoning models that reject temperature != 1 and don't
expose /v1/completions. Map the aliases to gpt-4.1 / gpt-4.1-mini
(modern non-reasoning OpenAI models) instead — keeps user-facing
aliases preserved while picking a current underlying that still
supports the parameters/endpoints the tests exercise.
|
||
|
Yuneng Jiang
|
a4a1726d99 |
test(proxy): add endpoint-level regression for /config/update double-encryption
Adds test_update_config_env_var_round_trip_not_double_encrypted, which drives the real /config/update handler: first write plaintext, then re-POST the stored ciphertext (the Admin UI round-trip) and assert the value is not stacked with a second encryption layer and untouched keys stay byte-identical. Verified to fail against the pre-fix handler and pass after. Also tightens the unit test to exactly three ciphertext re-feeds. |
||
|
Yuneng Jiang
|
0d8c9137fb |
fix(proxy): make /config/update env-var encryption idempotent
A single decrypt-then-encrypt chokepoint (_encrypt_env_variables_for_db) now backs both update_config and save_config. Re-submitting a value the Admin UI read back from /get/config/callbacks as ciphertext no longer stacks a second encryption layer, which previously decrypted to garbage and silently broke the callback. The chokepoint decrypts with the pure _decrypt_db_variables (no os.environ mutation on the write path) and encrypts exactly once; update_config merges only the sent keys so untouched env vars keep their stored ciphertext byte-for-byte. |
||
|
|
f9ba70d357 |
fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpo… (#27976)
* fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint (#27943) * docs: add one-line docstring to _disable_debugging (#27894) Squash-merged by litellm-agent from oss-agent-shin's PR. * Add jp. Bedrock cross-region inference profile for claude-sonnet-4-6 (#27831) Squash-merged by litellm-agent from Cyberfilo's PR. * Sanitize empty text content blocks on /v1/messages (#27832) Squash-merged by litellm-agent from Cyberfilo's PR. * fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint The bedrock-mantle gateway (Claude Mythos Preview) serves the Anthropic Messages API at /anthropic/v1/messages; /v1/messages returns 404 Not Found. Both AmazonMantleConfig (chat/completions caller route) and AmazonMantleMessagesConfig (anthropic-messages caller route) hardcoded the wrong path, so every Mantle request 404'd before reaching the model. Per the Anthropic docs: "[Claude in Amazon Bedrock] uses the Messages API at /anthropic/v1/messages with SSE streaming." https://platform.claude.com/docs/en/api/claude-on-amazon-bedrock Confirmed independently against the live endpoint: /v1/chat/completions -> 200 OK /v1/messages -> 404 Not Found (what litellm used) /anthropic/v1/messages -> 200 OK (Claude only) Adds a regression test asserting both Mantle configs build the /anthropic/v1/messages path, and updates the existing assertions that encoded the wrong path. --------- Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> * fix: sanitize empty text blocks in sync anthropic_messages_handler path Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: João Costa <13508071+jpv-costa@users.noreply.github.com> Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
|
50df072d95 |
feat: add weighted-routing failover (#27980)
* Feat: Add Weighted-Routing Failover * test(router): cover weighted failover helper functions Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): align weighted failover deployment list type with mypy Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): address greptile review on weighted failover - Narrow exception swallowing in `_maybe_run_weighted_failover` to `openai.APIError` so model failures defer to the regular fallback while programming bugs (AttributeError/KeyError/TypeError) surface. - Note async-only limitation of `enable_weighted_failover` in the Router constructor docstring. - Make the weighted distribution test less flaky (1000 iterations, looser bound) and make the non-simple-shuffle test deterministic by failing both deployments instead of relying on the latency strategy's first pick. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): ensure weighted failover metadata persists in kwargs The previous `kwargs.setdefault(metadata_variable_name, {}) or {}` returned a brand-new dict whenever the existing metadata was falsy (empty dict or None), so writes to `_failover_excluded_ids` never made it back into `kwargs`. Multi-hop weighted failover then re-selected previously failed deployments and exhausted `max_fallbacks` prematurely. Explicitly assign a fresh dict into kwargs when metadata is missing so mutations are visible to subsequent failover hops. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(router): regression for weighted failover metadata persistence Asserts kwargs["metadata"]["_failover_excluded_ids"] is populated after _maybe_run_weighted_failover, proving the metadata dict written by the helper is the same object that lives in kwargs (no disconnected copy). Pairs with the prior fix that replaced `setdefault(..., {}) or {}` with an explicit get/assign so writes survive across hops. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): harden weighted failover error/state handling - Catch RouterRateLimitError (ValueError) alongside openai.APIError in _maybe_run_weighted_failover so an exhausted intra-group retry falls through to the regular cross-group fallback path instead of bubbling out and bypassing configured fallbacks. - Stop mutating the shared input_kwargs dict; build a local copy with the weighted-failover keys so the entry (with _excluded_deployment_ids) cannot leak into later fallback paths reading the same dict. - _get_excluded_filtered_deployments now returns an empty list when the exclusion filter removes every healthy deployment, instead of falling back to the original list. The original-list behavior risked re-picking the just-failed deployment; callers already handle the empty case by raising their no-deployments error, which weighted failover now catches and converts into a normal cross-group fallback. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(router): fall through to rpm/tpm when total weight is zero When the weight metric's total is zero (e.g. after weighted-failover exclusion leaves only zero-weight backups), continue to the next metric (rpm/tpm) instead of returning a uniform random pick immediately. This lets rpm/tpm still drive routing when present, and only falls back to the uniform random pick at the end if no metric provides a positive total weight. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(router): skip weighted failover when remaining deployments are all in cooldown _maybe_run_weighted_failover was computing 'remaining' from all_deployments (every deployment in the model group, including those in cooldown). This meant that when all non-excluded deployments were in cooldown the method still invoked run_async_fallback unnecessarily, which propagated into async_get_healthy_deployments, found no eligible deployments, and raised RouterRateLimitError — only safely caught thanks to the earlier exception-broadening fix. The fix: before computing 'remaining', fetch the current cooldown set via _async_get_cooldown_deployments and subtract it from all_ids. This allows _maybe_run_weighted_failover to return None immediately (skipping the run_async_fallback call entirely) when every non-failed deployment is in cooldown, letting the caller fall through to the correct cross-group fallback path without the wasteful extra round-trip. Tests added: - unit: _maybe_run_weighted_failover returns None without calling run_async_fallback when all remaining deployments are in cooldown - unit: _maybe_run_weighted_failover still calls run_async_fallback when at least one healthy (non-cooldown) deployment is available - integration: end-to-end fallthrough to cross-group fallback when remaining deployments are in cooldown Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> |
||
|
Sameer KankuteandGitHub
|
106b2f2da8 |
Merge pull request #27977 from BerriAI/litellm_mcp_internal_delegate_pkce
fix(mcp): delegate PKCE bypass for internal MCP servers |
||
|
Sameer KankuteGitHubCursorgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>Sameer Kankute |
c2efe9e422 |
fix(vertex-ai): fix zero cost/usage on completed Vertex AI batch jobs (#27912)
* fix(vertex-ai): fix zero cost/usage on completed Vertex AI batch jobs Vertex batch jobs recorded 0 spend and 0 tokens after PR #25627 added automatic transformation of GCS predictions.jsonl to OpenAI format. Two bugs fixed: 1. batch_utils.py: the Vertex-specific cost/usage reader (calculate_vertex_ai_batch_cost_and_usage) was always invoked and reads raw usageMetadata fields that no longer exist in the OpenAI-shaped output. Now the reader is only used when disable_vertex_batch_output_transformation=True; otherwise the generic path handles the already-transformed OpenAI-shaped content. 2. cost_calculator.py: batch_cost_calculator skipped the global litellm.get_model_info() lookup when a model_info dict was passed in, even when that dict had no pricing fields (e.g. deployment metadata with only id/db_model). It now falls back to the global pricing table when the provided model_info has no pricing data. Co-authored-by: Cursor <cursoragent@cursor.com> * Update litellm/cost_calculator.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(cost-calculator): use not-any guard for pricing fallback in batch_cost_calculator Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cost-calculator): treat explicit zero batch pricing as set in model_info The fallback to litellm.get_model_info() used truthy checks on pricing fields, so 0.0 was treated as missing and replaced by global rates. Use `is not None` like elsewhere in cost calculation. Add regression test. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> |
||
|
|
cbdc70d544 |
fix(managed_batches): convert raw output_file_id to managed ID in CheckBatchCost poller (#27984)
* fix(managed_batches): convert raw output_file_id to managed ID in CheckBatchCost poller CheckBatchCost bypasses async_post_call_success_hook, causing raw provider output_file_ids to be persisted in LiteLLM_ManagedObjectTable. This fix converts output_file_id and error_file_id to managed base64 IDs before the DB write. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(check_batch_cost): persist managed file before mutating response and propagate team_id - Move setattr after store_unified_file_id so the response only receives the managed ID once the DB record is successfully written. Avoids serializing an orphaned managed ID into file_object when the store call fails. - Populate team_id on the minimal UserAPIKeyAuth from job.team_id so the managed file record is created with the correct team ownership, allowing other team members to access the batch output file via /files/{id}/content. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(managed_batches): extend test to cover error_file_id conversion Co-authored-by: Cursor <cursoragent@cursor.com> * fix managed file test --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
|
fe755ee02a |
feat(proxy): fix vector store retrieve/list/update/delete without model (#27929)
* feat(proxy): fix vector store retrieve/list/update/delete routing without model Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): remove unchecked query-param injection in vector store management endpoints Co-authored-by: Cursor <cursoragent@cursor.com> * test(proxy): use subset assertion for vector store route test to allow extra kwargs like shared_session Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
|
4e2b2d9d1f |
fix(mcp): expose delegate_auth_to_upstream in MCP server list rows (#27936)
_build_mcp_server_table omitted delegate_auth_to_upstream, so GET /v1/mcp/server always returned the default false while the registry kept the DB value. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Sameer Kankute
|
d855e56333 |
chore(mcp): warn on internal + upstream PKCE delegate
Log verbose_logger.warning when loading oauth2 interactive servers with available_on_public_internet=false and delegate_auth_to_upstream=true (config + DB). Dashboard Alert for the same combo. CLAUDE note for operators. Tests for log and M2M skip. |
||
|
Sameer Kankute
|
5aabfccf57 |
fix(mcp): allow delegate PKCE bypass for internal MCP servers
Remove available_on_public_internet gating from delegate-auth-to-upstream paths so oauth2 + delegate_auth_to_upstream interactive servers behave the same when marked internal. Keeps M2M exclusion. Updates tests. |
||
|
yuneng-jiangandGitHub
|
16bd81985d |
Merge pull request #27795 from BerriAI/litellm_vcr-cache-observability-and-fixes-c5bc
test(vcr): classify cache verdicts, surface cost leaks, and fix the two biggest leakers |
||
|
|
39e1831e84 |
Emit native web_search_tool_result blocks for Anthropic clients (Claude Desktop / Cowork citations) (#27886)
* feat(custom_logger): add async_post_agentic_loop_response_hook Lets a CustomLogger shape the response returned by the agentic-loop follow-up call without bypassing the loop's safety / observability machinery (depth tracking, fingerprinting, etc.). Default returns the response unchanged. Used by websearch_interception to inject Anthropic-native web_search_tool_result blocks when the originating client requested a native web_search_* tool. * feat(llm_http_handler): call post-agentic-loop hook on the originating callback In _execute_anthropic_agentic_plan, after anthropic_messages.acreate returns, call the originating callback's async_post_agentic_loop_response_hook so it can mutate the final response (e.g. inject native tool_result blocks). Pass the callback through from _call_agentic_completion_hooks. Exceptions in the post-hook are caught and logged so a buggy callback can't kill the request. * feat(websearch_interception): add is_anthropic_native_web_search_tool Identifies tools the Anthropic-native clients (Claude Desktop, the Anthropic SDK, the Anthropic Console) use to request native search: type starts with "web_search_" (e.g. web_search_20250305). Rejects the LiteLLM standard tool, the OpenAI-function variant, the bare "WebSearch" legacy name, and the bare "web_search" Claude Code shape. This lets us decide per-request whether the client expects web_search_tool_result content blocks in the response, without renaming any existing constants or touching native-provider skip logic. * feat(websearch_interception): add build_web_search_tool_result_block Produces the Anthropic-native web_search_tool_result content block from a structured SearchResponse. Anthropic-native clients use this block to populate citations / source links — the existing text-blob flatten path only feeds readable evidence to the model and discards the structure, so this builder gives us the missing piece. Shape matches https://docs.anthropic.com/en/api/web-search-tool — web_search_result items carry url, title, page_age, encrypted_content (empty string when the search provider doesn't supply one). * feat(websearch_interception): emit native web_search_tool_result blocks When the originating client request carried a native Anthropic web_search_* tool, the final response now also carries web_search_tool_result content blocks alongside the model's text answer — so Claude Desktop / Anthropic SDK clients can populate the citations panel and replay conversation history with structured search evidence. Wiring: - Pre-request hooks (both deployment + Anthropic path) set a flag on kwargs when they see a native web_search_* tool, so the signal survives the conversion-to-litellm_web_search step regardless of which hook fires first. - _execute_search now returns (text, SearchResponse) so the structured results aren't lost when the text is flattened for the follow-up model call. - _build_anthropic_request_patch returns the parallel list of SearchResponse objects. - async_build_agentic_loop_plan pre-builds the web_search_tool_result blocks (one per tool_use_id) and stashes them on plan.metadata when the flag is set. - async_post_agentic_loop_response_hook reads the metadata and prepends the blocks to response.content. - _execute_agentic_loop mirrors the injection for the legacy path so both paths behave identically. Clients that send the LiteLLM standard tool keep the existing text-only behavior — no regression. * test(websearch_interception): cover native web_search_tool_result emission 18 tests across: - detector branches (native vs litellm-standard, OpenAI-function shape, Claude Desktop builtin WebSearch, bare web_search, missing type) - block-builder shape (results, none, empty) - pre-request hook flag-setting (native sets, standard does not) - async_build_agentic_loop_plan attaches blocks to plan.metadata when the flag is present, leaves metadata untouched when absent - post-hook injection into dict and object responses - legacy _execute_agentic_loop mirrors the injection so both paths return the same shape * test(websearch_short_circuit): keep _execute_search mocks in sync with new tuple return * test(websearch_thinking_constraint): keep _execute_search mocks in sync with new tuple return * feat(websearch_interception): emit native blocks from try_short_circuit_search The agentic-loop post-hook only fires when the model returns a tool_use block. Cowork / Claude Desktop on Bedrock actually make TWO requests per user turn: the main /v1/messages with their builtin tool, and a separate standalone /v1/messages whose only tool is web_search_20250305. That second request hits try_short_circuit_search — no agentic loop, no post-hook — and was returning text-only, leaving the citations panel empty. When the short-circuit input carries a native web_search_* tool, build a synthetic server_tool_use + web_search_tool_result pair (using the structured SearchResponse already returned by _execute_search) so the client gets the native shape it expects. The legacy text block is preserved so non-native short-circuit callers (Claude Code, github_copilot, etc.) see the same payload as before. Failure path still emits the native block pair (with empty results) plus the text-error block, so the client gets a well-formed response rather than a malformed half-shape. * test(websearch_native_blocks): cover short-circuit native-block emission Three new cases on top of the existing 18: - native web_search_20250305 short-circuit → [server_tool_use, web_search_tool_result, text], ids paired, urls/titles carried. - litellm_web_search short-circuit → text-only (no regression). - native short-circuit on search failure → still emits the native block pair (empty results) plus the text-error block, so the client never sees a malformed half-shape. * test(websearch_short_circuit): index assertions by block type, not by position Native short-circuit responses now have [server_tool_use, web_search_tool_result, text] when the input carries web_search_20250305 — find the text block by type rather than relying on content[0]. * fix(websearch_interception): gate legacy WebSearch name on schema absence Clients like Cowork / Claude Desktop ship a client-side tool named "WebSearch" with a full input_schema — they handle it themselves and expect to make a separate native web_search_20250305 sub-request for the actual search. Today is_web_search_tool matches the bare name regardless of other fields, which hijacks the client's tool server-side. The agentic loop fires on the main request, the model never gets to emit the client-side tool_use, and the separate native sub-request (where citation data flows) is never made. Net: citations panel empty. Real Anthropic client tools always carry input_schema (the API rejects them otherwise), so a bare {name: "WebSearch"} with no schema is the only thing that could be a legacy interception marker. Gate the match on schema absence: legacy callers (if any) keep working, real client-side WebSearch tools pass through untouched. * fix(websearch_interception): drop "WebSearch" from response-detection lists Post-conversion the model always sees ``litellm_web_search``, so the "WebSearch" entry in the response-side tool_use detection lists was dead at best. If a model ever did return ``tool_use(name="WebSearch")`` it would now (incorrectly) hijack the client's own ``WebSearch`` tool again — same Cowork problem we just fixed on the input side. Drop it. * test(websearch_native_blocks): cover the WebSearch legacy-name schema gate Three new cases: - {name: "WebSearch"} (bare interception marker) → still matched - {name: "WebSearch", input_schema: {...}} (Cowork client tool) → passes through untouched - {name: "WebSearch", description: "..."} (no schema) → still matched on the assumption it's a legacy marker rather than a malformed real client tool. --------- Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> |
||
|
|
9b6ab55c5f |
fix: allow for allowlisted redirect URIs (#27761)
* fix: allow for allowlisted redirect URIs * github comment addressing * Update litellm/proxy/_experimental/mcp_server/oauth_utils.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * harden oauth wildcard further * test: cover wildcard entry with dot-leading suffix rejection --------- Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> |
||
|
|
7a462a4220 |
fix(rate-limit): stop v3 limiter from leaking internal stash to provider body (#27913)
* fix(rate-limit): stop v3 limiter from leaking internal stash to provider body
PR #27001 (atomic TPM rate limit) introduced a reservation flow that
writes four LiteLLM-internal keys onto the request data dict:
_litellm_rate_limit_descriptors
_litellm_tpm_reserved_tokens
_litellm_tpm_reserved_model
_litellm_tpm_reserved_scopes
_litellm_tpm_reservation_released
These keys are forwarded as request body params to the upstream provider,
which rejects them as unknown fields:
OpenAI -> 400 'Unknown parameter: _litellm_rate_limit_descriptors'
(mapped by litellm to RateLimitError / 429, hiding the bug
behind a misleading 'throttling_error' code)
Anthropic -> 400 '_litellm_rate_limit_descriptors: Extra inputs are
not permitted'
Net effect: every chat completion against any real provider fails the
moment a virtual key has any tpm_limit / rpm_limit set — i.e. v3-enforced
key-level TPM/RPM limits are broken end-to-end. The v3 RPM/TPM check
itself still runs (raises 429 on over-limit), but the success path
poisons the upstream body.
Reproduced on litellm_internal_staging HEAD (
|
||
|
|
a6494e6fe3 |
perf: eliminate per-request callback scanning on proxy hot path (#27858)
- Introduce `_CallbackCapabilities` dataclass and `ProxyLogging._callback_capabilities()` static method that inspects `litellm.callbacks` once and caches capability flags keyed on (list length, member ids); invalidates automatically when the callback list mutates without per-request iteration overhead - Replace O(n) `litellm.callbacks` walks in `async_pre_call_hook`, `during_call_hook`, `async_post_call_streaming_iterator_hook`, `async_post_call_streaming_hook`, and `post_call_response_headers_hook` with fast-path exits when no relevant callbacks are registered - Add `needs_iterator_wrap()` and `needs_per_chunk_streaming_hook()` instance methods to decouple iterator-level wrapping from per-chunk hook execution; avoids `get_response_string` materialization per chunk when no guardrail or chunk-hook callback is active - Introduce `_fast_serialize_simple_model_response_stream()` using `orjson` for common single-choice text streaming chunks, bypassing the full Pydantic serializer; falls back to `model_dump_json` for tool calls, logprobs, usage, and provider-specific fields - Add early-return in `_restamp_streaming_chunk_model` when downstream model already matches the requested model, avoiding unnecessary string comparisons on every chunk - Fix stale zero-cost cache bug in `_is_model_cost_zero`: move the per-router `_zero_cost_cache` dict onto the `Router` instance and clear it in `_invalidate_model_group_info_cache` so in-place pricing updates via `upsert_deployment` immediately resume budget enforcement - Add `scripts/benchmark_chat_completions_perf.py`: standalone async benchmarking tool with a mock OpenAI provider, LiteLLM proxy process management, non-streaming RPS, streaming TTFT, and full-stream latency measurements with repeat/median run support - Add comprehensive unit tests covering capability detection, cache invalidation, fast-path correctness, zero-cost cache regression, and the no-callback streaming fast path Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> |
||
|
vladpolevoiandGitHub
|
65d6ad82ef |
feat(lasso): add tool-calling support to LassoGuardrail (#27648)
* feat(lasso): extend LassoGuardrail to support tool calling (RND-5748)
* fix(lasso): PR review followups for tool-calling guardrail (RND-5748)
* fix(lasso): handle object-style tool_calls in _update_tool_calls_from_masked (RND-5748)
* fix(lasso): use model role for tool_use blocks (RND-5748)
* test(lasso): add round-trip tests for message transformation (RND-5748)
* fix(lasso): remove unused imports, handle Responses-API input masking, flatten multimodal content (RND-5748)
* fix(lasso): inspect Responses-API input field (RND-5748)
* fix(lasso): guard text-cursor remap against Lasso count mismatch (RND-5748)
* fix(lasso): flatten list content in tool_result.content (RND-5748)
* fix(lasso): remap multimodal list content during masking (RND-5748)
Bug: _map_masked_messages_back counted list-content messages in
original_text_count but the remap loop only handled isinstance(str).
The positional text_cursor never advanced for list messages, causing
all subsequent masked texts to be written onto the wrong messages.
Fix: added elif isinstance(content, list) branch that replaces the
list with the masked text string and advances the cursor — mirrors
the existing string-content branch. Also handles the assistant +
tool_calls combo for list-content messages.
Test: test_map_masked_messages_back_list_content verifies a user
message with [text + image_url] followed by an assistant message
gets correct masked content on both (cursor stays aligned).
* refactor(lasso): extract _get_field and _extract_tool_call_fields helpers (RND-5748)
The dict-vs-object access pattern (x.get('y') if isinstance(x, dict)
else getattr(x, 'y', None)) was duplicated 14 times across 5 methods.
_get_field(obj, field) — single-point dict/Pydantic field access.
_extract_tool_call_fields(call) — returns (call_id, name, parsed_input)
with JSON argument parsing, replacing ~30 duplicate lines in both
async_post_call_success_hook and _expand_messages_for_classification.
Also simplified _update_tool_calls_from_masked, _prepare_payload tool
mapping, and _apply_masking_to_model_response call_id extraction.
Net ~60 lines removed. No behavior change — all 32 tests pass.
* fix(lasso): add count guard to _apply_masking_to_model_response (RND-5748)
_apply_masking_to_model_response used a bare text_cursor without
verifying 1:1 correspondence between text-bearing choices and masked
text entries. If Lasso returned a different number of text messages
than choices with content, masked text would be applied to the wrong
choice or silently skip choices.
Added the same count-mismatch guard pattern already used in
_map_masked_messages_back: count original text-bearing choices,
compare to masked_text length, skip text remap on mismatch with a
warning log. Tool_call masking via id-based lookup is unaffected.
Tests:
- test_apply_masking_to_model_response_multiple_choices: verifies
correct per-choice masked text with 2 choices
- test_apply_masking_to_model_response_count_mismatch: verifies
content is left unchanged when counts disagree
* fix(lasso): close two guardrail-bypass paths flagged in review (RND-5748)
* tool-call args: when function.arguments is malformed JSON or parses
to a non-object, preserve the raw string as {"arguments": <raw>} so
Lasso still inspects it instead of receiving input=None. Covers both
pre-call and post-call extraction (shared helper). Also resolves the
CodeQL empty-except warning since the except body now assigns parsed=None.
* Responses-API input: when a request carries both "messages" and
"input", inspect both. Previously a benign messages array let the
guardrail skip data["input"] entirely. The masking write-back is
split via a count boundary so masked messages flow back to
data["messages"] and masked input flows back to data["input"]
without cross-contamination.
Tests: malformed/non-object args round-trip, dual-field classification,
dual-field masking write-back split.
* chore(lasso): black formatting + comment on expand skip branch (RND-5748)
* black: wrap two long expressions in lasso.py and reformat dict
literals in test_lasso.py to satisfy CI lint.
* add a short comment in _expand_messages_for_classification
explaining why empty string and None content are intentionally
skipped (None is the OpenAI shape for a pure tool-call turn).
* fix(lasso): satisfy mypy in _handle_masking, _update_tool_calls_from_masked, _apply_masking_to_model_response (RND-5748)
* Narrow `response.get("messages")` into a local before slicing so
mypy doesn't see `Optional[List[Dict[str, str]]]` as non-indexable.
* Rename the two write-side `func` bindings in
`_update_tool_calls_from_masked` to `func_dict` / `func_obj` so
mypy doesn't unify the dict and Any|None branches.
* Rename the inner loop variable in `_apply_masking_to_model_response`
from `msg` to `masked_msg` to avoid clashing with the
`msg = choice.message` rebinding below.
No behavior change; resolves the 7 mypy errors from the CI lint job.
|
||
|
Kenan YildirimandGitHub
|
649eb2d176 | fix(guardrails): improve CrowdStrike AIDR input handling (#26658) | ||
|
|
410ce761dc |
fix: tighten budget field validation and authorization checks (#27897)
* fix: block NaN/Inf budget bypass and add missing non-admin guards
Addresses three security issues:
GHSA-wvg4-6222-3q4r: /user/update exposes max_budget, soft_budget, spend
to self-editing non-admin users with no server-side guard. Non-admin callers
now receive HTTP 403 if any of those fields appear in the update payload.
GHSA-q775-qw9r-2r4g: _enforce_upperbound_key_params returned early (no-op)
when upperbound_key_generate_params was absent from config, letting any
authenticated user generate a key with unlimited max_budget. Fix adds a
delegated-authority ceiling in _common_key_generation_helper: non-admins
cannot grant a key more budget than their own key carries.
GHSA-2rv4-xv66-fpjg: float('nan') passes every `value < 0` guard because
nan < 0 is False in Python, and spend >= nan is always False, permanently
disabling budget enforcement for any entity carrying a NaN max_budget.
All write-time budget guards now use `not math.isfinite(v) or v < 0`.
_enforce_upperbound_key_params validates finiteness unconditionally (before
the early-return). All spend-enforcement comparisons in auth_checks.py are
now guarded with math.isfinite(max_budget) as defense-in-depth.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: close budget ceiling bypass for callers with no max_budget (GHSA-q775)
Non-admin callers whose API key has no explicit max_budget (None) could
bypass the delegated-authority ceiling and create keys with arbitrary
budgets. Now blocks budget assignment when caller has no budget configured.
Also removes redundant inline import of LitellmUserRoles.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: only apply budget ceiling to explicitly requested max_budget
Capture the caller-supplied max_budget before _enforce_upperbound_key_params
can fill it with a default, so auto-filled defaults don't trigger the
ceiling guard for non-admin users with no budget on their own key.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: capture requested max_budget before any defaults are applied
Move _requested_max_budget capture before both default_key_generate_params
and upperbound_key_generate_params mutations, so auto-filled values don't
trigger the ceiling check for non-admin users.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: allow unlimited-budget callers to delegate any budget
Callers with max_budget=None (unlimited) can legitimately create
budget-capped keys. Only block when caller has an explicit budget
and the requested budget exceeds it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
||
|
|
de1747dca8 |
fix(spend-logs): redact echoed prompts in error_information (LIT-2992) (#27689)
Provider validation errors (e.g. OpenAI RateLimitError carrying 178
pydantic errors each with their own 'input': [...]) were stored verbatim
in LiteLLM_SpendLogs.metadata.error_information.error_message via
str(original_exception), producing rows >12 MB.
Sanitize before metadata is serialized:
- redact 'input'/'messages' values in both error_message and traceback
when store_prompts_in_spend_logs is False (back-door leak paths)
- always apply the MAX_STRING_LENGTH_PROMPT_IN_DB size cap to
error_message and traceback (DB-storage safeguard)
Value scanning uses a parser-based balanced-bracket walk that respects
string quoting, so multi-modal payloads ('messages': [{'content': [...]}])
and user text containing literal brackets ("secret[123") are handled
correctly instead of leaking past a depth-1 regex.
Scoped to the spend-log path so OTEL/Datadog/etc. callbacks still
receive the untruncated error per LITELLM_TRUNCATION_DB_SAFEGUARD_NOTE.
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Krrish DholakiaGitHubClaude Sonnet 4.6veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> |
8bbc61e03c |
fix: harden /key/update authorization checks (#27878)
* fix: patch Host-header auth bypass in get_request_route Starlette reconstructs request.url from the Host header. A malformed Host like `localhost/?x=1` causes Starlette to build the full URL as `http://localhost/?x=1/health`, which url-parses to path="/". Since "/" is in LiteLLMRoutes.public_routes, all protected routes became reachable without authentication. Fix: read scope["path"] (set by uvicorn from the HTTP request line, not derivable from headers) instead of request.url.path. Sub-path deployments are handled via scope["app_root_path"] / scope["root_path"], mirroring Starlette's own base_url construction logic. Affected variants confirmed fixed: Host: localhost/?x=1 Host: localhost:4000/?x=1 Host: localhost/#test Host: localhost:4000/#test Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * style: reduce comments in route fix Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block credential fields in RAG ingest vector_store options Credential fields (vertex_credentials, aws_access_key_id, api_key, etc.) in ingest_options.vector_store are now rejected at the API boundary with a 400 error. Credentials must be configured server-side. Previously any authenticated user could supply a vertex_credentials dict with type=external_account pointing credential_source.file at an arbitrary path (e.g. /proc/1/environ) and token_url at an attacker-controlled server. google-auth's identity_pool.Credentials refresh() would read the file and POST its contents to the attacker. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block /key/update self-escalation by assigned users Non-admin users who were assigned a key (created_by != caller) could update any non-budget field — models, rpm_limit, guardrails, etc. — without admin authorization, allowing privilege self-escalation. Gate: only the key creator (created_by == caller) may edit their own key without admin check; budget changes always require admin regardless of creator status. All other callers must pass _check_key_admin_access. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block user-controlled api_base in RAG ingest vector_store options A user-supplied api_base in ingest_options.vector_store caused the server to forward its configured provider credentials (Gemini, OpenAI) to an attacker-controlled endpoint via SSRF. Add api_base to the blocked credential params set alongside api_key and the existing credential fields. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: restrict /utils/transform_request to PROXY_ADMIN and apply body safety check Any authenticated internal_user could POST arbitrary provider config (aws_sts_endpoint, api_base, etc.) to /utils/transform_request and have the server forward its credentials to an attacker-controlled endpoint. - Gate the endpoint on PROXY_ADMIN role (403 for all other roles) - Call is_request_body_safe() to reject banned params even for admins - Convert ValueError from safety check to HTTP 400 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: apply banned-param check to /utils/transform_request Without is_request_body_safe(), any authenticated user could pass aws_sts_endpoint, api_base, or aws_web_identity_token to /utils/transform_request and have the server forward its configured provider credentials to an attacker-controlled endpoint during SDK credential resolution. Applies the same banned-param blocklist already used by LLM endpoints. Endpoint remains accessible to all authenticated users. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block SSRF via api_base in /prompts/test dotprompt YAML frontmatter Any frontmatter key not in ["model","input","output"] flowed into optional_params and was merged into the LLM call data dict, bypassing is_request_body_safe. An attacker with any bearer key could set api_base in YAML to redirect the outbound LLM request — including the provider API key — to an attacker-controlled host. Fix: call is_request_body_safe on the constructed data dict after optional_params are merged, before invoking ProxyBaseLLMRequestProcessing. ValueError from the banned-param check is surfaced as HTTP 400. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Update litellm/proxy/rag_endpoints/endpoints.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * fix: coerce nested config strings before banned-param check _NESTED_CONFIG_KEYS descent used isinstance(nested, dict) which silently skipped litellm_embedding_config when delivered as a JSON string via multipart/form-data. Banned params (api_base, aws_sts_endpoint, etc.) nested inside the stringified value were invisible to is_request_body_safe. _NESTED_METADATA_KEYS already used _coerce_metadata_to_dict which parses JSON strings before checking. Apply the same coercion to _NESTED_CONFIG_KEYS. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: replace substring match with prefix match in is_llm_api_route mapped_pass_through_routes used `_llm_passthrough_route in route` (substring) so any admin-only path whose URL contained a provider name (openai, anthropic, azure, bedrock, etc.) was misclassified as an LLM API route and bypassed the admin gate in non_proxy_admin_allowed_routes_check. Confirmed live: non-admin key could GET /credentials/by_name/openai (read masked provider API key) and DELETE /credentials/openai (delete credential). Fix: use exact match or startswith(prefix + "/") — the same pattern used everywhere else in RouteChecks — so only routes that actually start with a passthrough prefix are allowed through. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: stabilize PR #27878 test failures - key_management_endpoints: extend can_skip_admin_check to team keys so team members with /key/update permission can update non-budget fields. can_team_member_execute_key_management_endpoint already validates team membership + permission and raises if unauthorized; reaching the admin check on a team key means the caller was authorized. - test: set created_by on mock key in test_update_key_non_budget_fields_allowed_for_internal_user so caller_is_creator resolves correctly (MagicMock default ≠ user_id). - auth_utils.get_request_route: guard against non-dict request.scope (e.g. MagicMock in unit tests) to prevent a MagicMock leaking into UserAPIKeyAuth.request_route and failing Pydantic validation. - ci: assign test_multipart_bypass_repro.py to the proxy-runtime shard in test-unit-proxy-db.yml to satisfy the shard-coverage check. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(lint): add explicit str() cast in get_request_route for MyPy scope.get() returns Any|None which MyPy cannot coerce to str implicitly. Wrap both scope.get() calls in str() to satisfy the type checker. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: guard bare-/ root_path strip + make total_spend migration idempotent auth_utils.get_request_route: when Starlette sets scope["app_root_path"] to "/" (e.g. behind some middleware), the old stripping logic would remove the leading slash from every path ("/team/new" → "team/new"), breaking route matching and causing auth to misclassify protected routes. Skip stripping when root_path is bare "/". migration: add IF NOT EXISTS to total_spend ALTER TABLE so the migration is safe to replay when a prior partial run already created the column. Without this guard, prisma migrate deploy fails on CI DBs that were partially migrated, causing all subsequent DB operations (including /team/new) to 500. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: require creator still owns key for personal-key bypass in /key/update caller_is_creator now requires both created_by == caller AND user_id == caller. Previously checking only created_by let a demoted admin who originally created a key for another user continue editing non-budget fields on it after reassignment, bypassing _check_key_admin_access. Adds regression test: creator whose key was reassigned is blocked (403). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: extract auth checks to fix PLR0915 + broaden max_budget assertion internal_user_endpoints._update_single_user_helper exceeded 50 statements (PLR0915). Extract authorization checks into _check_user_update_authz helper to bring statement count under the limit. test_validate_max_budget: assert "negative" (substring of both the local "cannot be negative" and the CI "non-negative finite number" messages) so the test is stable regardless of which exact wording the function uses. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> |
||
|
yuneng-jiangandGitHub
|
0c4982042a |
Merge pull request #27892 from BerriAI/worktree-fix-mcp-byok-oauth
Worktree fix mcp byok oauth |
||
|
yuneng-jiangandGitHub
|
e3e5209f51 |
Merge pull request #27801 from stuxf/chore/get-instance-fn-runtime-s3-gate
chore(proxy): refuse remote-URL instance-fn loads outside config-file path |
||
|
user
|
626b768d25 |
chore(tests): drop redundant membership check; trim test comment
``test_azure_ad_token_is_in_banned_list`` only asserted tuple membership of a name the parametrized test already exercises end-to-end through ``is_request_body_safe``. Removed. Tightened the admin-opt-in test comment. |
||
|
user
|
5e56022553 |
chore(proxy): coerce stringified nested-config containers before descent
``_NESTED_CONFIG_KEYS`` descent used ``isinstance(nested, dict)``, so a caller sending ``extra_body`` as a JSON-encoded string instead of an object (the same shape multipart/form-data clients use for ``litellm_metadata``) skipped the banned-key check entirely. Switched to ``_coerce_metadata_to_dict`` so the JSON-string path is parsed before descent — mirrors the existing handling on ``_NESTED_METADATA_KEYS``. |
||
|
user
|
2519ac161e |
chore(proxy): cover extra_body + azure_ad_token in banned-params check
``extra_body`` is the OpenAI-SDK passthrough container. Provider modules read provider-auth fields out of it directly (Azure's ``extra_body.azure_ad_token``, Bedrock's ``extra_body.aws_web_identity_token``, etc.) without re-validating, so the boundary check has to walk it the same way it walks ``litellm_embedding_config``. Adding it to ``_NESTED_CONFIG_KEYS`` extends single-level banned-key descent into the container — top-level admin opt-ins (``allow_client_side_credentials`` / ``configurable_clientside_auth_params``) still apply. ``azure_ad_token`` was not in ``_BANNED_REQUEST_BODY_PARAMS`` despite being the bearer-token field the Azure transformer resolves through ``get_secret`` (same shape as ``aws_web_identity_token`` on the Bedrock STS path). Added so it can't be supplied per-request without an admin opt-in. |
||
|
|
1294165768 |
feat(mcp): support MCP access group names in URL-based namespacing (#27726)
* feat(mcp): support MCP access group names in URL-based namespacing
Extends dynamic_mcp_route to resolve /{name}/mcp requests where {name}
is an MCP access group tag or a comma-separated list of servers/groups,
matching what the documentation promised but the handler did not implement.
Resolution order: registered server alias → toolset → comma-separated
list → single access group tag (404 if none match).
Adds unit tests covering all four resolution paths plus 404 cases.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): address Greptile review comments on dynamic_mcp_route
- Move comma-separated check before toolset DB lookup so comma names
short-circuit without hitting the database
- Cache access-group DB lookups via user_api_key_cache to avoid a raw
find_many on every request (matches toolset caching pattern)
- Remove unused response_started variable from _forward_as_mcp_path
- Update tests to assert comma list skips toolset call and to mock cache
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(mcp): extract helpers to fix PLR0915 too-many-statements in dynamic_mcp_route
Extract _mcp_forward_as_path and _is_mcp_access_group_cached as
module-level helpers so dynamic_mcp_route stays under the 50-statement
limit. Update tests to patch the new module-level symbols directly.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Avoid caching missing MCP access groups
* fix(mcp): stream MCP responses via _stream_mcp_asgi_response instead of buffering
_mcp_forward_as_path previously accumulated the full response body in
memory before sending it. Replace the buffering custom_send pattern with
_stream_mcp_asgi_response, which uses an asyncio.Queue bridge so chunks
are yielded to the client as they arrive, preventing unbounded memory
growth on large or long-lived MCP responses.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): short-TTL negative cache for access-group existence lookup
An unauthenticated caller could repeatedly request /<unknown>/mcp and
force a fresh DB lookup for the access-group existence check on every
request (only positive results were cached). Cache negative results
for a short DEFAULT_MCP_ACCESS_GROUP_NEGATIVE_CACHE_TTL window (10s by
default) so the DB is shielded from flooding while a transient DB error
(which surfaces as an empty list) cannot hide a real group for long.
https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9
* fix(mcp): use plain int for access-group negative cache TTL
Drop the os.getenv wrapper around DEFAULT_MCP_ACCESS_GROUP_NEGATIVE_CACHE_TTL
to avoid the documentation_test_env_keys check failing on the new variable.
The negative-cache window is a small internal tuning constant, not a
user-facing knob, so a plain integer is clearer than an env override.
https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9
* fix(mcp): validate, dedupe, and cap CSV tokens in dynamic MCP route
For /{name1,name2,...}/mcp, validate every token resolves to a known
server alias or access group, dedupe case-insensitively, and cap at
DEFAULT_MCP_NAMESPACE_CSV_MAX_TOKENS=16 before forwarding.
- Bounds the per-request DB / cache fan-out an authenticated caller can
trigger by stuffing the path with tokens (raised by veria-ai).
- Returns 404 instead of forwarding when no token resolves, so the
downstream server filter cannot silently fall back to the full
allowed_mcp_servers list (raised by Cursor agentic security review).
- Forwards only the resolved subset, so unknown tokens cannot ride along
into the downstream filter.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(mcp): exact-match CSV token dedupe to preserve case-sensitive distinct tokens
Bugbot flagged that case-insensitive dedup on `MyGroup,mygroup` could
collapse to whichever case appeared first and silently drop the matching
casing if the downstream resolver is case-sensitive. Switch to exact-match
dedup so distinct casings survive; whitespace-only differences still
collapse via the .strip() before comparison.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: mateo-berri <mateo@berri.ai>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
|
||
|
user
|
aced4295bb |
chore(proxy): also scrub guardrail callbacks / module paths from DB overlay
A guardrail entry's ``callbacks`` list (v1: ``{name: {callbacks:[...]}}``,
v2: ``{guardrail_name, litellm_params: {callbacks: [...], guardrail:
"module.path"}}``) is iterated during config load and threaded through
``get_instance_fn``. A PROXY_ADMIN persisting
``litellm_settings.guardrails[*].callbacks: ["s3://..."]`` or
``litellm_settings.guardrails[*].litellm_params.guardrail: "s3://..."``
via ``/config/update`` was not covered by the previous scrub matrix.
Walk both v1 and v2 entry shapes and null out remote-URL callbacks /
module-path values before the merge. Adds four regression tests.
|
||
|
Krrish DholakiaandClaude Sonnet 4.6
|
f1d07c13e5 |
fix: block SSRF fields in RAG ingest vector_store config
aws_sts_endpoint, aws_web_identity_token, and aws_bedrock_runtime_endpoint in ingest_options.vector_store were passed directly to the Bedrock ingestion class, which reads them into boto3 STS client construction. Any authenticated caller could redirect AssumeRole calls to an attacker-controlled server, leaking the proxy's instance profile credentials. Calls is_request_body_safe() on ingest_options["vector_store"] before forwarding to litellm.aingest(). Same banned-params list and admin opt-in escape hatch (allow_client_side_credentials) as the /chat/completions path. ValueError from the safety check is caught and re-raised as HTTP 400. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> |
||
|
lmcdonald-godaddyandGitHub
|
baa68ebb12 |
fix(pricing): GPT-4o-Transcribe Pricing (#27875)
* Update gpt-4o-transcribe price * Update test for gpt-4o-transcribe pricing fix * Update gpt-4o-mini-transcribe price |
||
|
|
f028a622e2 |
fix(prometheus): emit litellm_remaining_tokens_metric for Bedrock and Vertex (#27705)
* fix(prometheus): emit remaining_tokens/requests gauges for bedrock + vertex (LIT-2719) Bedrock and Vertex AI never return x-ratelimit-remaining-* response headers, so litellm_remaining_tokens_metric / litellm_remaining_requests_metric only fired for OpenAI / Azure / Anthropic deployments even when tpm/rpm was configured on the router. Add a provider-agnostic fallback in PrometheusLogger.async_log_success_event that asks Router.get_remaining_model_group_usage() for the same model_group and emits the gauges with configured_limit - current_usage when the upstream provider didn't populate the headers itself. Existing OpenAI / Azure / Anthropic flows are unchanged because the fallback short-circuits when both header values are already present. Tests: 8 new tests covering bedrock + vertex emission, header short-circuit, partial-header fill, llm_router=None, missing model_group, empty router result, and router exception swallowing. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(prometheus): narrow except to ImportError, log router lookup failures via verbose_logger.exception Address greptile review: - The optional 'from litellm.proxy.proxy_server import llm_router' should guard against ImportError specifically, not all exceptions, so that unexpected errors (e.g. AttributeError from partially-initialized state) stay visible. - get_remaining_model_group_usage failures are now logged via verbose_logger.exception (with traceback) instead of debug, matching the PR description's intent and avoiding silent loss of router-cache errors in production. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(prometheus): subtract in-flight delta in router-remaining fallback The router's TPM/RPM counter is incremented by Router.deployment_callback_on_success, which fires alongside this prometheus callback in the success-log fan-out. Prometheus wins the race, so get_remaining_model_group_usage returns the pre-decrement counter for the current request — while vendor headers (OpenAI/Anthropic/Azure) are already post-decrement. That broke parity between providers on the same gauge: dashboards plotting litellm_remaining_requests_metric showed Bedrock/Vertex perpetually one request behind Anthropic for the same throughput. Replay the in-flight increment before emit: subtract total_tokens from remaining_tokens and 1 from remaining_requests. * Revert "fix(prometheus): subtract in-flight delta in router-remaining fallback" This reverts commit 001ce95ecdd952b4b5a23dd2b1e62c4562c932bc. * fix(router): post-decrement router-derived ratelimit headers Router.set_response_headers injects x-ratelimit-remaining-{tokens, requests} for providers that don't return them natively (Bedrock, Vertex). The values come from get_remaining_model_group_usage, which reads the router's TPM/RPM counter — incremented post-response by deployment_callback_on_success. So the headers reflected the counter state before the current request was counted: pre-decrement. Vendor headers from OpenAI/Anthropic/Azure are post-decrement (the vendor counted the request before responding). Same metric name, two semantics — dashboards plotting litellm_remaining_requests_metric showed Bedrock/Vertex perpetually one request behind for the same throughput, and the HTTP response headers exposed the same skew to clients. Subtract the in-flight delta before writing: 1 from remaining-requests, response.usage.total_tokens from remaining-tokens. Fixes both the response headers and (transitively) the prometheus gauges that read from standard_logging_payload.additional_headers. --------- Co-authored-by: cursor <cursor@example.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
user
|
3aef0642a8 |
chore(proxy): also scrub pass_through_endpoints[].target from DB overlay
A pass-through endpoint's ``target`` field is passed through ``create_pass_through_route`` into ``get_instance_fn`` during config load. A PROXY_ADMIN persisting ``target: "s3://attacker/m.i"`` via the DB-overlay ``pass_through_endpoints`` write path was not covered by the previous scrub matrix, so the remote module load would still reach the loader because the YAML-load chain has ``config_file_path`` set. Walk each entry in ``general_settings.pass_through_endpoints`` and null out any ``target`` that starts with ``s3://`` or ``gcs://``. The entry itself is preserved so the path-registration helper can choose how to handle a missing target (the existing code skips the route when ``target is None``). Adds two regression tests. |
||
|
user
|
7575df2549 |
chore(proxy): scrub remote-URL module loads from DB-overlay config
When ``ProxyConfig`` merges DB-persisted ``litellm_settings`` / ``general_settings`` on top of the YAML config, the merged dict is later iterated by ``load_config`` which threads ``config_file_path`` (the YAML path) into ``get_instance_fn``. The runtime gate that refuses ``s3://`` / ``gcs://`` modules when ``config_file_path`` is ``None`` therefore can't distinguish a YAML-sourced value from a DB-sourced one: both look the same to ``get_instance_fn``. Strip ``s3://`` / ``gcs://`` entries from the DB-overlay value for every field whose contents reach ``get_instance_fn`` during config load: - litellm_settings: ``callbacks``, ``success_callback``, ``failure_callback``, ``audit_log_callbacks``, ``post_call_rules``, ``custom_provider_map[].custom_handler`` - general_settings: ``custom_auth``, ``custom_key_generate``, ``custom_key_update``, ``custom_sso``, ``custom_ui_sso_sign_in_handler``, ``litellm_jwtauth.custom_validate`` The YAML config-file load path is unchanged — the documented operator flow (``callbacks: ["s3://bucket/module.instance"]`` in ``config.yaml``) still works. Only DB-overlay writes (e.g. via ``/config/update``) are stripped. Adds 16 regression tests covering the scrub matrix. |
||
|
Krrish DholakiaandClaude Sonnet 4.6
|
b95130eb32 |
fix: block client-side pricing injection via request body
Authenticated clients could supply CustomPricingLiteLLMParams fields (input_cost_per_token, output_cost_per_token, etc.) in the request body. These were forwarded to register_model() in main.py, permanently mutating the shared global litellm.model_cost dict for all users on the instance. Adds all CustomPricingLiteLLMParams fields to _BANNED_REQUEST_BODY_PARAMS so is_request_body_safe() rejects them before they reach completion(). New pricing fields added to CustomPricingLiteLLMParams are auto-covered. Admin opt-in via allow_client_side_credentials or configurable_clientside_auth_params still works as before. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> |
||
|
|
6274b4c217 |
fix(fireworks_ai): strip thinking_blocks from chat messages before Fireworks API call (#27881)
* fix(fireworks_ai): strip thinking_blocks from chat messages before API call Fireworks OpenAI-compatible ChatMessage schema uses additionalProperties:false and rejects Anthropic-style messages[].thinking_blocks (e.g. Claude Code replays), returning invalid_request_error. Remove the field in _transform_messages_helper alongside provider_specific_fields. Adds unit test test_transform_messages_helper_strips_thinking_blocks. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(fireworks_ai): drop inline comments from message sanitization Co-authored-by: Cursor <cursoragent@cursor.com> * docs(fireworks_ai): explain why provider_specific_fields and thinking_blocks are stripped Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
|
b593b88ec6 |
Ishaan - May 13th Staging LiteLLM (#27877)
* fix: strip Gemini thought-signature from tool_use.id in non-streaming path; example websearch config (#27873) - adapters/transformation.py: mirror the streaming path and strip the `__thought__<b64>` suffix off `tool_call.id` before building the AnthropicResponseContentBlockToolUse. Base64's `+ / =` characters violate Anthropic's `^[a-zA-Z0-9_-]+$` tool_use.id pattern, so when a conversation that flowed through Gemini is later replayed to an Anthropic-native provider (Bedrock or Anthropic API) the request 400s. - example_config_yaml/websearch_interception_config.yaml: register the interceptor under `callbacks:` not `success_callback:`. `success_callback` does not run pre-request hooks, so the tool-conversion step never fires on `/v1/messages` and the raw `web_search_20250305` tool is forwarded to Bedrock, which 400s. - adds a unit test pinning the non-streaming strip behavior and the surviving `^[a-zA-Z0-9_-]+$` shape of the resulting id. Co-authored-by: oss-agent-shin <279349115+oss-agent-shin@users.noreply.github.com> * Fix/azure image edit auth header (#27863) * fix(azure/image_edit): use api-key header instead of Authorization Bearer Delegate `AzureImageEditConfig.validate_environment` to `BaseAzureLLM._base_validate_azure_environment` so the image-edit route follows the same auth resolution as every other Azure provider: - prefer the Azure-native `api-key` header when an API key is available - fall back to `Authorization: Bearer <azure_ad_token>` only for AAD auth The previous implementation unconditionally set `Authorization: Bearer <api_key>`, which is the OpenAI-direct convention and is rejected by Azure OpenAI / APIM-fronted deployments with `401 Access denied due to missing subscription key`. Adds regression tests covering api_key kwarg, litellm_params.api_key, and the AAD-token fallback path. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(azure/image_edit): pin api-key precedence semantics + add regression test Address review feedback that the move to ``BaseAzureLLM._base_validate_azure_environment`` changed the relative priority of the positional ``api_key`` kwarg vs. ``litellm_params["api_key"]``. The new behavior — ``litellm_params["api_key"]`` wins, positional only fills in when ``litellm_params["api_key"]`` is empty — is intentional and matches every other Azure ``validate_environment``: ``AzureVideosConfig`` uses the exact same merge logic, while ``AzureVectorStoresConfig`` and ``AzureResponsesAPIConfig`` don't accept a positional ``api_key`` at all. The old ``or`` chain (positional wins) was the outlier and was part of the same OpenAI-vs-Azure convention drift that produced the original ``Authorization: Bearer`` bug. The only production caller (``llm_http_handler.image_edit``) sources both values from the same ``litellm_params.api_key``, so this change is behaviorally a no-op there. Document the precedence in the docstring and lock it in with an explicit test so future refactors can't quietly re-invert it. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: Adam Kirstein <adam.kirstein@disney.com> Co-authored-by: Cursor <cursoragent@cursor.com> * test(azure/image_edit): expect api-key header instead of Authorization Bearer PR #27863 fixed Azure image edit to use the Azure-native api-key header instead of OpenAI's Authorization: Bearer convention, but did not update test_azure_image_edit_litellm_sdk to match. The test still asserted 'Authorization' in headers, which now fails since the new code routes through BaseAzureLLM._base_validate_azure_environment and emits api-key when an api_key is provided. Update the assertion to pin the correct Azure behavior: api-key header present with the resolved key, and no Authorization header. --------- Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: oss-agent-shin <279349115+oss-agent-shin@users.noreply.github.com> Co-authored-by: Adam Kirstein <107421694+justalittleadam@users.noreply.github.com> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: Adam Kirstein <adam.kirstein@disney.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> |
||
|
|
bbeb094d00 |
Litellm agent oss staging 05 11 2026 (#27733)
* fix(ollama): Include provider in model list for ollama (#26135) * Include provider in model names for ollama * Fix unit tests * fix(ollama): process both thinking and content in same streaming chunk (#26098) * fix(health_check): skip max_tokens for image_generation mode (#26417) * fix(health_check): skip max_tokens for image_generation mode `_update_litellm_params_for_health_check` injected `max_tokens` for every deployment. OpenAI `/v1/images/generations` strictly rejects unknown fields, so health checks for dall-e-* and gpt-image-1 always failed with `400 "Unknown parameter: 'max_tokens'"` even though the actual image endpoint calls succeed. Skip the `max_tokens` injection when `model_info.mode == "image_generation"`. `messages` still gets injected (downstream `_filter_model_params` already strips it for non-chat handlers). * Switch to allow-list with per-deployment override Per @krrishdholakia review: deny-listing image_generation only re-introduces the same bug for every other non-chat mode (embedding, audio_*, rerank, video_generation, ocr, search, moderation, ...). Replace the single image_generation skip with `_MAX_TOKEN_SUPPORT_MODES = {chat, completion, responses}`. Missing `mode` is treated as chat for backward compatibility. New modes are safe by default. Add `model_info.health_check_supports_max_tokens` as an operator escape hatch — True forces injection on a non-listed deployment (operator wants to bound probe tokens), False suppresses it on a chat-style deployment behind a strict-schema provider. Tests: parametrize over 3 chat-style + 10 non-chat modes, plus override on/off and the no-mode legacy path. * fix(http_handler): handle RequestNotRead in MaskedHTTPStatusError for multipart uploads (#26718) Squash-merged by litellm-agent from dawidkulpa's PR. * fix(ollama): guard against double 'ollama/' prefix in live model listing Greptile flagged that Ollama servers can return names that already start with 'ollama/'. Check the prefix before prepending so we don't produce 'ollama/ollama/...'. Adds a regression test. * Fix Ollama empty reasoning stream chunks Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: James Myatt <james@jamesmyatt.co.uk> Co-authored-by: VHash <225398745+vhash0@users.noreply.github.com> Co-authored-by: hayden <sewhan.kim+@a-bly.com> Co-authored-by: dawidkulpa <84176950+dawidkulpa@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude <claude@anthropic.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
|
06c1c3497f |
fix: Fix Redis Sentinel client handling to solve authentication error… (#26302)
* fix: Fix Redis Sentinel client handling to solve authentication error with password protected sentinel (#25625) * fix Redis Sentinel authentication handling * test: cover Redis Sentinel auth routing * refactor: align Redis Sentinel kwargs threading * fix: avoid duplicate Redis Sentinel socket timeouts * Address review comments * refactor(_redis): return set from _get_redis_kwargs for O(1) lookup Align _get_redis_kwargs() with the cluster helper by returning a set instead of a list, so the sentinel connection-kwargs filter uses O(1) membership tests. Addresses Greptile review feedback on PR #26302. * fix(_redis): restore Azure-specific kwargs in cluster kwargs set The set-literal refactor of _get_redis_cluster_kwargs dropped four LiteLLM-custom Azure keys (azure_redis_ad_token, azure_client_id, azure_tenant_id, azure_client_secret) that the prior list form had explicitly appended. Because they are not in RedisCluster's argspec, they were silently stripped, breaking Azure IAM auth on cluster clients. Re-add them to the explicit include set. --------- Co-authored-by: Kristin Cowalcijk <kristincowalcijk@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: krrish-berri-2 <krrish-berri-2@users.noreply.github.com> Co-authored-by: claude <claude@anthropic.com> |
||
|
Yuneng Jiang
|
d6fa08307b |
fix(proxy): expose db status on public /health/readiness
External readiness probes consumed the legacy detailed payload's `db`
field to drive alerting and pod-rotation decisions. Stripping the body
to `{"status": "healthy"}` broke those probes silently — the HTTP code
still flipped to 503, but probes checking `body.db == "connected"`
treated the response as healthy.
Add `db` back to the unauthenticated payload. Keep the rest of the
diagnostic fields (litellm_version, callbacks, cache, log_level) gated
behind /health/readiness/details so the recon-leak gate from #26912
holds. Values match the legacy contract: "connected", "disconnected",
"Not connected".
|