store_in_memory_spend_updates_in_redis drained the in-memory queues
into local variables before the rpush pipeline. If rpush raised (cloud
Redis hiccup, timeout, connection blip), those already-drained
transactions were garbage-collected with the scheduler job, silently
losing all spend aggregated during that tick.
Wrap the rpush in try/except. On failure, re-enqueue the aggregated
transactions into their respective in-memory queues so the next
scheduler tick retries.
Add a unit test that seeds real queues, simulates an rpush failure,
and asserts the transactions land back in-memory.
Restore guardrail spend/UI event_type wiring, request_data on streaming
OUTPUT paths, and centralized match redaction after the upstream revert.
Made-with: Cursor
Previously, members added to a team without an explicit per-member budget were
all linked to the same `litellm_budgettable` row referenced by the team's
`metadata.team_member_budget_id`. Updating one member's budget via
`/team/member_update` mutated the shared row and silently changed every other
member's budget too.
Now both write paths produce a private, per-member budget:
- `add_new_member` clones the team's default budget into a fresh row when a
member is added without `max_budget_in_team`/`allowed_models`. If no team
default exists, the membership is created with no budget.
- `_upsert_budget_and_membership` detects when an existing membership still
points at the team's default budget id and clones-on-write, relinking the
membership to the new private budget before applying the update.
- `team_member_update` reads `team_member_budget_id` from team metadata and
passes it through so the helper can make this distinction.
Adds unit tests for clone-on-write, in-place update of a private budget, and
the no-default-no-budget add path.
Made-with: Cursor
Existing tests pinned exact kwargs on `PrismaManager.setup_database`,
but the opt-in v2 resolver added `use_v2_resolver=False` to every call.
Update the three assertions to reflect the new signature.
Fixes:
- TestHealthAppFactory::test_use_prisma_db_push_flag_behavior
- TestHealthAppFactory::test_startup_fails_when_db_setup_fails
Add regression tests that mock make_bedrock_api_request and verify
input_type=request uses source=INPUT with user messages, and
input_type=response uses source=OUTPUT with synthetic ModelResponse.
Made-with: Cursor
* fix: /health/readiness returns 503 when DB is unreachable due to handle_db_exception re-raising
handle_db_exception() re-raises the Prisma exception inside _db_health_readiness_check's
except block, which propagates out to health_readiness() and gets wrapped in a 503.
The health endpoint never reached the reconnect path and the service never recovered.
Fix:
- Remove handle_db_exception() call from _db_health_readiness_check — that helper is
for API request handlers (allow_requests_on_db_unavailable flag), not health checks
- Replace raw disconnect()+connect() with attempt_db_reconnect(), which uses the proper
lock, cooldown, escalation, and heavy-reconnect (recreate_prisma_client) machinery
* test: update health readiness tests for handle_db_exception removal
- Remove tests that expected handle_db_exception to re-raise (old buggy behaviour)
- Remove tests asserting disconnect()/connect() calls (replaced by attempt_db_reconnect)
- Add regression tests covering the 503 loop fix:
- transport errors never raise (ClientNotConnectedError, httpx.ConnectError, etc.)
- reconnect success path returns 'connected'
- reconnect failure path returns 'disconnected' without raising
- non-transport errors return 'disconnected', skip reconnect
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* refactor: new agentic loop event hook
simplifies how to create logic for tool based multi llm calls
* fix: compress - make it work on anthropic input as well
* fix(compress.py): working prompt compression for claude code
ensures claude code messages can run through proxy easily
* docs: add agentic loop hook guide
* docs: add agentic_loop_hook to sidebar
* fix: fix multiple arguments error
* fix: fix tool call loop for compression on streaming /v1/messages
* fix: fix linting errors
* fix: fix ci/cd errors
* feat(litellm_pre_call_utils.py): use claude code session for litellm session id
allows claude code logs to be stitched together, making it easy to know they were all part of the same conversation
* fix: suppress incorrect mypy warning rE: module
* revert: drop PR's changes to litellm/proxy/_experimental/out/
Restores the 34 HTML files under _experimental/out/ to their pre-PR
paths (X/index.html -> X.html). All renames are R100 (content
unchanged); no other files are touched.
* fix: address greptile review comments on PR #25729
- Skip ``kwargs["tools"] = []`` injection when compression is a no-op —
Anthropic Messages rejects empty tool arrays on requests that did not
originally declare tools.
- Move agentic-loop safety guards (fingerprint cycle / max depth) out of
the per-callback try/except so they propagate instead of being swallowed
by the generic exception handler. Extracted _check_agentic_loop_safety.
- Gate generic ``x-<vendor>-session-id`` capture behind the
LITELLM_CAPTURE_VENDOR_SESSION_HEADERS env var (off by default) to
preserve backwards compatibility; explicit x-litellm-* headers are
unaffected.
- Fix monkeypatch target in pre-call-hook test to patch the actual
module-level binding
(litellm.integrations.compression_interception.handler.compress).
- Add regression tests for empty-tools skip and opt-in session capture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* revert: drop LITELLM_CAPTURE_VENDOR_SESSION_HEADERS flag
Generic x-<vendor>-session-id header capture is a new feature and only
runs *after* the explicit x-litellm-trace-id / x-litellm-session-id
checks, so it does not change behavior for any existing caller that was
already using the LiteLLM headers — no backwards-incompatibility to gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(compress): replace input_type with CallTypes call_type
Drop the bespoke ``CompressionInputType`` literal and use the existing
``litellm.types.utils.CallTypes`` enum instead. ``litellm.compress()``
now takes ``call_type: Union[CallTypes, str]`` (default
``CallTypes.completion``) — no new concept to learn, and the enum is
already the way the rest of the codebase talks about request shapes.
Supported values: ``completion`` / ``acompletion`` (OpenAI chat-completions
shape) and ``anthropic_messages`` (Anthropic structured content blocks).
Updated: compress(), the compression_interception handler, tests, docs,
and the two eval scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Per-user OAuth MCP requests now only skip pre-emptive 401 when a stored token is available, preserving token-reuse behavior while restoring fast PKCE kickoff for first-time or missing-token users.
`should_create_missing_views()` had `and result[0]["reltuples"]` which is
falsy when reltuples=0. On a fresh empty PostgreSQL table, CREATE INDEX sets
reltuples=0, causing the guard to return False and skip view creation entirely.
Views like MonthlyGlobalSpendPerKey are never created, and the
/global/spend/logs endpoint returns 500.
Fix: change to `and result[0]["reltuples"] is not None` so reltuples=0
(empty table) and reltuples=-1 (unanalyzed table) both correctly return True.
Also harden test_vertex_ai.py to return None instead of crashing with
JSONDecodeError when the spend-logs endpoint returns a non-JSON 500 response,
and add unit tests covering all three reltuples branches (0, -1, positive).
test_virtual_key_max_budget_alert_check_per_key_overrides_global asserted
override semantics but the implementation does additive merge. Renamed test
and updated assertion to match: per-key and global thresholds are unioned,
not replaced.
Project-level model rpm/tpm limits stored in project_metadata were never
checked during rate limit enforcement — only model-level limits applied.
Adds _add_project_model_rate_limit_descriptor_from_metadata() to the v3
limiter (mirrors the existing team metadata path) and calls it in
async_pre_call_hook, creating a model_per_project descriptor keyed as
"{project_id}:{model}" with the project's configured limits.
Also extends get_model_rate_limit_from_metadata's Literal to accept
"project_metadata" and adds get_project_model_rpm/tpm_limit helpers.
Fixes: LIT-2317
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Guard empty recipients in _handle_multi_threshold_max_budget_alert:
log warning and skip instead of falling through to old path error loop
- Widen max_budget_alert_emails type to Dict[str, Union[str, List[str]]]
to match _parse_email_list runtime behavior (accepts comma-separated strings)
- Pre-filter asyncio.create_task with min threshold check to avoid
unnecessary task allocation on every request when spend is below
all configured thresholds
When litellm.max_end_user_budget_id is configured, implicitly-created end users
(via /chat/completions) have budget_id=NULL in the DB since the default budget
is only applied in-memory. The budget reset job filtered by budget_id, so these
users were never reset and eventually permanently blocked.
Fix: when the default budget is in the reset list, also query for and reset
end users with budget_id=NULL and spend > 0. This keeps the hot auth path
unchanged (no DB writes on every request).
Fixes#22019
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add `default_key_max_budget_alert_emails` litellm_settings config as
global fallback for all virtual keys (per-key metadata takes priority)
- Fix crash when key has no user_id/user_email by passing recipient email
to _get_email_params (same pattern as team soft budget path)
- Use owner email for greeting, falling back to key_alias or token
- Rename setting from default_max_budget_alert_emails to
default_key_max_budget_alert_emails for clarity
Users can set metadata.max_budget_alert_emails as a JSON map of threshold
percentages to email recipients on virtual keys. When configured, the email
handler loops over each threshold, checks per-threshold dedup cache, and
sends to the configured recipients (auto-including the key owner's email).
When no map is set, the existing single 80% threshold behavior is preserved
unchanged. Teams support is out of scope for this v0.
- url_utils.py: narrow sockaddr[0] from str|int to str via a helper with a
fail-closed isinstance check. Fixes the two mypy errors introduced by
the SSRF hardening without masking unexpected stdlib behavior.
- key_management_endpoints.py: restore the documented team member_permissions
path for /key/update. The cross-key admin check added to close the
cross-org rewrite attack was over-broad: it rejected non-admin team
members even when can_team_member_execute_key_management_endpoint had
already validated their team membership and /key/update grant. Now skip
the admin check when the key has a team_id and the change is non-budget
(membership + permission already enforced above). Budget/spend changes
still require team/org admin. The cross-org attack remains blocked:
an outside org admin fails the earlier team membership check.
- test_logging_redaction_e2e_test.py: rename and rewrite two parametrized
tests to assert that request-body turn_off_message_logging has no effect.
Reflects the intentional removal of turn_off_message_logging from
_supported_callback_params so the caller cannot override admin logging
policy via the request body.
- test_key_management_endpoints.py: add two tests covering the restored
team member permission path — one positive (non-budget update succeeds
for a team member with /key/update grant), one negative (max_budget
change still rejected without admin role).
Compute supports_reasoning once per non-wildcard health-check resolution path and update the stale default-max-tokens test docstring.
Made-with: Cursor
Apply reasoning-first precedence for background health-check max tokens, parse reasoning env as optional, and raise non-wildcard fallback max_tokens from 1 to 5 for better reliability.
Made-with: Cursor
Audit-B #2. _check_key_admin_access was gated on max_budget/spend
changes only, which meant a non-admin caller could blanket-rewrite
any OTHER field on any key (key_alias, models, tpm_limit, rpm_limit,
metadata, tags, allowed_routes, guardrails, blocked, duration,
permissions, auto_rotate, access_group_ids, object_permission, …)
as long as they avoided budget/spend. Example attack:
POST /key/update {
key: sk-victim-in-org-B,
models: [],
blocked: true,
organization_id: org-A,
}
The caller is org-admin of org-A, which satisfies the route gate;
the handler then wipes models and blocks the victim's key.
Policy after this fix:
- PROXY_ADMIN: always allowed.
- Key OWNER (matching user_id): allowed for non-budget fields;
budget/spend changes still require team/org admin.
- Everyone else: must pass _check_key_admin_access (PROXY_ADMIN /
key-owner / team-admin / org-admin of the key).
Regression test confirms a non-owner INTERNAL_USER cannot rewrite
key_alias/blocked on someone else's key; existing test covers the
owner-can-update-alias case; existing test covers
internal-user-cannot-modify-max-budget.
Continuation of Veria E3NpkuAd / Audit-B hardening. All three are the
same anti-pattern PR #25904 already addressed for _user_is_org_admin
and /user/delete: route-level gate trusts a caller-supplied scope
field, handler operates on a different scope.
1. /user/update no longer silently creates a user when the target
email doesn't exist. Pre-fix, an org admin could supply a fresh
email + caller-chosen budget/models/metadata; the INSERT path
created the user with no org attachment, bypassing /user/new's
org/team authorization. Now require PROXY_ADMIN for the create
branch; return 404 otherwise. Also fixes /user/bulk_update because
it dispatches through the same _update_single_user_helper.
2. /team/bulk_member_add with all_users=true restricted to PROXY_ADMIN.
The flag pulls every user in the database into the target team,
ignoring org scope — any team admin could use it to capture every
user across every org into their team.
3. /team/update now verifies destination-org admin rights. When the
request carries an organization_id that differs from the team's
current org, an org admin of the caller's current org could
previously relocate the team into any other org (draining their
resources, or capturing a team they once administered). Require
PROXY_ADMIN or org-admin of the DESTINATION org for the relocation.
Regression tests for #1 and #3; #2 covered by the existing bulk_add
suite after the gate addition.
Greptile P2. The admin-inject gate only removed tags from data['metadata']
and data['litellm_metadata']; and the
policy engine read directly, so a caller without
allow_client_tags could still drive tag-based policy decisions by moving
tags to the body root. Also strip the root key in the same branch.
