* fix(proxy): honor object_permission for managed vector store access
* perf(proxy): preload team object_permission on UserAPIKeyAuth
Populate team_object_permission during virtual-key and JWT auth when the
team is loaded, so can_user_access_vector_store uses it in memory first
and only falls back to get_object_permission by id when missing.
Made-with: Cursor
* fix(team_endpoints): auto-add SSO team members to org for proxy admins
* test: proxy_admin vs team_admin security boundary for team→org move
* screenshots: before/after for team-org SSO fix
* fix(team_endpoints): restore staging security features dropped in SSO commit
Co-Authored-By: Ishaan Jaff <ishaan@berri.ai>
* style: black formatting for team_endpoints
MCP server CRUD endpoints (/v1/mcp/server*) were bundled with MCP
tool-call / passthrough endpoints under llm_api_routes, so setting
DISABLE_LLM_API_ENDPOINTS=true on admin-only nodes also blocked the
Admin UI from listing, adding, or attaching MCP servers.
Separate mcp_inference_routes (data-plane, gated by
DISABLE_LLM_API_ENDPOINTS) from mcp_management_routes (control-plane,
gated by DISABLE_ADMIN_ENDPOINTS). Keep mcp_routes as a union for
backward compat with allowed_routes=["mcp_routes"] virtual key configs.
Upgrade is_management_route to pattern-aware matching so
/v1/mcp/server/{path:path} resolves for concrete IDs.
Temporary MCP OAuth sessions were kept in process-local memory, so on
multi-instance/LB proxy deployments a session created on instance A could
not be found when the follow-up /server/oauth/{server_id}/... request
landed on instance B.
Persist temporary session records to Redis (encrypted with the existing
proxy encryption helpers) as a best-effort L2 cache alongside the current
in-memory L1. Convert get_cached_temporary_mcp_server to async and await
it from the authorize/token/register OAuth endpoints.
Made-with: Cursor
Vertex multi-region endpoints (e.g. us, eu) use the rep host pattern, not
{geo}-aiplatform.googleapis.com. Regional IDs still contain a hyphen.
common_utils.get_vertex_base_url centralizes the rule for SDK/API URL building.
Proxy pass-through duplicates the same branching in a local get_vertex_base_url
(with trailing slashes) to avoid importing from common_utils there; live
WebSocket passthrough uses the same multi-region host logic for wss://.
Tests cover us/eu for the common_utils helper.
Made-with: Cursor
Sibling tests were mutating litellm.proxy.proxy_server.master_key and
prisma_client with raw setattr. Values leaked across tests in the same
xdist worker, flipping the auth short-circuit in user_api_key_auth and
causing unrelated tests (e.g. test_ui_view_session_spend_logs_pagination)
to return 401 instead of 200.
Replace raw setattr with monkeypatch in the two offending files and add
an autouse conftest fixture that snapshots/restores the known-leaky
module globals for every proxy test.
Two fixes to proxy-db CI:
1. test_realtime_webrtc_endpoints.py's `proxy_app` fixture mutated the
module-global `proxy_server.master_key` without restoring it, leaking
state into any test that shared the same xdist worker. Under
--dist=loadscope with 2 workers (GHA proxy-endpoints), this caused the
google_endpoints tests to fail with "No api key passed in." because
user_api_key_auth saw a set master_key and a missing API key on the
test request. The fixture now saves and restores the original value.
2. Address the Greptile note that the semantic shard design has no
catch-all, so a new test file added to tests/proxy_unit_tests/ without
a matrix entry would silently skip CI. Adds an assert-shard-coverage
job that enumerates test_*.py files and fails the workflow if any are
not referenced by a matrix entry, with a clear message telling the
author which semantic shard to place it in. All proxy-db shards now
depend on this guard.
Two independent deflakes:
1. test_ui_view_spend_logs_unauthorized (unit) was returning 400 instead
of 401/403 when earlier tests in the file left proxy-auth globals
(prisma_client, master_key, user_custom_auth, general_settings,
user_api_key_cache) in a state that let invalid tokens pass auth and
fall through to the endpoint's own start_date/end_date validation.
Add an autouse fixture that pins those globals to their import-time
defaults for every test in the file. Harden the assertion to include
response body so future flakes are diagnosable.
2. test_basic_spend_accuracy (CI job proxy_spend_accuracy_tests) depends
on the Redis transaction buffer flushing spend to Postgres. The buffer
uses a single global pod-lock key (cronjob_lock:db_spend_update_job)
and a single global buffer list key. Pointing the proxy at the shared
remote Redis means concurrent CI pipelines contend for the same lock
and can drain each other's buffer into the wrong database. Add a
start_redis reusable command that boots a per-job redis:7-alpine
container (digest-pinned), and switch proxy_spend_accuracy_tests to
REDIS_HOST=host.docker.internal:6379 so lock and buffer state are
isolated per CI run.
The periodic budget-window reset job filtered keys/teams with
`where={"budget_limits": {"not": None}}`. The prisma-client-python
library does not support null-filtering on `Json?` columns (no
DbNull/JsonNull sentinel — upstream issue #714). The client drops the
`None` value during serialization and the engine rejects the query with
`MissingRequiredValueError: where.budget_limits.not: A value is
required but not set`, so neither the key nor team reset path runs.
Switch those two `find_many` calls to `query_raw` with
`WHERE budget_limits IS NOT NULL`, selecting only the PK and the
`budget_limits` column. Writes still go through the ORM. Add unit tests
covering the expired/unexpired paths for keys and teams, string-encoded
JSON payloads, empty payloads, error isolation between the two paths,
and a regression guard asserting the query still uses `IS NOT NULL`.
* fix(mcp_semantic_tool_filter): match canonical tools that arrive with
a client-side namespace prefix.
`SemanticMCPToolFilter._get_tools_by_names` matched by exact equality
between the canonical name stored in the router
(`<server><MCP_TOOL_PREFIX_SEPARATOR><tool>`) and the name in the
incoming `tools[]` list. MCP clients such as opencode wrap every tool
name with their own additive alias prefix
(`<client_alias>_<canonical>`), so the two never matched, the filter
dropped every tool to zero, and the proxy forwarded `tools: []` with
`tool_choice: auto` — which strict upstream providers reject with a 400.
The fix adds anchored suffix matching with a separator check: the
canonical must form the complete tail of the incoming name and be
preceded by `_` or `-`. Exact matches still win over suffix matches,
incoming tools are returned at most once, and the original tool object
is passed through unchanged so the client-facing name survives for
tool-call round-trips.
Seven unit tests in a new TestGetToolsByNames class cover exact
match, underscore- and dash-prefixed variants, non-separator-anchored
suffixes (which must not match), exact-wins-over-prefixed precedence,
deduplication when two canonicals suffix-match the same incoming tool,
and ordering-follows-router-output.
Fixes#26078
* review: strengthen the suffix-fallback tie-breaker and the
deduplication regression test (Greptile comments on #26117)
- test_same_tool_not_returned_twice now passes two distinct canonicals
("read_file" and "file") that both suffix-match the same incoming
tool, rather than the same canonical twice, so the assertion
actually exercises the used_ids dedup path instead of the
duplicate-input-list path.
- The suffix fallback in _get_tools_by_names now prefers the shortest
incoming name that still qualifies under the separator-anchored
match. In the one-prefix-per-client opencode scenario this is a
no-op, but in multi-namespace configurations the shortest qualifying
name is the least-wrapped one and is the most defensible deterministic
choice, replacing the dict-insertion-order fallback.
- Adds test_suffix_fallback_prefers_shortest_candidate covering the
new tie-breaker directly.
Still 15 tests passing locally (was 14).
* review(#26117): gate suffix-matching on canonical containing MCP_TOOL_PREFIX_SEPARATOR
@krrish-berri-2 flagged a possible collision in the suffix fallback:
a local user function whose name happens to end in a bare canonical
substring (e.g. my_firecrawl_scrape vs canonical firecrawl_scrape)
would be spuriously selected.
Server-registered MCP tools are always emitted as
<server_name><MCP_TOOL_PREFIX_SEPARATOR><tool_name> via
add_server_prefix_to_name, so a canonical without the separator is
not a namespaced MCP tool and does not warrant suffix matching.
Added that guard to _name_matches_canonical with a regression test
(test_does_not_collide_with_local_function_on_unprefixed_canonical)
that reproduces the collision before the fix and is pinned after.
Pre-existing TestGetToolsByNames fixtures that relied on bare
canonicals (get_weather, search, read_file, write/delete/read) were
switched to realistic server-prefixed ones so they continue to
exercise the suffix-fallback path under the new guard. The opencode
scenario (client prefix on already-server-prefixed canonical) is
unchanged.
---------
Co-authored-by: sakenuGOD <sakenuGOD@users.noreply.github.com>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
store_in_memory_spend_updates_in_redis drained the in-memory queues
into local variables before the rpush pipeline. If rpush raised (cloud
Redis hiccup, timeout, connection blip), those already-drained
transactions were garbage-collected with the scheduler job, silently
losing all spend aggregated during that tick.
Wrap the rpush in try/except. On failure, re-enqueue the aggregated
transactions into their respective in-memory queues so the next
scheduler tick retries.
Add a unit test that seeds real queues, simulates an rpush failure,
and asserts the transactions land back in-memory.
Restore guardrail spend/UI event_type wiring, request_data on streaming
OUTPUT paths, and centralized match redaction after the upstream revert.
Made-with: Cursor
Keep anthropic_messages as the logged call_type for non-Anthropic /v1/messages adapter paths and add a regression test to prevent fallback to completion/acompletion.
Made-with: Cursor
Previously, members added to a team without an explicit per-member budget were
all linked to the same `litellm_budgettable` row referenced by the team's
`metadata.team_member_budget_id`. Updating one member's budget via
`/team/member_update` mutated the shared row and silently changed every other
member's budget too.
Now both write paths produce a private, per-member budget:
- `add_new_member` clones the team's default budget into a fresh row when a
member is added without `max_budget_in_team`/`allowed_models`. If no team
default exists, the membership is created with no budget.
- `_upsert_budget_and_membership` detects when an existing membership still
points at the team's default budget id and clones-on-write, relinking the
membership to the new private budget before applying the update.
- `team_member_update` reads `team_member_budget_id` from team metadata and
passes it through so the helper can make this distinction.
Adds unit tests for clone-on-write, in-place update of a private budget, and
the no-default-no-budget add path.
Made-with: Cursor
Existing tests pinned exact kwargs on `PrismaManager.setup_database`,
but the opt-in v2 resolver added `use_v2_resolver=False` to every call.
Update the three assertions to reflect the new signature.
Fixes:
- TestHealthAppFactory::test_use_prisma_db_push_flag_behavior
- TestHealthAppFactory::test_startup_fails_when_db_setup_fails
Adds total_spend column to LiteLLM_TeamMembership that accumulates
continuously and is not zeroed by the budget cycle reset job. This
enables UI surfaces to distinguish current-cycle spend (the existing
spend column, which resets) from lifetime spend per team member.
Also exposes budget_reset_at on LiteLLM_BudgetTable so /team/info
callers can see when a member's budget window next resets. The field
was already stored in the DB but stripped by the response Pydantic
model.
Includes regression tests that:
- Guard the reset job against ever writing total_spend: 0
- Verify the spend writer increments both spend and total_spend in
one UPDATE statement.
Add regression tests that mock make_bedrock_api_request and verify
input_type=request uses source=INPUT with user messages, and
input_type=response uses source=OUTPUT with synthetic ModelResponse.
Made-with: Cursor
* fix: /health/readiness returns 503 when DB is unreachable due to handle_db_exception re-raising
handle_db_exception() re-raises the Prisma exception inside _db_health_readiness_check's
except block, which propagates out to health_readiness() and gets wrapped in a 503.
The health endpoint never reached the reconnect path and the service never recovered.
Fix:
- Remove handle_db_exception() call from _db_health_readiness_check — that helper is
for API request handlers (allow_requests_on_db_unavailable flag), not health checks
- Replace raw disconnect()+connect() with attempt_db_reconnect(), which uses the proper
lock, cooldown, escalation, and heavy-reconnect (recreate_prisma_client) machinery
* test: update health readiness tests for handle_db_exception removal
- Remove tests that expected handle_db_exception to re-raise (old buggy behaviour)
- Remove tests asserting disconnect()/connect() calls (replaced by attempt_db_reconnect)
- Add regression tests covering the 503 loop fix:
- transport errors never raise (ClientNotConnectedError, httpx.ConnectError, etc.)
- reconnect success path returns 'connected'
- reconnect failure path returns 'disconnected' without raising
- non-transport errors return 'disconnected', skip reconnect
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* refactor: new agentic loop event hook
simplifies how to create logic for tool based multi llm calls
* fix: compress - make it work on anthropic input as well
* fix(compress.py): working prompt compression for claude code
ensures claude code messages can run through proxy easily
* docs: add agentic loop hook guide
* docs: add agentic_loop_hook to sidebar
* fix: fix multiple arguments error
* fix: fix tool call loop for compression on streaming /v1/messages
* fix: fix linting errors
* fix: fix ci/cd errors
* feat(litellm_pre_call_utils.py): use claude code session for litellm session id
allows claude code logs to be stitched together, making it easy to know they were all part of the same conversation
* fix: suppress incorrect mypy warning rE: module
* revert: drop PR's changes to litellm/proxy/_experimental/out/
Restores the 34 HTML files under _experimental/out/ to their pre-PR
paths (X/index.html -> X.html). All renames are R100 (content
unchanged); no other files are touched.
* fix: address greptile review comments on PR #25729
- Skip ``kwargs["tools"] = []`` injection when compression is a no-op —
Anthropic Messages rejects empty tool arrays on requests that did not
originally declare tools.
- Move agentic-loop safety guards (fingerprint cycle / max depth) out of
the per-callback try/except so they propagate instead of being swallowed
by the generic exception handler. Extracted _check_agentic_loop_safety.
- Gate generic ``x-<vendor>-session-id`` capture behind the
LITELLM_CAPTURE_VENDOR_SESSION_HEADERS env var (off by default) to
preserve backwards compatibility; explicit x-litellm-* headers are
unaffected.
- Fix monkeypatch target in pre-call-hook test to patch the actual
module-level binding
(litellm.integrations.compression_interception.handler.compress).
- Add regression tests for empty-tools skip and opt-in session capture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* revert: drop LITELLM_CAPTURE_VENDOR_SESSION_HEADERS flag
Generic x-<vendor>-session-id header capture is a new feature and only
runs *after* the explicit x-litellm-trace-id / x-litellm-session-id
checks, so it does not change behavior for any existing caller that was
already using the LiteLLM headers — no backwards-incompatibility to gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(compress): replace input_type with CallTypes call_type
Drop the bespoke ``CompressionInputType`` literal and use the existing
``litellm.types.utils.CallTypes`` enum instead. ``litellm.compress()``
now takes ``call_type: Union[CallTypes, str]`` (default
``CallTypes.completion``) — no new concept to learn, and the enum is
already the way the rest of the codebase talks about request shapes.
Supported values: ``completion`` / ``acompletion`` (OpenAI chat-completions
shape) and ``anthropic_messages`` (Anthropic structured content blocks).
Updated: compress(), the compression_interception handler, tests, docs,
and the two eval scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Per-user OAuth MCP requests now only skip pre-emptive 401 when a stored token is available, preserving token-reuse behavior while restoring fast PKCE kickoff for first-time or missing-token users.
`should_create_missing_views()` had `and result[0]["reltuples"]` which is
falsy when reltuples=0. On a fresh empty PostgreSQL table, CREATE INDEX sets
reltuples=0, causing the guard to return False and skip view creation entirely.
Views like MonthlyGlobalSpendPerKey are never created, and the
/global/spend/logs endpoint returns 500.
Fix: change to `and result[0]["reltuples"] is not None` so reltuples=0
(empty table) and reltuples=-1 (unanalyzed table) both correctly return True.
Also harden test_vertex_ai.py to return None instead of crashing with
JSONDecodeError when the spend-logs endpoint returns a non-JSON 500 response,
and add unit tests covering all three reltuples branches (0, -1, positive).
test_virtual_key_max_budget_alert_check_per_key_overrides_global asserted
override semantics but the implementation does additive merge. Renamed test
and updated assertion to match: per-key and global thresholds are unioned,
not replaced.
