Conflict resolution for #26968 dropped the `Iterator` typing import
(NameError at module load), left a dead `fallback_models = cast(...)`
block, and the new tests called `_enforce_key_and_fallback_model_access`
without the now-required `request` kwarg.
* fix(proxy): re-validate user_id ownership after /user/info re-parses query
The route-level access check in `RouteChecks.non_proxy_admin_allowed_routes_check`
reads `request.query_params.get("user_id")`, which decodes literal `+` to
spaces. The endpoint then re-parses the raw query string with `urllib.unquote`
in `get_user_id_from_request` to preserve `+` characters (so plus-addressed
emails work as user_ids). Those two paths produce different ids: a caller
who registered a user_id containing a literal space could pass the route
check and then read another user's row by sending the encoded `+` form.
Add `_enforce_user_info_access` and call it after `_normalize_user_info_user_id`
returns the final id. Proxy admin / view-only admin still bypass; everyone
else must match the resolved user_id (or have no user_id, which falls back
to the caller's own id later in the handler).
Tests cover the admin bypass, owner-match path, and the cross-user lookup
that this change blocks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(proxy): apply user_info ownership check to PROXY_ADMIN_VIEW_ONLY
`_enforce_user_info_access` was bypassing both PROXY_ADMIN and
PROXY_ADMIN_VIEW_ONLY, but the upstream route check in
`RouteChecks.non_proxy_admin_allowed_routes_check` only treats
PROXY_ADMIN as a true admin for the `/user/info` route — view-only
admins go through the `user_id == valid_token.user_id` enforcement
along with regular users. Mirroring that asymmetry left the same
encoded-`+` bypass open for view-only admins whose user_id contains a
literal space.
Drop the PROXY_ADMIN_VIEW_ONLY exemption so the post-decode re-check
matches the upstream rule. Update tests: a view-only admin must now
be blocked from cross-user lookups but still allowed to read their
own row.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(batches): count non-chat tokens and validate every model in batch file
Two security control bypasses on POST /v1/batches:
1. `_get_batch_job_input_file_usage` only summed tokens for
`body.messages` (chat completions). Embedding (`input`) and text
completion (`prompt`) batches reported zero, letting massive
non-chat workloads slip past TPM rate limits. Extend the counter
to handle string and list shapes for both fields.
2. The batch input file was forwarded to the upstream provider
without inspecting the models named inside the JSONL — only the
outer `model` query parameter was checked against the caller's
allowlist. A caller restricted to gpt-3.5 could submit a batch
targeting gpt-4o and the upstream would execute it under the
proxy's shared API key.
Add `_get_models_from_batch_input_file_content` (returns the
distinct `body.model` values) and call it from
`_enforce_batch_file_model_access` in the pre-call hook, which runs
each model through `can_key_call_model` so the same allowlist
semantics (wildcards, access groups, all-proxy-models, team aliases)
the proxy enforces on `/chat/completions` apply here too. Any
unauthorized model raises a 403 before the file is forwarded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(batches): count pre-tokenized prompt/input shapes, classify 403 logs
Two follow-ups from the Greptile review on the batch validation PR:
1. P1 TPM bypass via integer token arrays. The OpenAI batch schema
accepts ``prompt`` and ``input`` as ``list[int]`` (a single
pre-tokenized prompt) or ``list[list[int]]`` (multiple) in addition
to the string and ``list[str]`` shapes. Pre-fix only the string
shapes were counted, so a caller could submit a batch with hundreds
of millions of pre-tokenized tokens and the rate limiter would
record zero. Extract the per-field logic into
``_count_prompt_or_input_tokens`` and count each int as one token.
2. P2 access-denial logs were indistinguishable from I/O failures.
``count_input_file_usage`` caught every exception under a generic
"Error counting input file usage" message, so an intentional 403
from ``_enforce_batch_file_model_access`` looked the same in the
logs as a missing file or a Prisma timeout. Catch ``HTTPException``
separately and log 403s at WARNING level with a security-relevant
message before re-raising.
Tests cover the new shapes: single ``list[int]``, ``list[list[int]]``
(the worst-case bypass vector), and embeddings ``input`` with
pre-tokenized arrays.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reset the policy ID index during policy engine test cleanup so stale policy versions cannot leak between tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
Mark runtime-created policies and attachments initialized so global policy attachments created from the policy builder apply immediately without requiring a restart.
Co-authored-by: Cursor <cursoragent@cursor.com>
When a team grants /key/list via team_member_permissions, non-admin members
should see all keys for that team — same as a team admin. Previously the
classification in list_keys() only checked admin status, so permitted
members fell into the service-account-only path and could not see other
members' personal keys. Routes those members into the full-visibility set.
Greptile P1: the unsafe-method branch of `_check_proxy_admin_viewer_access`
ended with a blanket `if route in management_routes: return`. That set is a
mix of reads (info/list — handled via the safe-method GET branch above) and
writes. The fallback let Admin Viewer POST to write endpoints not enumerated
in `_ADMIN_VIEWER_BLOCKED_WRITE_ROUTES`, including:
- /team/block, /team/unblock, /team/permissions_update
- /jwt/key/mapping/{new,update,delete}
- /key/bulk_update
- /key/{key_id}/reset_spend
Remove the fallback. The two remaining allow sets (admin_viewer_routes and
global_spend_tracking_routes) are both read-only, so removal does not affect
the legitimate POST-as-read cases (e.g. /spend/calculate, which is in
spend_tracking_routes ⊂ admin_viewer_routes).
Tests:
- 8 new parametrized cases pinning each previously-leaking management write
endpoint to 403 on POST for PROXY_ADMIN_VIEW_ONLY.
The reservation path (PR #26845) atomically pre-fills `spend:user:{user_id}`
and admits at the strict-`<` boundary. The legacy `_PROXY_MaxBudgetLimiter`
pre-call hook re-reads the same counter with `>=`, so a reservation that
fills the counter to exactly `max_budget` (e.g. a request without a
`max_tokens` cap that falls back to reserving the smallest remaining
headroom) is rejected by the hook even though the reservation already
admitted it.
Skip the hook when the request's active `budget_reservation` covers
`spend:user:{user_id}`. The reservation is the source of truth for that
counter cross-pod; the legacy `>=` path remains in place for requests
without a reservation (e.g. paths that bypass the reservation entirely).
Reproduces as `tests/otel_tests/test_prometheus.py::test_user_budget_metrics`
on a fresh user with `max_budget=10` calling `fake-openai-endpoint` without
`max_tokens`. Adds focused unit coverage in
`tests/test_litellm/proxy/hooks/test_max_budget_limiter.py`.
Greptile flagged two follow-ups on the OpenAPI/local-registry pre-call
check:
1. **P1 runtime crash via None proxy_logging_obj.**
`kwargs.get("proxy_logging_obj")` is `None` on the MCP entry path,
and `pre_call_tool_check` calls `proxy_logging_obj._create_mcp_request_object_from_kwargs`
unconditionally after the security checks, which would have crashed
every legitimate call with `AttributeError`. Source the logging
object from `litellm.proxy.proxy_server` the same way
`_handle_managed_mcp_tool` already does.
2. **P2 authorization-bypass window when mcp_server is None.**
Previously the new check was guarded by `if mcp_server is not None`,
so any local tool whose registry entry had no resolvable server (a
startup-race window before `_initialize_tool_name_to_mcp_server_name_mapping`
completes, or an orphaned registry entry) ran without the security
check. Tools registered via openapi_to_mcp_generator are always tied
to a server, so a missing one is a configuration/timing fault — fail
the call with 503 instead of dispatching unguarded.
Tests: existing two pass with an added assertion that
`proxy_logging_obj` is non-None at the call site, plus a new test that
covers the 503 deny branch when the tool→server mapping is missing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`execute_mcp_tool` dispatches in two ways: managed MCP servers go
through `_handle_managed_mcp_tool`, which calls
`MCPServerManager.pre_call_tool_check` to enforce allowed/banned tool
lists, key/team `object_permission` tool grants, and parameter
validation. OpenAPI-backed tools, however, were resolved via
`global_mcp_tool_registry` and dispatched directly to
`_handle_local_mcp_tool` — entirely skipping `pre_call_tool_check`.
A caller could invoke any registered OpenAPI tool regardless of their
key/team permissions, including administrative or destructive
operations on the upstream API.
Run `pre_call_tool_check` before the local-registry dispatch whenever
the resolved server is set (the same condition used to surface server
context to the managed path). Honor any guardrail-modified arguments
the hook returns. Errors raised by the hook propagate up before
`_handle_local_mcp_tool` runs.
Tests cover both directions: the pre-call hook fires when the local
tool resolves alongside a server, and a hook-raised HTTPException
prevents the local handler from being invoked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related authorization gaps in management endpoints:
1. `/project/update` evaluated permission against the team_id supplied in
the request body. By passing `data.team_id` pointing at a team they
admin, a caller could hijack any project — `_check_user_permission_for_project`
was given the attacker's team_object and happily checked admin
membership against that. Drop the team_object kwarg so the helper
re-fetches the existing project's team. Also require admin rights on
the destination team when reassigning a project across teams, so a
team admin cannot shed projects into another team's namespace.
2. `/key/update` accepted any `organization_id` and only checked that
the org existed before applying limits. A caller could thereby point
their key at an arbitrary org. Add `_validate_caller_can_assign_key_org`
which enforces the same membership rule already applied on the
`/key/list` filter path (`validate_key_list_check`); proxy admins and
no-change updates skip the check.
Tests cover both helpers in isolation: existing-team-admin allow,
unrelated-team admin deny, proxy-admin shortcut, org-member allow,
non-member deny, missing user_id deny, no-memberships deny.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The GitHub merge conflict resolver concatenated both test sets but left
`from litellm.proxy._lazy_openapi_snapshot import _normalize_operation_ids`
stranded between functions instead of at the top of the file.
A non-admin scoped to ["model-a"] could call /health?model_id=id-b
(where id-b belongs to a deployment outside their scope) and the
background-cache code path would return id-b's cached health entry. The
helper returned {model_id} unconditionally, so the cache filter was
driven by an unvalidated id and the global cache leaked the entry — the
ternary `targeted_ids if not None else allowed_model_ids` skipped any
intersection with the caller's allowed deployments.
Make _resolve_targeted_model_ids walk the supplied model_list for both
the model and model_id branches. Callers pass an already-scoped list
(filtered to allowed model_names for non-admins, full list for admins),
so an out-of-scope model_id resolves to an empty set and the cache
filter drops every entry — matching the live path's existing behavior.
When JWT auth is enabled but `JWT_AUDIENCE` is unset, `auth_jwt`
disabled audience verification entirely. Tokens minted by any other
application that shared the same IdP signing keys (Azure AD, Okta,
etc.) were accepted as long as their signature checked out, even
though their `aud` and `iss` claims pointed at unrelated apps. The
proxy then fell into the no-team / no-user branch where access checks
default-allow.
This change:
1. Adds support for the `JWT_ISSUER` env var. When set, PyJWT verifies
the token's `iss` claim — turning on the same defense for tokens
that share an audience but come from a different IdP tenant.
2. Refactors the duplicated `jwt.decode` calls (RSA/EC/OKP path and
x509 path) into a single `_build_decode_kwargs` helper that
computes audience, issuer, and the corresponding `verify_*` opt-outs
once per call.
3. Logs a single startup-time warning when JWT auth is enabled but
neither `JWT_AUDIENCE` nor `JWT_ISSUER` is configured, so operators
running the insecure default see a flag in their logs without
getting spammed per-request.
Default behavior (no env vars) is preserved for backward compatibility.
Setting `JWT_AUDIENCE` and/or `JWT_ISSUER` opts into the verification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per review: `assert response.status_code != 503` is satisfied by 404,
500, or any other non-503 code, so a regression that returned the wrong
non-503 status would slip through. Switch to `== 200` so the assertions
verify the actual expected status, not just the absence of one specific
failure.
When use_background_health_checks is enabled, /health?model=foo returned
the full cached aggregate across every model — so an unhealthy foo
combined with any other healthy deployment kept healthy_count > 0 and
the targeted-503 path never fired.
Resolve the targeted model/model_id to a deployment-id set first
(mirroring perform_health_check's match-on-model_name-or-litellm_model
semantics) and narrow the cache to those IDs before _post_process
evaluates healthy_count, so the 503 contract holds for both the live
and cache code paths.
/health?model=foo and /health?model_id=foo previously returned HTTP 200
even when zero endpoints were healthy, forcing monitoring systems to
parse the JSON body to detect failure. /health/readiness similarly
returned 200 even when a configured Prisma DB was unreachable, leaving
unhealthy pods in rotation.
Both endpoints now flip to HTTP 503 in the failure case while keeping
the JSON response body identical, so existing parsers continue to work
and orchestrators can rely on the HTTP status alone.
If atomic_check_and_increment_by_n returns overall_code=OVER_LIMIT but no
status entry matches a descriptor key the dynamic limiter dispatcher knows
how to translate into a 429 (`model_saturation_check` or `priority_model`),
the for-loop previously exited cleanly and execution fell through to the
priority-tracking increment + the data["litellm_proxy_rate_limit_response"]
write — silently admitting an over-limit request.
This is the fail-open path a future contributor would hit by wiring a new
descriptor type into enforced_descriptors without updating the dispatcher.
Refuse the request with a generic 429 carrying the offending descriptor
metadata so the operator can see what slipped past, and emit an error log
to surface the wiring gap.
Adds a regression test (test_dynamic_rate_limiter_v3_fails_closed_on_unknown_descriptor)
that drives the limiter with a synthetic OVER_LIMIT response carrying an
unrecognized descriptor_key and asserts a 429 is raised.
Tests: 65 passed (1 skipped), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: add expected_base_url origin check to get_litellm_gateway_api_key
* fix: scope get_stored_api_key and save base_url on login
* fix: pass base_url to get_stored_api_key in CLI entrypoint
* fix: scope ProxyClient stored key to base_url
* test: add expected_base_url coverage for get_stored_api_key
* fix: initialize self.http with resolved api_key not raw param
* fix: black formatting in client.py and test_auth_commands.py
- Lua script now reads time via redis.call('TIME') instead of a client-supplied
timestamp. Prevents window-reset divergence across replicas with skewed
wall-clocks, which could otherwise reopen the cross-replica TOCTOU window.
- Per-descriptor window_size is now plumbed through both the Lua ARGV layout
and the in-memory fallback. Previously the in-memory path used the global
self.window_size while Lua honored the per-descriptor override, so a
descriptor with a custom window would be enforced inconsistently between
Redis-available and Redis-unavailable code paths.
- Lua-failure fallback path now logs at error severity and explicitly
documents the in-memory ↔ Redis state divergence risk so operators can
alert on it. Prior `warning` log understated the impact.
- Coarse-granularity lock is now documented inline with the conditions under
which a per-descriptor sharded lock would be worth introducing.
- New regression test: zero-token batch consumes RPM only and is properly
capped by the RPM ceiling (validates the asymmetric quota path that arises
from `inc_amount <= 0: continue`).
Tests: 64 passed (1 skipped), 0 regressions. Multi-instance Redis loadtest
re-verified: chat 20/80 success @ RPM=20, batches 3/20 @ TPM=200.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
