mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-17 08:25:03 +00:00
80cf50dedbcd55b3ddd7809b47f6df63b5a90d57
39473
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 michelligabrieleandGitHub
|
80cf50dedb | fix(v3 limiter): cap no-max_tokens TPM floor at smallest configured limit (#28805) | ||
|
yuneng-jiangandGitHub
|
8b16b61114 | chore: update Next.js build artifacts (2026-05-31 02:04 UTC, node v20.20.2) (#29366) | ||
|
michelligabrieleandGitHub
|
117136cccc | fix(openai-moderation): wire streaming flags through to unified dispatcher (#27324) | ||
|
yuneng-jiangandGitHub
|
f0ebfb2a1b |
fix(ui): break logout redirect loop across origins (#29360)
When the user has visited both the dev UI (e.g. localhost:3000) and the
proxy UI (e.g. localhost:4000) in the same tab, logging out from the dev
origin produced an infinite logout/login redirect.
The proxy-side LoginPage's "is the user still authenticated?" check
was reading getCookie("token"), which falls back to sessionStorage when
document.cookie has no token. The cross-origin clearTokenCookies() call
from the dev origin can clear cookies on the shared hostname, but cannot
reach sessionStorage on the proxy origin (sessionStorage is per-origin),
so the fallback returned a stale token and LoginPage interpreted the
user as logged in, redirecting back to the dev origin. Dev origin then
saw no cookie and redirected to LoginPage, repeating ~20x per second.
This change introduces getCookieFromDocument(), a cookie-only read with
no sessionStorage fallback, and uses it in LoginPage's already-logged-in
check. The HttpOnly-reverse-proxy defense from PR #23532 is unaffected:
storeLoginToken still writes both the JS cookie at /ui and the
sessionStorage backup, and getCookie still falls back for callers that
want the full read path.
|
||
|
ryan-crabbe-berriandGitHub
|
a9cc6ed68c |
test(e2e): cover PROXY_LOGOUT_URL redirect on Logout (#29080)
* test(e2e): cover PROXY_LOGOUT_URL redirect on Logout Env-gated spec mirroring the existing serverRootPathRedirect pattern: when the proxy is booted with PROXY_LOGOUT_URL set, clicking Logout in the navbar must navigate to that external URL. The standard run_e2e.sh exports an empty value so the rest of the suite is unaffected; this spec self-skips unless the env var is populated. * test(e2e): run PROXY_LOGOUT_URL spec in the suite + harden logout assertions Boot the e2e proxy with PROXY_LOGOUT_URL set (job-level env in CircleCI and run_e2e.sh) so proxyLogoutUrl.spec.ts actually runs instead of self-skipping. Nothing else in the suite performs a logout, so this only affects the behavior under test. Harden the spec to verify the logout flow rather than a URL substring: - wait for /sso/get/ui_settings before clicking so logoutUrl is populated (otherwise window.location.href = "" silently reloads same-origin) - assert a token cookie exists first, and is cleared after logout - locate the dropdown via getByRole instead of internal antd CSS classes - stub the external destination and assert on URL origin + path prefix * test(e2e): assert exact PROXY_LOGOUT_URL on logout redirect Replace the origin + startsWith(pathname) checks with a single normalized href comparison. With PROXY_LOGOUT_URL=https://www.example.com the path was "/", so startsWith("/") matched any path and left path/query/hash unchecked. Comparing normalized hrefs pins scheme, host, port, path, query and hash while still tolerating the browser's trailing-slash/default-port normalization. |
||
|
yuneng-jiangandGitHub
|
7d1bd9d9f4 |
fix(reset_budget): write only {spend, budget_reset_at} and stop pre-zeroing counter (#29358)
* fix(reset_budget): write only {spend, budget_reset_at} and stop pre-zeroing counter
ResetBudgetJob's batched update_data path shipped the full key/user/team
model on each reset. Prisma rejects object_permission_id and budget_limits
on the update input type, so any row carrying those fields detonated the
entire batch -- spend never reset, budget_reset_at never advanced. After
v1.84.0 started populating object_permission_id on UI-created keys, this
fires routinely.
_reset_budget_common also zeroed the cross-pod spend counter before the
DB write, so failed resets left enforcement reading 0 from the counter
while the DB still held the over-budget spend, admitting requests past
the cap until the counter naturally re-saturated from new reservations.
Switch the write to per-row narrow updates ({spend, budget_reset_at})
via db.batch_, and move the counter invalidation out of
_reset_budget_common so it only fires after the DB write commits. On
DB-write failure the counter is left untouched, enforcement continues
to block, and the next scheduler tick can retry without leaving a
bypass window.
Fixes #27730.
* fix(reset_budget): address Greptile review on #29358
- Strengthen the bypass-half regression test: replace the for-loop over
call_args_list (vacuously true when empty) with assert_not_called(),
so the test would actually flag a re-introduction of counter-zeroing
via any code path.
- Add the same explanatory docstring on _write_user_reset_updates and
_write_team_reset_updates that _write_key_reset_updates already has,
so all three helpers point future maintainers at #27730.
* test(reset_budget): update test_proxy_budget_reset for new batch-write path
Same shape as the previous test_reset_budget_job.py update: keys/users/teams
now write through prisma.db.batch_().<table>.update, not update_data, so the
tests need a batcher mock and updated assertions. Adds:
- _wire_batcher_for_test helper that returns a list which accumulates per-row
batch updates captured from prisma_client.db.batch_().
- _attrify helper that wraps dict fixtures so getattr(item, "token") works
alongside the dict item-access the fake_reset_* mocks rely on. The new
narrow-write helpers use getattr to pull out the row's id, and would
silently skip plain dicts otherwise.
- Updates 3 partial_failure tests to assert against the batch-call list
(rows by id, payload contains only {spend, budget_reset_at}) instead of
update_data.assert_awaited_once + data_list inspection.
- Updates test_reset_budget_continues_other_categories_on_failure: only
budget + enduser still flow through update_data; key/user/team go through
the batch path now.
- Wires the batcher mock into 3 service_logger_*_success tests so commit()
is actually awaitable and the success hook fires.
These tests were silently passing locally only because the editable install
in .venv pointed at the main repo, not the worktree — running pytest with
PYTHONPATH overridden to the worktree (matching CI) reproduces the failures.
|
||
|
ryan-crabbe-berriandGitHub
|
90b5104475 |
feat(mcp/auth): additive key access-group grants + opt-in member assignment (#29313)
* fix(mcp): make key.access_group_ids grants additive over team ceiling A key whose unified access_group_ids grant a private MCP server was having that grant intersected against its team's MCP ceiling, so a key in a team scoped to other servers (or with no own scope) lost the granted server entirely. Resolve access_group_ids once as ungated additive grants and union them on top of the key/team ceiling instead of folding them into the key scope that gets intersected. * test(mcp): align key access-group tests with additive-grant model The previous commit moved key.access_group_ids resolution out of the intersected key ceiling (_get_allowed_mcp_servers_for_key) and into the ungated additive grant path (_get_key_access_group_mcp_server_extras), unioned on top of the team ceiling. Five tests from #28890/#29195 still asserted the old gated / in-key-scope contract and failed: - _get_allowed_mcp_servers_for_key now returns the object_permission ceiling only and never resolves access_group_ids; two tests now assert the group resolver is not called from that path (with and without an object_permission present). - The extras path is ungated, so a group whose assigned_team_ids / assigned_key_ids exclude the caller still contributes its servers. - The end-to-end test asserts the grant surfaces via the extras path rather than the base key path. - Dropped test_key_access_group_ids_empty_returns_no_extras; the empty case is already covered by the extras family's no-groups test. * feat(auth): gate member access-group assignment on keys behind opt-in Non-admin team members could attach access_group_ids to keys they create or update, letting them self-grant resources (MCP servers/models) the team admin never intended. Add an opt-in KEY_ACCESS_GROUP_ASSIGNMENT team-member permission (default-deny) enforced at /key/generate and /key/update; proxy and team admins bypass. Surfaces automatically as a checkbox in the team Member Permissions UI. * fix(auth): gate access-group assignment on /key/regenerate too RegenerateKeyRequest inherits access_group_ids and prepare_key_update_data persists it, so a non-admin key owner could self-grant access groups by regenerating. Apply the same opt-in member gate using the existing key's team. * test(auth): cover member access-group gate and additive MCP grants Add unit tests for enforce_member_can_assign_access_groups (deny without opt-in, allow with opt-in, and proxy-admin / team-admin / non-team-key bypasses) and for _get_key_access_group_mcp_server_extras (no-auth and no-resolved-servers return empty, resolved ids are expanded, errors degrade to no grants). |
||
|
|
dc4f5b12ef |
fix(proxy): enforce allowed_passthrough_routes for auth=true pass-thr… (#29256)
* fix(proxy): enforce allowed_passthrough_routes for auth=true pass-through
Pass-through endpoints with auth=true were injected into openai_routes,
so teams with openai_routes access bypassed per-team allowed_passthrough_routes.
Gate auth-enforced pass-through at JWT, virtual-key, and non-admin route checks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): clarify JWT passthrough denial
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): make pass-through auth checks method-aware
Prevent allowlist bypass when the same path is registered with different auth settings per HTTP method.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix passthrough route auth checks
* fix(proxy): reject unregistered pass-through HTTP methods
Enforce method-aware JWT checks and return 405 when stale FastAPI routes accept requests outside the current pass-through registry.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): remove duplicate request_method in JWT team lookup
Fixes SyntaxError on proxy startup caused by passing request_method twice to find_team_with_model_access.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix passthrough route auth enforcement
* fix(proxy): raise passthrough-specific 403 directly in virtual-key path
* fix(proxy): load team for RBAC role-claim JWT passthrough gating
* Revert "chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)" (#29326)
This reverts the Bedrock CI account migration (#28728). The original account
(888602223428) was put under an AWS security restriction after a leaked key
and has since been reactivated, while the replacement account (941277531214)
lacks access to several models the suites exercise (legacy Bedrock Claude 3
models, Cohere, Nova Canvas image gen, Bedrock batch inference, and flagship
Opus). Pointing CI back at the reactivated account restores that coverage.
This is the exact inverse of #28728: all hardcoded 941277531214 references go
back to 888602223428 (provisioned/imported-model ARNs, AgentCore runtime ARNs
and their suffixes, batch execution role ARN, and the example proxy config),
the S3 buckets revert to litellm-proxy and load-testing-oct, the guardrail IDs
revert to wf0hkdb5x07f and ff6ujrregl1q, the SageMaker endpoint and Knowledge
Base revert to their original ids, and the live-call tests go back to the
legacy model strings. The grid_spec fail_reason workaround for the unentitled
Opus cells is dropped while keeping the unrelated bedrock_effort_ceiling field
added after the migration.
The CircleCI AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars still point at
941277531214 and must be set to the reactivated account's fresh credentials
separately via the CircleCI API; AWS_REGION_NAME stays us-west-2.
(cherry picked from commit
|
||
|
|
ba2699740c |
feat(pass_through): extend passthrough_managed_object_ids to Azure (#29160)
* feat(pass_through): extend passthrough_managed_object_ids to Azure
Adds managed ID minting/resolution for Azure passthrough endpoints
(/azure/...) alongside the existing OpenAI passthrough support.
Key changes:
- pass_through_endpoints.py: detect azure/azure_ai custom_llm_provider
(string or enum) to set _is_managed_id_provider and _managed_id_provider;
both INPUT and OUTPUT rewrite blocks now fire for Azure.
- llm_passthrough_endpoints.py: forward custom_llm_provider into
create_pass_through_route so it reaches pass_through_request (was None).
- managed_id_rewriter.py: extend _PASSTHROUGH_PREFIX_RE and _canonical_path
to strip /azure/openai prefix and add /v1/ for Azure paths that omit it;
add ("azure", method, path) entries to BUILTIN_OUTPUT_ID_FIELD_MAP for
files and batches endpoints.
- managed_id_codec.py / types/utils.py: supporting codec and enum constant.
- proxy_server.py: register llm_passthrough_router before batches_router to
prevent route collision for /openai_passthrough/* paths.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(pass_through): remove unused imports for ruff F401
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(pass_through): satisfy mypy for optional parsed_body
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(pass_through): compute query params string after managed-ID rewrite
Move requested_query_params_str computation to after the managed-ID
input rewrite block so logging_url reflects the rewritten raw-provider
query params actually sent upstream, instead of the original
managed IDs.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Add support for managed ids for passthrough responses api
* Add support for list batches and list files
* style: run Black on passthrough managed ID files
Fix CI formatting for managed_id_rewriter.py and pass_through_endpoints.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(passthrough): parse json file_object and implement before-cursor pagination
- Parse row.file_object via json.loads when Prisma returns it as a string;
mirrors openai_files_endpoints/common_utils.py so list responses keep all
stored detail fields (status, timestamps, etc.).
- Implement the previously-parsed-but-unused 'before' cursor for list
pagination by flipping fetch order to ascending with a 'gt' bound on
created_at, then reversing rows so the response stays newest-first.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Remove logger
* refactor: split list_passthrough_ids_from_db to fix PLR0915
Extract pagination, fetch, and serialization helpers so the main list
function stays under the statement limit without changing behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: scope passthrough managed ID dedup and list by provider
Validate embedded provider before reusing deduped file/object rows so
OpenAI and Azure cannot share the same managed ID for an identical raw ID.
Filter list responses to rows whose managed IDs decode to the current
provider, with over-fetch scanning when needed.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(managed_id_rewriter): cap pagination trim at effective limit
When raw_limit > 100, fetch_limit is capped at 101 (one extra row to
detect has_more), but trimming with rows[:raw_limit] failed to drop
the sentinel row. Use the capped effective limit instead.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: cross-provider object collision and fail-closed list error handling
Greptile P1: move provider check before access check in _mint_or_reuse_object
so a cross-provider raw ID collision (OpenAI and Azure share the same batch_
ID) falls through to mint a new provider-scoped row instead of raising 404.
Veria-ai medium: _fetch_provider_scoped_list_rows now always returns
(page, has_more) — DB errors break out of the scan loop and return matched
rows so far. list_passthrough_ids_from_db never returns None for a recognised
list route, so the caller can never fall through to the upstream provider.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: namespace passthrough model_object_id by provider to prevent unique violation
Store model_object_id as 'passthrough:{provider}:{raw_id}' instead of the
bare raw ID so OpenAI and Azure can each own a row for the same raw batch ID
without hitting the @unique constraint. Dedup lookup uses the same namespaced
key so it is implicitly provider-scoped and the _managed_id_matches_provider
check is no longer needed on the object path.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: gate list interception on managed_files hook like input/output rewrites
Without the hook no managed IDs are minted so the DB is empty. Intercepting
GET /v1/files without the hook returned an empty list and hid the caller's
real upstream files/batches. Matches the guard used by the input and output
rewrite blocks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: set has_more=True when scan cap is hit with a full final DB batch
When max_scans (20) is exhausted and the last DB page was full-sized,
there are almost certainly more rows beyond the scan window. Track
last_batch_full across iterations so the scan_cap_hit condition sets
has_more=True in that case, preventing silent pagination truncation in
high-mixed-provider pools.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(managed_id_rewriter): scope pagination cursor lookup to caller-owned rows
Prevent a cross-tenant timing oracle by constraining the after/before
cursor row lookup to the caller's owner_filter, and cover the real Azure
responses path form (no /v1/) in tests.
* fix(managed_id_rewriter): align passthrough list metadata with direct GET
Persist upstream file metadata when minting a managed file ID and rewrite
nested batch file IDs before snapshotting the object, so DB-served file/batch
list responses return the same fields and managed IDs as a direct endpoint
GET.
* fix(managed_id_rewriter): degrade to raw id on cross-owner object collision
The OUTPUT (mint) path of _mint_or_reuse_object raised HTTPException(404)
when a dedup hit on the namespaced model_object_id belonged to a different
owner, converting a successful upstream batch/response creation into a 404
for the caller. Two upstream accounts under one provider name can issue the
same raw id, so this is reachable in multi-tenant deployments.
Return the caller's raw id unmanaged instead: the upstream create already
succeeded, a new managed row can't be minted (model_object_id is @unique),
and reusing the other owner's managed id would later fail the access check.
* fix(managed_id_rewriter): scope list cursor by provider and cap body-rewrite recursion depth
* perf(managed_id_rewriter): push batch list provider scope to DB and anchor canonical-path prefix
Object (batch) list rows store model_object_id as passthrough:{provider}:{raw},
so the provider filter is now applied at the indexed DB column, collapsing the
application-layer multi-scan to a single query for that table. File rows keep
the decode-based scan since they have no provider column.
Anchor the canonical-path prefix regex at a path boundary so routes such as
/openai_realtime/... are no longer mis-stripped.
* fix(managed_id_rewriter): refresh stored batch snapshot on reuse
The dedup-reuse path in _mint_or_reuse_object returned the existing managed id
without updating the stored file_object, so DB-served list responses kept the
creation-time snapshot and showed null output_file_id/error_file_id even after
the batch completed. Refresh the snapshot when an owned row is reused so the
list reflects the batch's latest state.
* fix(managed_id_rewriter): deny cross-owner object access on retrieve/cancel/delete
Returning the raw id when can_access_resource fails only made sense for create
responses, where the caller's own upstream create succeeded under a raw id that a
different owner already holds. On retrieve/cancel/delete the caller reaches that
branch only by supplying another tenant's raw id (which bypasses the managed-id
input gate), so echoing the upstream object back leaked it cross-tenant. Restrict
the raw fallback to create routes and return 404 otherwise.
* fix(managed_id_rewriter): deny cross-owner file access on retrieve/delete
_mint_or_reuse_file scoped the raw file dedup lookup to the current caller, so a
raw file-... id belonging to another tenant was never found and the OUTPUT path
minted a fresh managed id for that same upstream file under the caller. A raw id
only reaches this path by skipping the managed-id input gate (raw provider ids
are opt-out), so a different-owner row means the caller is touching someone
else's file. Look up flat_model_file_ids globally and run can_access_resource;
deny with 404 on retrieve/delete and leave the raw id unmanaged on create, which
mirrors the cross-owner handling already in _mint_or_reuse_object.
* fix(managed_id_rewriter): deterministic provider-scoped file dedup
Replace the unscoped find_first in _mint_or_reuse_file with a find_many
ordered by created_at and an application-layer provider filter. The file
table has no provider column, so a raw file id shared across OpenAI and
Azure could map to one row per provider; find_first then picked a row
non-deterministically and, on a provider mismatch, minted a fresh managed
row on every call, accumulating duplicates. Selecting the oldest matching
same-provider row the caller can access keeps reuse stable and prevents
duplicate rows while preserving the cross-tenant deny/leave-raw behaviour.
* refactor(pass_through): scope passthrough managed IDs on the explicit provider
Move the openai/azure detection out of pass_through_request into
resolve_passthrough_managed_id_provider in llms/base_llm/managed_resources,
and key managed-ID rewriting on the forwarded custom_llm_provider rather than
the upstream URL's EndpointType. The helper documents why azure and azure_ai
collapse to one "azure" scope (they share the same Azure OpenAI files/batches
surface, so an ID minted on one must resolve on the other) and returns None for
any other provider so a third-party OpenAI-compatible endpoint never triggers
managed-ID minting.
Add TestManagedIdProviderScope covering the azure_ai -> azure collapse and the
non-openai/azure exclusion.
* test(log_db_metrics): assert sanitized event_metadata contract
test_log_db_metrics_success still asserted the legacy event_metadata
shape (function_name/function_kwargs/function_args), which #28909
intentionally removed so that live Prisma clients, OTel spans, and
secrets never land on a service-log span. The decorator now emits only
a sanitized payload: None when no table_name is present, and
{"table_name": ...} when it is. Update the test to verify both branches
of that contract.
* fix(managed_id_rewriter): page provider-scoped file list by offset
The file list scan advanced its cursor with a strict created_at boundary.
When several rows shared a created_at timestamp and a non-matching provider
row sat on the page boundary, the next query skipped the remaining rows at
that timestamp, dropping matching files from the response. Page by a stable
offset over a total order (created_at plus the unique id column) so tied
rows are never skipped or repeated.
* fix(managed_id_rewriter): push file-list provider scope to the DB
The file-list helper had no provider column to query, so it scanned the
table and filtered by decoding each managed ID in the application layer,
capped at 20 pages. For an admin with a large mixed-provider file pool
that cap could truncate a page.
Mint now writes a _passthrough_provider:{provider} marker into
flat_model_file_ids, giving the file table the same DB-queryable provider
scope object rows already get from the namespaced model_object_id. The
list helper pushes the scope into the query so a single round-trip serves
the page. The scan loop, the offset paging, and the cap are gone, so pages
can no longer truncate, leak the other provider, or skip rows that share a
created_at timestamp.
* fix(managed_id_rewriter): deny raw provider IDs that map to another tenant's managed resource
Clients only ever receive managed IDs on passthrough, so a raw file/batch/response ID for another tenant's managed object can only be recovered by decoding that tenant's managed ID. Raw IDs were forwarded upstream untouched (deliberate opt-out), which on a retrieve/cancel/delete executed upstream before the response-side ownership check ran, leaking a cross-tenant action.
Guard raw provider IDs on the input path: when a raw file-/batch_/resp_ ID resolves to a managed row the caller cannot access, return 404 before forwarding. Genuinely unmanaged raw IDs (no DB row) and IDs the caller owns are left untouched, preserving the opt-out.
* test(managed_id_rewriter): cover azure_ai and pre-versioned azure passthrough paths
* fix(managed_id_rewriter): fall back to raw id when persistence fails
A DB persistence failure after a successful upstream create left the
client holding a minted managed ID with no backing row, so every later
resolve returned 404 and the resource was permanently unreachable. Mint
the managed ID only when the row is stored; on persistence failure return
the raw provider id, matching the no-persistence-available fallback, so
the freshly-created resource stays reachable.
* fix(managed_id_rewriter): log only rewritten query param keys
* fix(managed_id_rewriter): use compound (created_at, id) list cursor boundary
A timestamp-only lt/gt cursor boundary skips list rows that share the
cursor row's created_at across a page boundary, silently dropping them.
Compare the unique id (the secondary sort key) alongside created_at so
the page walk stays complete when timestamps tie.
* fix(managed_id_rewriter): converge concurrent object creates on one managed id
* fix(managed_id_rewriter): bound raw-id guard DB lookups per request
The INPUT guard fired one DB lookup for every file-/batch_/resp_ prefixed
string in the path, query, and body. The file-id guard is an unindexed
array-containment scan over LiteLLM_ManagedFileTable, so an authenticated
caller could amplify a single passthrough request into thousands of
full-table scans by packing a body with id-shaped strings.
De-dupe raw ids within a request and cap the distinct guard lookups,
failing closed with 400 instead of skipping the guard. Legitimate callers
hold managed ids (resolved via an indexed unified_*_id lookup, not the
guard), so the cap only trips under abuse.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
|
7ca796beb1 |
fix(proxy): restrict vector store index create/delete to proxy admins (#29202)
* fix(proxy): restrict vector store index create/delete to proxy admins Prevent non-admin API keys from registering indexes via POST /v1/indexes or deleting Azure AI Search indexes through managed pass-through routes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): tighten vector store index lifecycle checks Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
|
4c3efe9c7c |
fix(guardrails): return HTTP 400 for litellm content filter blocks (#28418)
* fix(guardrails): return HTTP 400 for litellm content filter blocks Align litellm_content_filter hard rejects with the standard guardrail block status code so clients receive 400 instead of 403. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): return HTTP 400 for custom code guardrail blocks Pre-call custom code guardrail blocks now raise HTTPException(400) instead of using the passthrough ModifyResponseException path that returned a synthetic 200 response. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): preserve custom code passthrough blocks Keep standalone custom code guardrail blocks on the passthrough contract while covering policy pipeline block handling for passthrough-style guardrail interventions. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Mateo WangandGitHub
|
152b1177e5 |
test(reasoning-effort-grid): cover Claude Opus 4.8 across provider routes (#29327)
* test(logging): align DB metrics event_metadata assertions with safe redaction PR #28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped. * test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI account, so they were marked xfail. xfail keeps reporting them as expected failures even after access is granted, so the wire translation never gets verified again. Now the cell makes the call and skips only when Bedrock replies "is not available for this account"; the moment the model is entitled the same cells run their full assertions with no edit. A focused unit test pins the tolerance predicate so any other failure still surfaces loudly and the available path still runs the assertions. * test(reasoning-effort-grid): add claude-opus-4-8 across provider routes Adds claude-opus-4-8 to the anthropic, azure, vertex and bedrock-converse routes (275 cells total) so the reasoning-effort wire translation is covered for the new model. The bedrock opus-4-8 and opus-4-7 cells reuse the self-heal path: they run the call and skip only on Bedrock's "is not available for this account" reply, then assert in full once the model is entitled. The azure and vertex opus-4-8 cells stay xfail until a Foundry deployment exists and Vertex availability is confirmed. The shared xhigh+max capability set is renamed to _CAPS_XHIGH_MAX now that more than one model uses it. |
||
|
yuneng-jiangandGitHub
|
94a043efb2 |
refactor(proxy/auth): normalize Bearer prefix in safe-hash helper (#29343)
* refactor(proxy/auth): normalize Bearer prefix in safe-hash helper
UserAPIKeyAuth._safe_hash_litellm_api_key now strips a leading
"Bearer "/"bearer " prefix before its existing sk-/JWT classification, so
the helper produces the same hashed output regardless of whether the
caller stripped the Authorization header prefix or passed the header
value through unchanged.
* refactor(proxy/auth): make Bearer-prefix strip case-insensitive
Per RFC 7235 the HTTP authorization scheme token is case-insensitive.
Replace the two-prefix loop with a single case-insensitive check so the
helper normalizes "Bearer ", "bearer ", "BEARER ", and any mixed-case
variant before classifying the remainder as sk- or JWT. The contract
test gains coverage of "BEARER " and "BeArEr ".
* test(mcp): align auth-handler test expectations with safe-hash helper
The two MCP auth tests asserted that UserAPIKeyAuth(api_key="Bearer ...")
retained the raw header bytes on the api_key field. _safe_hash_litellm_api_key
now normalizes that input — stripping the Bearer prefix and hashing the
resulting sk- key — so the expectations move to the normalized form:
the bare token in the parametrize case, and hash_token("sk-...") in the
backward-compat assertion. This matches what the real auth flow produces
(the builder strips Bearer and the DB stores the hashed token), so the
mocks now line up with production rather than with the un-normalized
validator output.
|
||
|
Mateo WangandGitHub
|
bfbb5d2375 |
fix(ci): make litellm_internal_staging green (logging test + Bedrock Opus 4.7 self-heal) (#29344)
* test(logging): align DB metrics event_metadata assertions with safe redaction PR #28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped. * test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI account, so they were marked xfail. xfail keeps reporting them as expected failures even after access is granted, so the wire translation never gets verified again. Now the cell makes the call and skips only when Bedrock replies "is not available for this account"; the moment the model is entitled the same cells run their full assertions with no edit. A focused unit test pins the tolerance predicate so any other failure still surfaces loudly and the available path still runs the assertions. |
||
|
|
2d4c13c00f |
fix(mcp): preserve omitted fields on PUT /v1/mcp/server partial updates (#29253)
* fix(mcp): preserve omitted fields on PUT /v1/mcp/server partial updates Use model_dump(exclude_unset=True) for updates so schema defaults (transport=sse, allow_all_keys=false, etc.) are not written when callers omit them. Serialize JSON fields from the filtered dict and only force is_byok on create. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): allow explicit alias=None on MCP server partial updates Snapshot caller-provided fields before normalization so omitted alias is not written while an intentional alias=None still clears the stored value. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
|
ace3c65ab3 |
fix(mcp): preserve source_url in GET /v1/mcp/server list responses (#29249)
* fix(mcp): preserve source_url in GET /v1/mcp/server list responses
The list endpoint builds responses from the in-memory registry, but
source_url was dropped during the DB-to-registry roundtrip even though
GET /v1/mcp/server/{id} returned it correctly from the database.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests/mcp): set source_url on MagicMock table records
MagicMock auto-creates source_url as a mock object, which fails MCPServer
Pydantic validation after source_url was wired through build_mcp_server_from_table.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Mateo WangandGitHub
|
f11c12d157 |
Revert "chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)" (#29326)
This reverts the Bedrock CI account migration (#28728). The original account (888602223428) was put under an AWS security restriction after a leaked key and has since been reactivated, while the replacement account (941277531214) lacks access to several models the suites exercise (legacy Bedrock Claude 3 models, Cohere, Nova Canvas image gen, Bedrock batch inference, and flagship Opus). Pointing CI back at the reactivated account restores that coverage. This is the exact inverse of #28728: all hardcoded 941277531214 references go back to 888602223428 (provisioned/imported-model ARNs, AgentCore runtime ARNs and their suffixes, batch execution role ARN, and the example proxy config), the S3 buckets revert to litellm-proxy and load-testing-oct, the guardrail IDs revert to wf0hkdb5x07f and ff6ujrregl1q, the SageMaker endpoint and Knowledge Base revert to their original ids, and the live-call tests go back to the legacy model strings. The grid_spec fail_reason workaround for the unentitled Opus cells is dropped while keeping the unrelated bedrock_effort_ceiling field added after the migration. The CircleCI AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars still point at 941277531214 and must be set to the reactivated account's fresh credentials separately via the CircleCI API; AWS_REGION_NAME stays us-west-2. |
||
|
|
967fed1fa1 |
feat(enterprise): add RESEND_FROM_EMAIL for self-hosted Resend sends (#28830)
Allow self-hosted installs to override the default LiteLLM sender address via RESEND_FROM_EMAIL, matching SendGrid's SENDGRID_SENDER_EMAIL pattern. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
|
4cc3dd7aad |
feat(context_management): compact_20260112 polyfill for non-Anthropic providers (#28868)
* feat(anthropic/messages): in-gateway context_management polyfill for non-Anthropic providers
- Add `context_management/` module with `clear_tool_uses_20250919` editor
dispatched before chat-completions translation on `/v1/messages`
- Hard-protect most-recently completed tool_result from being cleared
- Attach `context_management.applied_edits` to both non-streaming and
streaming (final `message_delta`) responses
- Bedrock Converse: forward `context_management`; filter to
`compact_20260112`-only edits with `compact-2026-01-12` beta header
- token_counter: guard Anthropic-format tools (no `function` key) to
prevent AttributeError during polyfill token counting
- Streaming: handle empty-choices usage-only trailing chunks
- Skip polyfill when `litellm.drop_params = True`
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): pop None context_management before sending to Bedrock Converse
If context_management is forwarded as None (e.g. when mapping returns
None for an invalid format), _filter_context_management_for_bedrock_converse
previously returned early without removing the key, leaving
"context_management": null in the request and causing a validation
error. Pop the key when the value is not a dict.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(bedrock/converse): pop None context_management; extract helpers to fix PLR0915
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(anthropic/messages): check per-request drop_params alongside global
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(anthropic/messages): preserve drop_params for downstream and respect explicit False
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: lazy debug logging in clear_tool_uses; remove unused context_management constants
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic/messages): guard context_management polyfill with try/except
Wrap apply_context_management() in a try/except so any failure (e.g.
litellm.token_counter raising on an unknown tokenizer or unexpected
message format) is logged but does not crash the underlying LLM
request. The polyfill is a best-effort additive feature; on failure we
forward the original messages without applied edits.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(token_counter): guard None input_schema in Anthropic tool fallback
Use `or {}` instead of `.get(..., {})` so explicit null parameters do not
raise AttributeError when formatting function definitions for token counting.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: minimize context_management polyfill threading
- Use None (not empty list) for polyfill_applied_edits when context
management isn't requested, so semantics of 'feature not requested'
vs 'feature requested but no edits applied' are distinct.
- In the streaming iterator, only pass applied_edits to the per-chunk
translator on the final (finish_reason) chunk; intermediate chunks
ignore it anyway, and this makes intent explicit on both sync and
async paths.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(context_management): align tool_use counts and normalize list spec
- _count_tool_uses now requires a string id, matching _collect_tool_use_ids_in_order so the tool_uses trigger can't fire on blocks that aren't clearable.
- apply_context_management dispatcher now accepts the OpenAI list form and normalizes it via AnthropicConfig.map_openai_context_management_to_anthropic, so the polyfill path no longer silently no-ops on list input.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(context_management): add compact_20260112 polyfill for non-Anthropic providers
Implements an in-gateway compaction polyfill that summarizes long conversations
using a configurable model when `compact_20260112` is requested for non-Anthropic
targets (e.g. OpenAI, Gemini), matching Anthropic's context management beta
behaviour for those providers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(compact): skip tool_result-only user turns; bedrock: elif for context_management
- compact_20260112 Phase D: when keeping the last user turn after a full
summary, skip role=user turns whose content is exclusively tool_result
blocks. Such turns translate to OpenAI tool-role messages with no
preceding assistant tool_calls (those got summarized away), which
non-Anthropic providers reject. Fall back to a synthetic continuation
prompt if no eligible user question exists, so the downstream call
always has a non-empty user message.
- bedrock converse: chain the context_management param as elif so it
follows the same if/elif pattern as the surrounding thinking/
reasoning_effort checks.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic): post-compaction question selection, system type, sync stream merge
- compact.py: select last user question from effective_messages (post-compaction slice) instead of raw messages, so prior summarized turns aren't reintroduced
- handler.py: widen _prepare_completion_kwargs system parameter type to Union[str, List[Dict]] matching PolyfillResult.system
- streaming_iterator.py: mirror async hold-and-merge logic in sync __next__ so context_management is attached to the final merged message_delta when stop_reason and usage arrive in separate chunks
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic/messages): apply context_management on sync path; clear held stop_reason chunk in async iterator
- Sync `anthropic_messages_handler` was silently dropping the
`context_management` kwarg via `ANTHROPIC_ONLY_REQUEST_KEYS` after the
polyfill was moved into the async handler. Bridge to the async
dispatcher with `run_async_function` so `litellm.messages.create()`
callers keep working (regressed e.g. `clear_tool_uses_20250919`).
- In the streaming iterator's `__anext__` `StopIteration` handler, clear
`self.holding_stop_reason_chunk` after capturing it (matches `__next__`)
so a subsequent call doesn't re-emit the same chunk.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(bugfixes): bedrock None context_mgmt; stream per-instance queue; sync polyfill; trailing-chunk passthrough
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic): silently drop trailing chunks after usage; remove dead _polyfill_result key
- streaming_iterator: in sync __next__, after the usage chunk has been
merged and emitted, silently consume any trailing provider events
via 'continue' instead of forwarding them through the queue. Trailing
chunks would translate to content_block_delta or message_delta and
violate Anthropic SSE ordering after the final message_delta. The
async __anext__ already drops these via 'if not self.queued_usage_chunk:'
gating, so this aligns sync and async behavior.
- handler: drop unused '_polyfill_result' from ANTHROPIC_ONLY_REQUEST_KEYS.
PolyfillResult is passed as an explicit arg to the adapter methods, never
through extra_kwargs, so the entry was dead code.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* refactor(anthropic): extract usage-merge helper; guard empty slice-only compaction result
- Extract the duplicated hold-and-merge usage logic from the sync __next__ and
async __anext__ paths into a shared _merge_usage_into_held_stop_reason_chunk
helper so the subtle cache-token / context_management attachment lives in
exactly one place.
- In the compact_20260112 slice-only path, fall back to _select_last_user_question
when _strip_compaction_blocks produces an empty list (e.g. messages ending on
an assistant turn whose only content was the compaction block) so the
downstream API never receives an empty messages array.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* refactor(anthropic/context_management): streaming iterator compaction fixes and compact polyfill improvements
- Extract usage-merge helper; guard empty slice-only compaction result
- Silently drop trailing chunks after usage; remove dead _polyfill_result key
- Fix bedrock None context_mgmt; stream per-instance queue; sync polyfill; trailing-chunk passthrough
- Apply context_management on sync path; clear held stop_reason chunk in async iterator
- Fix post-compaction question selection, system type, sync stream merge
- Skip tool_result-only user turns; bedrock: elif for context_management
- Add streaming iterator compaction test suite
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(html): restore flat *.html naming in _experimental/out
Reverses the accidental rename from *.html → */index.html introduced in
|
||
|
Yassin KortamandGitHub
|
3be3c1dea1 | feat(otel): add team_metadata, http.route, and model names to inference spans (#29319) | ||
|
ryan-crabbe-berriandGitHub
|
37e6e2da1c |
test(e2e): assert internal-user navbar identity is scoped to that user (#29077)
* test(e2e): assert internal-user navbar identity is scoped to that user The existing login.spec.ts only checks the admin's navbar identity. This adds the symmetric check for the internal user — verifying the account button + dropdown surface the internal user's email, id, and role, and that no admin-scoped values leak through. * test(e2e): harden navbar identity test per review feedback Locate the user dropdown panel by a data-testid on the popupRender div instead of Ant Design internal + Tailwind class names, so styling refactors no longer risk breaking the identity-scoping assertions. Source the seeded user emails/ids from shared constants (match seed.sql) instead of hardcoding them inline. |
||
|
ryan-crabbe-berriandGitHub
|
892838963c |
test(e2e): cover Internal User create-key flow when in no teams (#29083)
* test(e2e): cover Internal User create-key flow when in no teams The seeded e2e-internal-user is in two teams, so the "no team" branch of the Create Key modal — where the team dropdown must render empty — was unreachable. Seeds a noteam@test.local user and adds a spec that logs in fresh, opens the modal, and asserts the dropdown has zero options. * test(e2e): harden no-team dropdown assertion + add with-teams counterpart Replace the one-shot count() check with a settled-empty assertion: wait for the dropdown's loaded "No teams found" state before asserting zero options, so the test can't pass on a transient empty frame while the team-options request is still in flight. Add internalUserWithTeams.spec.ts as the differential partner; it logs in as the seeded e2e-internal-user (two team memberships) and asserts the dropdown lists exactly those teams. Without it, the no-team spec's zero-options assertion would still pass against a regression that empties the dropdown for every user. |
||
|
|
12d29a38a7 |
tests(proxy_server): surface current behavior in tests (#29309)
* test(proxy/proxy_server): pin forwarding routes (PR2) (#28887) * test(proxy): pin proxy_server.py forwarding-route behavior PR2 of the proxy_server.py behavior-pinning project: fills the 12 forwarding-route test files added by the harness PR with happy + error pins for all 52 LLM-facing routes (models, chat/completions, completions, embeddings, moderations, audio, assistants, threads, utils, model-info, model-metrics, queue). Every happy-path test asserts the full response dict via normalize() so the gate enforces real shape pinning rather than status codes. * test(proxy): drop task-plumbing comments from PR2 test files * test(proxy): tighten PR2 error-path status-code pins Apply the same review feedback Greptile gave on PR1 (#28856) and PR3 (#28850) to PR2's forwarding-route tests: - Replace permissive `>= 400` / `in (X, Y)` status assertions with the exact 500/405 the handler actually returns, so a regression that silently shifts the code now fails the pin. - Add a body-presence check alongside each tightened status assertion to satisfy _pin_check.py's no-status-only rule. --------- Co-authored-by: Claude <noreply@anthropic.com> * test(proxy): pin proxy_server.py non-route surface behavior (PR1) (#28856) * test(proxy): pin proxy_server.py non-route surface behavior (PR1) Fills the 7 PR1 placeholder files under tests/test_litellm/proxy/proxy_server/ with behavior pins for the non-route surface of proxy_server.py: lifecycle/init/shutdown, ProxyConfig class methods, DB-overlay config scrubbers, spend counters, background-health helpers, OpenAPI customization, exception handlers, and streaming-generator helpers. 233 tests cover 101 pin-list symbols (1+ happy + 1+ error each). New-tests-only coverage on litellm/proxy/proxy_server.py: 32.80% line / 20.91% branch (PR1 gate: 25% line / 18% branch). Full directory runs in ~22s with -n 4. Plan: https://www.notion.so/Plan-Pin-proxy_server-py-behavior-2026-05-25-36c43b8acdab81ee845fd5365128a2fc * test(proxy): address Greptile review comments on test_lifecycle.py - test_initialize_signature_is_async_with_expected_params: hard-code expected_param_count so a signature change actually trips the gate (previously both sides of the comparison were len(sig.parameters)). - test_check_request_disconnection_invalid_when_connected_times_out: patch asyncio.sleep so the test no longer spins for ~1.2 s of real wall-clock; timeout lowered to 0.05 s. --------- Co-authored-by: Claude <noreply@anthropic.com> * test(proxy/proxy_server): pin control-plane routes (PR3) (#28850) * test(proxy/proxy_server): pin misc routes (PR3, partial) Adds happy + error tests for the misc control-plane routes: GET /, /routes, /adaptive_router/state, /get_logo_url, /get_image, /get_favicon. Also gitignores .pin_list.txt (used by the pin gate). * test(proxy/proxy_server): pin login/SSO routes (PR3, partial) Adds happy + error tests for the 5 login/SSO control-plane routes: GET /fallback/login, POST /login, POST /v2/login, POST /v3/login, POST /v3/login/exchange. Mocks authenticate_user and create_ui_token_object at their imported location. * test(proxy/proxy_server): pin onboarding routes (PR3, partial) Adds happy + error tests for the 2 onboarding control-plane routes: GET /onboarding/get_token, POST /onboarding/claim_token. Wires a MagicMock async context manager for prisma_client.db.tx() and signs the onboarding JWT with the patched master_key. * test(proxy/proxy_server): pin model_cost_map reload routes (PR3, partial) Adds happy + error tests for the 5 model-cost-map control-plane routes: POST /reload/model_cost_map, POST|DELETE|GET /schedule/model_cost_map_reload(/status), GET /model/cost_map/source. Attaches litellm_config to mock_prisma per-test (the table is not in the default _PRISMA_TABLES fixture). * test(proxy/proxy_server): pin anthropic_beta_headers reload routes (PR3, partial) Adds happy + error tests for the 4 anthropic-beta-headers control-plane routes: POST /reload/anthropic_beta_headers, POST|DELETE|GET /schedule/anthropic_beta_headers_reload(/status). Stubs db.litellm_config (not in default _PRISMA_TABLES) and monkeypatches reload_beta_headers_config so no network calls fire. * test(proxy/proxy_server): pin invitation routes (PR3, partial) Adds happy + error tests for the 4 invitation control-plane routes: POST /invitation/new, GET /invitation/info, POST /invitation/update, POST /invitation/delete. Patches _user_has_admin_privileges / _user_has_admin_view to avoid extensive get_user_object mocking. * test(proxy/proxy_server): pin config CRUD routes (PR3, partial) Adds happy + error tests for the 8 config-CRUD control-plane routes: POST /config/update, POST|GET /config/field/update|info, GET /config/list, POST /config/field/delete, POST /config/callback/delete, GET /get/config/callbacks, GET /config/yaml. Attaches litellm_config to mock_prisma per-test. * test(proxy/proxy_server): tighten pin assertions per review - test_routes_misc.py: `b"" in response.content` is trivially true; replace with `len(response.content) > 0` so an empty 405 body trips the gate. - test_routes_login_sso.py: `len(response.content) >= 0` is trivially true; tighten to `> 0`. - test_routes_anthropic_beta.py: replace brittle string-literal checks on the serialized JSON (`'"interval_hours": 12' in payload`) with `json.loads` + dict access so the assertion survives any serializer spacing. - test_routes_config.py: `assert status_code in (404, 500)` was too permissive; the handler re-raises HTTPException(404) verbatim, so pin 404 strictly. --------- Co-authored-by: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Yassin KortamandGitHub
|
d82eb33a60 | feat(otel): typed semconv-aligned OpenTelemetry instrumentation (#28909) | ||
|
Mateo WangandGitHub
|
581c30f1e8 | [internal copy of #29089] fix: duplicate claude code traces (#29311) | ||
|
ryan-crabbe-berriandGitHub
|
c754c560dd |
fix(proxy): link passthrough success spans to the SERVER root OTEL span (#29315)
* fix(proxy): link passthrough success spans to the SERVER root OTEL span Passthrough requests never wired user_api_key_dict.parent_otel_span into the logging metadata, so on success the litellm_request span orphaned into its own trace and the "Received Proxy Server Request" root span was never ended. Setting it once in _init_kwargs_for_pass_through_endpoint fixes both the non-streaming and streaming paths, since update_environment_variables copies that metadata onto the logging object's model_call_details, which is what the OTEL handler reads. Resolves LIT-3443 * fix(proxy): set passthrough parent span after client metadata merge Greptile flagged that litellm_parent_otel_span was assigned before the _metadata.update() calls that merge request-body metadata, so a client body mirroring the internal key could overwrite the real span with a JSON scalar and null the fix for that request. Move the assignment after the merge and add a regression test that fails on the old ordering. * fix(proxy): also set user_api_key after client metadata merge Per Greptile, user_api_key had the same clobber window as the parent span: a passthrough request body mirroring the key could overwrite the authenticated value in the logged metadata. Move it into the same post-merge block and add a deterministic contract test asserting both internal keys resist client-supplied metadata. |
||
|
|
af17400c38 |
feat(a2a): well-known agent-card discovery + LangGraph Platform mode (#28860)
* feat(a2a): well-known agent-card discovery + LangGraph Platform mode Adds a registration-time discovery flow so admins can paste an upstream agent URL, see its skills/capabilities, pick what to expose, and have the proxy front it with a LiteLLM-shaped agent card. Backend (new litellm/proxy/a2a/ module): - fetch_well_known_card walks /.well-known/agent-card.json, /.well-known/agent.json, /agent.json by default. langgraph_platform mode hits the canonical path with ?assistant_id=<id> (LangGraph serves one shared endpoint per deployment). - merge_agent_card overlays LiteLLM overrides on the upstream card: drops upstream url, forces protocolVersion=1.0, replaces securitySchemes with LiteLLMKey bearer, emits supportedInterfaces pointing at the proxy, filters capabilities to a small allowlist, strips non-v1.0 fields. - POST /v1/a2a/discover returns the raw upstream card (admin-only) so the UI can render skills/capabilities for selection. - create/update/patch agent endpoints pre-generate the agent_id and run merge_agent_card before storing, so DB.agent_card_params already embeds the proxy-fronted URL. UI (ui/litellm-dashboard): - New AgentCardDiscovery component with a parent-driven plan: discovery_mode + params + display URL. For LangGraph the parent composes (api_base, assistant_id); for pure A2A it uses the url field. Component hides the manual URL input when the parent drives. - add_agent_form wires discovery for every non-custom agent type and overlays the user's selections onto agent_card_params at submit, fixing the bug where dynamic agent forms ignored discovery picks. Completion-bridge fixes (paired): - Add kind: "message" to A2A response messages and unwrap result so it's a Message directly per spec (matches a2a SDK SendMessageResponse validation). - Forward A2A metadata to LangGraph runs via extra_body.metadata. * fix(a2a): preserve agent url, fix streaming chunk envelope, and protect forwarded metadata - Streaming chunk: move final out of the message object into the result envelope per the A2A spec. - Agent card merge: keep upstream url on the stored card so the runtime invocation path can locate the upstream backend; the public well-known endpoint already rewrites this field to the proxy URL before exposing it to clients. - Completion bridge: apply A2A forward metadata after merging litellm_params so an agent-configured extra_body cannot overwrite the forwarded metadata. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(a2a): fix legacy streaming chunk, agent card test, and metadata merge - providers/litellm_completion: move 'final' out of the message object into the result envelope per the A2A spec (matches the bridge fix). - agent endpoints test: the runtime invocation path now preserves the top-level 'url' on the stored card, so update the assertion to match. - completion bridge metadata: when forwarding A2A metadata via extra_body.metadata, merge into any existing extra_body.metadata instead of replacing it, so an agent-configured metadata block is preserved (forward metadata still wins on key conflicts). Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(a2a): remove dead duplicate transformation dir; drop SSRF-prone headers field from /v1/a2a/discover Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(a2a): revert accidental html→index.html rename from |
||
|
|
83fcacad08 |
fix(mcp): ignore stale ids on key save (#29128)
* fix(mcp): ignore stale server ids during key permission validation Prevent virtual key save failures when object_permission still includes MCP server IDs that were deleted from the registry. The validator now drops stale IDs before team scope checks and adds coverage for stale-ID and active unauthorized-ID behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): use DB find_many to detect stale server IDs, add tool-permission test Addresses greptile feedback on PR #29128: P1 – Race-condition / authoritative-signal concern: Extract _get_stale_mcp_server_ids() helper that performs a single DB find_many (LiteLLM_MCPServerTable) when prisma_client is available, making server-existence checks authoritative even during in-memory registry warm-up. Falls back to the in-memory registry only when no DB client is present (config-only deployments / unit tests). Pass prisma_client through both call sites in key_management_endpoints. P2 – Missing test coverage for mcp_tool_permissions: Add test_validate_stale_ids_in_mcp_tool_permissions_silently_dropped to confirm stale server IDs that appear only as keys in mcp_tool_permissions are also stripped without raising 403. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): also check in-memory registry when DB is present to avoid misclassifying config-file servers as stale Config-file MCP servers are never written to LiteLLM_MCPServerTable, so a DB-only query would classify them as stale and silently drop their IDs from the authorization check, allowing unauthorized key access. Fix: treat a server ID as stale only when it is absent from BOTH the DB and the in-memory registry (which holds config-file servers). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): normalize server aliases on key save Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: remove unused stale MCP helper and capture normalized object_permission on key generation Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(mcp): keep validate_key_mcp_servers_against_team stub in sync with return contract validate_key_mcp_servers_against_team now returns the (normalized) object_permission, and the key-generation helper assigns that return value back into the request data. The two key-generation tests stubbed the function with a bare AsyncMock, whose default MagicMock return value clobbered object_permission and skipped permission-record creation. Make the stubs pass object_permission through unchanged. * fix(mcp): preserve provided object_permission fields on key update The key-update path reconstructs data.object_permission from a full model_dump(), which marks every field as set. Downstream model_dump(exclude_unset=True) then emits models/blocked_tools/ search_tools as None, and those are non-nullable array columns, so the Prisma write fails whenever the UI submits object_permission with only a subset of fields populated (e.g. a TPM/RPM-only edit). Build the dict with exclude_unset=True so the normalized object_permission keeps the caller's original field set. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
1d9095f914 |
fix(bedrock): support tool search results + chat annotations (#29120)
* Fix overiding of fastapi_response headers * fix(bedrock): support tool search results and surface citations as annotations Add an optional tool-message search_results path that maps directly to Bedrock toolResult.searchResult blocks, and convert Converse citationsContent into chat completion annotations for user-facing citation metadata. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(format): align bedrock prompt factory with black Reformat the updated bedrock prompt template conversion file so CI black --check passes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(bedrock): harden citations, search_results mapping, and token counting Resolve mypy issues in citation parsing, only attach url_citation annotations when citation text is stitched into content, fall back to tool content when search_results is empty, and count search_results text in token/TPM preflight paths. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(bedrock): extract tool result helpers to satisfy PLR0915 Refactor _convert_to_bedrock_tool_call_result into smaller helpers so lint passes without changing Bedrock tool result behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(bedrock): count all forwarded search_results fields in token estimates Include source, title, content text, and citations when estimating tokens so large metadata cannot bypass TPM preflight checks. Reformat factory.py with black. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(managed-files): skip content blocks without a type key in get_file_ids_from_messages * fix(bedrock): stitch citations for any punctuation-only text block * fix(bedrock): map null citation source/title to empty annotation strings * fix(bedrock): advance citation offset for text-only citationsContent blocks * fix(bedrock): complete citation TypedDicts for grounding annotations --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
c792df64d2 |
feat(mcp): support stateless and stateful clients via session-id routing (#26857)
* feat(mcp): support stateless and stateful clients via session-id routing - Add session_manager_stateful (stateless=False) alongside stateless - Route by mcp-session-id: has ID → stateful, initialize (no ID) → stateful, else → stateless - Peek POST body to detect initialize for routing; replay via wrapped receive - Handle stale session IDs for both managers - Add test_mcp_routing_initialize_to_stateful_no_session_to_stateless - Update test_valid_mcp_session_id_is_preserved, test_concurrent_initialize_session_managers Made-with: Cursor * fix(mcp): respect stateful routing and harden initialize detection Ensure streamable MCP requests are dispatched via the computed target session manager, and guard initialize detection against non-object JSON bodies. Update stale-session test patches to target the stateful manager so routing assertions remain correct. Made-with: Cursor * test(mcp): patch stateless/stateful managers in concurrency init test Update concurrent session-manager initialization test to patch session_manager_stateless and session_manager_stateful directly, matching initialize_session_managers() behavior and preventing NameError from undefined mocks. Made-with: Cursor * Fix tests * Fix tests * Fix MCP stateful routing edge cases * Fix stateful MCP auth context refresh * Fix MCP stateful session cleanup * fix(mcp): bind stateful sessions to creator and reject hijacks Stateful mcp-session-id was usable by any authenticated proxy caller. Track the session creator's hashed API key (or user_id) when a new session is issued and reject mismatched callers with 403 before _set_or_update_auth_context overwrites the stored MCPAuthenticatedUser. Also formats nested with-statements in test_mcp_stale_session.py and fixes a pre-existing AsyncMock mismatch in test_stale_mcp_session_id_is_stripped. * fix(mcp): serialize concurrent requests on same stateful session Bugbot's 'Concurrent requests share context' finding: _update_auth_context mutates the single MCPAuthenticatedUser stored per session in place on every request, so two requests sharing one mcp-session-id can overwrite each other's mcp_servers / auth headers / oauth state / client_ip while in-flight callbacks are still reading the same object. Owner-binding alone narrows this to same-principal racing, but the in-place mutation race remains. Add a per-session asyncio.Lock around handle_request so concurrent same-session requests run sequentially. The lock is allocated on demand and torn down with the rest of the session state on DELETE / idle expiry. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(mcp): include OAuth2 bearer in stateful session owner fingerprint UserAPIKeyAuth() for OAuth2 passthrough has no api_key/user_id, so every OAuth caller fingerprinted to "anonymous" and could hijack another OAuth caller's mcp-session-id. Hash the upstream Authorization header into the fingerprint as oauth:<sha256>. * fix(mcp): don't hold stateful session lock for streaming GETs The per-session lock wraps handle_request, so a long-lived GET (SSE stream held open for the life of the session) would block every subsequent POST on the same mcp-session-id. Only POST/DELETE mutate the shared MCPAuthenticatedUser, so it's sufficient to serialize those — GETs run lock-free and stream concurrently. * fix(mcp): allow None user_api_key_auth in MCPAuthenticatedUser The set_auth_context / _set_or_update_auth_context / _update_auth_context helpers in server.py all accept Optional[UserAPIKeyAuth] and pass it straight into MCPAuthenticatedUser, but the dataclass-style constructor typed user_api_key_auth as required UserAPIKeyAuth. Mypy flagged this on the stateful-routing branch: server.py:3227: error: Incompatible types in assignment (expression has type "UserAPIKeyAuth | None", variable has type "UserAPIKeyAuth") server.py:3255: error: Argument "user_api_key_auth" to "MCPAuthenticatedUser" has incompatible type "UserAPIKeyAuth | None"; expected "UserAPIKeyAuth" Widen the parameter type to Optional[UserAPIKeyAuth] to match the call sites. Runtime behavior is unchanged. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * style: replace with new alias * fix(mcp): fall back to client_ip in stateful session owner fingerprint Addresses Greptile review on PR #26857: when no API key, user_id, or OAuth bearer is available (e.g. unauthenticated/passthrough callers), the owner fingerprint collapsed to a single 'anonymous' value, allowing two unrelated callers to drive each other's stateful MCP sessions. Fold client IP into the fingerprint as a fallback identity signal so distinct anonymous sources do not share an owner identity. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * Fix active stateful MCP session cleanup * test(mcp): cancel leaked stateful auth-context cleanup task initialize_session_managers() spawns a real asyncio.create_task running _cleanup_expired_stateful_session_auth_contexts(). The test_concurrent_initialize_session_managers test was saving and restoring the session-manager context-manager globals but did not save, cancel, or restore _stateful_auth_context_cleanup_task. Because pyproject.toml sets asyncio_default_fixture_loop_scope=session, the event loop is shared across tests in the same session, so the leaked task kept running against module-level dicts for the rest of the test run. Save and cancel the task in the finally block so the test fully cleans up after itself. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * Fix stateful MCP session fingerprinting * Hash MCP session user owner fingerprints * Fix stale MCP session DELETE cleanup * fix(mcp): harden owner fingerprint hashing for non-str api keys _owner_fingerprint_for assumed api_key/user_id supported .encode(); MagicMock-based tests (and any non-str truthy values) crashed with TypeError before routing. Only hash str/bytes secrets; fall through otherwise so MCP routing and session tests behave correctly. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix MCP stateful cleanup loop resilience * Fix stateful MCP initialize auth capture * fix(mcp): drop orphan per-session lock when auth context absent Defensive cleanup for _stateful_session_locks entries created on sessions that never enter _stateful_session_auth_contexts. The periodic cleanup loop only iterates auth_context_last_seen, so such locks would otherwise live forever. Add a test that reproduces the leak and verifies the request finalizer pops the lock. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(mcp): trim verbose comment on lock cleanup Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * Fix stateful MCP delete failure tracking * fix test * fix(mcp): cap routing-peek body size to bound pre-dispatch memory Authenticated clients that POST without an mcp-session-id forced the proxy to buffer the entire request body before routing, since the peek loop drained every body chunk to decide whether the JSON-RPC method was 'initialize'. Cap the peek at 4 KB (more than enough for an initialize envelope) and let the remainder stream through wrapped_receive into the downstream handler. * test: replace dall-e-3 with gpt-image-1 in health check and router tests (#27813) OpenAI returns 'The model dall-e-3 does not exist' for the test account, breaking test_openai_img_gen_health_check and test_image_generation. Switch to gpt-image-1, matching the existing TestOpenAIGPTImage1 pattern. * fix(tests): drop dall-e-only test classes; route live image tests via gpt-image-1 Second wave of failures from the 2026-05-12 DALL-E shutdown: - tests/image_gen_tests/test_image_edits.py::TestOpenAIImageEditDallE2 and tests/image_gen_tests/test_image_generation.py::TestOpenAIDalle3 are explicitly named for the deprecated models and can't pass; remove. gpt-image-1 coverage already exists in sibling classes. - tests/local_testing/test_router.py image gen tests use dall-e-3 only as a routing example; swap to gpt-image-1. - tests/local_testing/test_custom_callback_input.py image_generation success/failure paths swapped to gpt-image-1. * Fix MCP initialize session active tracking Co-authored-by: Yassin Kortam <yassin@berri.ai> * Fix MCP reinitialize session tracking Co-authored-by: Yassin Kortam <yassin@berri.ai> * Fix MCP reinitialize auth context aliasing Co-authored-by: Yassin Kortam <yassin@berri.ai> * Apply black formatting after merge Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * Run owner-binding 403 before consuming POST body Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * Harden MCP routing peek bound and stateful purge race Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * Remove inadvertently committed Next.js build artifacts Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * Run owner check before stale MCP session cleanup Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(mcp): reverse cleanup ordering to terminate transport before clearing owner Reverses _purge_expired_stateful_session_auth_contexts so the transport is popped from server_instances and terminated BEFORE owner/auth tracking is cleared. The previous order left a window where _stateful_session_owners was already empty but server_instances still served the session, so a concurrent request would observe expected_owner is None and bypass the owner-binding check. Addresses Greptile review on PR #26857. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(mcp): fully reset stateful session tracking in auth-context refresh test Use _remove_stateful_session_tracking in teardown so the test no longer leaks _stateful_session_auth_context_last_seen and _stateful_session_locks between tests, matching the cleanup used by the sibling stateful tests. * fix(mcp): cap concurrent stateful sessions per caller to bound memory --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: Sameerlite <sameerlite@users.noreply.github.com> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Claude Babysitter <claude@anthropic.com> Co-authored-by: mateo-berri <mateo@berri.ai> |
||
|
|
70d2748d80 |
fix(proxy): map stripped batch body.model to proxy alias for auth (#29264)
* fix(proxy): map stripped batch body.model to proxy alias for auth replace_model_in_jsonl rewrites JSONL body.model to the provider id before upload; batch file access checks must resolve that id back to model_name so keys granted the proxy alias are not rejected with 403. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): surface resolved proxy alias in batch file 403 detail --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
0ffab87da4 |
fix(router): enforce deployment budgets for dynamically added models (#29273)
* fix(router): enforce deployment budgets for dynamically added models Register deployment max_budget/budget_duration when models are added via upsert_deployment (e.g. /model/new) so RouterBudgetLimiting matches startup model_list behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): address CI lint and router coverage for budget sync helpers Remove unused RouterBudgetLimiting import and add router unit tests for deployment budget helper methods required by router_code_coverage. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): clear stale deployment budget on upsert without limits Unregister deployment budget config when max_budget/budget_duration are removed, including upsert replace paths. Hoist provider budget logger lookup outside the provider loop. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix mypy * fix black --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
ryan-crabbe-berriandGitHub
|
909a5f597a |
fix(mcp): resolve key.access_group_ids → MCP servers (ungated) (#29195)
* fix(mcp): resolve key.access_group_ids → MCP servers (ungated) A teamless virtual key whose unified access_group_ids grant an MCP-granting access group now sees and can call that server instead of getting an empty list / 403. The key path previously read only the legacy object_permission; this folds key.access_group_ids into the key's base scope (ungated), mirroring can_key_call_model's fallback. The gated assigned_*-checked override is unchanged. * fix(mcp): expand name/alias entries in key access-group servers access_mcp_server_ids may hold server names/aliases, not just ids. The new ungated key path now runs them through expand_permission_list at the source, so the early-return and union branches both surface resolved ids — matching the legacy object_permission path and the gated extras path. |
||
|
ryan-crabbe-berriandGitHub
|
5eafe1c1fc |
test(e2e): cover navbar Logout flow as proxy admin (#29076)
* test(e2e): cover navbar Logout flow as proxy admin
The Logout button under the navbar User dropdown was an uncovered
manual-QA step. This test signs in as admin, opens the dropdown,
clicks Logout, then navigates to a protected page and asserts the
redirect to /ui/login — proving the session was cleared.
* test(e2e): fix logout dropdown trigger and account-menu selector
The button never rendered the literal text "User" (it shows initials +
display name), and the antd Dropdown uses trigger={["click"]}, so the
synthetic mouseover/mouseenter never opened the popup. Open it with a real
click on the button's aria-label ("Account menu — ...").
|
||
|
ryan-crabbe-berriandGitHub
|
fcd5760891 |
test(e2e): cover Internal User key modal, team info, key page (#29074)
* test(e2e): cover Internal User key modal, team info, key page Three previously-uncovered manual-QA paths for the Internal User role: - Create Key modal — confirm the team dropdown is populated with the user's teams (verifying the role-scoped UI flow exists). - Team info page — confirm the Settings/Members tabs are hidden for a regular team member; only the read-only tabs render. - Virtual Keys page — confirm the proxy's internal litellm-dashboard team keys never leak into an internal user's table. * test(e2e): share clickTeamId helper, strengthen key-filter assertion Address review feedback on the Internal User e2e spec: - Extract clickTeamId into helpers/navigation.ts; import in both internalUser and teams specs instead of duplicating it. - Anchor the litellm-dashboard absence check on the user's own seeded key so it cannot pass vacuously against an empty table. - Drop redundant dismissFeedbackPopup calls (navigateToPage already dismisses internally). |
||
|
ryan-crabbe-berriandGitHub
|
10bda4456a |
test(e2e): cover Internal Viewer nav, key, and team-info gating (#29075)
* test(e2e): cover Internal Viewer nav, key, and team-info gating Three previously-uncovered manual-QA paths for the Internal Viewer role: - Nav only renders the read-only sections; admin-only items (Internal Users, Organizations, Models + Endpoints) stay hidden. - Virtual Keys page hides Create New Key, and the key detail view hides Regenerate / Reset Spend / Delete actions. - Team info page hides Members and Settings tabs for the viewer. * test(e2e): scope viewer nav to sidebar, strengthen tab assertions Address review feedback on the Internal Viewer e2e spec: - Scope the nav test to the sidebar complementary landmark and match items by link role + accessible name. The prior CSS nav, aside selector grabbed the top bar (the sidebar is a complementary landmark, not a <nav> tag), so the assertions never hit the real nav links. - Land via navigateToPage so the networkidle wait settles the role-gated nav before asserting. - Assert the Virtual Keys tab is visible (was only commented). - Use toHaveCount(0) for hidden team tabs to match the nav block; tabs are conditionally rendered, not CSS-hidden. - Drop redundant dismissFeedbackPopup calls (navigateToPage already dismisses internally). |
||
|
Mateo WangandGitHub
|
a55817cbc6 |
fix(anthropic): stop injecting unsupported output_config.effort=xhigh for Claude Code on Sonnet/Opus 4.6 (#29304)
* fix(anthropic): don't inject output_config.effort=xhigh on models without xhigh The legacy-thinking translator on the /v1/messages route mapped any thinking.budget_tokens >= 24000 to effort=xhigh and injected it into output_config without checking model support. Claude Code's default thinking budget (31999) hit this bucket, so Sonnet 4.6 (and Opus 4.6) on Bedrock/Vertex started returning 400 output_config.effort: Input should be 'low', 'medium', 'high' or 'max' Gate the xhigh choice on _supports_effort_level(model, "xhigh"), the same capability check the reasoning_effort path already uses. Models that advertise xhigh (Opus 4.7) keep it; everything else falls to high. Fixes #29282 * test(anthropic): pin Opus 4.6 in legacy-thinking xhigh-clamp regression test Opus 4.6 (bare, bedrock/invoke, vertex_ai) has supports_adaptive_thinking but no supports_xhigh_reasoning_effort, so it hits the same clamping path as Sonnet 4.6. It was named in the PR scope but lacked a pinned regression guard; add the three variants to the parametrize list. |
||
|
michelligabrieleandGitHub
|
68852ef165 | fix(teams): expose keys_count on /v2/team/list and wire UI Resources badge (#28502) | ||
|
|
f27df8d516 |
docs: hand-written CLAUDE.md; point GEMINI.md and AGENTS.md at it (#29252)
* docs: replace generated CLAUDE.md with hand-written guidance, remove AGENTS.md Swap the auto-generated CLAUDE.md for a concise hand-written version that captures how we actually want agents to work in this repo: minimal comments, simplicity first, meaningful tests with a high mutation kill rate, PRs based off litellm_internal_staging rather than main, and curl against a live proxy as proof of fix instead of pasted pytest output. Remove AGENTS.md so there is one source of truth for agent guidance. The customer and company name confidentiality policy, along with the MCP available_on_public_internet note, are carried over from the previous CLAUDE.md. * fix: further clarify communication guidelines * docs: point GEMINI.md at CLAUDE.md instead of duplicating guidance Replace the standalone GEMINI.md copy, which had already drifted from the new CLAUDE.md, with a one-line pointer so Gemini reads the same single source of truth. * docs: simplify PR template test checklist item Replace the rigid "at least 1 test is a hard requirement" checklist line with "I have added meaningful tests", which matches the testing guidance in CLAUDE.md, and tidy a comma into a semicolon in the scope-isolation item. * docs: point AGENTS.md at CLAUDE.md instead of deleting it Keep AGENTS.md so tools that read it still resolve guidance, but collapse it to the same one-line pointer to CLAUDE.md used by GEMINI.md, keeping a single source of truth. * fix: make AI-generated rules more concise * fix: spelling Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: make the .env usage more careful * docs: restore MCP available_on_public_internet note to CLAUDE.md The PR description states this note was carried over verbatim from the previous CLAUDE.md, but it was dropped in the rewrite. Restore it so the file matches the description and the team guidance is not lost. * docs: restore browser storage and CI supply-chain safety notes to CLAUDE.md These security-relevant rules were dropped in the rewrite. Restore the sessionStorage-over-localStorage (XSS) guidance and the CI supply-chain rules (no curl|bash, pin versions, verify checksums) so agents editing UI or CI code are still steered away from those pitfalls. * docs: move area-specific guidance into nested CLAUDE.md files The MCP, browser-storage, and CI supply-chain notes are scoped to particular parts of the tree, so move each into a nested CLAUDE.md that Claude Code loads on demand when those files are touched: the MCP note under the mcp_server gateway, the browser-storage rule under the UI dashboard, and the CI supply-chain rules under .circleci. Keeps the root CLAUDE.md focused on general guidance while the area notes surface where they are relevant. * docs: keep CI supply-chain note in root CLAUDE.md CI guidance applies beyond .circleci (it also covers downloads in GitHub workflows and any CI script), and CI work does not reliably touch a single subtree, so a nested file under .circleci would not surface it dependably. Keep it in the always-loaded root instead. The MCP and browser-storage notes stay nested where they map cleanly to one area of the tree. * fix: make it clear we prefer httpOnly * chore: make ci rule more concise * chore: make concise Fix formatting and punctuation in MCP note. * fix: don't include Claude attribution --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berriandGitHub
|
2bfbf14882 |
test(e2e): cover Team Admin view + member + key flows (#29072)
* test(e2e): cover Team Admin view + member + key flows
Adds a new spec exercising the previously-uncovered team-admin manual-QA
items: viewing all team keys (including other members'), adding a member,
removing a member, and creating a team key with All Team Models. Also
seeds a dedicated invitee user so the add-member test can run in parallel
with the proxy-admin invite test without colliding on the team roster.
* test(e2e): harden team-admin member specs per review feedback
Address Greptile feedback on the Team Admin spec:
- locate the delete action via getByTestId("delete-member") instead of
the fragile svg/img .last() selector
- match the seeded removable member by user_id (members_with_roles stores
no email, so the roster renders user_id)
- assert exact success-toast strings rather than broad regexes that could
match unrelated "success" text
|
||
|
ryan-crabbe-berriandGitHub
|
9918a9c78c |
fix(guardrails): persist disable_global_guardrails on keys (#29233)
* fix(guardrails): restore disable_global_guardrails persistence for keys The per-key/team "Disable Global Guardrails" toggle silently stopped working after #17042, which removed `disable_global_guardrails` from the key/team request models and from the premium metadata allowlist. Without those, the UI's top-level field was dropped by pydantic and never folded into key `metadata`, so the runtime gate always read False and global default_on guardrails kept running. Restore the request-model fields (KeyRequestBase, NewTeamRequest, UpdateTeamRequest) and the `LiteLLM_ManagementEndpoint_MetadataFields_Premium` entry so the flag is promoted into metadata again. Because the key edit form always submits the flag (false by default), guard the UI so it is only sent when it actually changed (edit) or is enabled (create) — this keeps the premium gate on enabling intact while not 403-ing non-premium users who edit unrelated key fields, mirroring how guardrails/tags are already stripped. * test(guardrails): cover disable_global_guardrails toggle-off + clarify premium field comment Add a prepare_metadata_fields case asserting `disable_global_guardrails: False` overwrites an existing `True`, and rewrite the PREMIUM_METADATA_FIELDS comment to explain why boolean premium fields are excluded from the empty-value strip loop. |
||
|
|
01e83e2537 |
fix(ci): restore real Bedrock batch S3 bucket and role in oai_misc_config (#29245)
The OSS-staging sync (
|
||
|
Mateo WangandGitHub
|
bae04591b2 |
feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238)
* feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags Register claude-opus-4-8 across the anthropic/bedrock/vertex/azure cost-map entries, BEDROCK_CONVERSE_MODELS, and the setup-wizard provider list. Prune two reasoning-effort fields from the cost map: - Drop supports_minimal_reasoning_effort from the Claude fleet (58 entries). "minimal" is not a real Anthropic effort level (the API accepts only low/medium/high/xhigh/max), so LiteLLM degrades it to "low" regardless; the flag was inert and misleading on Anthropic. - Remove tool_use_system_prompt_tokens everywhere (103 entries). It is not in the ModelInfo type and is read by no production code. Update the affected config/schema tests; the reasoning-effort registry tests now assert the Claude fleet omits supports_minimal. * fix(anthropic): recognize output_config effort after minimal-flag prune Pruning supports_minimal_reasoning_effort from the Claude fleet removed the only "supports effort param" marker from 11 Opus 4.5 / mythos-preview map entries that lack supports_output_config. _model_supports_effort_param then returned False for them, so output_config was wrongly dropped under drop_params=True -- regressing test_anthropic_model_supports_effort_param_recognizes_supporting_models for claude-opus-4-5-20251101 and the mythos preview. - _model_supports_effort_param now treats supports_output_config as a sufficient signal, matching the bedrock-invoke call sites that already check supports_output_config OR a reasoning-effort flag. Shared map lookup extracted into _supports_model_capability. - Add supports_output_config: true to the 11 Opus 4.5 / mythos entries that lost their only marker, restoring prior effort-forwarding behavior without re-adding the inert minimal flag. |
||
|
yuneng-jiangandGitHub
|
ffc113b428 |
chore(ci): bump version (#29242)
* bump: version 1.87.0 → 1.88.0 * uv lock |
||
|
yuneng-jiangandGitHub
|
6b23d32ea0 |
chore(cookbook): bump Go directive to 1.26.3 in gollem example (#29234)
Updates the gollem_go_agent_framework example to the current Go release. Clears stale Go stdlib advisories reported by osv-scanner against the older 1.25.1 directive. No source changes; the single pinned dependency (gollem v0.1.0) is backward compatible. |
||
|
yuneng-jiangandGitHub
|
76f56c3283 |
fix(tests/vcr): mint Google OAuth tokens live to prevent stale-token replay (#29229)
The Redis-backed VCR layer was recording and replaying the Google OAuth2/STS token-mint call. The replayed ya29.* access token is long-expired, but its recorded expires_in keeps credentials.expired False, so litellm never refreshes it and sends the stale token to a live Vertex/Gemini endpoint, which returns 401 ACCESS_TOKEN_EXPIRED. This broke live partner-model tests whose completion call is not itself cassette-backed (e.g. test_vertex_ai_llama_tool_calling). Force credential-exchange hosts to pass through live (never recorded, never replayed) by returning None from before_record_request, mirroring the existing telemetry passthrough, so a fresh token is minted each run. Regression from #28826, which added OAuth-token matcher tolerance plus TTL-refresh-on-read so a stale token episode matched and never expired. |
||
|
|
5e2d75d75d |
bump deps (#29208) (#29226)
* fix(deps): bump vulnerable proxy dependencies (starlette/fastapi, granian, pyarrow, semantic-router) Resolve known CVEs flagged by osv-scanner/grype against uv.lock. All bumped versions verified to resolve, install, and pass the proxy auth/route/middleware unit suites (717 tests) plus an import smoke on the new stack. - starlette 0.50.0 -> 1.1.0 (CVE-2026-48710 "BadHost", GHSA-86qp-5c8j-p5mr): versions <1.0.1 reconstruct request.url from the unvalidated Host header, poisoning request.url.path. Required raising fastapi 0.124.4 -> 0.136.3, which dropped fastapi's starlette<0.51.0 cap; an explicit starlette>=1.0.1 floor blocks regression to a vulnerable transitive resolution. The proxy's own auth already reads scope["path"] via get_request_route, but the locked starlette still flagged in container scanners and left other request.url consumers exposed. - granian 2.5.7 -> 2.7.4 (CVE-2026-42544, unauthenticated DoS via WebSocket subprotocol header panic; CVE-2026-42545, WSGI response-header-panic DoS). granian is a selectable proxy server (proxy_cli). - pyarrow 22.0.0 -> 23.0.1 (CVE-2026-25087 / PYSEC-2026-113). - semantic-router 0.1.12 -> 0.1.15: 0.1.12 was yanked (CVE-2026-42208 — its unbounded litellm pin could resolve a credential-exfiltrating litellm==1.82.8 wheel). Not fixable by bump: diskcache 5.6.3 (CVE-2025-69872, unsafe pickle deserialization) has no upstream fix and is left pinned; exploiting it requires write access to the local cache directory. Relock side effect: sse-starlette 3.4.2 -> 3.4.4. * deps: relax exact pins in optional extras to compatible ranges The proxy/optional extras exact-pinned every dependency, which (1) forces downstream `pip install litellm[proxy]` consumers into version lockstep and (2) blocks them from pulling transitive security patches without forking — the structural cause behind needing a litellm release to clear the starlette CVE in the previous commit. Convert the ordinary extras deps to `>=current,<next_major` ranges, mirroring the core [project].dependencies style. Reproducibility for litellm's own Docker/CI is unaffected: images install via `uv sync --frozen`, and the lock re-resolves to the identical versions (no locked version changed). Kept exact-pinned: - litellm-proxy-extras, litellm-enterprise — litellm's own sub-packages, versioned in lockstep with the release. - opentelemetry-api/sdk/exporter-otlp — must resolve to matching versions. - grpcio — supply-chain-pinned to a vetted, aged release. Also corrects the stale comment claiming the extras are exact-pinned for Docker reproducibility (the images use the lock, not these pins). * fix(ci): resolve license-check lookup version from the floor for ranged deps check_licenses.py derived the PyPI lookup version with `next(iter(req.specifier))`, which returns an arbitrary specifier clause. For a range like `>=0.12.1,<1.0` it picked the upper bound (`1.0`) — a version that doesn't exist on PyPI — so the license lookup 404'd and the package was flagged as having an unknown license. The previous commit's switch from exact pins to ranges exposed this for soundfile, pyroscope-io, redisvl, diskcache, and mlflow (the ranged deps not already in liccheck.ini's allowlist). Prefer a lower-bound/exact version (a real released version) for the lookup. * fix(proxy): set strict_content_type=False on the FastAPI app Starlette 1.0 / FastAPI 0.13x flipped the default to strict_content_type=True, which refuses to parse a JSON request body when the client omits the Content-Type header. The proxy previously accepted those requests, so the fastapi/starlette bump in this PR would silently break clients that don't send a Content-Type. Restore the prior lenient behavior explicitly. Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> |
||
|
|
47c92669c2 |
feat(helm): split per-component ServiceAccounts for gateway, backend, and UI (#28712)
* feat(helm): split per-component ServiceAccounts for gateway, backend, and UI Replace the single shared serviceAccount with three separate serviceAccounts (gateway, backend, ui) so operators can attach different IRSA / Workload Identity annotations per component without granting data-plane credentials to the UI pod. Key changes: - values.yaml: rename serviceAccount → serviceAccounts with gateway/backend/ui sub-keys; UI defaults to automount: false - _helpers.tpl: replace litellm.serviceAccountName with three component-scoped helpers (litellm.gateway/backend/ui.serviceAccountName) - serviceaccount.yaml: create up to three separate ServiceAccount objects with component labels and per-SA automountServiceAccountToken - gateway/backend deployments: use their respective SA helpers - ui deployment: use litellm.ui.serviceAccountName + explicit automountServiceAccountToken: false on the pod spec so the projected token is absent even when the SA itself allows it - migrations-job: share the backend SA (both need DB write access) Resolves LIT-3171 https://claude.ai/code/session_01QPy362WnjmEpeNuJaPUqmF * fix(helm): enforce automountServiceAccountToken on all pod specs; fix leading --- in serviceaccount.yaml - gateway/backend deployments: add explicit automountServiceAccountToken on the pod spec so serviceAccounts.*.automount is honoured regardless of whether the SA is chart-created or operator-supplied (previously the flag only took effect on the SA object when create: true, creating an asymmetry with the UI which already enforced it at pod-spec level) - serviceaccount.yaml: use a $prev sentinel to emit --- only between documents, preventing a leading --- when gateway SA is skipped but backend or ui SA is created (avoids lint/GitOps warnings from strict YAML parsers and tools like ArgoCD) https://claude.ai/code/session_01QPy362WnjmEpeNuJaPUqmF --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
michelligabrieleandGitHub
|
928f09f8a4 |
fix(datadog): drain cost-management queue + opt-in FinOps tag allowlist (#28487)
* fix(datadog): drain cost-management queue + opt-in FinOps tag allowlist * fix(datadog): guard non-dict callback_specific_params + log empty aggregation * fix(datadog): block user-controlled tags from overwriting reserved cost-attribution dimensions * fix(datadog): cast metadata to dict[str, Any] to satisfy mypy |
||
|
|
69afcd09d0 |
fix(vertex-ai): use DB credentials in video handlers + implement Veo video edit (#29098)
* fix(vertex-ai): pass litellm_params to validate_environment in video handlers and implement video edit for Veo - Pass litellm_params to validate_environment in 11 video handler call sites (remix, create_character, get_character, edit, extension, delete) so DB-stored Vertex AI credentials are used instead of falling back to ADC - Implement transform_video_edit_request/response for VertexAI: fetches source video via fetchPredictOperation then submits a new predictLongRunning request with the video bytes/gcsUri + edit prompt Co-authored-by: Cursor <cursoragent@cursor.com> * fix(vertex-ai): hoist fetchPredictOperation into handlers to avoid blocking event loop - Add get_video_edit_prefetch_params() to BaseVideoConfig (returns None) - VertexAI overrides it to return the fetchPredictOperation URL/body - Both sync and async video_edit handlers call this and use their shared httpx client for the fetch, passing the result as prefetched_source_data - transform_video_edit_request is now a pure transform with no HTTP calls - Fix extra_body.pop() mutation by working on a shallow copy Co-authored-by: Cursor <cursoragent@cursor.com> * fix(vertex-ai): include prefetch call inside _handle_error try/except block Co-authored-by: Cursor <cursoragent@cursor.com> * fix(videos): add prefetched_source_data param to all transform_video_edit_request overrides Co-authored-by: Cursor <cursoragent@cursor.com> * fix(video_edit): keep transform/pre_call outside try so validation errors propagate Move transform_video_edit_request and logging_obj.pre_call outside the try/except that wraps HTTP calls in (async_)video_edit_handler so that ValueError validation errors (e.g. 'source video not complete yet') are not silently wrapped as 500s by _handle_error. The prefetch HTTP call keeps its own try/except so its errors are still mapped through the provider's error handler. Matches the pattern used by video_extension_handler and video_remix_handler. Co-authored-by: Yassin Kortam <yassin@berri.ai> * refactor(vertex_ai): delegate get_video_edit_prefetch_params to status retrieve Co-authored-by: Yassin Kortam <yassin@berri.ai> * Fix varia review * fix(video_edit): route transform errors through _handle_error Wrap transform_video_edit_request and pre_call in the same try/except as the HTTP call in sync and async handlers so validation failures (e.g. source video not complete) return typed LiteLLM exceptions. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |