* Use auth key name if there are no app id in in headers or in extra_data
* use key alias instead of key name
* Fix
* last priority key alias
* Fix
* Add tests
* [Feat] Day-0 support for GPT-5.5 and GPT-5.5 Pro (#26449)
* feat(openai): day-0 support for GPT-5.5 and GPT-5.5 Pro
Add pricing + capability entries for the new GPT-5.5 family launched by
OpenAI on 2026-04-24:
- gpt-5.5 / gpt-5.5-2026-04-23 (chat): $5/$30/$0.50 per 1M
input/output/cached input
- gpt-5.5-pro / gpt-5.5-pro-2026-04-23 (responses-only): $60/$360/$6
per 1M input/output/cached input
Other fees (long-context >272k, flex, batches, priority, cache
discounts) follow the same ratios as GPT-5.4, with context window
retained at 1.05M input / 128K output.
No transformation / classifier code changes are required:
OpenAIGPT5Config.is_model_gpt_5_4_plus_model() already matches 5.5+ via
numeric version parsing, and model registration is driven from the
JSON. The existing responses-API bridge for tools + reasoning_effort
(litellm/main.py:970) already covers gpt-5.5-pro.
Tests:
- GPT5_MODELS regression list now covers gpt-5.5-pro and dated variants
- New test_generic_cost_per_token_gpt55_pro cost-calc test
- Updated test_generic_cost_per_token_gpt55 for long-context fields
* fix(openai): mirror reasoning_effort flags onto gpt-5.5 dated variants
gpt-5.5-2026-04-23 and gpt-5.5-pro-2026-04-23 were missing the
supports_none_reasoning_effort, supports_xhigh_reasoning_effort, and
supports_minimal_reasoning_effort flags that their non-dated
counterparts define. Reasoning-effort routing in OpenAIGPT5Config is
fully capability-driven from these JSON flags — since an absent flag
is treated as False for opt-in levels (xhigh), users pinning to a
dated snapshot would silently lose xhigh support and diverge from the
base alias on logprobs + flexible temperature handling.
Copy the flags onto both dated variants so every dated snapshot
inherits the base model's reasoning-effort capability profile.
Adds a parametrized regression test that asserts
supports_{none,minimal,xhigh}_reasoning_effort parity between each
dated variant and its non-dated counterpart, preventing future drift
when new snapshots are added.
* [Feat] Add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants) (#26361)
* feat(azure): add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants)
Azure variants of OpenAI's GPT-5.5 family. Microsoft has not yet
shipped GPT-5.5 on Azure OpenAI (latest GA on the Foundry models page
is GPT-5.4 as of 2026-04-24), but adding the entries day-0 mirrors the
established precedent for azure/gpt-5.4* (which were in the cost map
before the Azure rollout) so cost tracking and capability flags work
the moment customers deploy.
Schema follows the existing azure/gpt-5.4* shape:
- Same base/long-context pricing as openai/gpt-5.5*: $5/$30 chat,
$60/$360 pro per 1M, with priority tier 2x base
- Azure variants drop the flex/batches keys (Azure has no flex tier)
but keep priority pricing, matching gpt-5.4* precedent
- mode=chat for the thinking model, mode=responses for pro
reasoning_effort capability flags mirror the OpenAI variants exactly
since Azure proxies the same API contract: minimal rejection on both
chat and pro, low/none rejection on pro. Once #26456 (which sets
supports_low_reasoning_effort + minimal=false on openai/gpt-5.5*)
lands, OpenAI and Azure flag profiles align.
Tests pin entry presence + pricing for all four Azure variants and
verify the live-API-derived reasoning_effort flags.
* test: register supports_low_reasoning_effort in cost-map JSON schema
azure/gpt-5.5-pro and azure/gpt-5.5-pro-2026-04-23 added in this branch
carry supports_low_reasoning_effort=false. The strict
'additionalProperties: false' schema in
test_aaamodel_prices_and_context_window_json_is_valid rejected the new
key. Register it alongside the other supports_*_reasoning_effort
entries.
Note: the runtime side of this flag (code that reads it) lands in
#26456. Until that PR merges the flag is inert for both Azure and
OpenAI pro entries, but having the schema accept it lets cost-map
tests pass on either merge order.
* Use sanitize deep copy style to replace deepcopy usage
* Added test checking error is not happening anymore
* Added warning log when json copy failed
* Reduce to one change
* Fix spaces
---------
Co-authored-by: Ido Lavi <ido@noma.security>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: TomAlon <tom@noma.security>
tool_calls on assistant messages were translated to OllamaToolCall format
but never copied into the outgoing OllamaChatCompletionMessage, so Ollama
received {role: assistant, content: ''} with no tool_calls. The model
then had no record of having made a tool call, causing it to re-issue
the identical call on every turn (infinite loop).
Similarly, tool_call_id on role:tool messages was silently dropped.
Ollama uses this field to resolve the tool name from conversation history.
Also add tool_call_id to OllamaChatCompletionMessage TypedDict.
Fixes#26094
* fix(proxy): invoke post-call guardrails on pass-through endpoint responses (#20270)
Wire post_call_success_hook into non-streaming pass-through response path,
gated on explicit guardrail config (opt-in only, no backwards-compat break).
- Call post_call_success_hook after reading non-streaming response body
- Build enriched hook_data with guardrails metadata and litellm_logging_obj
at call site (avoids mutation of _parsed_body which is shared by logging)
- Handle ModifyResponseException with provider-agnostic error envelope,
post_call_failure_hook, and defensive try/except
- Strip stale content-length when guardrail modifies response body
- Move ModifyResponseException to litellm.exceptions to break cyclic import;
re-export from custom_guardrail for backwards compat
- Add call_type fallback in UnifiedLLMGuardrails for pass-through endpoints
using CallTypes.pass_through.value enum
* test: add unit tests for pass-through post-call guardrails
5 tests covering the post-call guardrail invocation on pass-through endpoints:
- post_call_success_hook fires when guardrails configured
- post_call_success_hook skipped when no guardrails (backwards compat)
- ModifyResponseException returns 200 with provider-agnostic error
- UnifiedLLMGuardrails resolves call_type from logging_obj for pass-through
- ModifyResponseException re-export from custom_guardrail stays in sync
Bedrock enforces non-increasing TTL ordering across cache_control blocks
(tools → system → messages). The tool cache_control TTL was being
unconditionally dropped to the default 5m, while system blocks preserved
the user-specified TTL for Claude 4.5+ models. This mismatch caused
"a ttl='1h' block must not come after a ttl='5m' block" errors when
users set ttl='1h' on both tools and system.
Converse path: add_cache_point_tool_block() now accepts a model param
and preserves TTL for Claude 4.5+, matching _get_cache_point_block().
Invoke path: _remove_ttl_from_cache_control() now also processes tools
(was only processing system and messages).
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(redis): cache GCP IAM token to prevent async event loop blocking
## Problem
GCPIAMCredentialProvider.get_credentials() calls _generate_gcp_iam_access_token
on every Redis connection establishment. This function performs synchronous HTTP
and gRPC calls (google-auth + google-cloud-iam) which block Python's asyncio
event loop while running.
Under concurrent load (e.g. connection pool warm-up, parallel health checks),
multiple connections are established simultaneously, each triggering an
independent blocking IAM token refresh. These refreshes serialise behind each
other inside the single-threaded event loop, causing individual Redis spans to
take 20-25 seconds instead of milliseconds.
Observed in production via Datadog APM: a single INCRBYFLOAT Redis span took
25.6 seconds (90% of a 28.4s trace), with GCP metadata + GenerateAccessToken
gRPC calls visible inside the span. This cascaded into aiohttp SocketTimeoutError
on upstream LLM API calls — not because the upstream was slow, but because the
event loop was frozen and the 30-second sock_read timer fired on a connection
that was never given CPU time.
## Fix
Add a module-level token cache (dict keyed by service account, value is
(token, expiry_monotonic)). _get_cached_gcp_iam_token() returns the cached
token on cache hit (no I/O), and refreshes only when expired using
double-checked locking so only one thread performs the network round-trip.
GCP IAM tokens are valid for 1 hour; the cache TTL is set to 55 minutes
(_GCP_IAM_TOKEN_TTL_SECONDS = 3300) to refresh safely before expiry.
The cache is shared across all GCPIAMCredentialProvider instances for the same
service account, so N concurrent Redis connections on the same pod share a
single token and avoid N concurrent blocking refreshes.
get_credentials_async() already used asyncio.to_thread (non-blocking), and is
updated to call _get_cached_gcp_iam_token so it also benefits from caching.
## Tests
- Updated existing test that expected a fresh token on every call to reflect
the new caching behaviour.
- Added tests for: cache hit (no redundant I/O), cache expiry and refresh,
and cache sharing across multiple provider instances.
- Added autouse fixture to clear the module-level cache between tests.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(redis): remove unused Optional import from _redis_credential_provider.py
* refactor(redis): improve documentation for GCPIAMCredentialProvider class
Updated the docstring for the GCPIAMCredentialProvider class to clarify its purpose and the caching mechanism for GCP IAM tokens. The changes enhance readability and maintainability by providing a more concise explanation of the token caching strategy and its benefits for Redis authentication.
* refactor(redis): improve documentation for GCPIAMCredentialProvider class
Updated the docstring for the GCPIAMCredentialProvider class to clarify its purpose and the caching mechanism for GCP IAM tokens. The changes enhance readability and maintainability by providing a more concise explanation of the token caching strategy and its benefits for Redis authentication.
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(memory): jsonify metadata before Prisma writes on /v1/memory
The POST/PUT memory endpoints handed bare dicts (and bare `None`) to
prisma-client-python for the `Json?` `metadata` column, which the client
rejects with `MissingRequiredValueError` / `DataError: metadata should
be of any of the following types: NullableJsonNullValueInput, Json`.
Both the create and upsert paths now route writes through the existing
`jsonify_object` helper used elsewhere in the proxy for `Json?` columns
(e.g. `LiteLLM_VerificationToken.budget_limits`), and omit metadata
when None so the column defaults to SQL NULL via the schema.
Explicit `metadata: null` on PUT is now a no-op for the column to match
how the rest of the proxy handles nullable JSON fields (no
`JsonNull`/`DbNull` sentinel exists in prisma-client-python — see
RobertCraigie/prisma-client-py#714). A payload with only `metadata: null`
returns 400 instead of a misleading 200.
Made-with: Cursor
* fix(memory): JSON-encode non-dict metadata before Prisma writes
`jsonify_object` only stringifies dict values, so list-shaped metadata
still hit Prisma as raw Python objects and triggered the same
DataError this PR is meant to fix. `metadata` is typed `Optional[Any]`
so list payloads are valid input. Replace `jsonify_object` with a
local `_serialize_metadata_for_prisma` helper that always `json.dumps`
non-string values, applied at all three write sites
(POST create, PUT update, PUT-create). Adds regression tests for
list metadata on each path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): always json.dumps metadata, not just non-strings
The str-passthrough in `_serialize_metadata_for_prisma` left plain
Python strings (e.g. `metadata: "hello"`) unencoded — Postgres `jsonb`
rejects bare-word strings as invalid JSON, reproducing the same
DataError this PR is meant to fix. Always `json.dumps` regardless of
input type so all `Optional[Any]` shapes (dict, list, scalar, str)
become valid JSON. Adds a regression test for plain-string metadata.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): encode explicit metadata:null as JSON null to clear field
prisma-client-python has no JsonNull/DbNull sentinel for writing a
true SQL NULL on `Json?` columns (RobertCraigie/prisma-client-py#714),
so an earlier iteration of this PR treated `PUT {"metadata": null}`
as a no-op. That doesn't match the natural caller expectation that
explicit-null clears the field.
Encode it as the JSON literal `null` instead — stored as Postgres
`jsonb 'null'`, which prisma deserializes back to Python `None` on
read. Subsequent reads return `metadata: null`, so the field is
effectively cleared from the caller's perspective. Strict SQL NULL
remains unreachable via the typed client and would require raw SQL.
Also clean up stale `jsonify_object` references in test mock comments
(replaced by `_serialize_metadata_for_prisma`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(memory ui): use shared DeleteResourceModal for memory deletion
Swap the imperative `Modal.confirm` in MemoryView for the shared
`DeleteResourceModal`, so memory deletion matches the rest of the
dashboard: type-to-confirm guard on the key, in-flight loading state
on the OK button, cancel disabled while the request is pending, and
the modal stays open on error so the user can retry.
Made-with: Cursor
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): jsonify metadata before Prisma writes on /v1/memory
The POST/PUT memory endpoints handed bare dicts (and bare `None`) to
prisma-client-python for the `Json?` `metadata` column, which the client
rejects with `MissingRequiredValueError` / `DataError: metadata should
be of any of the following types: NullableJsonNullValueInput, Json`.
Both the create and upsert paths now route writes through the existing
`jsonify_object` helper used elsewhere in the proxy for `Json?` columns
(e.g. `LiteLLM_VerificationToken.budget_limits`), and omit metadata
when None so the column defaults to SQL NULL via the schema.
Explicit `metadata: null` on PUT is now a no-op for the column to match
how the rest of the proxy handles nullable JSON fields (no
`JsonNull`/`DbNull` sentinel exists in prisma-client-python — see
RobertCraigie/prisma-client-py#714). A payload with only `metadata: null`
returns 400 instead of a misleading 200.
Made-with: Cursor
* fix(memory): JSON-encode non-dict metadata before Prisma writes
`jsonify_object` only stringifies dict values, so list-shaped metadata
still hit Prisma as raw Python objects and triggered the same
DataError this PR is meant to fix. `metadata` is typed `Optional[Any]`
so list payloads are valid input. Replace `jsonify_object` with a
local `_serialize_metadata_for_prisma` helper that always `json.dumps`
non-string values, applied at all three write sites
(POST create, PUT update, PUT-create). Adds regression tests for
list metadata on each path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): always json.dumps metadata, not just non-strings
The str-passthrough in `_serialize_metadata_for_prisma` left plain
Python strings (e.g. `metadata: "hello"`) unencoded — Postgres `jsonb`
rejects bare-word strings as invalid JSON, reproducing the same
DataError this PR is meant to fix. Always `json.dumps` regardless of
input type so all `Optional[Any]` shapes (dict, list, scalar, str)
become valid JSON. Adds a regression test for plain-string metadata.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Code review noted the previous test reimplemented the proxy's
try/except/finally around post_call_success_hook, so it would not catch a
regression that re-introduced the duplicate-log bug in the production code
path. Extract the gating logic into
`ProxyBaseLLMRequestProcessing._flush_deferred_async_logging` so tests
exercise the production helper directly.
The proxy finally block becomes a single call to the helper. Tests now
invoke the helper itself and additionally assert (via inspect.getsource)
that base_process_llm_request continues to delegate to the helper rather
than inlining the gate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a non-streaming request had any post-call guardrail registered, the
proxy deferred the async-success logging closure until after
post_call_success_hook ran. The finally block fired that closure even when
the hook raised — the propagating HTTPException then routed through
post_call_failure_hook → _handle_logging_proxy_only_error, which writes its
own failure spend log via async_failure_handler. The result was two spend
log rows per blocked request: one Success exposing the blocked LLM response
and one Failure. Reproduces with both pre and post bedrock guardrails
configured for a team when the post-call OUTPUT scan blocks the response.
Gate the deferred closure on _exception_raised so the failure path remains
the single source of truth for blocked requests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(guardrails): apply team-level guardrails alongside global policy guardrails
Two bugs prevented team-direct guardrails from being automatically applied
when using a team-scoped API key:
1. Auth caching: `valid_token.team_metadata` was never refreshed from the
freshly-fetched team object at the "Check 6" step in
`_user_api_key_auth_builder`. Guardrails added to a team after the key
was first cached were therefore invisible to `move_guardrails_to_metadata`.
Fix: propagate `_team_obj.metadata` → `valid_token.team_metadata` after
every "Check 6" team fetch (user_api_key_auth.py).
2. Guardrail execution: `get_guardrail_from_metadata` checked
`data["litellm_metadata"]` before `data["metadata"]`. When a request
carried a non-empty `litellm_metadata` without a "guardrails" key, the
merged guardrail list written to `data["metadata"]` by
`move_guardrails_to_metadata` was shadowed and the guardrail received an
empty requested-guardrails list (custom_guardrail.py).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix merge conflict
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(bedrock guardrail): dedupe post-call log when only post_call is configured
When a Bedrock guardrail runs with only post_call configured, the post-call
trace section showed the same guardrail twice (one entry per parallel
INPUT/OUTPUT API call). Add skip_logging param to make_bedrock_api_request
and pass it for the INPUT scan so the OUTPUT scan stands as the single
canonical post_call log entry. INPUT exceptions still propagate, so the
input-side blocking behavior is preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(bedrock guardrail): post_call only scans OUTPUT, not INPUT
post_call is the response-validation hook by definition — input scanning
belongs to pre_call / during_call. The previous code ran an extra INPUT
scan in post_call when no pre/during hook was configured, which produced
a duplicate "post-call" entry in the trace and was semantically wrong
for a "post-call" event.
Drops the should_validate_input branch and parallel asyncio.gather in both
async_post_call_success_hook and async_post_call_streaming_iterator_hook
in favor of a single OUTPUT scan. Reverts the now-unneeded skip_logging
parameter on make_bedrock_api_request introduced in the previous commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Fall back to email match when looking up the caller in
members_with_roles — email-onboarded members may have user_id=None on the
stored entry, which caused a false 404 for valid members. (P1)
- Replace 3 raw Prisma queries with get_team_object / get_team_membership /
get_user_object so the endpoint reuses the cache + retry layer the rest of
the proxy uses. (P2)
- Allow internal_user role to reach /team/{team_id}/members/me by adding the
route to LiteLLMRoutes.self_managed_routes (the handler already enforces
member-of-team access).
- Return null from the UI fetch on 404 instead of throwing, so a proxy admin
who isn't a team member sees the existing empty state rather than an error
string in the always-visible tab. (P2)
- Move the fetch out of networking.tsx into a colocated React Query hook
(useMyTeamMember) next to MyUserTab; TeamInfo now passes only teamId.
- Tooltip + empty-state copy on Model Scope: drop "(all team models)"
parenthetical and the redundant tooltip line.
- Tests: build real LiteLLM_TeamMembership / LiteLLM_BudgetTableFull
fixtures (with created_at) so the Pydantic Union resolves to the Full
variant; add an assertion that budget_reset_at survives end-to-end; add a
test for the email-only member match path.
* feat(azure): add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants)
Azure variants of OpenAI's GPT-5.5 family. Microsoft has not yet
shipped GPT-5.5 on Azure OpenAI (latest GA on the Foundry models page
is GPT-5.4 as of 2026-04-24), but adding the entries day-0 mirrors the
established precedent for azure/gpt-5.4* (which were in the cost map
before the Azure rollout) so cost tracking and capability flags work
the moment customers deploy.
Schema follows the existing azure/gpt-5.4* shape:
- Same base/long-context pricing as openai/gpt-5.5*: $5/$30 chat,
$60/$360 pro per 1M, with priority tier 2x base
- Azure variants drop the flex/batches keys (Azure has no flex tier)
but keep priority pricing, matching gpt-5.4* precedent
- mode=chat for the thinking model, mode=responses for pro
reasoning_effort capability flags mirror the OpenAI variants exactly
since Azure proxies the same API contract: minimal rejection on both
chat and pro, low/none rejection on pro. Once #26456 (which sets
supports_low_reasoning_effort + minimal=false on openai/gpt-5.5*)
lands, OpenAI and Azure flag profiles align.
Tests pin entry presence + pricing for all four Azure variants and
verify the live-API-derived reasoning_effort flags.
* test: register supports_low_reasoning_effort in cost-map JSON schema
azure/gpt-5.5-pro and azure/gpt-5.5-pro-2026-04-23 added in this branch
carry supports_low_reasoning_effort=false. The strict
'additionalProperties: false' schema in
test_aaamodel_prices_and_context_window_json_is_valid rejected the new
key. Register it alongside the other supports_*_reasoning_effort
entries.
Note: the runtime side of this flag (code that reads it) lands in
#26456. Until that PR merges the flag is inert for both Azure and
OpenAI pro entries, but having the schema accept it lets cost-map
tests pass on either merge order.
Adds a new "My User" tab on the team detail page (between Overview and
Virtual Keys) so non-admin team members can see their own spend, budget,
budget reset date, rate limits, model scope, and team role.
Backend
- New `GET /team/{team_id}/members/me` endpoint that resolves the caller
from the API key and returns only their own LiteLLM_TeamMembership row
plus minimal team context (alias, role, email). Returns 404 if the
caller is not a member of the team. Avoids exposing other members'
data, which would happen if we filtered `/team/info` client-side.
- New `TeamMemberInfoResponse` Pydantic model.
Frontend
- New `MyUserTab` component (antd) — read-only summary cards.
- New `teamMemberMeCall` helper in networking.tsx.
- Tab is visible to all team members (including non-admins).
Mirrors the existing vertex_credentials handling for the newer
vertex_ai_credentials field, and extends the dynamic masker's sensitive
pattern set to recognize the plural form so other plural-named credential
fields are also covered.
Bring `/server/oauth/{server_id}/authorize`, `/token`, and `/register`
in line with `fetch_mcp_server`: the helper that resolves the server now
also applies the per-caller access policy. Admin-view callers are
unrestricted; non-admins must have the server in their allowed-servers
set; servers resolved from the admin-only `/server/oauth/session`
temporary cache reject non-admins.
dimensions=1 was silently dropped before #24415 wired the param through
to Vertex. Now that it's forwarded, Vertex rejects it (valid values:
128, 256, 512, 1408). Switch the test to a valid dimension.
The asyncio.gather in `_run_centralized_common_checks` ran with
`return_exceptions=False` and a single bare `except HTTPException`
arm, so an HTTPException from any one fetch (the realistic case is a
404 from `get_team_object` when a token references a deleted team)
zeroed out the user, end-user, project, and global-spend contexts in
addition to falling back the team object. That silently skipped the
user budget, end-user budget, and project enforcement passes inside
`common_checks` for the unrelated contexts that had actually fetched
fine.
Switch to `return_exceptions=True` and apply per-fetch fallback
(matches the pre-refactor per-fetch try/except pattern in the builder):
- ProxyException / BudgetExceededError still propagate as authz failures.
- HTTPException on the team fetch reconstructs from the token; on the
other fetches it nulls only that one context.
- Successful fetches always reach `common_checks` intact.
Adds two unit tests covering the team-404 and user-404 cases to lock
the per-fetch isolation in. Drops the inaccurate `PROXY_ADMIN tokens
short-circuit` claim from the docstring — admin tokens still flow
through `common_checks`; admin status is only honored where the
underlying check exempts it.
- test_prepare_key_update_data: replace bare MagicMock with
MagicMock(spec=LiteLLM_VerificationToken) and explicitly set
existing_key_row.metadata = {}, so reserved-field reads return real
values instead of MagicMock-returning-MagicMock. Fixes a regression
surfaced by the new reserved-metadata preservation logic.
- test_key_management_endpoints.py: black-format-only changes from
recent edits.
The previous version appended NULLS LAST to every ORDER BY, which would
silently change DESC semantics for any nullable sort column added to the
whitelist later. Today the existing sort columns (spend, total_tokens,
startTime, endTime, request_duration_ms, model) are all non-null in the
result set, so the clause is a no-op for them — but the broader form is
misleading.
Apply NULLS LAST only when sorting by ttft_ms (the only column whose
computed expression actually produces NULLs). Update the model test to
assert the clause is absent for non-nullable columns.
The routes in `global_spend_tracking_routes` (e.g. /global/spend/report,
/global/spend/teams, /global/spend/keys) return spend aggregated across
every team, customer, and api_key in the proxy. They were included in
`internal_user_routes` and `internal_user_view_only_routes`, so non-admin
roles could read proxy-wide spend.
Drop them from both non-admin route lists. PROXY_ADMIN and
PROXY_ADMIN_VIEW_ONLY access is preserved through their existing branches
in route_checks.py, and the `get_spend_routes` permission opt-in
continues to grant access for keys that need it.
Updates two pre-existing test parametrizations whose expected results
flip from True to False, and adds parametrized coverage over every
route in `global_spend_tracking_routes` for: PROXY_ADMIN_VIEW_ONLY
allowed, INTERNAL_USER blocked, INTERNAL_USER_VIEW_ONLY blocked,
INTERNAL_USER + get_spend_routes permission allowed.
Extend the /spend/logs/ui sort_by whitelist to accept "model" and
"ttft_ms", and wrap the Model and TTFT (s) column headers with the
existing SortableHeader component so users can sort by either.
TTFT has no stored column, so it is computed inline as
(completionStartTime - startTime) milliseconds. Non-streaming rows
(completionStartTime null or equal to endTime) yield NULL and the
ORDER BY uses NULLS LAST so they always sort to the bottom regardless
of direction, matching the existing "-" display in the UI.
Adds parametrized backend tests for sort_by=model and a dedicated test
covering streaming + non-streaming TTFT ordering in both directions.
Drops the team_create and model_add disable flags that were added in the
prior feat commit. Only disable_key_generate_for_org_admin remains, with
the helper renamed check_org_admin_can_generate_keys for clarity. Team
and model creation can be added back the same way later if needed.
* feat(proxy): add /v1/memory CRUD endpoints with user/team scoping
New LiteLLM_MemoryTable stores user/team-scoped key/value entries with
optional JSON metadata. Value is a String (LLM-readable text) and metadata
is an optional Json? envelope, matching the Letta + mem0 hybrid model so
future structured fields can be added without a schema migration.
Endpoints:
POST /v1/memory - create
GET /v1/memory - list (caller-scoped; admins see all)
GET /v1/memory/{key} - fetch one
PUT /v1/memory/{key} - upsert
DELETE /v1/memory/{key} - delete
Non-admin callers cannot set a user_id/team_id other than their own.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(proxy/memory): omit metadata field when None on create
Prisma's Python client rejects `metadata=None` on a `Json?` field with
"A value is required but not set" — the field must be omitted from the
`data` dict entirely to store SQL NULL. Build the create payload
conditionally in both `create_memory` and the PUT-create branch of
`upsert_memory`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ui): add Memory page to view/manage /v1/memory entries
Adds a new "Memory" sidebar item under Tools so users can see what their
agents have stored. Lists all memories visible to the caller (scoped by
the backend), with a key-search filter, preview column, scope tags, and
view/edit/delete actions. Create modal accepts optional JSON metadata.
- networking.tsx: fetchMemoryList / createMemory / updateMemory / deleteMemory
wired to the /v1/memory CRUD endpoints.
- MemoryView + MemoryEditModal: new antd-based components (per CLAUDE.md:
use antd for new UI, not tremor).
- page.tsx + leftnav.tsx: wire the "memory" route + sidebar entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(memory): add key_prefix filter + promote Memory to AI GATEWAY nav
Backend:
- GET /v1/memory now accepts `key_prefix` for Redis-style namespace
scans (e.g. `?key_prefix=user:`). When both `key` and `key_prefix`
are passed, `key_prefix` wins.
- Prefix filter sits under the visibility filter in the Prisma where
clause, so it can never leak rows across user/team scopes.
- New tests: prefix match, and cross-scope isolation (another user's
`user:*` rows must not appear in the caller's results).
UI:
- Memory moved from a Tools submenu to a top-level AI GATEWAY item
(alongside Agents, MCP Servers, Skills) — it's an API primitive,
not a tool-management surface.
- Search box now drives prefix search, matching the Redis mental
model ("type the namespace, see everything under it").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): enforce unique key per scope by using NULLS NOT DISTINCT
The unique constraint `(key, user_id, team_id)` on LiteLLM_MemoryTable
silently allowed duplicates when user_id or team_id was NULL, because
Postgres treats every NULL as distinct by default (ANSI semantics). A
caller with no team_id could POST the same key three times and get
three rows.
Migration:
1. Dedupe existing rows, keeping the most recent per (key, user_id,
team_id), using `IS NOT DISTINCT FROM` so NULL == NULL.
2. Drop the old unique index.
3. Recreate it with `NULLS NOT DISTINCT` (Postgres 15+).
No code change: POST already returns 409 on unique-violation error
messages — it just wasn't firing before because the constraint didn't
catch the NULL-team case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): make key globally unique, 409 on any duplicate
Switches from the compound unique `(key, user_id, team_id)` to a simple
`key @unique`. The compound form silently allowed duplicates when
user_id or team_id was NULL (Postgres treats each NULL as distinct), so
callers could POST the same key repeatedly. Globally-unique key means
one row per key, period — any duplicate create → 409.
- schema.prisma (×3): `key String @unique`, drop `@@unique(...)`.
- initial add_memory_table migration: unique index on (key) only.
- Remove the now-unused follow-up NULLS NOT DISTINCT migration.
- Endpoint error message simplified ("already exists" — no "for this scope").
- Test fake's create() now enforces global key uniqueness.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ui/memory): full-width layout + user/teams-style columns
- Add `w-full` to the MemoryView outer div so the page fills the
flex-flex-1 container (was collapsing to intrinsic width).
- Replace the combined "Scope" column with separate User ID / Team ID
columns, matching the layout of the Users / Teams pages: ID, Name,
Preview, User ID, Team ID, Updated, Actions.
- IDs render with a truncated mono label + copy-to-clipboard button,
same pattern as view_users.
- Detail drawer now shows Memory ID / User ID / Team ID as separate
fields instead of stacked color tags.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ui/memory): use clean MCP-style ID pill, drop copy icons
The ID / User ID / Team ID columns showed a mono text blob with a
copy-to-clipboard icon next to each value — too busy compared to the
MCP Servers page. Swap the renderer for MCP's pill style:
- Truncated mono ID inside a blue Tailwind pill
(`font-mono text-blue-600 bg-blue-50 ... rounded-md border`).
- No copy icon. Full ID surfaces via tooltip.
- ID column is a button that opens the detail drawer on click;
user/team ID pills are static (not clickable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): address greptile review feedback
Addresses 5 greptile findings (3/5 → higher confidence target):
1. Identity-less orphan rows (P1): non-admin callers with no user_id AND
no team_id could create rows that the visibility filter would never
match again. Now rejected up front with 400 — caller must authenticate
with a scoped key or act as PROXY_ADMIN.
2. Upsert race returning 500 (P1): PUT's check-then-create isn't atomic;
a concurrent writer could slip a row in between the 404-check and the
create call. Now catch unique-violation on create, re-read, and fall
through to update — PUT stays idempotent. If the conflicting row
belongs to a different scope, surface a 409 instead of 500.
3. PUT-create scope inconsistency (P2): PUT's create branch always used
the caller's own user_id/team_id, so admins couldn't bootstrap rows
scoped elsewhere via PUT (only POST). Now PUT-create calls the shared
`_resolve_scope()` helper, matching POST semantics.
4. Stale schema comment (P2): schema said "Keyed by (key, user_id,
team_id)" but `key` is globally unique. Updated all three schema
copies to reflect the actual design.
5. UI silently truncated at 200 (P2): MemoryView fetched pageSize=200
with no load-more. Swapped to real server-side pagination driven by
`data.total`; page size is now 50 and the pager is a real AntD
control.
Also extracts a shared `_resolve_scope()` helper and `_is_unique_violation()`
from create_memory so POST and PUT don't drift on the scope/error logic.
Tests: +3 new (identity-less 400, PUT admin bootstrap, PUT race →
update), 18/18 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): typed Prisma error + explicit-null metadata on PUT
Two more greptile threads from the last review:
- Unique-violation detection was string-matching "Unique"/"UniqueViolation"
in the exception message, fragile across Prisma/driver versions. Now
check the typed error `code == "P2002"` first, with string fallback.
- PUT could not distinguish "metadata omitted" from "metadata: null" —
both parsed as `None`, so callers had no way to clear stored metadata.
Switch to Pydantic v2's `model_fields_set` to tell which fields the
caller actually sent; explicit null now clears the column.
New tests:
- explicit null clears metadata
- omitted metadata preserves existing value
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ui/memory): send explicit null when user clears metadata
Addresses the remaining P1 from the last greptile review:
When the edit modal's metadata textarea was cleared and saved,
`metadataParsed` stayed `undefined`, `JSON.stringify` dropped the key
entirely, and the backend's `model_fields_set` guard therefore left
the stored metadata untouched — UI showed success but nothing changed.
Now: empty textarea on edit → send explicit `null` so the backend
sees `metadata` in `model_fields_set` and clears the column.
Empty textarea on create still maps to `undefined` (field omitted)
to avoid Prisma's `Json? = None` quirk on insert.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ui/memory): preserve slashes in key path encoding
The backend route `/v1/memory/{key:path}` supports keys with slashes,
but `encodeURIComponent` encoded `/` as `%2F`. Some proxies (nginx
default, CloudFlare, AWS ALB) reject or re-decode `%2F` mid-flight,
so UI update/delete calls on slash-containing keys could fail or
silently misroute.
New helper `encodeMemoryKeyForPath` splits by `/`, URL-encodes each
segment, then rejoins with literal `/`. Every other unsafe char
(spaces, `?`, `#`, `%`) stays encoded per-segment; slashes stay as
path delimiters, matching what the `:path` converter expects.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ui/memory): drop misleading client-side column sorters
With server-side pagination, client sorters on `key` and `updated_at`
only reorder the current page while pretending to sort the full
dataset — users would see "sorted by name" but only the visible 50
rows would actually be sorted.
Remove the sorters. The backend already returns rows in
`updated_at DESC` order (sensible default for a memory view), and
users can narrow the result with the key-prefix filter.
Greptile also flagged missing `@@map` on the new model as a
"consistency" issue, but only 1 of 59 tables in this repo uses
`@@map` — the dominant pattern is to rely on Prisma's default
(model name == table name). Skipping that finding as a
false-positive on convention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): compose visibility + key filters via explicit AND
Greptile P1 (filter-fragility): `where.update(vis)` was semantically
correct today, but dict-merging by key meant any future visibility
filter that grew a new top-level "OR" would silently clobber the
existing key filter.
Compose explicitly instead:
where = {"AND": [key_filter, vis]}
Applied to both `list_memory` and `_find_memory_for_caller`. When
either side is empty (admin has no visibility filter; list has no
key filter), skip the wrapper and use the non-empty side directly
to keep the generated SQL clean.
Test fake's `_matches` now understands top-level `AND` too.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(ui/memory): wrap write helpers with react-query useMutation
Previously the Memory view read via `useQuery` but called the raw
create/update/delete fetch helpers directly in handlers, tracking
loading state with a local `submitting` flag and invalidating state
via `refetch()`. That mixes two concerns:
- it skips react-query's mutation state (isPending / isError / isSuccess)
- `refetch()` only retouches the currently-mounted query instance, not
other cached pages, so navigating back to an older page could show
stale rows
Switch the three write paths to `useMutation`:
- `createMutation`, `updateMutation`, `deleteMutation` — each owns
the mutation fn, success toast, and error toast.
- Success handlers invalidate the whole `["memoryList", ...]` prefix
via `queryClient.invalidateQueries`, so every cached page refetches
(pagination + filter-aware).
- Refresh button now invalidates instead of `refetch()`, keeping all
behavior consistent.
- handleSave/handleDelete become thin adapters that call `.mutateAsync`;
their errors are swallowed locally since the mutation's onError has
already surfaced the toast.
Also tightened the edit modal's key-field tooltip to reflect the
actual global-unique semantics (was "Unique per user/team scope").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): close cross-user write gap + sanitize 500 errors (Veria)
Addresses two Veria findings:
**High — cross-user memory tampering via team membership.** The
visibility filter uses an OR (`user_id == caller OR team_id == caller`)
so team members can SEE each other's team-scoped rows. That's
intentional for list/get. But because PUT/DELETE used the same filter
to find the target row, any team member could overwrite or delete a
teammate's *personal* row whenever both `user_id` and `team_id` were
stamped on it — broader visibility was being silently treated as
broader authority.
New `_assert_write_access(row, caller)` enforces ownership for
mutations. Non-admin rules:
- The row's `user_id` must match the caller (personal ownership), OR
- The row has no `user_id` and its `team_id` matches the caller's
team (a "pure team row" intended for shared writes).
Admins bypass the check. The same gate runs in PUT (both regular
and post-race-recovery branches) and DELETE.
**Medium — DB internals leaked through 500 detail.** Every `except`
block was raising `HTTPException(500, detail=str(e))`, which surfaces
Prisma error strings (table/column names, host:port, error class
names) to API callers. New `_internal_error()` helper logs the real
exception server-side and returns a generic, caller-safe `detail`.
Applied to create, list, upsert (general fallthrough), and delete.
Also tightened the race-recovery 409 message to drop the "in a
different scope" wording — the caller never needs to know whose
scope it lives in.
Tests (+5):
- teammate cannot overwrite personal row → 403
- teammate cannot delete personal row → 403
- teammate CAN modify pure team row (no user_id stamped) → 200
- admin bypasses write-auth → 200
- 500 response never echoes Prisma internals (table/host/class names)
25/25 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): require team admin to modify pure team rows
Tightens the write-authorization rule for "pure team rows" (rows with
no user_id stamped, only team_id) to match the pattern used by
team-management endpoints (`_is_user_team_admin` + `_is_user_org_admin_for_team`):
- Plain team members can READ team rows via the OR visibility filter
(intentional, unchanged).
- Only PROXY_ADMIN, team admins of the row's team_id, or org admins
for the team's organization may MODIFY them. Plain members get 403.
`_assert_write_access` is now async and takes the prisma_client so it
can fetch the team and run the existing `_is_user_team_admin` /
`_is_user_org_admin_for_team` helpers from
`litellm.proxy.management_endpoints.common_utils`. The org-admin path
is best-effort: it calls `get_user_object`, which depends on the
proxy_server module being initialized, so any exception there is
treated as "not an org admin" rather than crashing the request.
Tests:
- team admin can modify pure team row → 200
- plain team member cannot modify pure team row → 403
- plain team member cannot delete pure team row → 403
Updates the test fake to add a tiny `litellm_teamtable.find_unique`
implementation and a `_make_team(team_id, admin_user_ids=[...])`
helper.
27/27 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: mypy + UI page-metadata sync for memory page
Two CI failures:
1. mypy: `_find_memory_for_caller` had `key_filter` inferred as
`dict[str, str]` (literal type) and the conditional `{"AND": [key_filter, vis]}`
returned `dict[str, list[...]]`, so the join site failed
`dict-item` typing. Annotate both intermediates as `dict` so mypy
widens the value type.
2. UI test (`page_utils.test.ts > should have descriptions for all
pages`): every leftnav entry must have a description in
`page_metadata.ts`, and `memory` was missing. Added a one-line
description, matching the style of neighboring entries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [Feat] Day-0 support for GPT-5.5 and GPT-5.5 Pro (#26449)
* feat(openai): day-0 support for GPT-5.5 and GPT-5.5 Pro
Add pricing + capability entries for the new GPT-5.5 family launched by
OpenAI on 2026-04-24:
- gpt-5.5 / gpt-5.5-2026-04-23 (chat): $5/$30/$0.50 per 1M
input/output/cached input
- gpt-5.5-pro / gpt-5.5-pro-2026-04-23 (responses-only): $60/$360/$6
per 1M input/output/cached input
Other fees (long-context >272k, flex, batches, priority, cache
discounts) follow the same ratios as GPT-5.4, with context window
retained at 1.05M input / 128K output.
No transformation / classifier code changes are required:
OpenAIGPT5Config.is_model_gpt_5_4_plus_model() already matches 5.5+ via
numeric version parsing, and model registration is driven from the
JSON. The existing responses-API bridge for tools + reasoning_effort
(litellm/main.py:970) already covers gpt-5.5-pro.
Tests:
- GPT5_MODELS regression list now covers gpt-5.5-pro and dated variants
- New test_generic_cost_per_token_gpt55_pro cost-calc test
- Updated test_generic_cost_per_token_gpt55 for long-context fields
* fix(openai): mirror reasoning_effort flags onto gpt-5.5 dated variants
gpt-5.5-2026-04-23 and gpt-5.5-pro-2026-04-23 were missing the
supports_none_reasoning_effort, supports_xhigh_reasoning_effort, and
supports_minimal_reasoning_effort flags that their non-dated
counterparts define. Reasoning-effort routing in OpenAIGPT5Config is
fully capability-driven from these JSON flags — since an absent flag
is treated as False for opt-in levels (xhigh), users pinning to a
dated snapshot would silently lose xhigh support and diverge from the
base alias on logprobs + flexible temperature handling.
Copy the flags onto both dated variants so every dated snapshot
inherits the base model's reasoning-effort capability profile.
Adds a parametrized regression test that asserts
supports_{none,minimal,xhigh}_reasoning_effort parity between each
dated variant and its non-dated counterpart, preventing future drift
when new snapshots are added.
* fix(schema): close LiteLLM_MemoryTable model brace dropped during merge
The rebase against `litellm_internal_staging` (which added
`LiteLLM_AdaptiveRouterState` / `LiteLLM_AdaptiveRouterSession`) left
the closing brace of `LiteLLM_MemoryTable` missing in all three
schema copies — the next model declaration ended up parsed as a field
of the memory table, surfacing as the CI prisma error:
error: This line is not a valid field or attribute definition.
--> schema.prisma:1250
|
1249 | // Per-(router, request_type, model) Beta posterior for the adaptive router.
1250 | model LiteLLM_AdaptiveRouterState {
Add the missing `}` (and the standard blank line) after the memory
table's `@@index([team_id])` in `schema.prisma`,
`litellm/proxy/schema.prisma`, and
`litellm-proxy-extras/litellm_proxy_extras/schema.prisma`.
`prisma generate --schema litellm/proxy/schema.prisma` now runs clean;
27/27 memory unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): infer team from DB when JWT has no team and user has one team
- When team_id is unset after JWT auth but the user row has exactly one
team, set team_id, team_object, and team_membership from DB.
- Skip when zero or multiple teams (ambiguous).
- Add parametrized unit tests in test_handle_jwt.py.
Made-with: Cursor
* fix(proxy): JWT single-team DB fallback: catch errors, tests match get_team_object
- Wrap get_team_object + get_team_membership in one try/except; log and skip on failure (stale/missing team id no longer fails auth).
- Parametrize tests: HTTP 404/500, membership error; use side_effect not return_value=None for missing team row.
Made-with: Cursor
* refactor(jwt): extract single-team fallback into _resolve_single_team_fallback helper
Made-with: Cursor
