* feat(ui): search teams by team ID alongside name
The Teams page search box only matched team_alias, so pasting a team UUID
returned zero results. Detect when the input is a full UUID and route it
to the team_id filter instead; otherwise keep the existing alias substring
search. Placeholder now reads "Search teams by name or ID...".
Resolves LIT-2648
* refactor: search teams via backend OR clause, drop client-side UUID detection
Adds a `search` query param to /v2/team/list that ORs across team_id
(exact) and team_alias (case-insensitive contains), so the search box
sends one param regardless of input format. Removes the isLikelyTeamId
helper and the client-side branching it fed.
* fix: invalidate cached tag object on tag budget reset (#27481) (#27572)
Squash-merged by litellm-agent from oss-agent-shin's PR.
* chore(mcp): tighten stdio server registration paths (#27570)
Squash-merged by litellm-agent from stuxf's PR.
* fix(proxy): clear MCP OpenAPI mappings on server eviction; widen budget cache invalidation
Evict OpenAPI tools from global_mcp_tool_registry and strip tool_name_to_mcp_server_name_mapping entries when a server leaves the runtime registry (remove_server and approval-status eviction). Invalidate user_api_key_cache for keys, orgs, and team members on budget-tier spend resets alongside tags.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): align update_server eviction with remove_server name fallback
Document budget-reset test assertion flip (cross-pod cache staleness).
Greptile: eviction now pops by server_id then server_name like remove_server;
test docstring explains assert_not_awaited -> assert_any_await change.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix org budget cache invalidation
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Greptile flagged that the strict isinstance(dict) guard in
_encryption_boundary_key would silently return None for any non-dict input,
including a LiteLLM_Params Pydantic instance, which exposes a custom .get()
method and is intended to be used dict-style in some router paths. If such
an instance ever flowed into healthy_deployments, the guard would drop every
candidate from boundary matching and fall through to the full deployment
pool, i.e. trigger the exact invalid_encrypted_content failure this check
exists to prevent.
Loosen the guard to accept any object exposing a callable .get(): plain
dicts (the common case) and LiteLLM_Params-style Pydantic instances. The
function still returns None for non-dict-like values (None, lists, strings,
ints, bare objects).
Adds regression tests covering:
- LiteLLM_Params Pydantic instance resolves to the same boundary tuple as
an equivalent plain dict
- non-dict-like values and dicts missing required fields still return None
When a Responses API follow-up switches model_name (e.g. gpt-5.3-codex ->
gpt-5.4, or to a LiteLLM-side alias of the same Azure deployment), the
router has already filtered healthy_deployments to the new group, so the
originating model_id is no longer present. The encrypted_content_affinity
check would log "decoded deployment not found" and fall back to the full
deployment pool, where simple-shuffle could land on a different Azure
resource and trip a 400 invalid_encrypted_content.
Fall back to pinning by the originating deployment's encryption boundary
(api_base + api_key) when the model_id miss is across model groups. The
encrypted_content travels with the Azure resource, not the model_name,
so any deployment on the same resource accepts it.
LIT-2531
* fix(tests): drop deprecated OpenAI-Beta realtime header
OpenAI deprecated the 'OpenAI-Beta: realtime=v1' header; the live
service now returns code 4000 invalid_beta with
"Unknown beta requested: 'realtime'.". Two integration tests in
tests/llm_translation/realtime/test_realtime_guardrails_openai.py
hardcoded the header and started failing across all PRs.
Library code is unaffected: the OpenAI realtime handler only
forwards 'OpenAI-Beta: realtime=v1' upstream when the proxy *client*
sends it (litellm/llms/openai/realtime/handler.py). Default proxy
behavior uses the GA protocol.
Connect to OpenAI without the deprecated header, and accept the GA
event name 'response.output_audio_transcript.delta' alongside the
beta-protocol name 'response.audio_transcript.delta' for the
transcript-delta assertion.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(logging): post_call tolerates non-JSON-serializable values
post_call() did json.dumps(original_response) without default=str, so
any provider passing a dict containing datetime/Decimal/etc. would
raise TypeError. Bedrock batch retrieval hits this with
get_model_invocation_job() responses that include datetime fields
(submitTime, lastModifiedTime, endTime), failing
tests/batches_tests/test_bedrock_files_and_batches.py::test_async_file_and_batch
across all PRs.
Pass default=str so non-serializable values fall back to str().
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(tests): mock boto3 in bedrock retrieve batch test
The test patched AsyncHTTPHandler.get, but the bedrock retrieve
handler uses boto3.client('bedrock').get_model_invocation_job
directly, so the real AWS call was being made on every run, failing
with AccessDeniedException because the hardcoded test ARN belongs to
a different AWS account.
- Mock boto3.client and BedrockBatchesConfig.get_credentials so the
test never touches AWS.
- Use status=Completed in the mock response so output_file_id is
populated (the handler intentionally leaves it None for
non-completed jobs).
- Assert the predicted per-job output object URI (matches what the
handler actually returns) instead of the bare output prefix.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* docs(tests): include GA event name in guardrail-block test docstring
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(bedrock): support retrieve for model-invocation-job batch ARNs
`bedrock.retrieve_batch` previously only handled `:async-invoke/` ARNs
(Twelve Labs Marengo embeddings). The `:model-invocation-job/` ARNs
returned by `CreateModelInvocationJob` (the bulk batch inference API
behind `bedrock.create_batch`) fell through and returned a misleading
data-plane error, leaving created jobs unretrievable through the
LiteLLM batches API.
The two ARN families live on different AWS service endpoints
(`bedrock-runtime` data plane vs `bedrock` control plane), so they need
distinct handlers. This adds:
* `BedrockBatchesHandler._handle_model_invocation_job_status` — calls
the control plane via boto3 (`bedrock:GetModelInvocationJob`),
reusing `BaseAWSLLM.get_credentials` for credential resolution so
model_list / env / role-assumption configs continue to apply. The
response is reshaped into a `LiteLLMBatch` with the same status
mapping `transform_create_batch_response` already uses.
* Output-file-URI prediction. Bedrock surfaces the user-supplied
`s3OutputDataConfig.s3Uri` *prefix* in `GetModelInvocationJob`, but
results actually land at `<prefix>/<job-id>/<basename(input)>.out`.
We compute that single-file URI client-side and surface it as
`output_file_id`, so OpenAI-style `client.files.content(...)` works
without an extra `ListObjectsV2` round-trip. The bare prefix stays
in metadata for callers that want the manifest.
* Dispatch in `litellm/batches/main.py` for the new ARN family,
alongside the existing async-invoke branch.
* Unit tests covering ARN parsing, output-URI prediction (incl. edge
cases), the full status mapping, region resolution precedence, and
failure-message propagation.
Note: `request_counts` is intentionally `(0, 0, 0)` —
`GetModelInvocationJob` does not report per-record counts; getting
accurate numbers requires parsing `manifest.json.out` from the output
S3 prefix, which is left to callers.
Made-with: Cursor
* fix(bedrock): address PR feedback on model-invocation-job retrieve
Addresses Greptile P2 findings on #26834:
1. Use the bare job id (not the full ARN) when constructing the
`api_base` URL for `pre_call` logging. Passing the full ARN double-
counts the `model-invocation-job/` segment and embeds colons in the
path, producing misleading log lines.
2. Drop the `or output_prefix` fallback when `_predict_output_file_uri`
returns None. A bare prefix is not a downloadable object and surfacing
it as `output_file_id` re-creates the very NoSuchKey bug this handler
exists to fix. The bare prefix is still preserved in
`metadata["output_s3_uri"]` for callers that want to do their own S3
listing or read `manifest.json.out`.
`metadata["output_file_uri"]` uses "" rather than None to satisfy the
OpenAI Batch metadata schema (`dict[str, str]`); callers should branch
on the typed `output_file_id` field instead.
Also expands test coverage on the new code path:
- new "stay None" regression test for the prediction-fail case
- pre_call/post_call logging hook assertions (incl. the bare-id URL)
- explicit cancelled_at / expired_at coverage
- _to_epoch type-handling matrix and the boto3 ImportError branch
- defensive _extract_region_from_bedrock_arn exception path
- empty-basename case for _predict_output_file_uri
Patch coverage on the changed lines is now 100% (the only remaining
uncovered lines in the file belong to the pre-existing
`_handle_async_invoke_status` method, which this PR does not touch).
Made-with: Cursor
* test(bedrock): cover retrieve_batch dispatch for both ARN families
Codecov flagged 8 uncovered lines on `litellm/batches/main.py` after
this PR refactored the Bedrock dispatch into a single guard with two
sub-branches (`async-invoke` + `model-invocation-job`). Existing tests
exercised the handlers directly but not the dispatch in `main.py`.
Adds `tests/test_litellm/batches/test_retrieve_batch_bedrock_dispatch.py`
with 6 mocked tests that exercise `litellm.retrieve_batch` end-to-end
for the dispatch logic:
- async-invoke ARN routes to `_handle_async_invoke_status`
- async-invoke ARN with no region falls back to "us-east-1" (preserves
prior behavior on this branch)
- model-invocation-job ARN routes to the new
`_handle_model_invocation_job_status` handler
- model-invocation-job ARN with no region forwards None (so the new
handler can sniff region from the ARN itself, rather than getting
silently routed to us-east-1)
- unrelated bedrock ARN family falls through to the generic
provider-config retrieve path (neither special handler invoked)
- non-bedrock batch ids skip the bedrock dispatch entirely
Both handlers are mocked at the import site so the tests don't hit
AWS — the focus here is purely the new dispatch logic in main.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(bedrock): move retrieve_batch dispatch test to tests/test_litellm/
The dispatch test landed under `tests/test_litellm/batches/`, a new
directory that no upstream `test-unit-*.yml` workflow's `test-path`
allow-list includes. As a result, the test was never executed in CI
and codecov reported `litellm/batches/main.py` patch coverage at
11.11% (8 lines uncovered) — the lines belonging to this PR's
dispatch refactor itself.
Move the file up one level so it matches the
`tests/test_litellm/test_*.py` glob that `test-unit-misc.yml`
already runs, and adjust `sys.path.insert` for the new depth.
The companion handler tests under
`tests/test_litellm/llms/bedrock/batches/test_handler.py` are
unaffected — they're picked up by the `llms` directory in
`test-unit-llm-providers.yml`.
Made-with: Cursor
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(responses): map chat tool_choice to Responses API when bridging from completions
OpenAI /v1/responses rejects tool_choice.function. Normalize forced-function
choice from chat shape to {type, name} in LiteLLMResponsesTransformationHandler.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(responses): strip tool_choice.function when top-level name is set
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Anthropic rejects tool_use/tool_result when tools is omitted. Always map
and attach the dummy tool in transform_request so CLIs work without
litellm.modify_params.
- Add unit test for transform_request dummy tool with modify_params off
- Adjust parallel function calling integration expectations: Bedrock
Converse still requires modify_params for this path
Co-authored-by: Cursor <cursoragent@cursor.com>
Reject fnmatch wildcards on non-scope claims when the claim string contains
whitespace so malformed iss values cannot match patterns like trusted.*.
Merge every entry when team_id_jwt_field resolves to a list instead of
keeping only the first element.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Extend responses_api_bridge_check when reasoning_effort + summary aliases
(including nested extra_body) without tools
- Merge summary into reasoning_effort for responses bridge; helpers in utils
- Strip summary aliases in GPT-5 chat mapping when not bridged
- Tests for bridge + merge behavior
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): forward extra_headers for OpenAPI MCP tools
OpenAPI-generated tools only applied static closure headers and BYOK
Authorization via ContextVar. Copy MCPServer.extra_headers from the
incoming MCP request into _request_extra_headers (set in server.py before
local tool dispatch), merge in openapi_to_mcp_generator via a small helper.
OAuth2 M2M: do not forward caller Authorization from raw_headers (same rule
as _prepare_mcp_server_headers for managed MCP).
Adds TestRequestExtraHeaders and clarifies mcp_server_manager registration
comment.
Fixes#26794
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(mcp): access has_client_credentials on MCPServer directly
Greptile: getattr default was redundant; property exists on MCPServer and
mcp_server is non-None inside the extra_headers forwarding block.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The previous detection treated any model with input_cost_per_image
or output_cost_per_image as image generation. Several chat and
embedding models carry those fields to price multimodal vision input,
not generated images:
- gemini-3.1-pro-preview (mode=chat) has output_cost_per_image=0.00012
alongside input/output token pricing.
- azure/gpt-realtime-* (mode=chat) has input_cost_per_image=5e-6.
- amazon.titan-embed-image-v1 (mode=embedding) has
input_cost_per_image=6e-5.
For these models the image-gen branch fired first and reserved a
fraction of a cent per request, short-circuiting the token-priced
path entirely. Long Gemini chats reserved 1 × $0.00012 instead of
the true token cost.
Gate strictly on mode in {"image_generation", "image_edit"}. All 197
real image_generation entries and all 31 image_edit entries
(Flux Kontext, Stability inpaint/outpaint, etc.) carry the right mode,
so the field-presence fallback was unnecessary.
Adds regression tests for the chat-model-with-image-cost-field case
and for image_edit reservation.
Image-generation routes (dall-e-3, flux, etc.) have no per-token output
cost so they fell through to the no-reservation read-time-only path.
Concurrent image requests against a depleted budget could all pass
common_checks (counter exactly at max_budget passes the strict-`>`
gate) and reach the provider before reconciliation caught up.
Add per-image reservation in _estimate_request_max_cost_for_model:
when the model has a per-image cost field, reserve `n × cost_per_image`
upfront. The atomic counter increment serializes concurrent admissions,
so the second request sees the post-first-reservation counter and
raises BudgetExceededError instead of silently leaking through.
Both `output_cost_per_image` and `input_cost_per_image` are honored —
naming is inconsistent across providers (OpenAI dall-e-3 uses
input_cost_per_image, aiml/dall-e-3 uses output_cost_per_image for
the same per-generated-image price).
Per-pixel pricing (DALL-E 2 size variants) and TTS/STT routes still
fall through to read-time enforcement; those are follow-ups.
- Introduce RoutingPrismaWrapper that transparently routes read operations (find_*, count, group_by, query_raw, query_first) to a reader endpoint while writes remain on the writer, enabling Aurora-style reader/writer endpoint splits
- Add IAMEndpoint dataclass and parse_iam_endpoint_from_url() to capture static connection fields from a reader URL so only the IAM token needs to rotate, avoiding the need for separate DATABASE_HOST_READ_REPLICA/etc. env vars
- Enhance PrismaWrapper with per-instance knobs (db_url_env_var, iam_endpoint, recreate_uses_datasource, log_prefix) so writer and reader wrappers are independent: the reader writes its fresh URL to DATABASE_URL_READ_REPLICA and passes datasource override to Prisma since Prisma only auto-reads DATABASE_URL
- Fix deadlock in PrismaWrapper.__getattr__: when called from inside a running event loop, schedule the token refresh as a background task instead of blocking with run_coroutine_threadsafe + future.result(), which would deadlock the loop thread waiting for a coroutine that needs the loop to run
- Fix botocore crash when DATABASE_PORT is unset by defaulting to "5432" in both proxy_cli.py and PrismaWrapper.get_rds_iam_token(); passing None caused botocore to embed the literal string "None" in the presigned URL
- Implement graceful reader degradation: reader connect/recreate failures are non-fatal; wrapper sets _reader_unavailable=True and silently routes reads to the writer to keep the proxy serving traffic during transient reader outages
- Add PrismaClient.writer_db property so the reconnect smoke-test always validates the writer engine specifically; query_raw on the routing wrapper would route to the reader and not verify the newly-recreated writer
- Expose DATABASE_URL_READ_REPLICA in Helm chart (values.yaml + deployment.yaml) via both plain value and secret key reference, and document the field in docker-compose.yml
- Add 887-line test suite covering routing logic, IAM token refresh paths, reader degradation scenarios, datasource override behavior, and the deadlock regression
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
reserve_budget_for_request fell back to reserving the entire remaining
team/key/user headroom whenever a request omitted max_tokens, which
pinned the spend counter at max_budget for the duration of the
in-flight request and false-positive-blocked every concurrent or
back-to-back request until the success callback reconciled. Surfaced
as an integration-test team being budget-blocked at its $2000 cap
while DB spend was $0.144.
Switch the missing-max_tokens path to a fixed default of 16384 output
tokens (mirrors parallel_request_limiter_v3's DEFAULT_MAX_TOKENS_ESTIMATE
precedent), and clamp explicit max_tokens at the model's
max_output_tokens for reservation accounting only. The outbound request
body is unchanged, so providers see whatever the caller actually sent;
only the local integer used to compute reservation cost is bounded.
This also prevents a hostile max_tokens=999999999 from inflating one
request's reservation up to the entire team headroom.
For Opus 4.7 (output $25/M, max_output 128K) on a $2000 budget the
worst-case per-request reservation drops from "everything left" to
$3.20, raising admittable concurrency from 1 to ~625.
