mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-02 18:21:56 +00:00
92d4bba58fb6def6975fbc20eebc3fa730cdf960
39379
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ryan-crabbe-berriandGitHub
|
92d4bba58f |
fix(ui/add-model): stop vertex_ai-anthropic_models from leaking under Anthropic (#28723)
`getProviderModels()` matched a model into a provider's dropdown when the model's `litellm_provider` string *contained* the provider key as a substring. The intent was to admit suffix variants (e.g. `anthropic_text`, `bedrock_converse`), but the substring check is too loose: it also pulls in unrelated providers whose name happens to contain the key, most visibly `vertex_ai-anthropic_models` matching `anthropic` and `vertex_ai-openai_models` matching `openai`. Replace `.includes()` with separator-anchored prefix matching (`startsWith(provider + "_")` / `startsWith(provider + "-")`). All legitimate variants in `model_prices_and_context_window.json` still match (`anthropic_text`, `azure_text`, `azure_ai`, `bedrock_converse`, `bedrock_mantle`, `cohere_chat`, `fireworks_ai-embedding-models`, `vertex_ai-*`, `vertex_ai_beta`), and the cross-provider leak is closed. Tests: update one assertion that pinned the buggy substring behavior (`custom_openai_endpoint` matching `openai` — not a real provider value); add 6 new tests covering the leak regressions and the variant-preservation contract for vertex_ai/bedrock/fireworks. |
||
|
yuneng-jiangandGitHub
|
5f73ad4fe7 |
fix(team): refresh team cache on team_model_add/delete (LIT-3244) (#28683)
* fix(team): refresh team cache on team_model_add/delete (LIT-3244)
team_model_add and team_model_delete wrote to the DB but did not
invalidate the in-memory LiteLLM_TeamTableCachedObj used by
common_checks. After the v1.83.14 common_checks centralization made
team.models authoritative on /v1/files and /v1/vector_stores/*,
adding a Team-BYOK model silently failed to grant the new public
model name to team members until the cache TTL expired (and a
removed model kept working until then on the symmetric path).
Extract the cache-refresh snippet from update_team into a small
helper and apply it consistently at all three team-write sites.
* test: also assert updated models in team-cache-refresh pin
Strengthens the LIT-3244 regression test to also assert
`call_kwargs["team_table"].models` matches the updated row,
not just `team_id`. Both `existing_team` and `updated_team`
share `team_id` in the test setup, so the previous assertion
would have passed even if the implementation accidentally cached
the pre-mutation row.
Greptile review feedback.
* fix(team): hydrate object_permission on cache-refreshing team updates
The Prisma update calls in update_team, team_model_add, and
team_model_delete returned a team row with object_permission_id set
but object_permission=None (the relation was not requested via
include=). _refresh_cached_team then wrote that to the in-memory
LiteLLM_TeamTableCachedObj, and the cache-hit path in get_team_object
returns the cached object without re-hydrating. Downstream consumers
(validate_key_search_tools_against_team, the MCP/agent authz paths)
treat a missing object_permission as no team-level restriction, so
a team-write op silently dropped object-permission enforcement until
the cache TTL expired or a DB-fetch path re-hydrated it.
Add include={"object_permission": True} to all three updates so the
refresh writes a complete cached team. Extend the LIT-3244 regression
test to pin both the cached object_permission and the include shape
on the Prisma call.
Surfaced in PR review of LIT-3244.
|
||
|
yuneng-jiangandGitHub
|
3bcfe41f05 |
test(model_prices): allow audio_transcription_config in schema (#28708)
The schema in test_aaamodel_prices_and_context_window_json_is_valid uses additionalProperties: false. The azure/speech/azure-stt entry added in #27482 introduced an audio_transcription_config field that the schema did not whitelist, so the test fails on every branch built on top of staging. Add the field as a string property. |
||
|
yuneng-jiangandGitHub
|
7c667b8797 |
fix(helm): drop main- prefix from default image tag (#28710)
* fix(helm): drop main- prefix from default image tag
The default image tag in the deployment + migrations-job templates was
`main-{{ .Chart.AppVersion }}`. The current release pipeline publishes
content tags without the `main-` prefix (e.g. `v1.85.1` / `1.85.1`,
`v1.86.0-rc.1` / `1.86.0-rc.1`), so the rendered ref points at a tag
that does not exist on GHCR or DockerHub and installs fail with
ImagePullBackOff.
- templates/deployment.yaml, templates/migrations-job.yaml: render
`.Chart.AppVersion` directly instead of `main-<AppVersion>`.
- Chart.yaml: bump stale `appVersion: v1.80.12` (not on either
registry) to `v1.85.1` so local-checkout installs also resolve.
- values.yaml: update the commented tag-override hint to match.
* fix(helm): use :latest in tag override example, not pinned version
Per review: ghcr.io/berriai/litellm-database:latest is a floating
alias for the most recent stable (same digest as :main-stable),
maintained by the release pipeline's UPDATE_LATEST advance step.
Better example than a pinned version that goes stale.
|
||
|
yuneng-jiangandGitHub
|
8513d7fc0c | chore: update Next.js build artifacts (2026-05-23 19:21 UTC, node v20.20.2) (#28707) | ||
|
ryan-crabbe-berriandGitHub
|
886e91b85e |
fix(otel): stamp http.response.status_code on all error responses (#28405)
* fix(otel): stamp http.response.status_code on all error responses
httpx.HTTPStatusError exposes status under .response.status_code, not as a
top-level attr, so unified-endpoint 5xx failures left the SERVER span without
a status. The admin hooks only wrote a child span and never stamped or ended
the parent at all, so admin 4xx/5xx (and success) responses were invisible
to dashboards. Adds a fallback to .response.status_code in get_error_information,
and ends the parent SERVER span in async_management_endpoint_{success,failure}_hook
with the same _record_exception_on_span helper the unified path uses.
Resolves LIT-3193
* test(otel): exercise httpx.HTTPStatusError through admin path
Pins the contract that get_error_information's response.status_code fallback
is reachable from any entry point — without this, a future refactor that
bypasses _record_exception_on_span in the admin hooks could regress for
httpx-wrapped exceptions while the unified suite still passes.
* chore(otel): trim verbose comments in LIT-3193 changes
Tighten docstrings and remove redundant section dividers/inline narration.
Behavior is unchanged.
* fix(otel): set span.status on management hook parent SERVER span
Mirror the unified failure path: stamp StatusCode.ERROR on the parent
SERVER span before recording the exception, and StatusCode.OK before
ending it on success. Without this, OTEL backends filtering on span
status (the idiomatic primitive) miss admin-endpoint failures even
though the http.response.status_code attribute is correct.
Extend assert_server_span_attrs to assert span.status.status_code
matches the expected outcome so the gap can't regress.
* fix(otel): close SERVER span on body-validation and unhandled errors
Stash the SERVER span on request.state in auth so FastAPI exception
handlers can finish it for failures that occur after auth but before
the route handler (e.g. /model/new TypeError, /key/generate
RequestValidationError). Without this, those requests left dangling
spans missing http.response.status_code.
Resolves LIT-3193
* fix(otel): generic 500 body, log exception details server-side
Don't leak str(exc) and type(exc).__name__ to clients on uncaught
exceptions. The full traceback is logged via verbose_proxy_logger and
the SERVER span still gets http.response.status_code=500.
Resolves LIT-3193
* fix(otel): stamp http.response.status_code on every SERVER span path
Closes three remaining gaps where the proxy SERVER span ended without
the http.response.status_code attribute:
1. ProxyException raised from _read_request_body (e.g. invalid JSON
body) bubbled out of user_api_key_auth before the SERVER span was
created, so the FastAPI handler had nothing to close and the trace
never reached the backend. Hoist the span creation to a new
idempotent _ensure_parent_otel_span_on_request_state helper called
at the top of user_api_key_auth; wire openai_exception_handler to
close the dangling span. Covers /v1/chat/completions, /v1/messages,
/v1/responses (shared handler).
2. /v1/responses success — _handle_success ends the proxy span before
async_post_call_success_hook fires on this path, so the hook's
set_response_status_code_attribute(200) silently no-op'd against an
ended span. Stamp 200 + set OK status at the close site in
_handle_success / _end_proxy_span_from_kwargs via a shared
_close_proxy_span_ok helper, so the attribute lands regardless of
which success hook runs first.
3. Failure path for exceptions without code/status_code (e.g. a bare
TypeError surfacing through _handle_llm_api_exception) — empty
error_information.error_code → _record_exception_on_span skips the
stamp → the hook ends the span. Default to 500 in
async_post_call_failure_hook so the attribute is always set.
Resolves LIT-3193
|
||
|
|
14c0a2b3e2 |
feat(prometheus): emit per-token-type detail metrics (LIT-3220) (#28372) (#28378)
* feat(prometheus): emit per-token-type detail metrics (LIT-3220) (#28372) Adds five sparse counter metrics that break out the token detail fields providers already report in `usage.prompt_tokens_details` and `usage.completion_tokens_details`: - litellm_input_cached_tokens_metric (provider prompt-cache reads) - litellm_input_cache_creation_tokens_metric (Anthropic prompt-cache writes) - litellm_input_audio_tokens_metric (audio input tokens) - litellm_output_reasoning_tokens_metric (reasoning tokens) - litellm_output_audio_tokens_metric (audio output tokens) These are additive — existing input/output/total counters are unchanged, so no dashboards break. Each new counter is only incremented when the underlying detail is populated and > 0, keeping scrape output sparse for providers that don't report a given field. Data is read from the canonical Usage dict that `get_standard_logging_object_payload` already attaches at `standard_logging_payload["metadata"]["usage_object"]`, so no new plumbing through the logging pipeline is required. Tests: 10 new unit tests covering registration, label-set parity, all-types increment, zero/None/negative skip behaviour, and the no-metadata/no-usage_object no-op paths. Closes LIT-3220 Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Claude <noreply@anthropic.com> * chore: remove proof folder image --------- Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> |
||
|
yuneng-jiangandGitHub
|
5e16f20962 |
test(proxy): phase-4 payload behavior pinning for tier-2/3 key + team management endpoints (#28681)
* test(proxy): phase-4 payload behavior pinning for tier-2/3 key + team management endpoints Extends the Phase 1–3 behavior-pin suite at tests/proxy_behavior/management/ with a second axis: payload-shape pinning. Phase 1–3 held payload minimal and pinned (actor, target) → status across 37 routes; Phase 4 holds the caller fixed at an authorized actor, varies the payload shape, and asserts the observable DB effect (on accept) or the named guard / row-unchanged (on reject). Faithfulness contract from Phase 1–3 is unchanged. Six families + one gap-closer (59 new scenarios, 620 → 679 total): * F1 — key budget / rate-limit (test_key_budget_limits.py, 18) * F2 — key↔team reassignment (test_key_team_change.py, 6) * F3 — team budget / rate-limit (test_team_budget_limits.py, 15) * F4 — member-info validation (test_team_member_info_validation.py, 5) * F5 — permission batching (test_team_permissions_bulk_update.py, 6) * F6 — org-scoped team access (+2 detail-string pins in existing files) * F7 — coverage gap-closer (test_f7_coverage_closeout.py, 7) Harness extensions in conftest.py (additive only): * create_scratch_org() seeder with its own scratch-prefixed budget row * budget / limit fields on create_scratch_team() * scratch teardown also sweeps litellm_organizationtable Coverage telemetry (behavior-suite-only): * key_management_endpoints.py 60 % → 65 % (+82 lines) * team_endpoints.py 62 % → 72 % (+137 lines, crosses 70 % stretch) Key lands under 70 % per plan §7 escape hatch — the gap is dominated by routes outside F1–F6 scope (key list/info v2 internals) and structurally dead org-budget guards (call sites at lines 889 + 2310 + 985 + 1751 load the org without include_budget_table=True, so org.litellm_budget_table is None at guard time and the aggregate guard no-ops). Pinned as observed no-op behavior so a future fix that flips the flag turns these into reds. Zero source-code changes; pyproject.toml diff is empty; test_route_coverage.py stays green untouched; G3 grep guards still green; local wall-time 14 s for the full suite (no coverage), 22 s with coverage. G4 regression-replay protocol executed against three representative fix-PR parents ( |
||
|
|
203b529c9d |
feat(azure): add speech transcription config support (#27482)
Co-authored-by: oss-agent-shin <279349115+oss-agent-shin@users.noreply.github.com> Co-authored-by: ishaan-berri <ishaan-berri@users.noreply.github.com> |
||
|
|
2eab9ee2c0 |
perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths (#28289)
* perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths
- Introduce pure-text fast-path in `_build_complete_streaming_response` that collapses O(N) `content_block_delta` events into a single equivalent SSE event before conversion, eliminating per-output-token Pydantic `ModelResponseStream` construction; non-text streams (tool_use, thinking, citations) fall back to the unchanged legacy path
- Skip agentic streaming wrapper entirely when no callback overrides `async_should_run_agentic_loop`; the wrapper buffered every chunk and rebuilt the SSE response only to call hooks that all return `(False, {})` — a pure no-op for the default config
- Serialize request body once (`json.dumps`) for both the pre-call log input and the wire, instead of twice; avoids a full O(payload) scan per request, significant for long-context Claude Code histories
- Add fast path in `async_streaming_data_generator` that bypasses the per-chunk `async_post_call_streaming_hook` coroutine await, response-string materialization, and cost-injection call when no callback/guardrail/cost-injection is active (the default config)
- Resolve `_DD_STREAMING_TRACE_ENABLED` once at import time; eliminate per-chunk `NullSpan` context manager allocation when Datadog tracing is disabled (the default)
- Memoize `get_type_hints(AnthropicMessagesRequestOptionalParams)` with `@lru_cache(maxsize=1)` — resolves once per process instead of once per `/v1/messages` request (~80µs each)
- Hoist `cost_injection_active` out of the per-chunk loop in `chunk_processor`; eliminates repeated `getattr` + endpoint-type checks on every streamed byte chunk
- Extract `_build_passthrough_logging_result` from `_route_streaming_logging_to_handler` as a standalone static method to facilitate future off-loop dispatch
- Convert `async_sse_data_generator` from an `async for: yield` trampoline to a direct return of the underlying generator, removing one async-generator layer per streamed chunk
- Skip redundant `strip_empty_text_blocks_from_anthropic_messages` scan in `anthropic_messages_handler` when the async wrapper already sanitized (signalled via `_litellm_messages_presanitized` sentinel, popped before reaching provider params)
- Gate debug log `f-string` evaluation behind `isEnabledFor(DEBUG)` in both the streaming generator and the transformation layer to avoid serializing entire message payloads on every request at non-debug log levels
- Add benchmark script (`scripts/benchmark_anthropic_messages_perf.py`) with a local mock Anthropic SSE provider for reproducible TTFT and TPM measurement across commits/branches
- Add parity tests asserting fast-path and legacy-path produce byte-identical logged/billed payloads, plus unit tests for agentic hook detection, pre-serialized body reuse, and memoized key resolution
* perf: address greptile review for anthropic streaming hot path
- Bail to legacy in `_collapse_pure_text_chunks` when content_block_delta
events from different block indexes are observed without an intervening
flush. Anthropic sends blocks strictly sequentially, but defensive bail
prevents silent text-merging if the protocol ever interleaves.
- Replace leaf-class `__dict__` check for `async_post_call_streaming_hook`
in `_callback_capabilities` with a function-identity comparison that
walks the MRO. A vendor base class can carry the override and the
registered class can add nothing else; before this PR the hook was
unconditionally invoked, so an inherited-override miss would silently
drop the hook on the streaming path.
- Add unit tests for both behaviors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(mypy): narrow model_name to str in cost-injection branch
The hoisted cost_injection_active flag in chunk_processor encodes the
`bool(model_name)` requirement but mypy can't track that invariant
through the local, so the per-chunk `_process_chunk_with_cost_injection(
chunk, model_name)` calls flagged Optional[str] vs str. Pin a typed
non-None local inside the cost-injection branch so mypy narrows
correctly without changing runtime behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
3b2ce201d8 |
encrypt callback_vars in key/team metadata at rest (#27141)
Co-authored-by: Michael Riad Zaky <michaelr@Michaels-MacBook-Air.local> Co-authored-by: Yuneng Jiang <yuneng@berri.ai> |
||
|
+11
|
492891cad8 |
CI: copy of #25177 (OCI GenAI: embeddings, streaming/reasoning fixes, model catalog) (#28223)
* fix(opentelemetry): JSON-serialize dict metadata fields for OTEL span attributes (#27451) (#27455)
Squash-merged by litellm-agent from Anai-Guo's PR.
* feat(dashscope): add embeddings and reranks(qwen3-rerank) support via OpenAI-compatible endpoint (#27508)
Squash-merged by litellm-agent from yimao's PR.
* fix(vertex_ai/gemini): raise BadRequestError when image_url or url fi… (#24550)
Squash-merged by litellm-agent from krisxia0506's PR.
* fix(vertex_ai): raise error on mid-stream 429/error chunks instead of silently swallowing (#23711)
Squash-merged by litellm-agent from krisxia0506's PR.
* fix: raise BadRequestError for file content blocks missing 'file' sub… (#24503)
Squash-merged by litellm-agent from krisxia0506's PR.
* Fix Gemini MIME detection for extensionless GCS URIs (#27278)
Squash-merged by litellm-agent from krisxia0506's PR.
* fix(vertex_ai/partner_models): drop unused vertexai SDK gate from count_tokens (closes #28084) (#28107)
Squash-merged by litellm-agent from voidborne-d's PR.
* feat(chart): add support for autoscaling behavior in HPA (#27990)
Squash-merged by litellm-agent from FabrizioCafolla's PR.
* feat(proxy): add blocked flag to models for pause/resume from the UI (#27927)
Squash-merged by litellm-agent from Cyberfilo's PR.
* fix: pass socket timeouts to Redis cluster clients (#27920)
Squash-merged by litellm-agent from tomdee's PR.
* Fix/cache token (#28009)
Squash-merged by litellm-agent from escon1004's PR.
* fix(deepseek): forward reasoning_content in multi-turn thinking mode conversations (#28080)
Squash-merged by litellm-agent from Divyansh8321's PR.
* fix(guardrails): return HTTP 400 instead of 500 for blocked requests (#27617)
* fix: reset org and tag budgets (#27326)
* reset org budgets
* reset tag budgets
---------
Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain>
* fix(ui): omit allowed_routes from key edit save when unchanged (#27553)
* fix(ui): omit allowed_routes from key edit save when unchanged
When a team admin opens Edit Settings on a key with key_type=AI APIs and
saves without changing anything, the UI re-sends the existing allowed_routes
value, which the backend's _check_allowed_routes_caller_permission gate
rejects for non-proxy-admins (LIT-2681).
Strip allowed_routes from the patch in handleSubmit when it deep-equals the
original keyData.allowed_routes. The backend treats absence as "leave alone,"
so no-op saves now succeed for non-admins. Admins explicitly editing the
field still send the new value.
* fix(ui): order-insensitive allowed_routes diff + cover null-original case
Address Greptile review:
- Switch the "is allowed_routes unchanged" check to a Set-based comparison so
a server-side reorder of the array doesn't register as a user edit and
re-trigger LIT-2681.
- Add two regression tests: (1) keyData.allowed_routes is null and the form
is untouched — patch should strip the field; (2) server returned routes in
a different order than the user originally entered — patch should still
recognize the value as unchanged.
* chore(ui): strip ticket refs and tighten comments in key edit fix
- Remove internal-tracker references from in-code comments
- Tighten the WHY comment in handleSubmit to two lines
- Drop redundant test-block comments — test names already describe the case
* fix(ui): annotate Set<string> generic in allowed_routes diff to fix tsc
* fix(guardrails): return HTTP 400 instead of 500 for guardrail-blocked requests
GuardrailRaisedException and BlockedPiiEntityError both lacked a
status_code attribute. When these exceptions reached the proxy
exception handler (getattr(e, 'status_code', 500)), the fallback
defaulted to HTTP 500 — making intentional guardrail blocks
indistinguishable from server errors and causing unnecessary client
retries.
Changes:
- Add status_code=400 (keyword-only) to GuardrailRaisedException
- Add status_code=400 (keyword-only) to BlockedPiiEntityError
- Update _is_guardrail_intervention() to recognize both exceptions
so downstream loggers record 'guardrail_intervened' instead of
'guardrail_failed_to_respond'
- Add 6 unit tests for default/custom status codes and getattr pattern
- Strengthen existing blocked-action test with status_code assertion
Fixes #24348
---------
Co-authored-by: Michael-RZ-Berri <michael@berri.ai>
Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
* fix(router/proxy): address Greptile P1+P2 review comments on PR #28161
- router: raise ServiceUnavailableError (503) instead of RouterRateLimitErrorBasic (429)
when a specifically-addressed deployment is administratively blocked; 429 misleads
retry-enabled clients into spinning forever against a paused model
- proxy_server: compute get_fully_blocked_model_names() once before both branches in
model_list() instead of duplicating the call in each branch
- deepseek: upgrade silent debug log to warning when injecting placeholder
reasoning_content so callers are clearly notified of degraded multi-turn quality
- tests: update two blocked-deployment assertions to expect ServiceUnavailableError
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: address bug detection findings (cache token order, mutable defaults)
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: address bugs in async pass-through, anthropic cache token detection, rerank tests
- async_get_available_deployment_for_pass_through: enforce blocked check on specific deployments
- cost_calculator: detect anthropic-style usage by attribute presence (not truthiness) to avoid mixing OpenAI cached_tokens into anthropic normalization when read=0
- dashscope rerank tests: pass request to httpx.Response constructions for consistency
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix code qa
* fix(vertex_ai/gemini): strip MIME parameters from GCS contentType
GCS object metadata's contentType field can include parameters such as
'text/html; charset=utf-8'. Strip them in _apply_gemini_mime_type_aliases
so downstream get_file_extension_from_mime_type sees a bare MIME type.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(vertex_ai/gemini): clarify mime-type error message string concatenation
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(oci): add embeddings, fix streaming/reasoning, expand model catalog
- Add OCIEmbedConfig with full Cohere embed support (7 models, batch up to 96)
- Fix sync streaming: split SSE events on \n\n before JSON parsing
- Fix reasoning models (Gemini 2.5, xAI Grok): make completionTokens and message
optional in OCIResponseChoice to handle max_tokens exhausted on reasoning
- Fix compartment_id resolution in chat transform to use resolve_oci_credentials
- Fix tool call id: make OCIToolCall.id optional, generate UUID fallback for
providers (Google via OCI) that omit it
- Add OCI_KEY env var support for inline PEM keys
- Fix datetime.utcnow() deprecation in request signing
- Expand model catalog: 29 OCI models including Llama 4, Gemini 2.5, xAI Grok,
Cohere Command A, and all Cohere embed variants
- Add 37 live integration tests: sync/async completions for Meta/Google/xAI/Cohere,
sync/async embeddings, tool use across all vendors, streaming, env var auth
- Add 23 embed unit tests covering all transform and validation paths
* fix(oci): remove dead OCI elif branch in utils.py, align async split_chunks with sync version
* test(oci): add unit tests for split_chunks fix and no-duplicate-OCI-branch guard
* fix(oci): address remaining bugs from issue #25082 — streaming signed body, Cohere stop sequences, hardcoded defaults
- Bug 1: sync and async streaming paths now use signed_json_body when provided
instead of re-serializing data with json.dumps() — the OCI RSA-SHA256 signature
covers the exact request body bytes, so re-serializing produces an invalid sig
- Bug 3: Cohere stop sequences now map to 'stopSequences' (was incorrectly 'stop')
- Bug 4: removed hardcoded Cohere defaults (maxTokens=600, temperature=1, topK=0,
topP=0.75, frequencyPenalty=0) that silently overrode user intent on every call
- Added 6 unit tests covering all three fixes
* fix(oci): comprehensive code quality pass — bugs, tests, schema accuracy
- Fix Cohere tool call IDs (was always call_0; now UUID per call)
- Fix TOOL_CALL finish reason mapping in both sync and streaming paths
- Fix Cohere stop parameter mapping (stop → stopSequences)
- Remove hardcoded Cohere defaults (maxTokens/topK/topP/frequencyPenalty)
- Fix content[0] safety guard against empty content arrays
- Fix streaming signed body used consistently (not re-serialized)
- Raise OCIError (not bare Exception/ValueError) throughout
- Centralize OCI_API_VERSION constant; import uuid at module level
- Fix embed get_complete_url to strip trailing slashes from api_base
- Fix OCIEmbedResponse schema: add inputTextTokenCounts (actual OCI field)
- Fix embed usage computed from inputTextTokenCounts (sum of per-input counts)
- Fix Cohere toolCallId included in tool result messages
- Add OCIToolCall.id as Optional (absent in Google/xAI streaming chunks)
- Update tests to reflect correct behavior (no hardcoded defaults, UUID ids,
deferred credential validation, OCIError vs ValueError, real response schema)
* test(oci): move integration tests to tests/llm_translation/
Addresses greptile P1: tests/test_litellm/ is for mock-only unit tests
(make test-unit target). Real-network OCI tests now live in the correct
location alongside other provider integration tests.
* fix(oci): align types and transformation with official OCI SDK
- Remove OCIVendors.GEMINI — apiFormat="GEMINI" is invalid; all non-Cohere
models use apiFormat="GENERIC"
- Add toolChoice, logitBias, logProbs to OCIChatRequestPayload so params
present in the mapping are no longer silently dropped by Pydantic
- Exclude n→numGenerations from Cohere param map (not a Cohere API field)
- Fix CohereToolResult: change callId/result to call/outputs matching
the OCI SDK's CohereToolResult structure
- Fix CohereToolMessage: replace non-existent toolCallId with toolResults
list; update adapt_messages_to_cohere_standard to build proper tool-result
history entries by resolving tool call name+params from preceding assistant
messages
- Map generic-model stream finish reasons to OpenAI convention
(COMPLETE→stop, MAX_TOKENS→length, TOOL_CALLS→tool_calls), consistent
with the existing Cohere streaming path
- Add optional id field to OCIEmbedResponse so valid API responses
carrying an id are not rejected by the Pydantic model
* fix(oci): use 'output' key in Cohere tool result outputs (matches reference impl)
* fix(oci): port schema/type utilities from langchain-oracle reference impl
- Add resolve_oci_schema_refs: inline $ref/$defs — OCI rejects JSON Schema refs
- Add resolve_oci_schema_anyof: flatten Optional[T] anyOf (Pydantic v2 emits these)
- Add sanitize_oci_schema: strip title, normalise null types, ensure array items
- Add OCI_JSON_TO_PYTHON_TYPES: Cohere expects Python type names (str/int/float),
not JSON Schema names (string/integer/number)
- Add enrich_cohere_param_description: embed enum/format/range/pattern constraints
into description since CohereParameterDefinition has no dedicated fields
- Apply all of the above in adapt_tool_definitions_to_cohere_standard and
adapt_tool_definition_to_oci_standard
- Fix toolChoice conversion: map OpenAI string ('auto','none','required') to OCI
dict form ({"type":"AUTO"} etc.) — the API rejects plain strings
- Update unit test expectations to match correct Python type names and enriched
descriptions
* refactor(oci): split transformation.py into cohere.py and generic.py
transformation.py was 1 243 lines doing too many jobs. Split along the
same boundaries as the langchain-oracle reference (providers/cohere.py,
providers/generic.py):
chat/cohere.py — Cohere message/tool building, response + stream parsing
chat/generic.py — Generic message/tool building, response + stream parsing
transformation.py — thin OCIChatConfig orchestrator + OCIStreamWrapper
Public symbols (OCIChatConfig, OCIStreamWrapper, adapt_messages_to_*,
OCIRequestWrapper, version, …) remain importable from transformation.py
for backward compatibility. OCIStreamWrapper gains delegating shims for
_handle_cohere_stream_chunk and _handle_generic_stream_chunk so existing
test call sites keep working unchanged.
transformation.py: 1 243 → 620 lines
* refactor(oci): principal-level code quality pass
- Remove _extract_text_content duplication — single definition in cohere.py,
imported where needed; instance method on OCIChatConfig eliminated
- Move cryptography imports to module level with _CRYPTOGRAPHY_AVAILABLE flag
and _require_cryptography() guard; no more re-import on every signing call
- Move litellm version import to module level via litellm._version; remove
inline import inside validate_oci_environment
- sign_with_manual_credentials now returns Tuple[dict, bytes] matching
sign_with_oci_signer — asymmetry eliminated, Optional[bytes] guards removed
throughout stream wrappers (signed_json_body: bytes = b"")
- Rename _openai_to_oci_cohere_param_map → openai_to_oci_cohere_param_map
for consistency with openai_to_oci_generic_param_map
- Remove double-key bug in map_openai_params where responseFormat was stored
under both OCI and OpenAI key names simultaneously
- Remove delegating shims (adapt_messages_to_cohere_standard,
adapt_tool_definitions_to_cohere_standard, _handle_generic_stream_chunk)
from OCIChatConfig/OCIStreamWrapper; tests now import directly from
cohere.py and generic.py where symbols live
- Trim __all__ to 7 genuine public symbols; remove the 13-symbol list that
existed only to support test imports
- Collapse per-model integration test classes into pytest.mark.parametrize;
CHAT_MODELS list is the single source of truth for model-specific config
- Black + Ruff clean across all OCI files
* fix(oci): address PR review findings
- types/llms/oci.py: add "TOOL_CALL" to CohereChatResponse.finishReason
Literal so Pydantic does not raise ValidationError on non-streaming
Cohere tool-use calls (Greptile P1)
- test_oci_cohere_tool_calls.py: add test covering TOOL_CALL finish reason
- model_prices_and_context_window.json: remove 6 duplicate oci/cohere.embed-*
keys that were silently overridden by the more complete entries already
present in the file (Greptile P1)
- common_utils.py: move OCI_API_VERSION here from chat/transformation.py
so embed/transformation.py does not need to import chat/transformation;
change Protocol stub body from ... to pass (CodeQL "statement no effect");
add comment to sha256_base64 clarifying it implements OCI HTTP signing
spec, not password hashing (CodeQL false positive)
- chat/transformation.py: import CustomStreamWrapper from
litellm_core_utils.streaming_handler instead of litellm.utils to reduce
import cycle depth (CodeQL cyclic import)
- chat/cohere.py, chat/generic.py: import Usage and
ChatCompletionMessageToolCall from litellm.types.utils instead of
litellm.utils for the same reason
- embed/transformation.py: import OCI_API_VERSION from common_utils
instead of chat/transformation (removes the embed→chat import edge)
* test(oci): add unit tests to improve patch coverage
- test_oci_common_utils.py (new): covers sha256_base64, build_signature_string,
OCIRequestWrapper.path_url, resolve_oci_credentials, get_oci_base_url,
validate_oci_environment, sign_with_oci_signer error paths, sign_oci_request
routing, load_private_key_from_file error paths, resolve_oci_schema_refs
(including circular ref and external $ref), resolve_oci_schema_anyof,
sanitize_oci_schema (all branches), enrich_cohere_param_description
- test_oci_generic_chat.py (new): covers content-message error paths (non-dict
item, unsupported type, non-string text, invalid image_url), tool-call
validation error paths, adapt_messages_to_generic_oci_standard error paths,
handle_generic_response (None message, text content, tool calls),
handle_generic_stream_chunk (finish reasons, streaming tool calls),
OCIStreamWrapper non-string chunk error
- test_oci_chat_transformation.py: add error paths for validate_environment
(empty messages), transform_request (missing compartment_id, Cohere without
user messages), transform_response (error key), map_openai_params
(unsupported param with and without drop_params), tool_choice string mapping
- test_oci_cohere_tool_calls.py: add edge cases for stream chunk finish
reasons (TOOL_CALL, MAX_TOKENS, unknown), _extract_text_content with
non-dict list items and non-string input,
adapt_messages_to_cohere_standard with malformed JSON tool arguments
* fix(oci): rename supports_streaming to supports_native_streaming in model prices
The JSON schema for model_prices_and_context_window.json uses
`supports_native_streaming` (not `supports_streaming`) and has
`additionalProperties: false`. Rename the field across all OCI
entries to pass the schema validation test.
* test(oci): add 67 tests targeting uncovered happy paths for coverage
Boost patch coverage on the four lowest-coverage OCI files:
- common_utils.py: sign_with_manual_credentials (oci_key / oci_key_file
paths), sign_oci_request routing, _require_cryptography
- generic.py: adapt_messages_to_generic_oci_standard (all roles),
adapt_tool_definition_to_oci_standard, adapt_tools_to_openai_standard,
handle_generic_stream_chunk text/finish-reason paths
- cohere.py: _extract_text_content, adapt_messages_to_cohere_standard
(all roles including tool results), handle_cohere_response /
handle_cohere_stream_chunk all finish-reason branches
- transformation.py: get_vendor_from_model, OCIChatConfig._get_optional_params
(toolChoice string→dict, responseFormat, tools for both vendors),
transform_request for GENERIC model, get_sync/async_custom_stream_wrapper
with mocked HTTP, OCIStreamWrapper.chunk_creator happy paths
* fix(oci): suppress CodeQL false positive on sha256_base64 (OCI HTTP signing, not password hashing)
* fix(oci): remove 6 duplicate model price entries and reconcile conflicting values
Six OCI chat model keys appeared twice in model_prices_and_context_window.json
with conflicting pricing/context data (JSON parsers silently discard the first).
Remove the first-occurrence entries and update the surviving entries:
- meta.llama-4-maverick / llama-4-scout: keep updated entries (free preview
pricing, larger context windows, vision support)
- meta.llama-3.1-70b: keep original pricing, restore supports_native_streaming
- google.gemini-2.5-{flash,pro,flash-lite}: keep OCI pricing page values,
restore supports_native_streaming
* fix(oci): route GPT-5 family to maxCompletionTokens
GPT-5 / GPT-5-mini / GPT-5-nano / GPT-5.5 on OCI reject "maxTokens"
with HTTP 400:
Invalid 'maxTokens': Unsupported parameter: 'maxTokens' is not
supported with this model. Use 'maxCompletionTokens' instead.
(Same convention as OpenAI's reasoning-API contract.)
Add a model-aware rename in OCIChatConfig._get_optional_params so the
request payload uses maxCompletionTokens when the model id starts with
openai.gpt-5. Regular Llama / Cohere / Gemini / GPT-4.x continue to use
maxTokens unchanged.
Also widen OCIChatRequestPayload to carry the new optional field so it
survives Pydantic serialization.
Verified live against OCI us-chicago-1:
- openai.gpt-5, gpt-5-mini, gpt-5-nano, gpt-5.5 all return 200
- Full feature sweep on gpt-5.5 (basic, system, multi-turn, streaming,
tools, usage) all green
- meta.llama-3.3-70b-instruct still uses maxTokens (no regression)
4 new unit tests cover the helper, the routing in both pre- and
post-translation states, and Pydantic serialization.
* ci(oci): fix CI failures — black formatting + recursive_detector ignore
- Run black on litellm/llms/oci/common_utils.py + 3 OCI test files
that drifted out of black-compliance during the rebase.
- Add the three bounded recursive functions in oci/common_utils.py
(`_resolve`, `resolve_oci_schema_anyof`, `sanitize_oci_schema`) to
the recursive_detector IGNORE_FUNCTIONS list. All three are bounded:
`_resolve` uses a `resolving_stack` cycle guard; the other two are
bounded by JSON-schema tree depth (no cycles in well-formed input),
matching the pattern of the existing OCI/Vertex schema walkers
already on the list.
* fix(oci): silence MyPy errors in cohere.py — typed-dict access
Two errors flagged by `lint` CI:
llms/oci/chat/cohere.py:73: "object" has no attribute "__iter__"
llms/oci/chat/cohere.py:119: No overload variant of "get" of "dict"
matches argument types "object", "CohereToolCall"
Both stem from `msg.get("tool_calls")` / `msg.get("tool_call_id")`
returning `object` per the AllMessageValues TypedDict union. Bind to
`Any` locally for the iteration and coerce the lookup key with `str()`,
removing the now-unused `# type: ignore` on those lines.
No behaviour change — pure type-narrowing for the type checker.
* fix(oci): silence CodeQL py/weak-sensitive-data-hashing on sha256_base64
CodeQL's taint analysis traces request bodies back to environment-loaded
secrets and flags `hashlib.sha256(body).digest()` as
`py/weak-sensitive-data-hashing` — even though SHA-256 is the algorithm
mandated by the OCI HTTP request signing spec for the
`x-content-sha256` header (not a password/secret hash).
The previous suppression used legacy `# lgtm[...]` syntax which the
modern CodeQL action ignores. Switch to Python's standard
`hashlib.sha256(..., usedforsecurity=False)` (Python 3.9+) which CodeQL
honours as a non-security declaration. Behaviour unchanged.
* feat(oci): add reasoning_effort passthrough — only true missing primitive
OCI's GenericChatRequest exposes a reasoningEffort field
(NONE/MINIMAL/LOW/MEDIUM/HIGH) that's the single biggest cost knob for
reasoning-capable models on the service:
- GPT-5 family
- Gemini 2.5
- Grok reasoning variants (3-mini, 4-fast, 4.20)
- Cohere Command-A-Reasoning
Setting reasoning_effort=LOW typically cuts reasoning-token spend 5-10×
vs the default. Without exposing this, litellm users had no way to tune
cost-vs-quality on these models.
The other GenericChatRequest fields (verbosity, parallel_tool_calls,
logit_bias, n, metadata, web_search_options, prediction) are not
exposed because they are not missing primitives — they either duplicate
prompt-engineering, framework-level controls, or are too niche to
justify the maintenance surface. We only ship what users genuinely
can't accomplish another way.
Excluded from the Cohere v1 param map: CohereChatRequest has no
reasoningEffort field, and Cohere reasoning models
(cohere.command-a-reasoning) use COHEREV2 which is a separate request
type not covered by this PR.
Verified live: GPT-5.5 + reasoning_effort="HIGH" sends
{"reasoningEffort": "HIGH"} on the wire and OCI accepts the request.
* feat(oci): reasoning_effort + reasoning_tokens for OCI GenAI
Three small additions for OCI reasoning models, requested by users
testing the PR in production fork builds:
1. **reasoning_effort param mapping (GENERIC vendors).** OCI expects
uppercase levels ("LOW"/"MEDIUM"/"HIGH"/"NONE") on `reasoningEffort`,
but OpenAI-compatible clients send lowercase. Mapped + uppercased in
`_get_optional_params`. Marked unsupported on Cohere V1/V2 since OCI
Cohere has no reasoning models (avoids Pydantic validation failure
on CohereChatRequest).
2. **"disable" → "NONE" mapping.** OpenAI uses "disable" to turn off
reasoning; OCI uses "NONE". Without this, callers get a 400.
3. **reasoning_tokens propagated to Usage.** OCI returns
`completionTokensDetails.reasoningTokens` but it wasn't being passed
to LiteLLM's Usage object. Now flows through to
`Usage.completion_tokens_details.reasoning_tokens` so callers can
track reasoning token consumption for cost/observability.
Tests: 7 new unit tests in TestOCIReasoningEffort covering upper/lower
case, "disable"→"NONE", Cohere drop/raise paths, and reasoning_tokens
extraction (with and without completionTokensDetails). 5 new live
integration tests against xai.grok-3-mini in us-chicago-1 verifying the
full request/response loop end-to-end. Existing
test_transform_response_simple_text assertion that
completion_tokens_details was None has been updated to assert
reasoning_tokens flows through.
Verified live on xai.grok-3-mini: reasoning_effort=low → OCI accepts
"LOW", returns reasoningTokens=316 in usage. reasoning_effort=disable
→ OCI accepts "NONE". Full suite: 370/370 unit + 51/51 integration.
* fix(codeql): re-scope py/weak-sensitive-data-hashing exclusion to OCI signing file
CodeQL's taint analysis re-fires the `py/weak-sensitive-data-hashing`
alert at `litellm/llms/oci/common_utils.py:103` whenever upstream code
paths into the OCI signing module change (touching `transformation.py`
opens new flow paths that CodeQL re-evaluates from scratch). The
`hashlib.sha256(..., usedforsecurity=False)` declaration silences the
direct-call form of the query but not the taint-flow form.
SHA-256 here is mandated by the OCI HTTP signing specification for the
x-content-sha256 content-integrity header — not for password storage:
https://docs.oracle.com/en-us/iaas/Content/API/Concepts/signingrequests.htm
CodeQL has no per-query path filter and GitHub Code Scanning ignores
inline lgtm/codeql comments, so path-ignoring this single ~560-line
signing utility file is the narrowest available suppression. All other
files retain full coverage of py/weak-sensitive-data-hashing — including
litellm/proxy/utils.py where the rule legitimately applies.
This restores the NEUTRAL CodeQL state the PR had on prior commits
(see `2111c98af7` for the same approach on the previous branch
evolution that the cherry-pick was rebased onto a different baseline).
* fix(oci): drop duplicate text on Cohere streaming terminal chunk
OCI Cohere's terminal SSE event re-sends the full assembled response in
`text` alongside a populated `chatHistory`. Emitting that text as another
delta concatenates the entire response onto the already-streamed output
(e.g. "How can I help?How can I help?").
Use `chatHistory is not None` as the discriminator for the consolidated
terminal event — `finishReason` is a weaker signal that could in principle
appear on a non-consolidated chunk. The two coincide today; this preserves
correctness if OCI ever ships finishReason on an incremental chunk.
Adds a live-OCI integration regression test that compares streamed vs
non-streamed length and asserts the response prefix appears only once.
Verified to fail under the previous code with the exact reported
reproduction: 'Hello! How can I help you today?Hello! How can I help you today?'.
Reported by @gotsysdba on PR #25177.
* fix(oci): buffer SSE stream across HTTP read boundaries
The old split_chunks helper split each individual HTTP read on "\n\n",
which assumed SSE event boundaries always aligned with read boundaries.
In practice the OCI streaming endpoint delivers events that may:
- straddle two reads (chunk_creator gets a truncated JSON and crashes)
- arrive separated by a single "\n" instead of "\n\n"
- share a read with multiple complete events
Replace the inline split with module-level helpers _iter_sse_events
(sync) / _aiter_sse_events (async) that maintain a buffer across reads,
split on any newline, and yield only complete "data:" lines.
Add 25 regression tests covering event-split-across-reads, tiny-chunk
reads, single-newline separators, keepalive/comment lines, trailing
partial events flushed at EOF, "\r\n" line endings, and an end-to-end
smoke test that feeds an awkwardly-chopped payload through the splitter
into OCIStreamWrapper.chunk_creator.
Reported by John Lathouwers.
* test(oci): repoint TestOCIKeyNormalization to sign_with_manual_credentials
The signing helper moved from OCIChatConfig._sign_with_manual_credentials
to a module-level sign_with_manual_credentials in common_utils.py. Four
tests in TestOCIKeyNormalization still called the old method:
- 2 failed outright with AttributeError
- 2 passed by accident because they used pytest.raises(Exception),
which happily caught the AttributeError instead of exercising the
intended OCIError path
Repoint all four to the new module-level function so they exercise the
actual oci_key type-validation branch.
* fix(oci): validate oci_region before URL interpolation to prevent SSRF
Anchor oci_region to ^[a-z][a-z0-9-]{0,30}[a-z0-9]$ inside get_oci_base_url
so user-supplied regions that would redirect the signed request to an
attacker-controlled host (e.g. 'evil.com/#') fail with HTTP 400 before
the URL or signature is built. Empty string still falls back to the
us-ashburn-1 default, so existing callers are unaffected.
* test(audio): skip when gpt-4o-audio-preview is unavailable upstream
OpenAI retired `gpt-4o-audio-preview` (404 model_not_found in CI as of
2026-05-19), and the existing try/except in these tests only re-raised
on 'openai-internal' errors. Other exceptions were silently swallowed,
so the next line ran with an unbound `response`/`completion` and
failed with an unrelated UnboundLocalError that masked the real cause.
Extend the skip condition to also cover model_not_found / 'does not exist'
so the suite reports the upstream outage cleanly, matching the pattern
used in
|
||
|
|
7270f723de |
fix(mcp): forward upstream initialize instructions on cold gateway init (#28231)
Prefetch upstream InitializeResult.instructions before merging gateway initialize options when YAML/DB do not set instructions, so clients receive upstream server text on the first MCP initialize without list_tools. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
+53Sameer KankuteGitHubYuneng JiangCursor AgentSameer KankuteYassin KortamMateo WangclaudeClaudeKenan YildirimvladpolevoiYassin Kortamryan-crabbe-berriMateo WangDennis Henryveria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>ishaan-berriIshaan Jaffergreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>shin-berriJoão Costaoss-agent-shinFilippo MenghiMichael-RZ-BerriShivam RawatTai AnVincentKris Xiad 🔹Fabrizio CafollaTom Denhamescon1004Divyansh Singhalrobin-fiddlerMichael Riad ZakyKrrish DholakiaNoah NistlerFelipe Rodrigues Gare Carnielliharish-berrimilan-berriRyanClaudeTorvaldUtnemubashir1osmaniIshacwang-ottoRoman Pushkinboarder7395stuxfDibyo MukherjeeKevin ZhaoMatthew LapointeElon AzoulayafoninskyJoseph BarkerMaruti AgarwalCursor BugbotGreptile <greptile-apps[bot]@users.noreply.github.com>Greptile Reviewerdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Felipe GaréwithomasmicrosoftAditya Singh |
f35e7eb2f6 |
feat(guardrails): add Microsoft Purview DLP guardrail (#24966)
* feat(guardrails): add Microsoft Purview DLP guardrail * fix(guardrails/purview): raise_for_status on HTTP errors, cap scope cache, reuse executor * fix(guardrails/purview): propagate litellm_call_id as correlation_id to Purview * chore: fixes * refactor(guardrails): delegate get_user_prompt to get_last_user_message PurviewGuardrailBase duplicated AzureGuardrailBase (and OpenAIGuardrailBase) user-prompt extraction. The same logic already lived in common_utils.get_last_user_message; wire guardrail bases to that helper, fix the helper docstring, and drop its redundant self-import of convert_content_list_to_str. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix(purview): make protection scope cache true LRU on hits OrderedDict.get() does not update insertion order; call move_to_end on TTL-valid cache hits so popitem(last=False) evicts least-recently-used users instead of FIFO by first insert. Add a regression test with a small max cache size. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * Fix mypy * fix(guardrails/purview): harden user-id resolution and broaden DLP text Prefer API key and proxy-injected metadata over client metadata for Entra identity. Scan full message transcript pre-call and all completion choices post-call. Align logging-only hook with the same user-id rules. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails/purview): scan /v1/completions prompt and TextChoices Normalize text-completion prompts (string or list of strings); skip token-id-only prompts. Run post-call DLP on TextCompletionResponse choices. Extend logging_only hook for text_completion. Add tests and completion_prompt_to_str helper. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(purview-dlp): return data after DLP pass; per-call executor; dedupe text extraction async_pre_call_hook now returns the request dict after a successful check so callers match skip-path behavior. logging_hook uses a fresh ThreadPoolExecutor per invocation like Presidio to avoid single-worker starvation. Response text extraction is centralized in _completion_response_text_parts. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix(purview): fix LRU cache refresh position and add Responses API scanning Two fixes to the Microsoft Purview DLP guardrail: 1. LRU cache bug (base.py): When a stale scope cache entry was re-fetched, the assignment updated the value but Python's OrderedDict.__setitem__ preserves the original insertion order for existing keys. This left the refreshed entry near the front of the dict, making it the first candidate for LRU eviction via popitem(last=False). Fix: call move_to_end(user_id) after every write to an existing key. 2. Responses API coverage gap (purview_dlp.py): Requests to /v1/responses use an 'input' field instead of 'messages' or 'prompt', so the pre-call hook returned without scanning the content. Similarly, post-call hook did not handle ResponsesAPIResponse.output. Fix: add _responses_api_input_to_str() helper and handle 'responses'/'aresponses' call types in async_pre_call_hook, async_post_call_success_hook (via _completion_response_text_parts), and async_logging_hook. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix(purview): message separator, non-blocking logging_hook, TextChoices type error Three bugs fixed in the Microsoft Purview DLP guardrail: 1. get_prompt_text_for_dlp message separator (base.py) - Previously called get_str_from_messages() which concatenated all message texts with NO separator, so 'end of msg1' + 'start of msg2' became 'end of msg1start of msg2'. - Now joins per-message text with '\n\n' via convert_content_list_to_str(), preserving DLP pattern detection accuracy across message boundaries. 2. logging_hook blocking the event loop thread (purview_dlp.py) - Previously called future.result() which blocked the calling thread (often the event loop thread) for the entire round-trip of two sequential Microsoft Graph API calls (_compute_protection_scopes + _process_content). - Now fires and forgets: when called inside a running loop, schedules the coroutine with loop.create_task(); otherwise spawns a daemon thread. Returns (kwargs, result) immediately in both cases. - Removes unused concurrent.futures.ThreadPoolExecutor import; adds threading. 3. Incompatible assignment type error (purview_dlp.py:180) - mypy inferred 'choice' as TextChoices from the first loop body, then flagged the assignment in the second loop as incompatible with Choices. - Fixed by using distinct loop variable names: text_choice (TextChoices) and chat_choice (Choices). Tests: 7 new tests added covering the separator fix (TestGetPromptTextForDlp) and the non-blocking logging_hook (TestLoggingHookNonBlocking). Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix(purview): suppress API errors in logging-only mode and scan tool-call arguments Three issues fixed: 1. _check_content except block re-raised unconditionally even when block_on_violation=False. The docstring promised 'log only - do not raise' but network/API errors always propagated. Fixed by checking block_on_violation before re-raising; when False, log a warning and continue. 2. async_logging_hook used a single try/except wrapping both the prompt and response audit calls. When the first _check_content (uploadText) raised due to an API error the second call (downloadText) was silently skipped. Fixed by giving each audit call its own try/except so both always run independently. 3. convert_content_list_to_str() only reads message.content, so tool_calls[].function.arguments and function_call.arguments were invisible to the Purview pre-call and post-call scans. An authenticated caller could embed sensitive text in tool-call arguments and bypass DLP. Fixed by: - Adding PurviewGuardrailBase._extract_tool_call_args_from_message() which handles both dict and object-style messages, covering both tool_calls[] arrays and the legacy function_call field. - Updating get_prompt_text_for_dlp() to include those arguments alongside message content (request/prompt path). - Changing _completion_response_text_parts() from @staticmethod to an instance method and adding tool-call argument extraction for ModelResponse choices (response path). Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * chore(ui): restructure pre-built Next.js output to directory-based routing Flat page files (e.g. guardrails.html) replaced by directory-based index.html equivalents (e.g. guardrails/index.html) matching the Next.js App Router output format. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix(purview): comprehensive security hardening — identity spoofing, streaming bypass, token-id gap Four security issues addressed: 1. end_user_id kwargs fallback missing in _resolve_user_id_from_logging_kwargs user_id already fell back to kwargs.get("user_api_key_user_id") when absent from metadata, but end_user_id only checked md.get("user_api_key_end_user_id") with no kwargs-level fallback. Added or kwargs.get("user_api_key_end_user_id"). 2. Streaming responses bypassed post_call blocking async_post_call_success_hook only runs on assembled non-streaming responses. For streaming requests the proxy already delivered all content before the hook ran, so raising HTTPException there had no effect. Added async_post_call_streaming_iterator_hook which buffers the entire stream, assembles it via stream_chunk_builder, runs the Purview DLP check, and only then re-yields chunks via MockResponseIterator. If a violation is detected the exception is raised before any bytes reach the client. The proxy automatically skips async_post_call_success_hook for guardrails that define this method, preventing duplicate scans. 3. Caller-controlled Purview user identity in blocking modes When a LiteLLM API key has no bound user_id the guardrail fell back to metadata[user_id_field], which is supplied by the caller. A caller could set this to any Entra object ID whose Purview policies are more permissive and bypass DLP. Added _resolve_trusted_user_id() that only returns identities from the proxy auth system (user_api_key_dict.user_id, end_user_id, or proxy-injected metadata["user_api_key_user_id"]). Added _resolve_user_id_for_blocking() used by all blocking-mode hooks: tries trusted sources first; if only caller-supplied is available, logs a SECURITY WARNING and still proceeds (backward compat); if nothing resolves, skips with a warning. 4. Token-id prompt DLP bypass When /v1/completions received a pure token-id array prompt, completion_prompt_to_str() returned None and the pre_call hook silently skipped the Purview scan. An authenticated caller could tokenize blocked text and send it without DLP evaluation. The hook now detects this case (raw_prompt present but prompt_text None) and logs a WARNING while letting the request pass through — token-id payloads are opaque at the text layer and cannot be scanned. This makes the gap explicit rather than silent. Tests: 94 total, all passing. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * Revert "chore(ui): restructure pre-built Next.js output to directory-based routing" This reverts commit c70c4303b735bb3885732bd4a0e01997e9571f56. * fix(purview): fail closed on identity spoofing, token prompts, and path encoding Encode Entra user IDs in Graph paths, guard caches with asyncio.Lock, scan Responses API instructions with string input, reject caller-only metadata and token-id completion prompts in blocking mode, and revert unrelated UI HTML restructure from the PR branch. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(purview): use threading.Lock and getattr for LitellmParams - Replace asyncio.Lock with threading.Lock in PurviewGuardrailBase. The cache lock is acquired both from the proxy's main event loop and from short-lived event loops created by the logging_hook thread fallback. In Python 3.10+ an asyncio.Lock is bound to the first event loop that acquires it, so the second loop would silently break audit logging with RuntimeError. All critical sections are in-memory dict ops with no awaits, so a synchronous lock is safe. - Use getattr() on LitellmParams in initialize_guardrail() instead of .get(), which does not exist on Pydantic BaseModel instances and would raise AttributeError at runtime. Tests updated to construct Mock objects with spec= so they reflect the real interface. Co-authored-by: Yassin Kortam <yassin@berri.ai> * refactor(purview): dedupe trust-level user resolution and drop dead code - _resolve_user_id now delegates levels 1-3 to _resolve_trusted_user_id so blocking and non-blocking paths share a single source of truth. - Drop redundant event_hook override in MicrosoftPurviewDLPGuardrail.__init__ (initialize_guardrail already forwards event_hook=litellm_params.mode). - Drop unused self._logging_only attribute; blocking is controlled by the block_on_violation argument passed to _check_content. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview): fail-closed on responses API transform error; avoid duplicate audit calls Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview): fail-closed blocking DLP; revert directory-based UI HTML Blocking hooks now require UserAPIKeyAuth user_id/end_user_id only (no spoofable metadata), re-raise Responses API transform errors, scan streamed text completions, and reject requests with no bound identity. Reverts the accidental directory-based Next.js output from cc47081 (c70c4303b7). Co-authored-by: Cursor <cursoragent@cursor.com> * Remove dead code in purview_dlp: _resolve_user_id_for_blocking never returns falsy The method either returns a non-empty trusted user id or raises HTTPException, so the 'if not user_id' guards in async_pre_call_hook and async_post_call_success_hook were unreachable. Tighten the return type to str and drop the dead checks to make the fail-closed behavior explicit. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview): exclude caller-controlled end_user_id from blocking DLP Blocking Purview checks now use only API-key/JWT-bound user_id, not end_user_id populated from request user/metadata/safety_identifier. Co-authored-by: Cursor <cursoragent@cursor.com> * style(purview): apply Black formatting to base.py Co-authored-by: Cursor <cursoragent@cursor.com> * fix(purview): use post-await timestamp for cache TTL Capture the timestamp after the network call completes when storing it as the cache freshness marker, so the effective TTL reflects when the response was actually received rather than when the request started. Under high network latency the previous behavior shortened the effective cache lifetime. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview_dlp): fail closed when stream_chunk_builder returns None stream_chunk_builder can return None (e.g., when ChunkProcessor filters all chunks), causing both isinstance checks to fail and the buffered chunks to be released without DLP scanning. Explicitly fail closed in that case by raising an HTTPException so the streaming DLP guardrail does not bypass policy enforcement. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview_dlp): resolve user_id before buffering stream Co-authored-by: Yassin Kortam <yassin@berri.ai> * merge main (#28629) * test(vcr): classify cache verdicts, detect live calls, surface cost leaks Convert the per-test VCR verdict line from a single 'NOOP / HIT / MISS / PARTIAL' tag into a classified outcome that distinguishes the cases that silently bill the live API on every CI run from the ones that don't: HIT pure replay PARTIAL mixed replay + new recordings MISS:RECORDED new cassette saved to Redis (cached next run) MISS:OVERFLOW cassette > MAX_EPISODES_PER_CASSETTE; persister refused to save; re-bills every run MISS:NOT_PERSISTED test failed; save_cassette skipped; re-bills NOOP VCR-marked but no HTTP traffic (mocked elsewhere) UNMARKED:LIVE_CALL test bypassed VCR AND opened a TCP connection to a known LLM provider host -> wasted spend UNMARKED:NO_TRAFFIC test bypassed VCR but didn't call out The UNMARKED:LIVE_CALL signal is what converts 'this test probably hits live' into 'this test connected to api.openai.com'. We install a socket.connect / socket.create_connection wrapper for the duration of each non-VCR-marked test and record any outbound TCP to a known LLM provider hostname. The probe sits below the httpx layer so vcrpy and respx (which both patch above the socket) are unaffected. Replace the file-level _RESPX_CONFLICTING_FILES blacklists in the llm_translation and local_testing conftests with per-item respx detection in apply_vcr_auto_marker_to_items. A test now skips VCR when it actually carries @pytest.mark.respx or has respx_mock in its fixture chain - not just because some other test in the same file imports MockRouter. Items skipped by skip_files are split into respx_conflict (real conflict, the module wires up respx) vs file_opt_out (dead skip- list entry whose module never touches respx) so the session summary makes pruning obvious. Stabilize the AWS SigV4 fingerprint: the Authorization header on Bedrock requests rotates its Credential date and Signature on every call, which previously pushed every Bedrock test past the 50-episode overflow threshold. Extract the access-key id only ('aws-sigv4:AKIA...') so two requests with the same identity match. Always emit verdict logging when VCR is active (set LITELLM_VCR_VERBOSE=0 to opt back into the legacy quiet mode). Add a session-end classification summary that lists overflow tests, unmarked live-call tests, and the skip-reason breakdown. Wire the live-call probe + summary hook into every test directory that already uses the Redis-backed VCR cache (audio_tests, guardrails_tests, image_gen_tests, litellm_utils_tests, llm_responses_api_testing, llm_translation, local_testing, logging_callback_tests, ocr_tests, pass_through_unit_tests, router_unit_tests, search_tests, unified_google_tests). Add tests/llm_translation/test_vcr_classification.py covering the verdict classifier, skip-reason tagging, AWS SigV4 fingerprint stability, live-host classification, and session summary rendering. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(vcr): drop dead 'from respx import MockRouter' imports These seven test files were on _RESPX_CONFLICTING_FILES, which made the auto-marker skip them entirely. Inspecting the source shows the only respx artifact is a top-level 'from respx import MockRouter' that no test ever uses - no @pytest.mark.respx, no respx_mock fixture, no respx.mock context manager. The import is dead code left over from a previous mocking pattern. Now that apply_vcr_auto_marker_to_items detects respx per-item via the marker / fixture chain ( |
||
|
yuneng-jiangandGitHub
|
574ee7526d |
test(streaming): tolerate Vertex 429 wrapped in MidStreamFallbackError (#28669)
Streaming 429s are wrapped in MidStreamFallbackError so the Router can fall back; the existing 'except litellm.RateLimitError: pass' in test_vertex_ai_stream no longer matches, causing the generic pytest.fail branch to fire when upstream Vertex returns 429. Add a sibling except for MidStreamFallbackError that only swallows it when e.original_exception is a RateLimitError, so unrelated streaming failures still fail the test. |
||
|
|
1b141bc588 |
fix(bedrock): decouple STS region from Bedrock aws_region_name (#28245)
* fix(bedrock): decouple STS region from Bedrock aws_region_name STS AssumeRole now resolves signing region from aws_sts_endpoint (parsed host) or AWS_REGION/AWS_DEFAULT_REGION instead of aws_region_name, fixing air-gapped cross-region Bedrock setups and endpoint/signature mismatches. Co-authored-by: Cursor <cursoragent@cursor.com> * test(bedrock): add regression coverage for _build_sts_client_kwargs Parametrize _resolve_sts_region and _build_sts_client_kwargs matrix cases, and assert IRSA/web-identity paths use aligned STS endpoint and region_name. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(bedrock): tighten STS region helpers and drop redundant web-identity endpoint synthesis Co-authored-by: Cursor <cursoragent@cursor.com> * test(bedrock): cover FIPS, GovCloud, and China STS endpoints Addresses greptile P2: regex sts(?:-fips)? supported sts-fips hosts but was not exercised by the parametrized parse test. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
|
a3c953ed4e |
style: apply black formatting to fix lint CI (LIT-3274) (#28639) (#28641)
* fix(bedrock): strip bedrock/ prefix and URL-encode ARNs in get_bedrock_model_id for invoke path The invoke path (used by /v1/messages → Anthropic SDK / Claude Code) called get_bedrock_model_id() which, when falling back to the raw model string, did not strip the 'bedrock/' routing prefix and did not URL-encode ARNs. For a model like: bedrock/arn:aws:bedrock:us-east-1:<ACCOUNT>:inference-profile/global.anthropic... the URL built was: /model/bedrock/arn:aws:bedrock:…/invoke-with-response-stream ❌ Bedrock returned a JSON error body. LiteLLM's AWSEventStreamDecoder passed those bytes into botocore's EventStreamBuffer which expects binary event-stream framing. Checksum validation failed on the JSON prelude (0x223a7b22 == ':{"') producing a misleading botocore.eventstream.ChecksumMismatch instead of the actual Bedrock error. Fix: strip 'bedrock/' (and 'invoke/') routing prefix from model string, then URL-encode if the result is an ARN — matching what the converse path already does in converse_handler.py. Fixes: LIT-3274 * fix(bedrock): use strip_bedrock_routing_prefix to handle compound prefixes Address greptile review: the original fix used a loop with break, so bedrock/invoke/arn:... only stripped bedrock/ leaving invoke/arn:... which is not an ARN → fell through to .replace('invoke/','',1) → bare unencoded ARN → same malformed-URL bug. strip_bedrock_routing_prefix() iterates without break, correctly stripping bedrock/ then invoke/ in sequence. Also adds test case for the compound-prefix scenario. * style: apply black formatting to fix lint CI (LIT-3274) --------- Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: LiteLLM Bot <bot@berri.ai> |
||
|
|
9600fda2cc |
fix(sagemaker): send native Cohere embed payload to Cohere SageMaker endpoints (#28613)
* fix(sagemaker): use Cohere embed payload for Marketplace endpoints
SageMaker embedding only special-cased Voyage; every other endpoint received
HuggingFace TGI `{"inputs": [...]}`. AWS Marketplace Cohere containers expect
the native Cohere embed payload (`texts`, `input_type`) and reject the HF
shape with `422 EmbedReqV2.inputs is of type string but should be of type
Object`.
Add `SagemakerCohereEmbeddingConfig` that reuses Bedrock/Cohere request and
response transforms, and route SageMaker endpoint names containing `cohere`
or a Cohere embed model fragment (`embed-multilingual`, `embed-english`,
`embed-v3`, `embed-v4`) to it. Supports `input_type`, `dimensions`, and
`encoding_format`. Voyage and HuggingFace SageMaker endpoints are unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(sagemaker): simplify cohere detection and align with file conventions
- Detect Cohere SageMaker endpoints with a single `"cohere" in model.lower()`
check, mirroring the existing Voyage branch instead of a separate helper
function and marker constant.
- Drop instance caches of sub-configs; instantiate `BedrockCohereEmbeddingConfig`
/ `CohereEmbeddingConfig` per call to match the existing pattern in
`BedrockCohereEmbeddingConfig._transform_request`.
- Match `SagemakerEmbeddingConfig`'s signatures, defaults, and `Any` typing for
`logging_obj`; collapse the input-normalization helper inline.
- Inline `transform_embedding_response` input lookup; no behavior change.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(sagemaker): restore provider-supported embedding params after map
Cohere input_type is advertised in get_supported_openai_params but was
filtered out of non_default_params by OPENAI_EMBEDDING_PARAMS before
map_openai_params ran. Merge supported params from passed_params after
map (same path Greptile flagged). Handle input_type explicitly in
SagemakerCohereEmbeddingConfig.map_openai_params and add an integration
test through get_optional_params_embeddings.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(embeddings): only restore non-OpenAI supported params after map
The post-map restore loop must skip OPENAI_EMBEDDING_PARAMS so mapped
fields (e.g. dimensions -> output_dimension) are not duplicated under
their OpenAI names. Align SageMaker embedding import order with sibling
files and add a regression test for dimensions mapping.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(sagemaker): avoid double post_call on Cohere embedding response
Greptile review on #28613 caught that `CohereEmbeddingConfig._transform_response`
calls `logging_obj.post_call` internally. The SageMaker embedding handler
already calls `post_call` once before invoking the transform, so the Cohere
SageMaker path fired callbacks, cost calculators, and log handlers twice
per request.
Extract the parsing body of `_transform_response` into
`_populate_embedding_response` (pure extract-method, no behavior change
for existing Cohere direct or Bedrock Cohere paths, which keep calling
`_transform_response`). Have `SagemakerCohereEmbeddingConfig` call the
new helper directly so it parses the response without re-logging.
Add a regression test asserting `logging_obj.post_call` is not invoked
by the SageMaker Cohere transform.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
ryan-crabbe-berriandGitHub
|
643989989f |
chore(test): remove dead old Playwright e2e suite (#28632)
The Playwright suite under tests/proxy_admin_ui_tests/e2e_ui_tests/ is no longer wired into CI (only test_*.py is globbed) and every active spec is duplicated by ui/litellm-dashboard/e2e_tests/tests/ (login, auth redirect, search users, internal user list). team_admin.spec.ts was entirely commented out. Removing the directory plus its only-used-here playwright config, package.json/lock, and utils/login.ts keeps the canonical suite under ui/litellm-dashboard/e2e_tests/ as the single source of truth. |
||
|
yuneng-jiangandGitHub
|
f62ae93e13 |
test(proxy): behavior-pinning matrix for tier-2/3 key + team management endpoints (#28620)
* test(proxy): add create_scratch_actor harness helper
Adds create_scratch_actor() to the management behavior-suite conftest and
extends create_scratch_team() with team_member_permissions / models kwargs,
needed by the PR3 team-key-permission and team-model matrices. The new
helper mints a scratch-prefixed user + verification token (+ org
memberships), all reclaimed by the existing scratch-prefix teardown.
* test(proxy): pin /key block, unblock, health, aliases behavior
Adds behavior-pinning matrices for POST /key/block, POST /key/unblock,
POST /key/health, and GET /key/aliases. Pins that the management-route gate
401s ORG_ADMIN-role callers before _check_key_admin_access runs, the
block/unblock round-trip on the blocked column, missing-key 404, and the
_apply_non_admin_alias_scope visibility rules for /key/aliases.
* test(proxy): pin /key/bulk_update + /team/key/bulk_update behavior
Adds behavior-pinning matrices for POST /key/bulk_update (PROXY_ADMIN-only;
ORG_ADMIN stopped 401 at the route gate, INTERNAL_USER-role 403 at the
handler) and POST /team/key/bulk_update (team-member-permission gate keyed
on KEY_UPDATE). Pins batch semantics: empty/over-cap 400, per-key failure
isolation into failed_updates, all_keys_in_team broadcast, and no-keys 404.
Adds an optional key_alias arg to create_scratch_key for multi-key scenarios.
* test(proxy): pin /key SA-generate, v2-info, reset-spend behavior
Adds behavior-pinning matrices for POST /key/service-account/generate
(team-membership + team-member-permission gating; SA keys carry no user_id),
POST /v2/key/info (per-key _can_user_query_key_info silently drops invisible
keys), and POST /key/{key}/reset_spend (PROXY_ADMIN or team admin only;
missing key 404, reset-value 400). Pins that ORG_ADMIN-role callers are
stopped 401 at the management-route gate on the two non-info routes.
* test(proxy): close PR1/PR2 key-side deferred coverage gaps
Closes the four key-side gaps deferred from PR1/PR2:
- 404 on missing key for /key/update and /key/delete (not 401/403)
- denied /key/update leaves max_budget/tpm_limit/rpm_limit untouched
- /key/regenerate enforces litellm.upperbound_key_generate_params (#26340)
- /key/list key_alias substring vs exact (admin-only) + team_id filter,
and a non-admin filtering a foreign team is 403
* test(proxy): pin /team block, unblock, available, filter/ui, members/me
Adds behavior-pinning matrices for POST /team/block + /team/unblock
(management-route gate fronts _verify_team_access; reachable only by
PROXY_ADMIN and an org admin of the team's own org), GET /team/available
(default empty path), GET /team/filter/ui (route-gated PROXY-ADMIN-only
despite the handler having no gate), and GET /team/{team_id}/members/me
(caller resolves its own membership; non-member 404, no-user_id key 400).
* test(proxy): pin /team model add/delete + permissions endpoints
Adds behavior-pinning matrices for POST /team/model/add + /team/model/delete
(route-gated PROXY-ADMIN-only; missing team 404), GET /team/permissions_list +
POST /team/permissions_update (self-managed; proxy/team/org admin pass), and
POST /team/permissions_bulk_update (PROXY_ADMIN-only). Pins the deliberate
divergence that the available-team self-join grants read access via
permissions_list but never write access via permissions_update.
* test(proxy): pin /team delete, bulk_member_add, v2/list, daily/activity
Adds behavior-pinning matrices for POST /team/delete (per-team
_verify_team_access; batch aborts whole on a missing id), POST
/team/bulk_member_add (route-gated PROXY-ADMIN-only; empty/over-cap 400),
GET /v2/team/list (_enforce_list_team_v2_access — bare query 401s regular
users, org-scoped for org admins) and GET /team/daily/activity (non-member
team_ids filter 404, the VERIA-43 fix).
* test(proxy): add route-coverage gate + close team org-relocation gap
Adds test_route_coverage.py (PR3.M1): parses every @router route literal
from the two management-endpoint source files and asserts each is exercised
by >=1 behavior-suite scenario — a permanent regression guard for future
routes. Closes the last PR1/PR2 deferred gap: the /team/update org-relocation
allowed branch, exercised by a dual-org-admin minted via create_scratch_actor.
test_team_model uses literal route URLs so the coverage parser resolves them.
* test(proxy): bound plain route params to one path segment in coverage gate
Plain path params ({team_id}) now compile to [^/?]+ instead of [^?]+, so a
parameter cannot span '/'. Starlette ':path' params still match across '/'.
Keeps the route-coverage guard from falsely reporting a future multi-segment
route as covered. All 37 routes remain covered.
|
||
|
yuneng-jiangandGitHub
|
985574b6be |
fix(check_licenses): read PEP 639 license-expression metadata (#28529)
The dependency license checker only read the legacy free-text `info.license` field from PyPI. Packages that adopt PEP 639 publish their license as an SPDX expression in `info.license_expression` and leave the legacy field null, so the checker reported "Unknown license" and failed CI for every newly-bumped PEP 639 dependency. `get_package_license_from_pypi` now resolves the license in order: `license_expression`, then legacy `license`, then the `License :: OSI Approved :: ...` trove classifiers. `is_license_acceptable` splits compound SPDX expressions on the uppercase OR/AND operators (case-sensitive, so the lowercase `-or-later` inside an identifier is not mistaken for an operator) and strips `WITH <exception>` suffixes, requiring every component to be acceptable. Free-text license blobs are detected and fall back to the original whole-string matching. The `black` and `pydantic-settings` entries in liccheck.ini that existed solely to work around this now resolve correctly on their own and have been removed. |
||
|
Mateo WangandGitHub
|
b0b25ae4b9 | Include team alias in CLI JWT token (#28621) | ||
|
+1
|
e9f0eddbd1 |
Litellm oss staging 2 (#28582)
* fix(anthropic): handle empty streaming tool calls (#28549) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * [Feature][Bug Fix] Decouple Azure OpenAI Deployment ID from model name via base_model to fix gpt5 model routing (#28490) * feat(azure): decouple deployment ID from model name via base_model Azure OpenAI deployments have arbitrary names (deployment IDs) that may not match the underlying model. Previously, model-type detection (o-series, gpt-5, etc.) relied on substring matching against the deployment name, causing misrouted configs and rejected params when deployment names were non-standard (e.g. 'my-deployment-id' for gpt-5.2). This change extends the existing base_model field to drive model-type detection, config selection, supported param resolution, and param mapping throughout the Azure call path: - _get_azure_config() uses base_model for is_o_series/is_gpt_5 checks - get_provider_chat_config() threads base_model for Azure - get_supported_openai_params() accepts and uses base_model - get_optional_params() accepts base_model and passes it to all Azure config method calls (get_supported_openai_params, map_openai_params) - azure.py completion handler uses base_model for GPT-5 detection - Config internal methods (e.g. is_model_gpt_5_2_model) now receive base_model so features like logprobs are correctly enabled Fully backward compatible - when base_model is unset, behavior is identical. Existing o_series/ and gpt5_series/ prefix workarounds continue to work. Usage in proxy config: model_list: - model_name: my-gpt5 litellm_params: model: azure/my-deployment-id model_info: base_model: azure/gpt-5.2 Fixes: non-standard deployment names like 'prefix-gpt-5.2' rejecting logprobs/top_logprobs despite the underlying model supporting them. * Addressing Greptile comments. * gemini-3.1-flash-lite pricing (#27933) * feat(model_prices): add gemini-3.1-flash-lite pricing with standard/batch/flex/priority tiers * fix pricing * add service tier --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> * fix(openai-responses): strip Anthropic cache_control from Responses API requests (#28431) Squash-merged by litellm-agent from cwang-otto's PR. * Treat None litellm_provider as wildcard in _check_provider_match (#28523) Squash-merged by litellm-agent from adityasingh2400's PR. * fix greptile * fix: use _azure_detection_model in default Azure branch of get_supported_openai_params Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(openai-responses): strip cache_control on compact endpoint as well Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Felipe Garé <90070734+FelipeRodriguesGare@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: withomasmicrosoft <withomas@microsoft.com> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: cwang-otto <chengxuan.wang@ottotheagent.com> Co-authored-by: Aditya Singh <60082699+adityasingh2400@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
|
21a21e01f7 |
fix(responses): use OpenAI SSEDecoder for Responses API streaming (#28566)
* fix(responses): use OpenAI SSEDecoder for Responses API streaming httpx aiter_lines() uses str.splitlines(), which splits on U+2028 inside JSON payloads and silently drops response.completed (no spend log). Use openai._streaming.SSEDecoder (bytes.splitlines before decode) instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(responses): drop redundant SSE prefix strip after SSEDecoder switch SSEDecoder already strips the 'data:' field prefix from each event, so the extra call to _strip_sse_data_from_chunk on sse.data was redundant and could incorrectly mangle payloads whose actual content starts with 'data:'. Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
|
50a3f10a92 |
feat(proxy): persist allowlisted OIDC claims in CLI SSO poll (#28463)
* feat(proxy): persist allowlisted OIDC claims in CLI SSO poll Map CLI_SSO_CLAIM_MAP sources into user metadata and return scalar attribution_metadata from /sso/cli/poll. Build SSOUserDefinedValues in cli_sso_callback so first-time CLI logins can upsert users. Add mock OIDC scripts and tests for claim extraction and poll exposure. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(proxy): document CLI SSO attribution_metadata in client README Co-authored-by: Cursor <cursoragent@cursor.com> * Delete scripts/mock_oidc_server_for_cli_sso.py * Delete scripts/test_cli_sso_claims_e2e.py * fix(ui_sso): preserve claim types and avoid metadata. prefix stripping - Replace _update_dictionary with a local recursive merge so string OIDC claim values that happen to look numeric are not silently coerced to int/float when persisting CLI SSO attribution metadata. - Use a local dot-path resolver in _extract_sso_claim_value so that source claim paths beginning with 'metadata.' are not silently stripped by get_nested_value (which is designed for LiteLLM JWT metadata, not arbitrary OIDC claims). Co-authored-by: Yassin Kortam <yassin@berri.ai> * Remove redundant metadata. prefix strip in _set_nested_metadata_value The _parse_cli_sso_claim_map already strips the metadata. prefix from dest keys before reaching the setter. The duplicate strip in _set_nested_metadata_value was a no-op in normal flow but could mis-place values for dest keys like metadata.metadata.foo. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Fix greptile * Fix ruff * Move CLI SSO user defined values build inside try/except for consistent error handling Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(proxy): enforce restricted SSO group on CLI SSO callback Apply verify_user_in_restricted_sso_group before CLI session completion and user upsert, matching the UI SSO path. Re-raise ProxyException so restricted-group denials return 403 instead of 500. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): replace recursive CLI SSO metadata helpers with iterative merge Use stack-based flatten/merge to satisfy recursive_detector CI. Fix mypy types for UserApiKeyCache and user_id on CLI SSO session completion. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: resolve nested CustomOpenID extra_fields in CLI SSO claim extraction When GENERIC_USER_EXTRA_ATTRIBUTES captures a parent object (e.g. org_info), extra_fields stores it as {"org_info": {"department": "..."}}. A CLI claim map entry using a dotted path like org_info.department would silently fail because the lookup only checked the exact flat key. Fall back to dotted-path resolution on extra_fields before model_dump(). Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(sso): update CLI SSO test for new received_response kwarg and remove redundant 'token' secret fragment Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
|
ef36e89638 |
feat(mcp): Add tool call and tool list support via UI for Oauth mcps (#28454)
* feat(mcp): cache OAuth token client-side so Tools tab loads without re-auth After a user creates an OAuth MCP server and completes the authorization flow, the resulting access token is now stored in sessionStorage keyed by server_id. The MCP Tools tab reads this cached token and includes it as an MCP auth header when listing and invoking tools, so the user never sees an empty tool list. When the session ends (tab close / new browser) an Authorize button re-triggers the flow without leaving the Tools screen. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(ui/mcp): surface listMCPTools 401 errors so auth gate reappears listMCPTools previously swallowed all errors (including HTTP 401) by returning a synthetic { tools: [], error: 'network_error', ... } payload. That made the useQuery retry-on-401 guard and mcpToolsError dead code, so expired OAuth tokens never re-triggered the auth gate. - Throw an enhanced Error with .status attached on non-2xx responses (still preserves the legacy shape for true network failures so the caller can render a generic message without crashing). - Clear the cached OAuth session token when the tools query fails with 401, mirroring callMCPTool's onError handler so the Authorize button is shown again. - Surface mcpToolsError in the existing error banner. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp-tools): stable onSuccess + reuse parsed flow state - Pass stable setOauthToken setter directly as onSuccess to avoid recreating useToolsOAuthFlow's resumeOAuthFlow on every render. - Reuse the already-parsed FLOW_STATE_KEY value (peeked) instead of re-reading and re-parsing sessionStorage in resumeOAuthFlow. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(ui/mcp): restore listMCPTools never-throws contract The previous fix made listMCPTools throw on HTTP errors while still returning a synthetic object on network errors. This inconsistent contract broke existing callers (MCPToolPermissions, MCPAppsPanel, MCPConnectPicker) which inspect result.error / result.message and expect the function to never throw. - Return a normalized { tools: [], error, message, status, ... } object on HTTP errors (instead of throwing) so all callers see a consistent shape and the user-visible error text from result.message is preserved. - Convert the returned error object into a thrown Error inside the one caller that needs it — the useQuery in mcp_tools.tsx — so the 401 retry/onError handlers still trigger and clear the cached OAuth token. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix greptile * fix(mcp): align OAuth header alias lookup with dashboard sanitization Backend auth header resolution now matches x-mcp-{alias} keys produced by the dashboard sanitizer, and the Tools tab re-syncs OAuth tokens when serverId changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): widen auth header lookup types for list_tools Accept legacy str | dict server auth maps and annotate list_tools server_auth_header as Union[str, dict] for mypy. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(ui): extract shared buildCallbackUrl/clearStorage for MCP OAuth hooks Hoist the duplicate buildCallbackUrl and clearStorage helpers out of useToolsOAuthFlow and useUserMcpOAuthFlow into a new shared module src/hooks/mcpOAuthUtils.ts so the two hooks cannot drift if the URL construction or storage cleanup logic needs to change. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(ui): don't gate M2M OAuth MCP servers behind interactive authorize M2M (client_credentials) OAuth servers share auth_type="oauth2" with interactive PKCE servers, but the backend fetches their token internally and they typically lack a user authorization endpoint. Gating tool listing on them rendered an Authorize button that would fail or redirect incorrectly. Detect M2M via the presence of token_url (matching the existing heuristic in mcp_server_edit.tsx) and skip the auth gate. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(ui/mcp): return error shape when listMCPTools JSON parse fails Restore the never-throws contract when response.json() fails on a 2xx body so callers do not receive null and crash on result.tools. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
|
7a93cceb9f |
Add error_description and hint for oauth flows (#28471)
* Add error_description and hint for oauth flows * Fix tests * fix(mcp-oauth): improve redirect_uri errors without leaking internal config Use NoReturn on _oauth_invalid_request, structured errors for BYOK loopback validation, and refactor validate_trusted_redirect_uri to satisfy PLR0915. Keep PROXY_BASE_URL and raw proxy_base_url in server logs only, not in the HTTP 400 body returned to unauthenticated callers. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp-oauth): stop leaking internal proxy origin in redirect_uri 400 body The trusted-redirect-uri rejection helper included the proxy's resolved scheme/host/port (e.g. http://litellm-internal:4000) in both the error_description and as a top-level proxy_origin field. Since the OAuth /authorize endpoint is unauthenticated, any caller could probe with a crafted redirect_uri and enumerate the internal network topology behind a reverse proxy. Keep full diagnostic detail in the server-side warning log (including the computed proxy base) but omit proxy-side values from the HTTP 400 body. Also drop the duplicated origin computation in _raise_trusted_redirect_uri_rejected now that those values are no longer needed by the response. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp-oauth): remove dead userinfo check in redirect_uri validation The first check combined missing netloc with userinfo presence, making the second userinfo-only check unreachable. Split into two distinct checks so each error message reflects the actual failure mode. Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Sameer KankuteandGitHub
|
d96e26064f | Fix conflicts and UI (#28477) | ||
|
harish-berriandGitHub
|
d04373f4ce |
Add granian as a ASGI compliant web server. Provider better throughput stability, (#26027)
* Add granian as a ASGI compliant web server. Provides better stability, 10-20 RPS improvement under standard LT conditions. TODO: Verify poetry lock details and add locust numbers to PR * Update granian version in license_cache.json and pyproject.toml to 2.5.7 * Enhance proxy CLI tests by adding SSL initialization checks for Granian server. Remove Python version skip conditions and implement tests to ensure SSL certificate and key are required for server initialization. * update uv lock to fix granian import error |
||
|
ryan-crabbe-berriGitHubClaude Opus 4.7greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
07bcd2c19e |
test(e2e): forward LITELLM_LICENSE to UI e2e proxy (#28398)
* test(e2e): forward LITELLM_LICENSE to UI e2e proxy The UI e2e job ran without LITELLM_LICENSE, so premium_user was always false in the issued login JWT and premium-gated UI surfaces (Team-BYOK Model switch, etc.) couldn't be driven through the UI. Forward the env var from run_e2e.sh and the CircleCI e2e_ui_testing job, and add a sanity test that decodes the admin storage state token and asserts premium_user=true so the wiring fails loudly if it ever regresses. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Update ui/litellm-dashboard/e2e_tests/tests/proxy-admin/license.spec.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
|
9fcd424318 |
chore(deps): bump deps (#28528)
* build(deps): bump next from 16.2.4 to 16.2.6 in /ui/litellm-dashboard (#27665) Bumps [next](https://github.com/vercel/next.js) from 16.2.4 to 16.2.6. - [Release notes](https://github.com/vercel/next.js/releases) - [Changelog](https://github.com/vercel/next.js/blob/canary/release.js) - [Commits](https://github.com/vercel/next.js/compare/v16.2.4...v16.2.6) --- updated-dependencies: - dependency-name: next dependency-version: 16.2.6 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump protobufjs in /tests/pass_through_tests (#28296) Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.6 to 7.6.0. - [Release notes](https://github.com/protobufjs/protobuf.js/releases) - [Changelog](https://github.com/protobufjs/protobuf.js/blob/protobufjs-v7.6.0/CHANGELOG.md) - [Commits](https://github.com/protobufjs/protobuf.js/compare/protobufjs-v7.5.6...protobufjs-v7.6.0) --- updated-dependencies: - dependency-name: protobufjs dependency-version: 7.6.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump ws from 8.20.0 to 8.20.1 in /tests/pass_through_tests (#28303) Bumps [ws](https://github.com/websockets/ws) from 8.20.0 to 8.20.1. - [Release notes](https://github.com/websockets/ws/releases) - [Commits](https://github.com/websockets/ws/compare/8.20.0...8.20.1) --- updated-dependencies: - dependency-name: ws dependency-version: 8.20.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
yuneng-jiangandGitHub
|
2a5dfcd5bc |
build(deps-dev): bump black to 26.3.1 and apply formatting (#28525)
* build(deps-dev): bump black 24.10.0 -> 26.3.1 * style: apply black 26.3.1 formatting * chore: authorize black 26.3.1 license in liccheck.ini |
||
|
|
0715ed3359 |
build(deps): bump next from 16.2.4 to 16.2.6 in /ui/litellm-dashboard (#27665) (#28524)
Bumps [next](https://github.com/vercel/next.js) from 16.2.4 to 16.2.6. - [Release notes](https://github.com/vercel/next.js/releases) - [Changelog](https://github.com/vercel/next.js/blob/canary/release.js) - [Commits](https://github.com/vercel/next.js/compare/v16.2.4...v16.2.6) --- updated-dependencies: - dependency-name: next dependency-version: 16.2.6 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
yuneng-jiangandGitHub
|
f1abe03ed6 |
fix(docker): restore npm to non_root builder image (#28519)
The non_root builder stage installs `nodejs` but not `npm`. Without `npm`
on PATH, prisma-python falls back to downloading a Node runtime via
nodeenv from nodejs.org, and that downloaded binary fails to load
`libatomic.so.1` — breaking `prisma generate` and the image build.
`npm` was dropped from this apk list in
|
||
|
yuneng-jiangandGitHub
|
3f953dfa96 |
test(vertex_ai): tolerate transient 500 in google maps grounding test (#28503)
test_gemini_google_maps_tool_simple makes live calls to Vertex AI's
Google Maps grounding backend, which intermittently returns
500 INTERNAL ("Please retry") — a transient Google-side failure, not a
LiteLLM bug. The request LiteLLM emits matches Google's published
googleMaps grounding spec field-for-field, and the maps-platform 500
only occurs after Vertex accepts the request.
The test already passes on RateLimitError; treat InternalServerError
the same way so transient Vertex-side failures don't fail CI.
|
||
|
yuneng-jiangandGitHub
|
67e6e5e1df |
test(proxy): behavior-pinning matrix for team management endpoints (#28441)
* test(proxy): behavior-pinning matrix for team management endpoints PR2 (Team Tier-1) of the management-endpoint behavior-pinning effort. Extends the tests/proxy_behavior/management/ harness PR1 built and adds the actor x target-resource authz matrix for the 7 team endpoints: /team/new, /team/info, /team/list, /team/update, /team/member_add, /team/member_delete, /team/member_update. Tests-only, no production code changes. Harness extensions: - actors.py: ORG_B_ADMIN actor (org admin of ORG_B) and TEAM_GAMMA (an ORG_A team with no actor members), so team-targeting endpoints get a clean own / same-org-other / cross-org target axis. - conftest.py: create_scratch_team() raw-seeds target teams without /team/new side effects; the scratch teardown now also strips dangling scratch-team refs from LiteLLM_UserTable.teams. 156 new scenarios; status codes pinned to observed handler behavior. * test(proxy): record mutmut run blockers in PR2 triage doc Attempted a scoped local mutmut run for G5; it did not complete. Record the three concrete blockers in mutmut_triage/pr2-team-tier1.md so the next attempt has a head start: 1. mutmut's mutants/ sandbox is import-shadowed by the worktree source. 2. the legacy mock suite and the real-DB behavior suite cannot share a pytest session (mock suite globally patches prisma_client). 3. the CI mutation-test.yml workflow starts no Postgres, so its stats phase now aborts on the behavior-suite tests PR1 added to tests_dir. mutmut stays a deferred follow-up (as in PR1); the binding pre-merge signal remains the behavior matrix (G1) and the G4 regression-replay. * test(proxy): drop suite README + triage doc, trim test comments Remove the two prose docs from the behavior suite (README.md and mutmut_triage/pr2-team-tier1.md) and tighten the comment blocks on the team test files + harness down to the load-bearing parts (the gate each matrix pins, plus genuinely surprising results). No behavior change — all 286 scenarios still pass. * test(proxy): remove mutmut tests_dir comment |
||
|
|
10bd7406e0 |
feat: add guardrail violation span attributes and fix missing spans on pre-call blocks (#28364)
- Fix missing guardrail child spans when a pre-call guardrail blocks the request before reaching the LLM provider; `async_post_call_failure_hook` now calls `_emit_guardrail_spans_from_request_data` to emit spans from `request_data["metadata"]` regardless of whether `_handle_failure` already fired - Add `guardrail_status`, `guardrail_action`, and `guardrail_violation_categories` as queryable top-level OTEL span attributes so trace backends can filter/group by violation type without parsing the redacted `guardrail_response` blob - Introduce `_emit_guardrail_spans_from_request_data` helper that constructs minimal kwargs from `request_data["metadata"]` and routes through `_create_guardrail_span`, sharing the same dedupe state to prevent double-emitting when both failure hooks fire - Extend `BedrockGuardrail` with `_build_tracing_detail` and `_extract_violation_category_names` which flatten BLOCKED assessments into human-readable category labels (topic names, content-filter types, PII entity types, named regex names) before redaction, and surface Bedrock's raw `action` field via `tracing_detail` - Security: violation category extraction deliberately omits `customWords.match` and unnamed regex `match` values because those fields carry the user-submitted content that triggered the rule; only operator-defined `name`/`type` labels are emitted - Add `violation_categories` and `guardrail_action` fields to `StandardLoggingGuardrailInformation` and `GuardrailTracingDetail` TypedDicts to carry the pre-redaction metadata through the logging pipeline - Add comprehensive test suite covering: guardrail span creation on failure, dedupe between `_handle_failure` and `async_post_call_failure_hook`, per-span status attributes for multi-guardrail sequences, Bedrock category extraction for all policy types, security leak prevention, and end-to-end `CustomGuardrail` violation path Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> |
||
|
|
b55749248d |
fix(logging): recalculate cost after router retry failures (#28476)
* fix(logging): recalculate cost after router retry failures Do not preserve response_cost=0 from failure_handler when processing a successful response; only keep pre-calculated costs > 0 (pass-through). Co-authored-by: Cursor <cursoragent@cursor.com> * test(logging): guard pass-through zero cost; use != 0 preserve check Use != 0 for pre-calculated cost preservation (Greptile feedback). Add tests for zero cost in _hidden_params and for hidden_params overriding failure 0. Co-authored-by: Cursor <cursoragent@cursor.com> * test(vertex): skip google maps tool test on transient upstream 500 The test test_gemini_google_maps_tool_simple calls real Vertex AI with the googleMaps tool, which depends on Google Maps Platform. CI has been failing on local_testing_part1 across many unrelated PRs (including this one and the litellm_internal_staging base) with an InternalServerError 500 from Maps Platform ('Internal server error. Please retry. ...maps- platform-support'), which is an external upstream flake unrelated to the change under test. Catch litellm.InternalServerError and skip (mirroring the existing RateLimitError handler) so transient upstream outages don't block CI. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
b60d4677cd |
fix(vertex_gemma): strip context_management from request body (#28438)
Vertex AI Gemma's chatCompletions wrapper does not understand the context_management parameter (an Anthropic / OpenAI Responses API concept). When callers route this field to a Gemma deployment (e.g. through allowed_openai_params or proxy passthrough), the upstream endpoint would reject the request with an unknown-field error. Drop context_management in VertexGemmaConfig.transform_request, matching the existing pattern used for stream and stream_options. Adds a direct transform_request unit test plus an acompletion-level test that exercises the realistic allowed_openai_params path. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
|
697a90ea77 |
fix: end user logs (#27758) (#28290)
* fix: end user logs
* fix(auth): address PR review feedback on end-user id validation
- Gate DB validation behind litellm.validate_end_user_id_in_db (default
False) so arbitrary client-supplied identifiers still pass through.
- Reuse get_end_user_object / get_user_object / _get_fuzzy_user_object
instead of issuing raw Prisma queries in the auth hot path.
- Consolidate: builder does the resolution once and stores it on the
auth obj; centralized checks reuse it, the outer user_api_key_auth
copy is removed.
- Preserve end_user_id when litellm.max_end_user_budget_id is set so
the default end-user budget can still apply to new customers.
* fix(auth): gate JSON-blob user-id rejection behind validate_end_user_id_in_db
Addresses PR review feedback: the JSON-encoded dict/list rejection in
_coerce_user_id_to_str was unconditionally applied, which would silently
stop tracking spend for deployments passing JSON-encoded user identifiers
on upgrade. Per the backwards-compatibility rule, default-path behavior
changes must be opt-in.
Now only strings that decode to a JSON object/array are dropped when
litellm.validate_end_user_id_in_db is True. Non-string dict/list/tuple
values are still always dropped, since stringifying them produces
unusable "{'device_id': ...}"-shaped spend-log rows.
* fix(auth): route email end-user lookup through get_user_object cache
The email-shaped end-user id branch called _get_fuzzy_user_object directly,
bypassing get_user_object's _should_check_db throttle and user_api_key_cache.
Every unique email would hit an unbudgeted raw Prisma query on the critical
auth path. Collapsing the two calls into one get_user_object invocation
with user_email=end_user_id routes through the cached helper per PR review
feedback.
* fix(auth): keep end-user safety net at user_api_key_auth tail
Krrish flagged that removing the tail-of-user_api_key_auth assignment
was a regression risk: ``_user_api_key_auth_builder`` has multiple
early-return paths (master_key=None, /user/auth, JWT short-circuits)
that bypass the end-user resolution block, so dropping the safety net
silently strips end-user attribution from those paths.
Restore the assignment but route it through resolve_and_validate_end_user_id
so the same validation rules apply. Skip the second pass when the builder
already set an id.
Adds two tests pinning the behaviour: one for the early-return safety
net and one verifying we don't double-resolve when the builder set the id.
Co-authored-by: Dennis Henry <dennis.henry@okta.com>
|
||
|
+10
|
b7e978a5c3 |
Litellm oss staging 04 21 2026 2 (#26569)
* fix(bedrock): use model info lookup for output_config support instead of hardcoded check Replace hardcoded _is_claude_4_6_model() string matching with supports_output_config flag in model_prices_and_context_window.json, accessed via _supports_factory(). This follows the project's established pattern for model capability checks (per AGENTS.md rule #8). Bedrock Invoke now conditionally preserves output_config for models that declare supports_output_config=true (currently Claude 4.6 models), while stripping it for older models to avoid request rejection. Ref: https://github.com/BerriAI/litellm/issues/22797 * fix(vertex_ai): single-flight credential refresh to prevent thundering herd (#26024) * fix(vertex_ai): single-flight credential refresh to prevent thundering herd When GCP credentials expire under high concurrency, all requests simultaneously call credentials.refresh() via asyncify, saturating the 40-thread anyio pool and blocking the proxy for 20+ seconds. This adds: - Per-credential asyncio.Lock in get_access_token_async for single-flight refresh (1 coroutine refreshes, others wait on the lock) - Background refresh when token_state is STALE (usable but near expiry), returning the current token immediately with zero added latency - threading.Lock on the sync get_access_token path - Uses google-auth's TokenState enum (FRESH/STALE/INVALID) instead of reimplementing expiry logic Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review comments - Use asyncio.create_task() instead of deprecated get_event_loop().create_task() - Track in-flight background refresh tasks to prevent duplicate refreshes when multiple STALE-path callers pass through the lock before the first background task completes - Add token validation in the STALE branch (consistent with FRESH/INVALID) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: lazy-import TokenState to avoid breaking when google-auth is not installed Also extract helper methods to bring get_access_token_async under the PLR0915 statement limit (50). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: apply Black formatting to test file and update uv.lock Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove user-provided project_id from log messages (CodeQL log injection) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: avoid leaking token value in error message, log type instead Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: restore uv.lock to match litellm_oss_branch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove project_id from remaining log message (CodeQL log injection) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove remaining project_id from log and error messages Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: reuse cached credentials in VertexAIPartnerModels (#26065) * fix: reuse cached credentials in VertexAIPartnerModels instead of creating new VertexLLM per request VertexAIPartnerModels.completion() was creating a throwaway VertexLLM() instance on every call to get an access token, bypassing the credential cache inherited from VertexBase. This caused a fresh token fetch for every single request, adding significant latency overhead. Fix: call super().__init__() to initialize VertexBase's credential cache, and use self._ensure_access_token() instead of a new VertexLLM instance. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: apply same credential caching fix to VertexAIGemmaModels and VertexAIModelGardenModels Same bug as VertexAIPartnerModels: both classes had `pass` in __init__ instead of `super().__init__()`, and created throwaway VertexLLM() instances per request instead of using self._ensure_access_token(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(fireworks): add glm-5p1 metadata and parallel_tool_calls (#26069) * fix(chatgpt): preserve responses routing and recover empty output (#25403) (#26219) - preserve existing shared backend `mode` when router deployment registration reuses a provider/model key already in `litellm.model_cost` (prevents alias with `mode: chat` from downgrading shared `chatgpt/gpt-5.4` from `responses` to `chat` and triggering 403s on /v1/chat/completions) - teach the ChatGPT Responses parser to recover `response.output_item.done` entries when `response.completed.output` is empty - add defensive /responses -> /chat/completions bridge fallback that reconstructs output items from raw SSE when `raw_response.output` is empty - regression coverage for shared alias routing, empty completed.output parsing, and SSE bridge recovery Closes #25403 Co-authored-by: afoninsky <andrey.afoninsky@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(deps): relax core runtime dependency pins from exact == to ranges When litellm migrated from Poetry to uv (PR #24905, v1.83.1), the core dependency specifications in pyproject.toml changed from Poetry bare-version strings (e.g. openai = "2.30.0") to PEP 621 exact pins (openai==2.24.0). Poetry bare-version strings are actually caret ranges (^X.Y.Z == >=X.Y.Z,<X+1), but PEP 621 == is exact. This means every downstream package that installs litellm as a library dependency is now forced to downgrade aiohttp, pydantic, openai, click, and 8 other common packages to exact old versions. Fix: restore range specifiers for the 12 core runtime dependencies. The optional extras (proxy, proxy-runtime, etc.) are consumed primarily by Docker images where exact pins are appropriate and are left unchanged. The uv.lock file continues to provide exact reproducibility for Docker builds and CI. Fixes: #26154 * Add Rubrik as officially-supported guardrail plugin (#25305) * Add Rubrik as officially-supported guardrail plugin Adds tool blocking and batch logging integration with an external Rubrik webhook service. The plugin validates LLM tool calls against a policy service (fail-open on errors) and batch-logs all requests/responses. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Update Rubrik docs: config.yaml as primary, env vars as fallback Restructures the Quick Start to present config.yaml as the recommended approach with tabbed UI, and environment variables as an alternative fallback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add Rubrik env vars to config_settings reference Fixes documentation validation by adding RUBRIK_API_KEY, RUBRIK_BATCH_SIZE, RUBRIK_SAMPLING_RATE, and RUBRIK_WEBHOOK_URL to the environment settings reference table. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add fallback message when blocking service returns empty explanation Prevents whitespace-only violation message when the tool blocking service blocks tools but returns an empty content field. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(ocr): add Reducto parse OCR support (#26068) * feat(ocr): add Reducto parse OCR support * fix(reducto): address OCR review feedback * chore: refresh uv lockfile * Revert "chore: refresh uv lockfile" This reverts commit 47200c0e603275108335aee852d0a96586165337. * Fix failing tests * Fix code qa * Replaced the async client violation * Replaced black formatting * Fix failing tests * Fix failing tests * Fix failing tests * Fix failing tests * Fix tests * Fix vertex ai cred test * Fix test * fix(xai): normalize usage total_tokens for prompt caching xAI can return total_tokens inconsistent with prompt_tokens + completion_tokens when caching is enabled. Align with OpenAI-style usage so shared LLM tests and downstream consumers see coherent totals. Apply to non-streaming responses and streaming usage chunks. Made-with: Cursor * Fix stale Vertex token refresh fallback * Fix OCR zero credit and Bedrock support checks * Fix OCR and Fireworks capability handling * fix: evict completed background refresh tasks from _background_refresh_tasks Completed asyncio.Task objects were never removed from _background_refresh_tasks. In long-running proxies with many distinct credential keys the dict grows indefinitely, retaining references to finished tasks and their results. Fix: - Pop the existing (done) entry before creating a replacement task. - Attach a done_callback to each new task that removes its entry from the dict once the task finishes (success or failure). Tests: - test_background_refresh_task_removed_after_completion: verifies the done-callback cleans up a single entry after the task completes. - test_background_refresh_tasks_no_accumulation_across_many_keys: drives 20 distinct credential keys and confirms the dict is empty after all background refreshes finish. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: guard asyncio.create_task in RubrikLogger.__init__ against missing event loop asyncio.create_task() raises RuntimeError when called outside a running event loop. Wrap the call in a try/except RuntimeError so that RubrikLogger can be instantiated in synchronous contexts (e.g. during startup, testing) without crashing. The periodic_flush background task simply won't start in those cases; it starts normally when the constructor is called inside an event loop. Add a test that verifies instantiation outside an event loop does not raise (does not patch asyncio.create_task). Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: preserve async batch and reauth coordination * Fix mypy * Fix xAI usage and Fireworks parallel tool params * Fix Rubrik batch drain and SSE recovery mutation * Fix router mode preservation and Rubrik batch flushing * fix(responses): merge text-only items with output items in SSE recovery When recovering output from raw SSE, OUTPUT_ITEM_DONE and OUTPUT_TEXT_DONE events were treated as mutually exclusive fallbacks. If a stream emitted OUTPUT_ITEM_DONE for some output indices and only OUTPUT_TEXT_DONE for others, the text-only items at the missing indices were silently dropped. Merge both dicts before returning, with OUTPUT_ITEM_DONE entries taking precedence at any shared index (preserving the existing behavior covered by test_transform_response_preserves_output_item_when_text_done_arrives_later). Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(rubrik): preserve events on batch send failure Previously, _log_batch_to_rubrik swallowed all HTTP errors and exceptions, and the parent flush_queue unconditionally drained the queue afterwards. On Rubrik 5xx responses, network errors, or timeouts the in-flight events were silently dropped without ever being delivered. - Re-raise from _log_batch_to_rubrik so failures surface to the caller. - In CustomBatchLogger.flush_queue, catch exceptions from async_send_batch and leave the queue intact for retry on the next flush. Existing loggers that override flush_queue (e.g. Datadog) or that swallow their own errors inside async_send_batch (e.g. Langsmith, GCS, Argilla) are unaffected. - Tests now assert events are preserved on HTTP errors, network errors, and that mid-flush appended events are also preserved on failure. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(chatgpt/responses): strip whitespace before parsing SSE chunks _parse_sse_json_chunk in ChatGPTResponsesAPIConfig passed the raw chunk directly to _strip_sse_data_from_chunk, which only matches the 'data:' prefix at position 0. Chunks with leading whitespace (e.g. ' data: {...}') were returned unchanged and silently failed JSON parsing, dropping the contained event. Mirror the existing fix in LiteLLMResponsesTransformationHandler._parse_raw_sse_chunk by calling chunk.strip() before stripping the SSE prefix. Adds a regression test using whitespace-padded data: lines and verifies that the response.output_item.done payload is recovered into the final ResponsesAPIResponse output. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(rubrik): override flush_queue so a single snapshot drives send and drain Previously RubrikLogger relied on CustomBatchLogger.flush_queue, which captured len(self.log_queue) separately from the snapshot taken inside async_send_batch. Although both happen without an intervening await today (so they agree in practice), they are semantically disconnected: a future refactor that adds an await between the two captures, or that changes the async_send_batch contract, could cause the parent to delete a different number of items than were actually sent and trigger duplicate deliveries to Rubrik. Override flush_queue on RubrikLogger so a single snapshot drives both the HTTP POST and the queue truncation. async_send_batch is preserved for direct callers/tests but no longer participates in the canonical flush path. Existing tests (including the one that explicitly invokes the base CustomBatchLogger.flush_queue path) still pass. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: register reducto/parse-v3 and reducto/parse-legacy in active model pricing file Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bedrock): restore output_config forwarding and black formatting Use model-map lookup with _model_supports_effort_param fallback so Bedrock Invoke keeps output_config for Claude 4.6/4.7 when pricing flags are missing. Revert custom_llm_provider=bedrock for supports_output_config checks, fix allowlist test model, and apply black to xai/vertex files failing lint CI. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(greptile): address remaining review concerns - fireworks: resolve supports_reasoning lookup for short model names by also trying the full accounts/fireworks/models/ path in model_cost - ocr_cost: drop reducto-specific guard in shared utility; treat missing pages_processed as zero cost when no per-page pricing is configured - docs: remove reducto/rubrik markdown stubs from this repo (canonical docs live in litellm-docs) * fix(model_prices): register mistral/ministral-8b-2512 Mistral's API now returns model='ministral-8b-2512' when 'mistral-tiny' is requested. Adding the entry so completion_cost can resolve the cost for that response. * fix(greptile): prune async refresh locks and lazy-start rubrik flush - vertex: back `_async_refresh_locks` with a WeakValueDictionary so a per-key Lock is auto-evicted once no coroutine holds it, preventing unbounded growth in deployments with many credential combinations while keeping single-flight semantics intact. - rubrik: defer the periodic flush task to the first log event when the logger is constructed without a running event loop, so low-traffic batches still get drained instead of being silently stranded by a swallowed RuntimeError. * Remove duplicate supports_max_reasoning_effort key in claude-opus-4-7 entries Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(vertex_ai): stabilize background refresh task tracking - Guard background refresh done_callback with an identity check so a stale callback cannot remove a newer task that already replaced it in the tracking dict (done_callbacks are scheduled via call_soon, so a fresh task can be stored for the same credential key before the old callback fires). - Replace WeakValueDictionary with a regular dict for _async_refresh_locks so the per-key asyncio.Lock identity is stable across concurrent callers; otherwise a lock can be GC'd between two coroutines arriving for the same key, breaking single-flight. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: surface OCR pricing gaps and recover OUTPUT_TEXT_DONE in ChatGPT SSE - cost_calculator.ocr_cost: log a warning when pages_processed is reported but no ocr_cost_per_page is configured, instead of silently billing zero via an implicit '(... or 0.0) * pages_processed' fallback. Behavior is preserved (zero cost) so free-tier / unpriced models still work, but configuration gaps are now visible in logs. - ChatGPTResponsesAPIConfig._extract_completed_response_from_sse: also collect response.output_text.done events into a text-only items map and merge them into the recovered output (OUTPUT_ITEM_DONE wins on duplicate output_index), mirroring the LiteLLMResponses handler. This recovers text content when a provider only emits OUTPUT_TEXT_DONE and the final response.completed event has an empty output list. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(cicd): drop obsolete async refresh locks auto-prune test Commit |
||
|
|
37ef8d9059 |
fix(proxy): hydrate wildcard discovery credentials (#28284) (#28419)
* fix(proxy): hydrate wildcard discovery credentials * fix(proxy): constrain wildcard credential hydration Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> |
||
|
yuneng-jiangandGitHub
|
79a5a7abad |
feat(tests): behavior-pinning harness + Key Tier-1 matrix (#28321)
* test(proxy_behavior): scaffold session-scoped async ASGI client + liveness smoke
Slice 2 of the management-endpoints behavior-pinning effort. New top-level dir
tests/proxy_behavior/management/ outside every existing pytest glob.
conftest.py initialises the proxy app once per session against the DATABASE_URL
the harness boots Postgres at, wraps it in httpx.AsyncClient via in-process
ASGITransport. The one smoke test asserts /health/liveliness returns 200, which
exercises the full FastAPI middleware stack against a real app — no mocks.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): connect prisma via real lifespan; key/generate de-risk
Slice 3 of the management-endpoints behavior-pinning effort. The fixture now
enters the real FastAPI lifespan (proxy_startup_event) instead of just calling
initialize() — that is where prisma_client is connected, password migration is
kicked off, and the rest of the startup wiring runs.
Tests pin the loop to the session scope so the AsyncClient created in the
session fixture and the prisma connection opened in the lifespan share the
same loop as the test bodies.
New de-risk smoke: POST /key/generate with the master key returns 200, the
returned sk- token resolves to a hashed row in LiteLLM_VerificationToken, and
the cleartext token is never stored. Proves auth + handler + helper + prisma
all wire together end-to-end against a real Postgres.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): seed 8-actor read-world for the authz matrix
Slice 4 of the management-endpoints behavior-pinning effort. New
``actors.py`` defines the actor enum + seeds an immutable world (2 orgs,
2 teams, 8 users, 8 verification tokens) under the ``behavior-pin-``
prefix so the rows are identifiable in psql and ``_wipe_world`` is
targeted.
Each actor key is created with its cleartext form generated locally and
its hashed form (via ``litellm.proxy.utils.hash_token``) stored in
``LiteLLM_VerificationToken`` — so the real ``user_api_key_auth`` accepts
the cleartext bearer token. Roles, ``team_id``, ``organization_id``, and
the service-account metadata flag are all set on the seeded rows so the
auth layer resolves the same scopes a real proxy would.
The session-scoped ``world`` fixture re-seeds at session start (idempotent
via wipe-then-create), and the smoke test confirms each of the 8 actor
keys can call ``/key/info`` on itself and receive its own row back.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): per-test scratch namespace + targeted delete_many teardown
Slice 5 of the management-endpoints behavior-pinning effort. Adds the
``scratch`` function-scoped fixture: each test gets a uuid4-derived
namespace prefix, tags writes with it (``key_alias``, ``team_alias``,
``user_id``, ``budget_id``), and the fixture teardown ``delete_many``-s
any row whose namespace column starts with that prefix.
Cleanup uses Prisma model methods only (no raw SQL, per CLAUDE.md) and
orders deletes children-before-parents to avoid FK conflicts. The Slice 3
de-risk smoke is migrated onto the same fixture so it stops accumulating
untagged tokens across repeated local runs.
Smoke proves both halves of the contract: one test writes a scratch-tagged
key and asserts it lands; a second test runs after the first's teardown
and asserts no rows in the scratch namespace survived.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): codify G3 (strict-import grep) as a pytest item
Slice 6 of the management-endpoints behavior-pinning effort. Two new tests
walk every .py file under tests/proxy_behavior/ and assert:
* no ``from litellm.proxy.management_endpoints`` import — the suite is
deliberately constrained to the HTTP boundary so it survives handler
refactors;
* no ``mock``/``patch`` on ``user_api_key_auth`` — mocking auth is the
structural failure mode of the existing 11k-line mock suite, and the
point of this harness is that the real auth layer runs.
Codifying G3 as a CI test removes the "did someone forget to check the
PR-description checklist" failure mode.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* style(proxy_behavior): apply black to G3 grep test
Follow-up to 6f588c753b — line-length fixes only, no behavior change.
* test(proxy_behavior): pin /key/generate authz matrix (18 scenarios)
Slice 7 of the management-endpoints behavior-pinning effort. Parametrized
matrix across two axes: actor (8 seeded) × target scope (self, team_alpha
in org_a, team_beta in org_b). 18 scenarios after dropping non-applicable
combos. Whole-suite wall-time stays at ~4.7s (well under the 10-min G2
budget for the eventual CI job).
While pinning, the test surfaced one seed gap: ``_get_user_in_team`` reads
``members_with_roles`` (a JSON list of ``{user_id, role}``), not the plain
``members`` String[]. Both columns are now populated in the seed to match
what the real ``/team/new`` handler would produce.
Expected status codes are intentionally heterogeneous (200, 400, 401)
because the current handler emits different statuses depending on which
check fails first (role gate, team-member-perm gate, "not assigned"
check). Pinning the *observed* codes — not what they "should" be — is
exactly the regression signal we want.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/info authz matrix (24 scenarios)
Slice 8 of the management-endpoints behavior-pinning effort. 8 actors ×
3 target keys (own, OWNER's key in org_a, CROSS_ORG_USER's key in org_b)
covering self-read, same-team-peer read, and cross-org read.
Notable pinned behaviors (intentionally surfaced for review, not "fixed"):
* ORG_ADMIN gets 403 on individual key info even within their own org
— visibility is scoped to "your own keys" + "your team's keys", not
"your org's keys".
* Same-team peers (INTERNAL_USER, UNRELATED_SAME_ORG, SERVICE_ACCOUNT)
DO see each other's keys. Whether that is desired is for the team
to decide; this PR only pins the existing behavior so unintentional
changes flip the matrix red.
Wall-time is unchanged (~4.3s for the slice on its own).
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/list default-visibility matrix (8 scenarios)
Slice 9 of the management-endpoints behavior-pinning effort. For /key/list
the response IS the matrix: each of the 8 seeded actors calls the endpoint
with default filters and the test asserts set-equality between the returned
visible-token set (filtered to seeded tokens only, so unrelated rows can't
flap the assertion) and a pinned expected actor-set.
Pinned default visibility:
* PROXY_ADMIN sees all 8 actors' keys.
* Every other actor sees only their own key — including ORG_ADMIN
(which had broader expectations going in but currently behaves
same-as-internal-user for /key/list defaults) and TEAM_ADMIN (no
team-aggregation without include_team_keys=true).
Future changes that broaden or narrow any single actor's default
visibility will turn this matrix red — exactly the regression signal we
want. Parameter-driven views (include_team_keys, filters) are deferred to
Slice 13 / PR2 follow-up.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/update authz matrix + mutation re-read (21 scenarios)
Slice 10 of the management-endpoints behavior-pinning effort. 8 actors ×
3 target shapes (self-owned, OWNER-scoped in org_a/team_alpha,
CROSS_ORG_USER-scoped in org_b/team_beta) = 21 applicable scenarios.
Each test:
1. Master-key-seeds a fresh scratch key with the target's (user_id,
team_id) scope (so the read-world stays untouched).
2. Has the actor under test POST /key/update flipping ``models`` to
a known marker list.
3. Asserts the status code AND the DB row's ``models`` field — present
when 200, unchanged otherwise — so a handler that silently mutates
on a denied response surfaces red.
Observed gating (pinned, not endorsed):
* PROXY_ADMIN bypasses every check.
* ORG_ADMIN is blocked by an early role gate, always 401.
* Every other (INTERNAL_USER-rolesed) actor hits one of three failure
modes — 403 "user can only create keys for themselves", 403
"only proxy admins, team admins, or org admins", or 401
"team_member_permission_error" — depending on whether they own the
target and whether they're a team admin / member of its team.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/regenerate authz matrix + rotation contract (22 scenarios)
Slice 11 of the management-endpoints behavior-pinning effort. 21 matrix
scenarios (8 actors × 3 target shapes, minus the cross_org/owner combo
that exists in the seed but isn't applicable) plus one smoke for the
``/key/{key:path}/regenerate`` route registration.
On 200 outcomes the test verifies the full rotation contract:
* the regenerate response key differs from the old cleartext,
* the OLD cleartext returns 401 on a follow-up ``/key/info``,
* the NEW cleartext returns 200 on a follow-up ``/key/info``.
On denied outcomes the test verifies the OLD cleartext still works —
catching any handler that mutates the token row on a failed call.
Pinned authz divergence vs /key/update: regenerate routes most denials
through the team-member-perm 401 path rather than the role-gate 403
path. The matrices for both endpoints are now in tree side-by-side, so
any future refactor that "harmonises" the codes will turn one of the two
red.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/delete authz matrix + post-delete contract (21 scenarios)
Slice 12 of the management-endpoints behavior-pinning effort. Mirrors
slices 10/11. On success: cleartext can no longer authenticate
(handles both hard-delete and soft-delete to LiteLLM_DeletedVerificationToken).
On denial: row survives and cleartext still authenticates.
Notable behavior gap with /key/update: same-team peers (internal_user,
unrelated_same_org, etc.) get 403 on /key/delete for OWNER's key — i.e.
cannot delete each other's keys — whereas they CAN read each other's
keys (Slice 8). Delete is stricter than read. Pinned as-is.
Cumulative whole-suite wall-time is 5.9s for all 128 tests on the local
runner — well under the 10-min G2 budget for the CI job in Slice 13.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* ci(proxy-mgmt-behavior): add PR-triggered workflow for the behavior suite
Slice 13 of the management-endpoints behavior-pinning effort. New
workflow ``test-unit-proxy-mgmt-behavior.yml`` fires ``on: pull_request``
for the same branch set every other proxy unit-test workflow watches
(main, litellm_internal_staging, litellm_oss_branch, litellm_**).
It delegates to the existing reusable ``_test-unit-services-base.yml``
with ``enable-postgres: true``, which already provisions a postgres:14
service container and runs ``prisma db push`` against it before pytest
collects. ``reruns: 0`` because a behavior-pinning matrix that needs
reruns is itself a regression — flakes are signal.
``timeout-minutes: 15`` gives generous headroom over the local 5.9s
whole-suite wall-time; the binding G2 budget is 10 min.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* docs(proxy_behavior): G4 regression-replay table for Key Tier-1
Slice 14 of the management-endpoints behavior-pinning effort. Documents
the regression-replay verification methodology + a 12-row table mapping
recent fix-PRs touching key_management_endpoints.py to the catching
scenarios in the PR1 matrix.
One canonical RED→GREEN cycle is captured verbatim —
|
||
|
yuneng-jiangandGitHub
|
e23d06dda4 |
test(realtime): expect session.created as xAI realtime initial event (#28424)
xAI's Grok Voice Agent API now sends session.created as its first realtime event (matching OpenAI), followed by conversation.created. The E2E canary pinned the old conversation.created value and failed. LiteLLM's xAI realtime path is a verbatim passthrough (provider_config is None, raw forwarding), so the event ordering is xAI's own — no transformation on our side. Update the pinned expected value and the now-stale comments to match the current API behavior. |
||
|
|
f99fb5f27f |
chore(ci): merge dev branch (#28314)
* chore(proxy): strict media-type match for form bodies (#27939) * chore(proxy): strict media-type match for form bodies ``_read_request_body`` and ``get_request_body`` routed on ``"form" in content_type`` / ``"multipart/form-data" in content_type``, which match any header containing the literal — ``application/form-json``, ``multiform/anything``, ``application/json; xform=1``. Starlette's ``request.form()`` returns an empty ``FormData`` for any non-canonical type without consuming the body, so the auth-time pre-read saw ``{}`` and skipped the banned-param check while the handler's later ``request.body()`` saw the original JSON payload. Parse the media type per RFC 7231 (substring before ``;``, trimmed, lowercased) and accept only ``application/x-www-form-urlencoded`` and ``multipart/form-data``. Replace both substring sites with the shared ``_is_form_content_type`` helper. Tests pin: case/whitespace/charset variants of the two real types match; ``application/form-json`` and similar substring-match traps fall through to the JSON parse path; real form POSTs continue to route through ``request.form()``. * chore(proxy): extract _is_json_content_type symmetric helper Mirror ``_is_form_content_type`` for the JSON branch of ``get_request_body`` so both classifications share the same media-type normalisation (strip params, trim, lowercase) and any future change to the parsing rules has one place to update. Adds tests for ``_is_json_content_type`` and for ``get_request_body`` covering the canonical JSON / form / unsupported / non-POST paths. * chore(proxy): surface form-parse failures instead of caching empty body Starlette's ``request.form()`` raises ``MultiPartException`` / ``ValueError`` / ``AssertionError`` on malformed multipart input (missing boundary, malformed chunk encoding, etc.). The outer ``except Exception: return {}`` swallowed every form-parse failure and cached an empty parsed body — auth-time pre-reads saw ``{}`` and skipped every banned-param check while a later raw-body re-read in the handler still saw the original payload. Same TOCTOU shape as the substring-match bypass: the auth gate and the handler don't agree on what the body is. Wrap ``request.form()`` in a narrow ``try`` that converts any parse failure to a 400 ``ProxyException``. The outer broad ``except`` is retained for unrelated unexpected errors but no longer covers form-parse-side bypass shapes. Adds a regression test parametrised over the exception classes Starlette can raise from ``request.form()``. * chore(proxy): drop redundant _is_json_content_type test class ``_is_json_content_type`` is a 3-line wrapper around the shared ``_normalize_media_type`` helper. Positive coverage lives in ``TestGetRequestBody.test_json_with_charset_param_parses_as_json``; negative coverage is covered transitively by ``TestIsFormContentType``'s non-form parametrize matrix (anything that isn't a form type falls through to the JSON branch). * chore(proxy): carry ASGI path into WebSocket auth synthetic Request (#27940) ``user_api_key_auth_websocket`` built a synthetic ``Request`` with a two-key scope (``type`` + ``headers``) and set ``request._url = websocket.url``. ``get_request_route`` reads ``scope.get("path", ...)`` and falls back to ``request.url.path`` only when ``path`` is absent. For the WebSocket flow that fallback fires and resolves to the Host-header-derived value (Starlette reconstructs ``websocket.url`` from the Host header), so a malformed Host collapses the resolved route and lets the auth gate compare against the wrong value. Carry the ASGI scope's ``path``, ``root_path``, and ``app_root_path`` into the synthetic scope so the lookup never reaches the fallback on the legitimate path. Regression test pins that the request handed to ``user_api_key_auth`` has ``scope["path"]`` equal to the ASGI scope's path. --------- Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> |
||
|
|
35520adb4f |
fix: serialize guardrail_response to JSON in OTEL traces (#28362)
* fix: serialize guardrail_response to JSON in OTEL traces Guardrail spans previously set the `guardrail_response` attribute via `safe_set_attribute`, which let dict payloads reach the OTEL exporter as Python repr strings. Downstream log pipelines could not parse those as JSON, breaking metric creation from guardrail traces. Serialize `guardrail_response` with `safe_dumps` before setting the attribute, matching how `masked_entity_count` is already handled. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: cover dict-serialization and None-skip for guardrail_response Address Greptile feedback on #28362 — add explicit coverage for the two behavioral guarantees of this fix: - Dict payloads (the OpenAI moderation case in the report) reach the span as a JSON string, not a Python repr. - ``None`` guardrail_response skips the attribute entirely, so no ``"null"`` leaks into traces. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
+6
|
988196911a |
Litellm oss staging 1 (#28337)
* feat: add Xiaomi MiMo-V2.5-Pro and MiMo-V2.5 OpenRouter model entries (#27700) Squash-merged by litellm-agent from TorvaldUtne's PR. * fix(ui): trim whitespace from MCP inspector tool call inputs (#28203) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * gemini-3.1-flash-lite pricing (#27933) * feat(model_prices): add gemini-3.1-flash-lite pricing with standard/batch/flex/priority tiers * fix pricing * add service tier --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> * fix: incorrect /v1/agents request example (#28131) * fix(anthropic): accept dict-shape reasoning_effort from Responses bridge (#28201) * fix(anthropic): accept dict-shape reasoning_effort from Responses bridge Issue #28196 — the Responses->Chat parser (transformation.py:184-200) keeps the full dict as reasoning_effort when summary is set; that branch was added in #25359. But the Anthropic transformation here still guarded on isinstance(value, str), silently dropping the param. Result: callers using the standard Reasoning(effort, summary) OpenAI-shaped object on Anthropic lose thinking entirely (0 reasoning_tokens, no thinking_blocks). Coerce dict -> string before mapping. Same shape tolerance that gpt_5_transformation._normalize_reasoning_effort_for_chat_completion already implements. summary is irrelevant for Anthropic's thinking_blocks. Adds two regression tests: one parametrized over string + dict shapes (with and without summary), one covering unparseable dict inputs (drops silently, no crash). * test(anthropic): add non-adaptive model coverage for dict-shape reasoning_effort Per Greptile feedback on PR #28198: the original regression test only exercised the adaptive (4.6+) path. Add a parametrized test for the non-adaptive branch (claude-sonnet-4-5) verifying that dict-shape reasoning_effort still maps to thinking.type='enabled' + budget_tokens, and that output_config is NOT set on pre-4.6 models. * test(anthropic): convert unparseable-dict test to @pytest.mark.parametrize Per @greptile-apps inline review on PR #28201 — matches the parametrize style of the two adjacent dict-shape tests and produces clearer failure messages (test ID per case instead of one collapsing for-loop). * feat: add pricing entry for openrouter/google/gemini-3.1-flash-lite (#28280) Squash-merged by litellm-agent from ro31337's PR. * fix(router): wrap aresponses streaming iterator for mid-stream fallbacks (#28215) Squash-merged by litellm-agent from cwang-otto's PR. * fix(router): unblock staging — mypy + coverage for aresponses streaming fallback (#28318) Squash-merged by litellm-agent from cwang-otto's PR. * fix(responses): forward timeout on completion transformation path (Anthropic, Bedrock, Vertex) (#28133) Squash-merged by litellm-agent from cwang-otto's PR. * feat(ui): add pause/resume Switch to the models table (#28151) Squash-merged by litellm-agent from Cyberfilo's PR. * fix(responses): merge sync completion kwargs to avoid duplicate keys Double-splatting litellm_completion_request and kwargs raised TypeError when metadata or service_tier were set. Match the async merge pattern. Co-authored-by: Cursor <cursoragent@cursor.com> * Use proxy base URL for CLI SSO form action (#28271) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(tests): add mistral/ministral-8b-2512 to cost map and backfill in conftest Mistral rotated the 'mistral/mistral-tiny' alias to return 'ministral-8b-2512' as the response model, which was missing from the cost map. This caused test_completion_mistral_api and test_completion_mistral_api_modified_input to fail in litellm.completion_cost lookup. - Add mistral/ministral-8b-2512 entry to both the in-tree model_prices_and_context_window.json and the bundled litellm/model_prices_and_context_window_backup.json (mirrors the existing openrouter/mistralai/ministral-8b-2512 pricing). - litellm.model_cost is loaded at import time from the URL pinned to main, so the new backup entry isn't visible at test runtime until it also lands on main. Backfill any entries missing from the remote-fetched map into litellm.model_cost in the local_testing conftest so cost-calculator lookups succeed on this branch. * fix(tests): drop unnecessary del of conftest backfill loop vars * fix(router): harden streaming fallback wrapper for bridge iterators - FallbackResponsesStreamWrapper now uses getattr fallbacks when copying attributes from the source iterator. The bridge path (LiteLLMCompletionStreamingIterator used by Anthropic/Bedrock/Vertex) does not call super().__init__ and is missing response, logging_obj (it uses litellm_logging_obj), responses_api_provider_config, start_time, request_data, call_type, and _hidden_params. Previously, wrapper construction raised AttributeError for any streaming fallback on the bridge path. - _aresponses_with_streaming_fallbacks now deep-copies the litellm_metadata (and metadata) dicts into fallback_kwargs. The primary attempt mutates this dict in place via _update_kwargs_with_deployment, so a shallow copy of kwargs was leaking primary-deployment fields (deployment, model_info, api_base) into the mid-stream fallback request. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(router): use safe_deep_copy for fallback metadata snapshot The ban_copy_deepcopy_kwargs CI check rejects copy.deepcopy() on any variable whose name contains 'kwargs' (incl. fallback_kwargs). Swap the two copy.deepcopy(fallback_kwargs[...]) calls for safe_deep_copy, which handles non-picklable values (OTEL spans, etc.) by per-key deepcopy with fallback to the original reference. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(ci): skip chronically flaky build_and_test integration tests Both tests have been failing on every recent run of build_and_test against this PR's HEAD (1686967, 1688402, 1689993, 1690877), and the same two tests also fail intermittently on unrelated commits and other branches, independent of any code change in this PR (which only touches router fallback wrappers, the Anthropic Responses bridge, and unrelated UI/cost-map files). - tests.test_spend_logs.test_spend_logs: /spend/logs?request_id=... returns 500 even after a 20s wait for the spend log to be written. Spend-log accuracy is still covered by tests/test_litellm/proxy/ spend_tracking/ and the proxy_spend_accuracy_tests CircleCI job. - tests.test_team_members.test_add_multiple_members: /team/info?team_id= ... intermittently returns 404/400 mid-loop after add_team_member calls in the same fixture-created team. Single-member coverage in test_add_single_member already exercises the same endpoints, and team-member CRUD has dedicated unit coverage under tests/test_litellm/proxy/management_endpoints/. Skipping unblocks the build_and_test job until the underlying race in the dockerized integration setup is root-caused. * fix: preserve explicit timeout=0 in responses API handler Use 'timeout if timeout is not None else request_timeout' instead of 'timeout or request_timeout' so an explicit timeout=0/0.0 isn't silently replaced by the default request_timeout. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(ui): guard model_info access in pause Switch with optional chaining * fix(ui): guard model_info access in pause Switch onChange handler Mirror the optional-chaining guard already applied to the isPausing check so a config-model row with a missing model_info cannot throw when the toggle's onChange fires. --------- Co-authored-by: TorvaldUtne <78661304+TorvaldUtne@users.noreply.github.com> Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com> Co-authored-by: cwang-otto <chengxuan.wang@ottotheagent.com> Co-authored-by: Roman Pushkin <roman.pushkin@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: boarder7395 <37314943+boarder7395@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Yassin KortamGitHubYassin KortamClaude Opus 4.7greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
2f9ac77b24 |
fix(proxy): expose Prisma idle/connect timeout + extra DB URL params (#28395)
* fix(proxy): expose Prisma idle/connect timeout + extra DB URL params Operators have reported large numbers of idle Prisma connections that never get closed. The proxy already forwards `connection_limit` and `pool_timeout` to the DATABASE_URL, but had no knob for capping idle or slow connections. Add three new `general_settings` keys that thread through to the DATABASE_URL / DIRECT_URL query string: - `database_connect_timeout` -> Prisma `connect_timeout` - `database_socket_timeout` -> Prisma `socket_timeout` (the main knob for closing idle connections from the LiteLLM side) - `database_extra_connection_params` -> untyped passthrough dict for any other Prisma URL param (`pgbouncer`, `statement_cache_size`, `sslmode`, ...); keys here override LiteLLM defaults. Refactors the duplicated DATABASE_URL/DIRECT_URL param dicts into a single `_build_db_connection_url_params` helper. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update litellm/proxy/proxy_cli.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
|
8acf64e16c |
fix(interactions): never drop streamed text deltas; always emit terminal completion (#28394)
* fix(interactions): never drop streamed text deltas; always emit terminal completion The interactions streaming bridge had two bugs flagged by Greptile on PR #28153: 1. The first OutputTextDeltaEvent (and the second, when no ResponseCreatedEvent precedes the deltas) was consumed to emit a synthetic interaction.created / step.start event, but the chunk's text payload was never forwarded as a step.delta. The text only reappeared in the terminal step.stop, which defeats the purpose of incremental streaming. 2. When the upstream Responses API stream ended via StopIteration without a ResponseCompletedEvent, the iterator emitted step.stop but never the terminal interaction.completed event carrying the full collected text. This refactors the iterator to translate each upstream chunk into a list of events (instead of a single event) and buffers them in a deque. A text delta now expands into [interaction.created, step.start, step.delta] on the first chunk so no token is dropped, and the StopIteration / StopAsyncIteration fallback always flushes a terminal interaction.completed event when one hasn't already been sent. Both behaviors are covered by new unit tests: - test_no_text_token_is_dropped_during_streaming - test_response_created_then_text_delta_emits_step_start_and_delta - test_stop_iteration_fallback_emits_completion_event - test_response_completed_emits_stop_then_completion (no double-emit) Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(interactions): correlate EOF terminal events with stream's interaction id The StopIteration fallback path previously built the terminal step.stop / interaction.completed events with id=None (legacy content.stop) and a memory-address fallback string (interaction.completed), neither of which matched the item_id used by the earlier interaction.created / step.start / step.delta events in the same stream. Downstream consumers correlating events by id would see a mismatch. Persist the interaction id derived from the first upstream chunk (item_id on an OutputTextDeltaEvent, or response.id on a ResponseCreatedEvent) and reuse it when flushing the terminal events on EOF. Author: mateo-berri <277851410+mateo-berri@users.noreply.github.com> * ci(windows): raise UV_HTTP_TIMEOUT to 300s for uv sync The using_litellm_on_windows job has been hitting flaky PyPI download timeouts during 'uv sync --frozen --group dev' — different packages on each rerun (six, pydantic-core), all surfacing the same uv error: Failed to download distribution due to network timeout. Try increasing UV_HTTP_TIMEOUT (current value: 30s). uv's default 30s per-request timeout is too tight for the Windows runner on this project (50+ deps, several multi-MB wheels), so bump it to 300s to let slow individual downloads complete instead of failing the build. * fix(interactions): correlate ResponseCompletedEvent terminal events with stream's interaction id When a stream starts directly with OutputTextDeltaEvent (no preceding ResponseCreatedEvent), interaction.created carries item_id while interaction.completed previously carried response.id from ResponseCompletedEvent. The two ids can differ, leaving consumers that correlate events by id unable to match the start and completion events. Fall back to self._interaction_id (set on the first chunk that derives an id) before response.id, mirroring the EOF terminal path. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
|
718c4637a8 |
feat(mcp): allow native MCP OAuth support for cursor (#28327)
* feat(mcp): allow native MCP OAuth redirect URIs (cursor://) Discoverable OAuth /authorize rejected cursor:// callbacks because validate_trusted_redirect_uri only accepted http/https. Add an allowlisted native path with a built-in Cursor default and optional MCP_TRUSTED_NATIVE_REDIRECT_URIS env for other clients. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): address Greptile native redirect URI review Lowercase paths in normalizer so env allowlist entries match case- insensitively. Tighten wildcard prefix matching to reject sibling paths (e.g. callback-2) unless the prefix ends with /. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): reject query params on native OAuth redirect URIs Greptile: normalization stripped query strings before allowlist compare, so cursor://.../callback?injected=... could pass validation. Reject any native redirect_uri with a query component (same as fragments). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(model_cost_map): add mistral/ministral-8b-2512 entry Mistral rotated the 'mistral/mistral-tiny' alias to return 'ministral-8b-2512' as the response model, which is not in the cost map. This caused test_completion_mistral_api and test_completion_mistral_api_modified_input to fail in completion_cost lookup. Add the entry mirroring the existing openrouter/mistralai/ministral-8b-2512 pricing. * fix(mcp): lowercase default native redirect URIs Make _parse_trusted_native_redirect_uris apply the same lowercasing to built-in defaults as it does to env-var entries. * fix(tests): backfill local model_cost into remote-fetched map litellm.model_cost is loaded at import time from the URL pinned to main, so pricing entries that exist only in this branch (e.g. mistral/ministral-8b-2512, freshly added because Mistral now returns this id from mistral-tiny) are absent at test time and completion_cost lookups raise. Backfill the in-tree backup so cassette-driven cost calculations resolve against the entries that ship with the branch under test. Fixes the local_testing_part1 failures on test_completion_mistral_api and test_completion_mistral_api_modified_input. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com> |