detection_model kept an empty-string base_model, so base_model: '' in a
deployment's litellm_params disabled the o-series/gpt-5 family and
response_format gates (fail-open: v2 served max_tokens where v1 routes
AzureOpenAIO1Config and emits max_completion_tokens). One-token fix to
'deps.base_model or model'; the docstring's 'exactly like v1' claim is
now true. Pinning corpus row uses base_model='' with the non-prefix
family substring deployment name prod-o3-mini per verifier-azure (the
shared openai prefix gate cannot catch it); row fails without the fix.
The old gate compared CHAR count to gemini's 1024-TOKEN cache minimum on
the false premise that one token spans at least one character; BPE tokens
floor at one BYTE, so 700 CJK or 400 emoji chars passed the gate while
v1's token_counter could reach 1024 and fire the cachedContents network
call (silent caching loss + wire-body divergence, flag-on). The gate now
computes a conservative upper bound on whatever v1 could count: UTF-8
byte length of every text/thinking/tool_result/tool_use-args surface
across the WHOLE request (a superset of v1's continuous cached block,
which spans first-to-last marked message inclusive), plus a fixed
per-message margin for token_counter's overhead, and fails closed on ANY
media block when a marker is present (v1 charges 250 tokens per image at
zero text bytes). Quirk rows pin the typed fallback for CJK, emoji, and
unmarked-image-beside-marker shapes (all three fail against the old
gate), and a sub-limit CJK quirk pins that small international markers
still serialize identically to v1.
_to_model_response_openai only handled the response.model-is-None arm;
v1's completion branch (convert_dict_to_response.py:699-711) also rewrites
a pre-set 'provider/model' to 'provider/{wire model}', and v1's openai
handler pre-sets exactly that for every non-openai compat consumer. The
differential was structurally blind (always passed a fresh ModelResponse),
so the corpus now pins both pre-set shapes: prefixed preset re-prefixed to
the wire model, slash-free preset kept verbatim. The prefixed row fails
without the seam fix (verified by stashing it).
The merged google provider uses urllib.parse.urlparse (pure URL string
parsing, no I/O) and the rule-hardener's bare 'import urllib' pattern
matches every submodule from-import semantically, so the gate flagged it.
Ban the actual client module instead; verified the rule still fires on
import urllib.request, from urllib import request, and
from urllib.request import urlopen, and not on urllib.parse.
The merged step() carries both branches' new event arms (wire_chunk +
chunk), tipping C901 to 11. Pull the two tool arms into one helper via an
or-pattern case; the helper narrows on the two-literal tag so no
placeholder arm is needed.
Clean merge (append-only Makefile/semgrep/ruff additions plus the two
gate scripts). CLAUDE.md freshness and file-size gates now run in
make lint-translation; verified against the merged tree in the next
baseline pass.
Both-add unions: pipeline registry rows (azure/azure_ai/azure_ai_anthropic
beside the google rows), ChunkDialect literal now carries azure + gemini,
the thinking-delta placeholder arm keeps gemini split from the
anthropic|openai|azure arm (all placeholder arms are removed by the
ChunkDialect split fix later in this branch). CLAUDE.md scope/tree
hand-unioned for ten providers. DIFFERENTIAL_REPORT.md regenerated on the
merged tree (0 divergent). RESPONSE_FORMAT_SPOOF_MODEL rename merged clean
(identical hunk on both branches).
Append-only registry rows (serializer/parser/dialect openai) plus a
_RAW_GUARDS table the pipeline consults before parse for same-family
providers. Request gate: 17 identical rows against v1 in-process
(map_openai_params + transform_request) and 17 asserted typed
fallbacks. Response gate: 5 identical rows against
convert_to_model_response_object (the live SDK-path normalizer) plus
loud-error rows for unreachable shapes. Stream gate: byte-identical
replays through the real CustomStreamWrapper over SDK chunks, an
SSE-line decode equivalence check, and the pinned usage-tail seam
contract. DIFFERENTIAL_REPORT.md regenerated: 0 divergent rows.
vertex_ai routes through v1's get_vertex_ai_model_route (gemini -> v2
vertex_ai, claude partner route -> v2 vertex_anthropic, everything else
stays v1); gemini (AI Studio) gets its own fork. The vertex OAuth token and
URL come from v1's own envelope helpers in the seam, the raw body carries
EVERY caller-set OpenAI param so unknown shapes fall back typed instead of
being dropped, and ambient globals the pure serializers cannot honor
(vertex_ai_safety_settings, custom_prompt_dict, modify_params) force v1.
parse_response mirrors convert_to_model_response_object (the SDK-path
normalizer; transform_response is dead code there) and rides the
normalized chat-completion body on ChatResponse.wire; the new openai
response dialect emits it unchanged. parse_event normalizes SSE chunks
to the SDK-dump shape and the openai chunk dialect reproduces the
wrapper's stateful decode (first-chunk role, role-strip
provider_specific_fields, empty-delta suppression, wire id pinning,
preserve_upstream extras, choices=[] usage passthrough). The seam
gains the openai usage style (verbatim Usage(**raw)) and wire
id/created/system_fingerprint adoption.
Body assembly = {model, messages, **params} with the original
max-tokens key re-emitted, recursive cache_control strip on tool
parameters (v1 filter_value_from_dict parity incl. the recursion cap),
and fail-closed gates: o-series/gpt-5 families, top_k/thinking/
reasoning_effort, the user param's model-list gate, and
response_format on gpt-4/gpt-3.5-turbo-16k.
Review fixes on the inherited work: typed map/seq narrowing in guard.py
(pyright strict), four new guard reasons for shapes the IR cannot
round-trip (empty tools/stop lists, empty assistant content list,
non-function tool_calls, tool_calls without string arguments), a
wire_chunk placeholder arm in the chunk fold, and lint formatting.
vertex_gemini / gemini / vertex_anthropic requests, responses, and streams:
snapshot == v1-at-HEAD == v2 (canonical JSON), plus a quirk corpus pinning
the 3-way structured-output fork, gemini-3 defaults and id forwarding,
multi-system parts, top_k passthrough on both routes, and the fail-closed
shapes (large cache markers, AI-Studio https media, xhigh effort, blocked
responses, mid-stream error objects).
One generateContent serializer family for the vertex_ai gemini route and AI
Studio (target = the researcher-2 drift list), the vertex claude wrapper on
the bedrock_invoke pattern (model pop, vertex anthropic_version, shared
json-tool spoof constant), gemini response/stream dialects on the inbound
side, and append-only IR extensions (composite stream chunk, per-modality
usage, UrlSource format).
The reference corpus (cases, recorded fixtures, canonical-JSON snapshots for
both bedrock chat routes) is copied verbatim from the translation
characterization branch; each request row asserts snapshot == v1-at-HEAD
(drift guard) == v2 (byte-equal). Responses and streams run full cycles
against v1's transform_response and the real stream decoders inside
CustomStreamWrapper at the parsed-event seam, with the corpus's
fastuuid/time freeze. A quirk corpus pins bedrock-only behaviors against v1
in-process: claude-4.5 cache ttl and parallel tool config, bedrock tool-name
normalization, thinking budget clamp, json_schema-with-effort dropping the
forced choice, tool_choice none gates, and the explained fallbacks
(pdf/file parts, non-Claude models, invoke response_format x reasoning).
Seam tests drive litellm.completion() through respx-intercepted SigV4 calls
for both routes. DIFFERENTIAL_REPORT.md gains the bedrock sections:
126 rows, 0 divergent.
bedrock_converse is the one genuinely distinct wire format: serialize_request
maps the same IR onto messages/system/inferenceConfig/toolConfig/
additionalModelRequestFields with v1's quirks reproduced bit-for-bit (bedrock
tool-name normalization, cachePoint ttl gates, thinking budget clamp,
forced-tool-choice-to-auto under reasoning, blank-assistant drop and
reasoning-first sort, toolUse/toolResult dedupe), plus parse_response (incl.
the json_tool_call properties-unwrap rewrite) and parse_stream pinned at the
parsed-event seam. bedrock_invoke is the anthropic serializer plus envelope
deltas only: pop model/stream, inject anthropic_version, spoof the mapping
model for response_format; response/stream parsers are anthropic re-exports.
The IR gains wire-reported total_tokens; the inbound response/stream
serializers gain a per-provider dialect because v1's outbound shapes differ
per provider (converse omits unset message keys, mirrors raw reasoningContent
deltas, stamps top-level provider_specific_fields on chunks). The engine's
wire_body now also strips converse's additionalModelRequestFields.stream
transform-seam marker, and fold_events joins fold_lines for providers whose
framing is botocore plumbing.
The seam forks completion()'s bedrock branch per BedrockModelInfo route:
AWS auth params split out of the body before parsing, SigV4 signs after
wire_body finalizes the bytes, and any unproven shape falls back to v1.
Off by default; enable via LITELLM_TRANSLATION_V2_PROVIDERS.
Phase 3 of the anthropic port (M3):
- ir.py gains the response IR (ChatResponse, ResponseUsage, finish reasons)
and stream-event union; providers/anthropic parse_response/parse_stream
reproduce v1 exactly for the v2-reachable surface (tool-name reverse
mapping recomputed from the request, json_tool_call rewrite, usage cache
folding, redacted thinking) and fail loudly on block types a v2-sent
request cannot trigger
- inbound/openai_chat serialize_response/serialize_stream mirror v1's
ModelResponse / CustomStreamWrapper shapes chunk-for-chunk (role on first
chunk, tool-index counting, thinking provider fields, finish chunk)
- engine/: pipeline.py (prepare -> send split so the fallback decision
happens before any I/O; async-first, sync is one asyncio.run wrapper at
the seam), http.py (injected HttpPort + ExecuteError values), stream.py
(the ONE accumulator: provider lines -> IR events -> chunk bodies)
- litellm/translation_seam.py (outside the package) adapts deps from litellm
ambient state, ModelResponse/ModelResponseStream envelopes, and owns the
completion() fork; litellm.translation_v2_providers allowlist global
seeded from LITELLM_TRANSLATION_V2_PROVIDERS (off by default, yaml-
configurable via the litellm_settings setattr fallback)
- main.py anthropic branch forks to the seam; streaming and modify_params
traffic stay on v1 (documented follow-ups); once sent, provider errors
raise the v1 exception contract, never a silent re-send
- differential gates: responses (4 full-cycle shapes incl. sanitized tool
names and json_tool_call), streams (3 SSE replays vs the real
CustomStreamWrapper), seam tests over respx (serve, fallback, 429
contract, async path); DIFFERENTIAL_REPORT.md committed as the merge
artifact: 53 rows, 0 divergent
- pyright strict scoped to the folder (litellm/translation/pyrightconfig.json),
0 errors: standardized the exhaustiveness pattern to match-on-Literal-tag
with assert_never AFTER the match (a 'case never:' capture arm is flagged
by strict as unmatchable), restructured narrowing-hostile spots instead of
suppressing
- ruff folder config (litellm/translation/ruff.toml): PLR1702 (preview), C901,
PLR0915, ANN incl. ANN401, PLW0603 + complexity caps; picked up by the
repo-wide ruff run through hierarchical discovery; folder also modernized
to PEP-604/builtin generics and black-formatted (CI formatter)
- semgrep tenet rules (.semgrep/rules/python/translation/): no raise outside
errors.py, no mutation methods/subscript/augmented assignment, no
module-level mutable state (the rule that would have caught audit F2);
rides the existing semgrep workflow; effort maps frozen behind
MappingProxyType and the name-collision loop rewritten without +=
- import-linter contracts in pyproject (forbidden v1-stack imports + downward-
only layers), import-linter==2.3 added to the dev group (uv.lock updated)
- CI: code-quality job gains translation_v2_pyright_strict and
translation_v2_import_contracts steps; 'make lint-translation' runs all
four gates plus the package tests locally in one command
- CLAUDE.md rewritten to match the real tree and conventions
JsonBlob values are created fresh at the parse boundary and each request
serializes once, so the per-tool_use deepcopy protected nothing and was
the largest remaining cost on tool-heavy histories. Bench now stable at
1.39-1.46x v1 at 601 messages (budget 1.5x).
The rewrite already replaced the tracer's O(n^2) folds; what remained was
per-object Expression overhead on the hot path. Three local fixes:
- ir.py gains a per-case fast constructor (precomputed tag/index/fields,
bit-identical instances) used by the of_* factories content blocks are
built through, ~4x cheaper than the stock tagged_union __init__
- the inbound converter returns plain 'value | TranslationError' unions
and builds local lists internally, lifting into Result/Block once at the
public boundary instead of allocating Ok wrappers per content part
- the anthropic serializer identity-checks the Nothing singleton instead
of class-pattern matching per block
Auditor's bench method (tool-call histories, median of 5): 0.82/2.82/8.19ms
vs v1 0.54/2.25/6.03ms at 41/201/601 messages = 1.36x at 601 (budget 1.5x;
tracer baseline was 9.9x).
Replaces the hand-rolled parser with frozen pydantic v2 wire models behind
boundary.parse (arktype calling convention: value or accumulated error,
never raises). Every inbound field is now accounted for: unknown or
unported shapes return a typed unsupported error so dispatch falls back to
v1, never a silent drop (audit F1/F13).
- IR gains cache_control, thinking/redacted_thinking blocks, reasoning
effort, response_format, user, parallel_tool_calls, top_k, url image
sources, tool-result part lists; opaque JSON rides as JsonBlob (no
freeze/thaw churn, audit P3)
- serializer reproduces v1 byte-for-byte on the supported surface:
model-map max_tokens default via injected deps (F3), tool-name
sanitize + history rewrite (F4), input_schema whitelist/coercion (F7),
tool_choice dict forms and https images (F5), empty-content placeholder
(F6), int wire fidelity (F9), drop_params stop/sampling gates (F10),
thinking budgets, response_format output_format vs json_tool_call
strategies including the json_mode transform-seam marker
- O(n^2) folds replaced with groupby/chain merges (P1/P2/P4)
- match-on-Literal-tag + assert_never throughout (F8)
- differential corpus grown 16 -> 46 shapes on current-generation model
ids; parse error table split into unsupported vs boundary classes
The dummy tool dict was inserted into returned bodies by reference, so a
downstream mutation of one response body changed every later request
(pattern-auditor F2, proven cross-request bleed). Build it fresh per call.
Also freeze the serializer registry behind MappingProxyType (F12).
The new tests/test_litellm/translation suite was in no CI shard, so it never
ran and codecov reported 0% patch coverage; add it to the misc unit-test
workflow and the Makefile group so it runs and reports
Allowlist the boundary JSON walkers freeze and thaw in recursive_detector, the
same treatment every other cycle-free JSON/schema walker gets (bounded by input
nesting depth, no cycles possible in JSON)
Drop the unused engine/http.py I/O port; it had no caller in a request-only
slice, so it was dead code with zero coverage. The injected port returns with
the response and stream increment that actually performs I/O
Add failures-as-values tests covering the parser's error branches (malformed
roles, content parts, images, tools, tool_calls, tool_choice) and four more
differential parity cases: system as array, stop as string,
max_completion_tokens, and a tool without a description
Introduce litellm/translation, the hub-and-spoke v2 of core LLM translation
from Mateo's Cycle 4 scope. Inbound parsers map a request schema into a frozen
IR, provider serializers map the IR onto one wire format, and dispatch.route is
the single v1/v2 fork carrying the same-family fast-path predicate
This first slice ports the OpenAI-chat-in to Anthropic-out request path and
proves it differential-green: a corpus runs the v1 AnthropicConfig chain
(map_openai_params then transform_request) and the v2 pipeline over the same
requests and asserts identical normalized JSON across text, system, multi-turn,
stop and stream, tools, every tool_choice form, tool-call round-trips, parallel
tool-result merging, assistant-text-plus-tool-call, images, and the max_tokens
default
Built on Expression (frozen Block and Map, Result, tagged unions); failures are
values, I/O is an injected port, and the package is import-isolated from the v1
stack, enforced by a test that stands in for an import-linter contract. The
anthropic flag stays off, and response and stream parsing, the other inbound
schemas and providers, and wiring route into completion are follow-ups
* feat(bedrock): support aws_bedrock_project_id for bedrock-mantle project association
Adds a litellm_params field to associate bedrock-mantle requests with an
Amazon Bedrock project, sent as the OpenAI-Project header on the
OpenAI-compatible chat and responses paths and as the anthropic-workspace
header on the Anthropic messages paths. This lets a single model entry opt
into a project-scoped data retention mode (e.g. provider_data_share for
Claude Fable 5) while the account-wide setting stays on default.
The param is carried via litellm_params only and is explicitly excluded
from optional_params so it can never leak into a request body.
Fixes#30070
* chore(ui): regenerate schema.d.ts for aws_bedrock_project_id
Generated with npm run gen:api after adding the field to LiteLLM_Params
* fix(proxy): ban client-supplied aws_bedrock_project_id in request bodies
The deployment pins aws_bedrock_project_id so the project's data
retention policy applies to its requests. Without this guard an
authenticated caller could supply the field in the request body and,
since client kwargs win the router merge, run requests under any
project reachable with the deployment's shared AWS credentials.
Adds the field to _BANNED_REQUEST_BODY_PARAMS so it is rejected at the
auth boundary by default while remaining available through the existing
admin opt-ins (allow_client_side_credentials proxy-wide or
configurable_clientside_auth_params per deployment).
* perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay
The GA realtime support added in #27110 made backend_to_client_send_messages
parse every backend frame up to three times for beta clients (OpenAI-Beta:
realtime=v1), build a discarded Pydantic object per frame for logging, and
re-serialize even frames that need no translation. For high-frequency
response.output_audio.delta frames carrying multi-KB base64 payloads, that
serialized CPU work on the hottest relay path drove the latency regression
between v1.83.14 and v1.88.1 for gpt-realtime-1.5 and gpt-realtime-2.
This parses each frame once via _parse_backend_event and threads the dict into
_handle_raw_backend_message, store_message, and _translate_event_to_beta;
short-circuits store_message before the Pydantic build for events not in the
logged set; returns the original event unchanged from _translate_event_to_beta
when no rename applies so the raw frame is forwarded without re-serialization;
and only json.dumps when the type is actually renamed.
* fix(realtime): widen store_message type hint to accept plain dict
The parse-once refactor passes the dict produced by _parse_backend_event into
store_message, but the parameter was typed as str | bytes | OpenAIRealtimeEvents
(a union of TypedDicts), which mypy does not consider compatible with a plain
dict. Add dict to the accepted union; the body already handles it.
---------
Co-authored-by: Miguel Armenta <maarmenta92@gmail.com>
* fix: coerce server_tool_use dict to ServerToolUse in Usage.__init__ (#26153)
* fix: coerce server_tool_use to ServerToolUse in stream_chunk_builder (#26153)
* fix: dict/pydantic-tolerant access in tool_call_cost_tracking (#26153)
* fix: dict/pydantic-tolerant access in anthropic cost_calculation (#26153)
* test: assert ServerToolUse type in existing stream_chunk_builder anthropic web search test
* test: regression test for #26153 (stream_chunk_builder server_tool_use type)
* test: dict/pydantic safety for tool_call_cost_tracking helper
* test: dict/pydantic safety for anthropic web_search cost
* refactor: consolidate _get_web_search_requests into shared cost-calc utils
* test(realtime): use gpt-realtime; openai retired gpt-4o-realtime-preview
OpenAI shut down the gpt-4o-realtime-preview family (incl. the undated
alias) on 2026-05-07, causing the live realtime test to fail with a
4000 invalid_request_error.invalid_model close. gpt-realtime is the GA
successor; switch the live-call tests to it, matching the base branch.
* refactor(types): drop redundant server_tool_use coercion in Usage.__init__
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
mateo-berri