mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-15 18:21:44 +00:00
f5b11b72a6dcc8f4e7a16f00a61bdd124cc171d2
99
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Mateo WangandGitHub
|
b4aee2c7dd |
test(vcr): close out the remaining VCR live-call leaks (#29603)
* Fix remaining VCR live-call leaks * test(vcr): dedupe live-test helpers and drop spurious kwargs Extract the duplicated isVertexQuotaError/runVertexRequestOrSkip Vertex quota-skip helpers into tests/pass_through_tests/vertex_test_helpers.js and the duplicated _skip_live_prompt_caching_test guard into tests/_live_test_helpers.py so each lives in one place. In test_aarun_thread_litellm, build a separate message_data carrying role/content for add_message and a thread_data without them for run_thread/run_thread_stream/get_messages, which no longer receive the spurious message fields. * test(overhead): assert mock transport is exercised in non-streaming and stream tests |
||
|
|
2bbdbfa5c3 |
fix: passthrough endpoints duplicate logs (#29598)
* fix duplicate cost callbacks for anthropic streaming pass-through Two bugs caused _PROXY_track_cost_callback to see stream=True + complete_streaming_response=None on every streaming pass-through request, making the dedup guard in dispatch_success_handlers permanently inactive: 1. pass_through_endpoints.py created the Logging object with stream=False for all requests. _is_assembled_stream_success short-circuits on self.stream is not True, so has_dispatched_final_stream_success was never set and any second dispatch went through unchecked. Fix: set logging_obj.stream = True after stream detection. 2. _create_anthropic_response_logging_payload set complete_streaming_response inside the try block after litellm.completion_cost(), so a pricing error caused an early return without setting it on model_call_details. Fix: set complete_streaming_response before the try block. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix stream * add stream to logging obj * test(pass_through): give mock logging object a real model_call_details dict The anthropic passthrough logging payload now records the assembled response on model_call_details before cost calculation, which requires model_call_details to support item assignment. In production it is always a dict; the existing unit test stubbed the logging object with a bare Mock whose attribute is not subscriptable, so the new assignment raised TypeError. Use a real dict to match the production logging object. * test(pass_through): cover streaming logging-obj stream flag The streaming branch of pass_through_request that marks the logging object as streaming (logging_obj.stream and model_call_details["stream"]) had no unit coverage, so the patch coverage gate flagged it. Add a regression test that drives a streaming pass-through request through pass_through_request and asserts the logging object is flagged as a stream before dispatch. * test(pass_through): cover SSE-response stream flag fallback branch The auto-detected streaming branch of pass_through_request (when a request that was not flagged as streaming returns a text/event-stream response) sets logging_obj.stream and model_call_details["stream"] but had no unit coverage, so the codecov patch gate failed at 60%. Drive a non-streaming pass-through request whose upstream response is SSE through pass_through_request and assert the logging object is flagged as a stream before dispatch. * fix(pass_through): gate complete_streaming_response on stream flag perform_redaction only scrubs complete_streaming_response when model_call_details["stream"] is True. Setting it unconditionally for non-streaming Anthropic pass-through responses left the assembled response unredacted in model_call_details, which is handed to logging callbacks as kwargs when message logging is disabled. Only record it for actual streaming responses so redaction always applies. --------- Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
|
ba2699740c |
feat(pass_through): extend passthrough_managed_object_ids to Azure (#29160)
* feat(pass_through): extend passthrough_managed_object_ids to Azure
Adds managed ID minting/resolution for Azure passthrough endpoints
(/azure/...) alongside the existing OpenAI passthrough support.
Key changes:
- pass_through_endpoints.py: detect azure/azure_ai custom_llm_provider
(string or enum) to set _is_managed_id_provider and _managed_id_provider;
both INPUT and OUTPUT rewrite blocks now fire for Azure.
- llm_passthrough_endpoints.py: forward custom_llm_provider into
create_pass_through_route so it reaches pass_through_request (was None).
- managed_id_rewriter.py: extend _PASSTHROUGH_PREFIX_RE and _canonical_path
to strip /azure/openai prefix and add /v1/ for Azure paths that omit it;
add ("azure", method, path) entries to BUILTIN_OUTPUT_ID_FIELD_MAP for
files and batches endpoints.
- managed_id_codec.py / types/utils.py: supporting codec and enum constant.
- proxy_server.py: register llm_passthrough_router before batches_router to
prevent route collision for /openai_passthrough/* paths.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(pass_through): remove unused imports for ruff F401
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(pass_through): satisfy mypy for optional parsed_body
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(pass_through): compute query params string after managed-ID rewrite
Move requested_query_params_str computation to after the managed-ID
input rewrite block so logging_url reflects the rewritten raw-provider
query params actually sent upstream, instead of the original
managed IDs.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Add support for managed ids for passthrough responses api
* Add support for list batches and list files
* style: run Black on passthrough managed ID files
Fix CI formatting for managed_id_rewriter.py and pass_through_endpoints.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(passthrough): parse json file_object and implement before-cursor pagination
- Parse row.file_object via json.loads when Prisma returns it as a string;
mirrors openai_files_endpoints/common_utils.py so list responses keep all
stored detail fields (status, timestamps, etc.).
- Implement the previously-parsed-but-unused 'before' cursor for list
pagination by flipping fetch order to ascending with a 'gt' bound on
created_at, then reversing rows so the response stays newest-first.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Remove logger
* refactor: split list_passthrough_ids_from_db to fix PLR0915
Extract pagination, fetch, and serialization helpers so the main list
function stays under the statement limit without changing behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: scope passthrough managed ID dedup and list by provider
Validate embedded provider before reusing deduped file/object rows so
OpenAI and Azure cannot share the same managed ID for an identical raw ID.
Filter list responses to rows whose managed IDs decode to the current
provider, with over-fetch scanning when needed.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(managed_id_rewriter): cap pagination trim at effective limit
When raw_limit > 100, fetch_limit is capped at 101 (one extra row to
detect has_more), but trimming with rows[:raw_limit] failed to drop
the sentinel row. Use the capped effective limit instead.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: cross-provider object collision and fail-closed list error handling
Greptile P1: move provider check before access check in _mint_or_reuse_object
so a cross-provider raw ID collision (OpenAI and Azure share the same batch_
ID) falls through to mint a new provider-scoped row instead of raising 404.
Veria-ai medium: _fetch_provider_scoped_list_rows now always returns
(page, has_more) — DB errors break out of the scan loop and return matched
rows so far. list_passthrough_ids_from_db never returns None for a recognised
list route, so the caller can never fall through to the upstream provider.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: namespace passthrough model_object_id by provider to prevent unique violation
Store model_object_id as 'passthrough:{provider}:{raw_id}' instead of the
bare raw ID so OpenAI and Azure can each own a row for the same raw batch ID
without hitting the @unique constraint. Dedup lookup uses the same namespaced
key so it is implicitly provider-scoped and the _managed_id_matches_provider
check is no longer needed on the object path.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: gate list interception on managed_files hook like input/output rewrites
Without the hook no managed IDs are minted so the DB is empty. Intercepting
GET /v1/files without the hook returned an empty list and hid the caller's
real upstream files/batches. Matches the guard used by the input and output
rewrite blocks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: set has_more=True when scan cap is hit with a full final DB batch
When max_scans (20) is exhausted and the last DB page was full-sized,
there are almost certainly more rows beyond the scan window. Track
last_batch_full across iterations so the scan_cap_hit condition sets
has_more=True in that case, preventing silent pagination truncation in
high-mixed-provider pools.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(managed_id_rewriter): scope pagination cursor lookup to caller-owned rows
Prevent a cross-tenant timing oracle by constraining the after/before
cursor row lookup to the caller's owner_filter, and cover the real Azure
responses path form (no /v1/) in tests.
* fix(managed_id_rewriter): align passthrough list metadata with direct GET
Persist upstream file metadata when minting a managed file ID and rewrite
nested batch file IDs before snapshotting the object, so DB-served file/batch
list responses return the same fields and managed IDs as a direct endpoint
GET.
* fix(managed_id_rewriter): degrade to raw id on cross-owner object collision
The OUTPUT (mint) path of _mint_or_reuse_object raised HTTPException(404)
when a dedup hit on the namespaced model_object_id belonged to a different
owner, converting a successful upstream batch/response creation into a 404
for the caller. Two upstream accounts under one provider name can issue the
same raw id, so this is reachable in multi-tenant deployments.
Return the caller's raw id unmanaged instead: the upstream create already
succeeded, a new managed row can't be minted (model_object_id is @unique),
and reusing the other owner's managed id would later fail the access check.
* fix(managed_id_rewriter): scope list cursor by provider and cap body-rewrite recursion depth
* perf(managed_id_rewriter): push batch list provider scope to DB and anchor canonical-path prefix
Object (batch) list rows store model_object_id as passthrough:{provider}:{raw},
so the provider filter is now applied at the indexed DB column, collapsing the
application-layer multi-scan to a single query for that table. File rows keep
the decode-based scan since they have no provider column.
Anchor the canonical-path prefix regex at a path boundary so routes such as
/openai_realtime/... are no longer mis-stripped.
* fix(managed_id_rewriter): refresh stored batch snapshot on reuse
The dedup-reuse path in _mint_or_reuse_object returned the existing managed id
without updating the stored file_object, so DB-served list responses kept the
creation-time snapshot and showed null output_file_id/error_file_id even after
the batch completed. Refresh the snapshot when an owned row is reused so the
list reflects the batch's latest state.
* fix(managed_id_rewriter): deny cross-owner object access on retrieve/cancel/delete
Returning the raw id when can_access_resource fails only made sense for create
responses, where the caller's own upstream create succeeded under a raw id that a
different owner already holds. On retrieve/cancel/delete the caller reaches that
branch only by supplying another tenant's raw id (which bypasses the managed-id
input gate), so echoing the upstream object back leaked it cross-tenant. Restrict
the raw fallback to create routes and return 404 otherwise.
* fix(managed_id_rewriter): deny cross-owner file access on retrieve/delete
_mint_or_reuse_file scoped the raw file dedup lookup to the current caller, so a
raw file-... id belonging to another tenant was never found and the OUTPUT path
minted a fresh managed id for that same upstream file under the caller. A raw id
only reaches this path by skipping the managed-id input gate (raw provider ids
are opt-out), so a different-owner row means the caller is touching someone
else's file. Look up flat_model_file_ids globally and run can_access_resource;
deny with 404 on retrieve/delete and leave the raw id unmanaged on create, which
mirrors the cross-owner handling already in _mint_or_reuse_object.
* fix(managed_id_rewriter): deterministic provider-scoped file dedup
Replace the unscoped find_first in _mint_or_reuse_file with a find_many
ordered by created_at and an application-layer provider filter. The file
table has no provider column, so a raw file id shared across OpenAI and
Azure could map to one row per provider; find_first then picked a row
non-deterministically and, on a provider mismatch, minted a fresh managed
row on every call, accumulating duplicates. Selecting the oldest matching
same-provider row the caller can access keeps reuse stable and prevents
duplicate rows while preserving the cross-tenant deny/leave-raw behaviour.
* refactor(pass_through): scope passthrough managed IDs on the explicit provider
Move the openai/azure detection out of pass_through_request into
resolve_passthrough_managed_id_provider in llms/base_llm/managed_resources,
and key managed-ID rewriting on the forwarded custom_llm_provider rather than
the upstream URL's EndpointType. The helper documents why azure and azure_ai
collapse to one "azure" scope (they share the same Azure OpenAI files/batches
surface, so an ID minted on one must resolve on the other) and returns None for
any other provider so a third-party OpenAI-compatible endpoint never triggers
managed-ID minting.
Add TestManagedIdProviderScope covering the azure_ai -> azure collapse and the
non-openai/azure exclusion.
* test(log_db_metrics): assert sanitized event_metadata contract
test_log_db_metrics_success still asserted the legacy event_metadata
shape (function_name/function_kwargs/function_args), which #28909
intentionally removed so that live Prisma clients, OTel spans, and
secrets never land on a service-log span. The decorator now emits only
a sanitized payload: None when no table_name is present, and
{"table_name": ...} when it is. Update the test to verify both branches
of that contract.
* fix(managed_id_rewriter): page provider-scoped file list by offset
The file list scan advanced its cursor with a strict created_at boundary.
When several rows shared a created_at timestamp and a non-matching provider
row sat on the page boundary, the next query skipped the remaining rows at
that timestamp, dropping matching files from the response. Page by a stable
offset over a total order (created_at plus the unique id column) so tied
rows are never skipped or repeated.
* fix(managed_id_rewriter): push file-list provider scope to the DB
The file-list helper had no provider column to query, so it scanned the
table and filtered by decoding each managed ID in the application layer,
capped at 20 pages. For an admin with a large mixed-provider file pool
that cap could truncate a page.
Mint now writes a _passthrough_provider:{provider} marker into
flat_model_file_ids, giving the file table the same DB-queryable provider
scope object rows already get from the namespaced model_object_id. The
list helper pushes the scope into the query so a single round-trip serves
the page. The scan loop, the offset paging, and the cap are gone, so pages
can no longer truncate, leak the other provider, or skip rows that share a
created_at timestamp.
* fix(managed_id_rewriter): deny raw provider IDs that map to another tenant's managed resource
Clients only ever receive managed IDs on passthrough, so a raw file/batch/response ID for another tenant's managed object can only be recovered by decoding that tenant's managed ID. Raw IDs were forwarded upstream untouched (deliberate opt-out), which on a retrieve/cancel/delete executed upstream before the response-side ownership check ran, leaking a cross-tenant action.
Guard raw provider IDs on the input path: when a raw file-/batch_/resp_ ID resolves to a managed row the caller cannot access, return 404 before forwarding. Genuinely unmanaged raw IDs (no DB row) and IDs the caller owns are left untouched, preserving the opt-out.
* test(managed_id_rewriter): cover azure_ai and pre-versioned azure passthrough paths
* fix(managed_id_rewriter): fall back to raw id when persistence fails
A DB persistence failure after a successful upstream create left the
client holding a minted managed ID with no backing row, so every later
resolve returned 404 and the resource was permanently unreachable. Mint
the managed ID only when the row is stored; on persistence failure return
the raw provider id, matching the no-persistence-available fallback, so
the freshly-created resource stays reachable.
* fix(managed_id_rewriter): log only rewritten query param keys
* fix(managed_id_rewriter): use compound (created_at, id) list cursor boundary
A timestamp-only lt/gt cursor boundary skips list rows that share the
cursor row's created_at across a page boundary, silently dropping them.
Compare the unique id (the secondary sort key) alongside created_at so
the page walk stays complete when timestamps tie.
* fix(managed_id_rewriter): converge concurrent object creates on one managed id
* fix(managed_id_rewriter): bound raw-id guard DB lookups per request
The INPUT guard fired one DB lookup for every file-/batch_/resp_ prefixed
string in the path, query, and body. The file-id guard is an unindexed
array-containment scan over LiteLLM_ManagedFileTable, so an authenticated
caller could amplify a single passthrough request into thousands of
full-table scans by packing a body with id-shaped strings.
De-dupe raw ids within a request and cap the distinct guard lookups,
failing closed with 400 instead of skipping the guard. Legitimate callers
hold managed ids (resolved via an indexed unified_*_id lookup, not the
guard), so the cap only trips under abuse.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
|
4cc3dd7aad |
feat(context_management): compact_20260112 polyfill for non-Anthropic providers (#28868)
* feat(anthropic/messages): in-gateway context_management polyfill for non-Anthropic providers
- Add `context_management/` module with `clear_tool_uses_20250919` editor
dispatched before chat-completions translation on `/v1/messages`
- Hard-protect most-recently completed tool_result from being cleared
- Attach `context_management.applied_edits` to both non-streaming and
streaming (final `message_delta`) responses
- Bedrock Converse: forward `context_management`; filter to
`compact_20260112`-only edits with `compact-2026-01-12` beta header
- token_counter: guard Anthropic-format tools (no `function` key) to
prevent AttributeError during polyfill token counting
- Streaming: handle empty-choices usage-only trailing chunks
- Skip polyfill when `litellm.drop_params = True`
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): pop None context_management before sending to Bedrock Converse
If context_management is forwarded as None (e.g. when mapping returns
None for an invalid format), _filter_context_management_for_bedrock_converse
previously returned early without removing the key, leaving
"context_management": null in the request and causing a validation
error. Pop the key when the value is not a dict.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(bedrock/converse): pop None context_management; extract helpers to fix PLR0915
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(anthropic/messages): check per-request drop_params alongside global
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(anthropic/messages): preserve drop_params for downstream and respect explicit False
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: lazy debug logging in clear_tool_uses; remove unused context_management constants
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic/messages): guard context_management polyfill with try/except
Wrap apply_context_management() in a try/except so any failure (e.g.
litellm.token_counter raising on an unknown tokenizer or unexpected
message format) is logged but does not crash the underlying LLM
request. The polyfill is a best-effort additive feature; on failure we
forward the original messages without applied edits.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(token_counter): guard None input_schema in Anthropic tool fallback
Use `or {}` instead of `.get(..., {})` so explicit null parameters do not
raise AttributeError when formatting function definitions for token counting.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: minimize context_management polyfill threading
- Use None (not empty list) for polyfill_applied_edits when context
management isn't requested, so semantics of 'feature not requested'
vs 'feature requested but no edits applied' are distinct.
- In the streaming iterator, only pass applied_edits to the per-chunk
translator on the final (finish_reason) chunk; intermediate chunks
ignore it anyway, and this makes intent explicit on both sync and
async paths.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(context_management): align tool_use counts and normalize list spec
- _count_tool_uses now requires a string id, matching _collect_tool_use_ids_in_order so the tool_uses trigger can't fire on blocks that aren't clearable.
- apply_context_management dispatcher now accepts the OpenAI list form and normalizes it via AnthropicConfig.map_openai_context_management_to_anthropic, so the polyfill path no longer silently no-ops on list input.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(context_management): add compact_20260112 polyfill for non-Anthropic providers
Implements an in-gateway compaction polyfill that summarizes long conversations
using a configurable model when `compact_20260112` is requested for non-Anthropic
targets (e.g. OpenAI, Gemini), matching Anthropic's context management beta
behaviour for those providers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(compact): skip tool_result-only user turns; bedrock: elif for context_management
- compact_20260112 Phase D: when keeping the last user turn after a full
summary, skip role=user turns whose content is exclusively tool_result
blocks. Such turns translate to OpenAI tool-role messages with no
preceding assistant tool_calls (those got summarized away), which
non-Anthropic providers reject. Fall back to a synthetic continuation
prompt if no eligible user question exists, so the downstream call
always has a non-empty user message.
- bedrock converse: chain the context_management param as elif so it
follows the same if/elif pattern as the surrounding thinking/
reasoning_effort checks.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic): post-compaction question selection, system type, sync stream merge
- compact.py: select last user question from effective_messages (post-compaction slice) instead of raw messages, so prior summarized turns aren't reintroduced
- handler.py: widen _prepare_completion_kwargs system parameter type to Union[str, List[Dict]] matching PolyfillResult.system
- streaming_iterator.py: mirror async hold-and-merge logic in sync __next__ so context_management is attached to the final merged message_delta when stop_reason and usage arrive in separate chunks
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic/messages): apply context_management on sync path; clear held stop_reason chunk in async iterator
- Sync `anthropic_messages_handler` was silently dropping the
`context_management` kwarg via `ANTHROPIC_ONLY_REQUEST_KEYS` after the
polyfill was moved into the async handler. Bridge to the async
dispatcher with `run_async_function` so `litellm.messages.create()`
callers keep working (regressed e.g. `clear_tool_uses_20250919`).
- In the streaming iterator's `__anext__` `StopIteration` handler, clear
`self.holding_stop_reason_chunk` after capturing it (matches `__next__`)
so a subsequent call doesn't re-emit the same chunk.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(bugfixes): bedrock None context_mgmt; stream per-instance queue; sync polyfill; trailing-chunk passthrough
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic): silently drop trailing chunks after usage; remove dead _polyfill_result key
- streaming_iterator: in sync __next__, after the usage chunk has been
merged and emitted, silently consume any trailing provider events
via 'continue' instead of forwarding them through the queue. Trailing
chunks would translate to content_block_delta or message_delta and
violate Anthropic SSE ordering after the final message_delta. The
async __anext__ already drops these via 'if not self.queued_usage_chunk:'
gating, so this aligns sync and async behavior.
- handler: drop unused '_polyfill_result' from ANTHROPIC_ONLY_REQUEST_KEYS.
PolyfillResult is passed as an explicit arg to the adapter methods, never
through extra_kwargs, so the entry was dead code.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* refactor(anthropic): extract usage-merge helper; guard empty slice-only compaction result
- Extract the duplicated hold-and-merge usage logic from the sync __next__ and
async __anext__ paths into a shared _merge_usage_into_held_stop_reason_chunk
helper so the subtle cache-token / context_management attachment lives in
exactly one place.
- In the compact_20260112 slice-only path, fall back to _select_last_user_question
when _strip_compaction_blocks produces an empty list (e.g. messages ending on
an assistant turn whose only content was the compaction block) so the
downstream API never receives an empty messages array.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* refactor(anthropic/context_management): streaming iterator compaction fixes and compact polyfill improvements
- Extract usage-merge helper; guard empty slice-only compaction result
- Silently drop trailing chunks after usage; remove dead _polyfill_result key
- Fix bedrock None context_mgmt; stream per-instance queue; sync polyfill; trailing-chunk passthrough
- Apply context_management on sync path; clear held stop_reason chunk in async iterator
- Fix post-compaction question selection, system type, sync stream merge
- Skip tool_result-only user turns; bedrock: elif for context_management
- Add streaming iterator compaction test suite
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(html): restore flat *.html naming in _experimental/out
Reverses the accidental rename from *.html → */index.html introduced in
|
||
|
Mateo WangandGitHub
|
581c30f1e8 | [internal copy of #29089] fix: duplicate claude code traces (#29311) | ||
|
|
1d9095f914 |
fix(bedrock): support tool search results + chat annotations (#29120)
* Fix overiding of fastapi_response headers * fix(bedrock): support tool search results and surface citations as annotations Add an optional tool-message search_results path that maps directly to Bedrock toolResult.searchResult blocks, and convert Converse citationsContent into chat completion annotations for user-facing citation metadata. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(format): align bedrock prompt factory with black Reformat the updated bedrock prompt template conversion file so CI black --check passes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(bedrock): harden citations, search_results mapping, and token counting Resolve mypy issues in citation parsing, only attach url_citation annotations when citation text is stitched into content, fall back to tool content when search_results is empty, and count search_results text in token/TPM preflight paths. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(bedrock): extract tool result helpers to satisfy PLR0915 Refactor _convert_to_bedrock_tool_call_result into smaller helpers so lint passes without changing Bedrock tool result behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(bedrock): count all forwarded search_results fields in token estimates Include source, title, content text, and citations when estimating tokens so large metadata cannot bypass TPM preflight checks. Reformat factory.py with black. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(managed-files): skip content blocks without a type key in get_file_ids_from_messages * fix(bedrock): stitch citations for any punctuation-only text block * fix(bedrock): map null citation source/title to empty annotation strings * fix(bedrock): advance citation offset for text-only citationsContent blocks * fix(bedrock): complete citation TypedDicts for grounding annotations --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
bb448b0031 |
fix(tests): stabilize image-edit VCR cassettes to stop live gpt-image-1 spend (#28110)
* fix(tests): stabilize image-edit VCR cassettes to stop live gpt-image-1 spend
The image-edit cassettes for ``gpt-image-1`` were accumulating >50
episodes and being refused by the persister
(``tests/_vcr_redis_persister.py``), so every CI run was hitting the
real OpenAI endpoint. The async parametrize was the clearest tell:
``test_openai_image_edit_litellm_sdk[True]`` cached to 1 entry, but the
``[False]`` (async) sibling grew to 51 entries and never replayed.
Two non-deterministic sources were fueling the growth, both fixed
here. After this patch, the cassettes settle at one episode per
unique call and replay for the 24-hour TTL like every other suite.
1. Pin httpx's multipart boundary at the source. The existing
``_normalize_multipart_boundary`` rewrites the boundary in the
``Content-Type`` header reliably, but on the async transport path
the body is not always a contiguous ``bytes`` object when
``before_record_request`` runs, so the body-side replacement
silently no-ops and the recorded cassette retains the random
``boundary=<hex>`` string. The next CI run gets a fresh random
boundary, the ``safe_body`` matcher misses, and
``record_mode="new_episodes"`` appends another episode. Wrapping
``httpx._multipart.MultipartStream.__init__`` so it always uses
``vcr-static-boundary`` when no boundary is supplied eliminates
the variance for both sync and async paths and leaves the normalizer
in place as a backstop. Exposed as
``pin_httpx_multipart_boundary`` so other multipart-heavy suites
(audio, ocr, batches) can adopt the same fixture later.
2. Pass raw ``bytes`` (not ``BytesIO`` streams) through the
image-edit fixtures. A ``BytesIO`` whose file pointer is at EOF
after the first multipart upload silently encodes an empty image on
the next SDK / Router retry — yet another divergent body that VCR
records as a new episode. ``bytes`` are immutable and position-less,
so retries re-encode an identical payload every time. This is also
a small production-correctness improvement: a customer passing
``BytesIO`` today would hit the same empty-body retry bug. The
BytesIO-specific smoke test
(``test_openai_image_edit_with_bytesio``) is preserved by giving
``get_test_images_as_bytesio`` its own factory instead of aliasing
the bytes one.
3. Add ``scripts/flush_image_edit_vcr_cassettes.py`` — a one-shot
Redis SCAN/DEL helper that clears the bloated pre-fix cassettes
under ``litellm:vcr:cassette:tests/image_gen_tests/test_image_edits/*``.
Without this, the next CI run still loads the existing 51-entry
cassette, the new fixed-boundary body still doesn't match any of
the stale entries, the persister still refuses to save, and the
bleed continues. Run once with the production
``CASSETTE_REDIS_URL`` after merge (dry-run by default).
* DIAGNOSTIC: log VCR body mismatches + per-episode body hashes
Temporary observability boost so we can root-cause why
``test_image_edits.py`` async parametrizes still record fresh
episodes on every CI run even though the multipart boundary is now
pinned (sync parametrizes cache cleanly as VCR HIT). The matcher
currently raises ``AssertionError("request bodies differ")`` with
zero context, so we cannot tell whether the live body genuinely
varies, the matcher is comparing a bytes object to a stream object,
or the normalizer is silently skipping the body because it is not
bytes/str.
Three logs added; the first two are worth keeping permanently, the
third is intended to be reverted after the diagnosis lands:
1. ``_safe_body_matcher`` now emits a structured stderr block on
mismatch (type of each side, length, SHA-256, first divergent
byte offset, ±100-byte window). Always-on -- mismatches are
signal, not noise, and the existing per-test verdict already
logs once per test. PERMANENT.
2. ``_normalize_multipart_boundary`` now logs to stderr when the
body type is not bytes/bytearray/str -- the silent ``else:
return`` branch was masking exactly the case we suspect is
firing on async (httpx ``MultipartStream`` handed to vcrpy
before the body is read). PERMANENT.
3. ``_RedisPersister.save_cassette`` now logs every episode's body
SHA-256, length, and 120-byte preview at save time. This lets
two consecutive CI runs be diffed: if the same test records a
different hash run-to-run, the live body genuinely varies; if
both runs record the same hash but the matcher still misses, the
bug is in the matcher itself. TEMPORARY -- revert once the
async variance is identified and fixed.
Once a single ``image_gen_testing`` CI run produces these logs,
revert this commit (or just the persister hash block) with a force
push so the cassette save path is not noisy in steady-state.
* DIAGNOSTIC: route VCR diagnostics through per-PID files (bypass xdist capture)
Re-push of the diagnostic logging from the previous commit, this
time wired so the output actually survives to the CI log. xdist
captures stdout/stderr from every passing test in the worker
process; the body-matcher and normalizer-skip diagnostics fire from
inside vcrpy machinery during the test, so for any test that
ultimately passes (which is all of them once the cassettes are
recorded), the diagnostic lines are silently swallowed.
Fix: write each diagnostic line to a per-PID file under
``test-results/vcr-diagnostics/<pid>.log`` instead of writing to
stderr. The controller's ``pytest_terminal_summary`` aggregates
those files and writes them through ``terminalreporter.write_line``,
which is not subject to per-test capture. As a bonus,
``test-results/`` is already collected by the ``store_test_results``
step in CircleCI, so the raw per-worker logs survive as build
artifacts even after the test session ends.
Three call sites updated:
1. ``_emit_body_mismatch_diagnostic`` (matcher) -- writes the
structured type/length/sha/window block via ``vcr_diag_write_line``.
2. ``_normalize_multipart_boundary`` -- logs the silent-skip path
(body not bytes/bytearray/str) the same way.
3. ``_maybe_log_episode_body_hashes`` (persister) -- replaces the
``_log.warning`` calls (which the root-logger config also
swallows in CI) with ``vcr_diag_write_line``.
Image-gen conftest is the only suite wired to dump the aggregated
log at session end. Other suites can opt in by adding
``emit_vcr_diagnostic_log(terminalreporter)`` to their own
``pytest_terminal_summary``. The diagnostic dir is cleared at the
start of each session (controller-only) so a local rerun does not
mix output from prior runs.
Same revert plan as the previous diagnostic commit: keep the
matcher + normalizer skip diagnostics permanently (they only fire
on signal events), revert the persister body-hash dump once the
async variance is identified.
* fix(tests): coalesce iterable request bodies before matching/recording
Root cause of the residual async image-edit cassette leak. The
diagnostic run for ``ba3915d9`` printed:
[vcr-safe-body-matcher] request body mismatch
body[a]: type='list_iterator' length=unknown sha256=N/A
body[b]: type='list_iterator' length=unknown sha256=N/A
httpx's async transport hands vcrpy a ``request.body`` that is a
``list_iterator`` over multipart chunks rather than a contiguous
``bytes`` blob. Two consequences:
1. ``_safe_body_matcher`` compares the two iterator objects with
``==``, which is identity comparison for arbitrary iterators -
semantically identical multipart bodies never compare equal, and
``record_mode="new_episodes"`` appends a new episode on every CI
run until the cassette crosses ``MAX_EPISODES_PER_CASSETTE`` and
the persister refuses to save (this is exactly what the OVERFLOW
warning has been catching).
2. ``_normalize_multipart_boundary`` short-circuits its
``else: return`` branch because the body is neither bytes nor
str, so any residual random boundary characters in the body bytes
are never rewritten.
Sync requests do not hit this code path: httpx's sync transport
hands vcrpy a single ``bytes`` body, so ``==`` works and the
boundary normalizer runs as intended. That is why
``test_openai_image_edit_litellm_sdk[True]`` records to ``entries=1``
and replays cleanly while ``[False]`` (async) kept growing by one
episode per run.
Fix: add ``_materialize_iterable_body`` which coalesces an iterable
``request.body`` into ``bytes`` in-place. Call it from two places:
* The top of ``_before_record_request``, so the boundary normalizer
and the cassette serializer both see bytes from then on.
* The top of ``_safe_body_matcher``, as defense in depth in case a
future vcrpy code path invokes the matcher without first going
through ``_before_record_request``.
The vcrpy ``Request`` is a wrapper used for matching and recording;
the underlying httpx transport sends its own request body
separately, so replacing the iterator on the vcrpy wrapper does
not starve the live HTTP send.
After this lands the async parametrizes should flip from
``[VCR MISS:RECORDED] entries=N+1`` to ``[VCR HIT] entries=N`` on
the next CI run, matching the sync side and dropping the residual
~$3/day to $0.
* fix(tests): handle bytes_iterator + never leave an exhausted body
Follow-up to 8e08272b. The previous attempt at coalescing iterable
request bodies bailed out (``return`` without writing
``request.body``) whenever it could not classify the chunk type.
That was the wrong failure mode for one critical case: vcrpy
sometimes presents the body as ``iter(some_bytes)``, whose Python
type is ``bytes_iterator`` and which yields ``int`` byte values
(0-255), not byte chunks. The old code saw an ``int`` chunk, hit
the ``else: return`` branch, and left ``request.body`` pointing at
the now-exhausted iterator.
The post-fix diagnostic run made this loud:
[vcr-safe-body-matcher] request body mismatch
body[a]: type='bytes_iterator' length=unknown sha256=N/A
body[b]: type='bytes_iterator' length=unknown sha256=N/A
Every async image-edit test then ballooned from entries=2 to
entries=10 in that single CI run -- the exhausted iterator meant
the live multipart upload went out as an empty body, OpenAI
returned 400, the SDK + flaky retries fired, each retry got a
fresh iterator that my hook exhausted again, and ``new_episodes``
recorded each failed attempt as a new cassette episode.
This patch:
* Recognizes ``bytes_iterator`` (chunks are ``int``) and
reconstructs the buffer via ``bytes(chunks)``.
* Keeps the existing ``list_iterator``-over-bytes-chunks handling
via ``b"".join(...)``.
* **Always writes a bytes value back to ``request.body`` after
consuming the iterator.** If the chunk shape is unrecognized,
``request.body`` is set to ``b""`` rather than left as an
exhausted iterator. That is wrong in the sense of "we lost the
body" but right in the sense of "the failure mode is now visible
(live API call sends empty body and fails fast) instead of
invisible (corrupt cassette grows silently)". Combined with the
matcher diagnostic, any future regression in this code path will
surface in the CI log immediately.
Local verification covers ``bytes_iterator``, ``list_iterator``
over bytes chunks, generator over bytes chunks, empty iterator,
already-bytes (idempotent), identical-content iterator equality
in the matcher (now matches), and differing-content iterator
inequality (still raises).
* fix(tests): clear vcrpy's sticky _was_iter flag so materialized bodies stay bytes
Actual root cause of the async image-edit cassette leak. The
previous diagnostic run produced this dead giveaway:
[vcr-episode-body-hash] ... episode[0]: body type='bytes_iterator'
is not bytes/bytearray/str -- cannot hash
[vcr-safe-body-matcher] request body mismatch
body[a]: type='bytes_iterator' length=unknown sha256=N/A
body[b]: type='bytes_iterator' length=unknown sha256=N/A
Both sides of the matcher were ``bytes_iterator`` **after** the
materializer had supposedly converted them to bytes. That made no
sense until I read vcrpy's ``Request`` class.
vcrpy's ``Request`` keeps two private flags that are set in
``__init__`` from the original body's type and **never cleared by
the setter**:
def __init__(self, method, uri, body, headers):
self._was_file = hasattr(body, "read")
self._was_iter = _is_nonsequence_iterator(body)
...
@property
def body(self):
if self._was_file: return BytesIO(self._body)
if self._was_iter: return iter(self._body)
return self._body
@body.setter
def body(self, value):
if isinstance(value, str): value = value.encode("utf-8")
self._body = value # <-- does NOT touch _was_iter / _was_file
So when httpx's async transport hands vcrpy an iterator body,
``_was_iter`` becomes ``True`` and stays there forever. Even after
``_materialize_iterable_body`` writes plain bytes via
``request.body = out``, the next read of ``.body`` re-wraps the
stored bytes in ``iter()`` -- producing a fresh ``bytes_iterator``
that compares unequal to any other ``bytes_iterator`` via object
identity. The matcher missed every time, the cassette grew by one
episode per run, and the persister saw the same iterator type when
trying to hash the body for the diagnostic log.
Fix: after writing the materialized bytes, also force
``_was_iter`` and ``_was_file`` to ``False``. vcrpy exposes no
public API for this, so we touch the private flags directly --
acknowledged as a pragmatic test-only hack with a clear unit
boundary (the only call site is ``_materialize_iterable_body``).
Local repro reproduces the exact production setup:
``Request('POST', url, iter(b'multipart-content'), {})`` on two
sides, runs the matcher, asserts HIT. Verified the matcher hits on
identical content and still raises on differing content.
Should be the last fix needed. Existing cassettes that contain
oddly-shaped bodies (lists of int chunks, etc. from the previous
``_was_iter=True`` save path) still match because the materializer
canonicalises both sides to bytes before comparison -- no fourth
re-flush required.
* revert(tests): drop the temp per-episode body-hash diagnostic
Removed now that 1c51ad13 has confirmed the root cause (vcrpy's
sticky ``_was_iter`` flag making the body getter re-wrap stored
bytes in ``iter()`` on every access). The hash dump did its job --
the post-1c51ad13 image_gen_testing run shows all five async
image-edit tests as ``[VCR HIT]`` with stable entry counts and
zero billing errors -- and is too noisy to keep on by default
(over 100 lines per session at steady state).
Kept permanently:
* ``_safe_body_matcher`` mismatch diagnostic in
``_vcr_conftest_common.py``. Only fires on a body mismatch,
which is signal worth surfacing whenever it happens.
* ``_normalize_multipart_boundary`` "skipped" log line. Same
rationale -- only fires when the body shape is something the
normalizer cannot rewrite in place.
* The ``test-results/vcr-diagnostics/<pid>.log`` per-PID file
plumbing (``vcr_diag_write_line`` /
``emit_vcr_diagnostic_log``). Useful for any future diagnostic
that needs to bypass xdist stdout/stderr capture; cheap to keep.
* chore(tests): delete unused flush script + wire VCR diagnostic dump everywhere
* Remove ``scripts/flush_image_edit_vcr_cassettes.py``. It was a
one-shot helper for the initial cassette flush; the iterator and
``_was_iter`` fixes mean no future flush should be required, and
the script was never run anywhere (the actual flushes happened
inside the CI conftest via the temp hacks that have since been
reverted).
* The matcher mismatch + normalizer skip diagnostics already write
per-PID files for every suite that imports the shared VCR
plumbing, but ``emit_vcr_diagnostic_log`` -- the controller-side
dump that surfaces those files into the CI log at session end --
was only wired into ``image_gen_tests``. Add the one-line call to
the 12 sibling conftests that already use VCR so the diagnostics
surface in any suite's terminal output if a body matcher ever
misses. No new output in steady state -- the dump is a no-op when
no diagnostics were recorded that session.
* chore(tests): trim non-essential comments per project comment policy
Strips docstrings, inline comments, and block comments that this PR
introduced where the code itself was already self-evident. Keeps the
few lines that document non-obvious behaviour (raw-bytes-not-BytesIO
rationale on the image fixtures, the per-PID-files-bypass-xdist note
on the diagnostic directory). Touches only comments this PR added --
no pre-existing comment is removed.
Net: -161 lines of comment/docstring across 3 files, no code
behaviour change.
* chore(tests): forward **kwargs in pin_httpx_multipart_boundary wrapper
Defensive against future httpx MultipartStream.__init__ adding new
optional kwargs. Without the forward, the wrapper would silently drop
them. No behaviour change today.
* chore(tests): canonicalize VCR matchers and surface shouldn't-happen branches
Bundles the "follow-up cleanup PR" into this one so it does not get
lost. Four small changes:
1. Introduce ``_canonical_body(req) -> (bytes, pre_type)`` and route
``_safe_body_matcher`` through it. The matcher now operates on
bytes by construction; the "compare two iterator objects via
``==`` and silently get object-identity semantics" failure mode
(which cost us this entire PR to diagnose) is structurally
impossible to reintroduce. ``pre_type`` is the body type *before*
canonicalization, surfaced by the mismatch diagnostic so a future
regression involving a new body shape is still visible.
2. Add a structured diagnostic to ``_key_fingerprint_matcher``. It
was previously raising a bare ``AssertionError("API key
fingerprints differ")`` with zero context -- exactly the
anti-pattern the body matcher had before this PR.
3. Surface "shouldn't-happen" branches via ``vcr_diag_write_line``:
* ``_strip_image_b64_payloads`` -- logs when ``response``,
``response['body']``, or ``response['body']['string']`` arrives
in an unexpected shape (vcrpy contract violation).
* ``_compute_key_fingerprint`` -- logs the ``"no-key"`` fallback
with the request method/URL so a stripped-auth-header bug is
visible instead of masked.
* ``_canonical_body`` -- logs its own empty-bytes fallback when a
body has a shape ``_materialize_iterable_body`` did not handle.
4. Re-introduce per-episode body-hash logging in
``_RedisPersister.save_cassette`` (was reverted in 927c5548 as
"noisy"). Quantified cost: ~25 KB of CI log per session at peak,
~ms-scale CPU, zero output in steady state (no save = no log).
Trade-off favours keeping it: lets two consecutive CI runs be
diffed by body hash, which is how we will spot the next regression
in the same class.
All call sites still work: local repro confirms iter==iter HIT,
iter!=iter raises, plain-bytes HIT, body-hash log emits via the same
per-PID file plumbing as the matcher diagnostics.
* chore(tests): symmetrize diag-log cleanup across every VCR-using conftest
``image_gen_tests/conftest.py`` was the only suite that cleared
``test-results/vcr-diagnostics/*.log`` at session start. The other 12
VCR-using conftests inherited any stale per-PID logs from a previous
local run and would dump them in the terminal summary -- harmless in
CI (fresh container) but confusing locally when running multiple
suites in sequence.
Extracts the cleanup into a ``reset_vcr_diag_dir`` helper in
``tests/_vcr_conftest_common.py`` and calls it from every VCR-using
conftest's ``pytest_configure``. Same single source of truth, no
inline duplication.
* fix(tests): gate body materialization on __next__ and strip PR comments
aiohttp/vcrpy stores the json kwarg as a dict; _materialize_iterable_body
was iterating it via __iter__ and joining the keys, replacing the request
body with concatenated key names ("textlanguageentities"). Gate on
__next__ so containers (dict/list/tuple) are left alone — only single-use
iterators like httpx's bytes_iterator / list_iterator are materialized.
Log diagnostic line when chunk type is unrecognized.
* fix(tests): JSON-encode dict bodies in canonical_body for stable matching
aiohttp stubs store the json kwarg as a dict; the fallback that compared
all dicts as b"" caused concurrent presidio analyze calls to be served
the wrong cassette episode. JSON-encode with sort_keys for stable bytes.
* fix(tests): guard emit_vcr_diagnostic_log against multi-conftest re-emission
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(tests): globalize multipart-boundary pin + stabilize whisper fixtures
Diagnostic shows audio_testing was silently re-recording 50+ live Whisper
episodes per CI run (over MAX_EPISODES_PER_CASSETTE, so the persister
refused to save). Two changes:
* Move the session-autouse _pin_multipart_boundary fixture into the
shared _vcr_conftest_common module so every VCR-using suite picks it
up via a single import. image_gen had it inline; the other 12 suites
silently lacked it.
* Replace the module-level open("rb") audio file handles in test_whisper
with cached bytes + a per-call (filename, bytes, mimetype) tuple,
mirroring the image_edits raw-bytes pattern. Stops the file-pointer-
at-EOF bug where the second test got an empty multipart body.
* chore(tests): drop per-episode body-hash dump and redundant emit guard
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
|
||
|
Mateo WangandGitHub
|
2c733c00f5 |
chore(ci): modernize model references in tests and configs (#27856)
* test: modernize models used in CircleCI e2e test suites
Replaces obsolete models (gpt-4o, gpt-4o-mini, gpt-3.5-turbo,
claude-3-5-sonnet-20240620, claude-sonnet-4-20250514) with current
equivalents across the e2e_openai_endpoints and
proxy_e2e_anthropic_messages_tests CircleCI jobs.
- gpt-4o -> gpt-5.5 (responses API e2e tests)
- gpt-4o-mini -> gpt-5-mini (websocket responses, oai_misc_config)
- gpt-4o-mini-2024-07-18 -> gpt-4.1-mini-2025-04-14 (fine-tuning,
still actively fine-tunable)
- gpt-4 / gpt-3.5-turbo target_model_names example -> gpt-5.5 /
gpt-5-mini
- bedrock claude-3-5-sonnet-20240620 batch entry -> haiku-4-5-20251001
(also aligning oai_misc_config model_name with what
test_bedrock_batches_api.py actually requests)
- bedrock claude-sonnet-4-20250514 (deprecated, retires 2026-06-15)
-> claude-sonnet-4-5-20250929
* test: point bedrock-claude-sonnet-4 alias at Sonnet 4.6, not 4.5
Greptile/Cursor flagged that after the previous commit, the
bedrock-claude-sonnet-4 alias collided with bedrock-claude-sonnet-4.5
(both pointed to claude-sonnet-4-5-20250929). Rename to
bedrock-claude-sonnet-4.6 and point it at the Sonnet 4.6 Bedrock ID
(us.anthropic.claude-sonnet-4-6, already in the litellm model
registry) so the alias name matches the underlying model version.
* test: modernize models across remaining CI-mounted configs & tests
Expands the modernization sweep to all CircleCI-mounted proxy configs
and to test directories where the model literal is a fixture/route key
(not the test's subject).
Config changes:
- proxy_server_config.yaml: bump gpt-3.5-turbo / gpt-3.5-turbo-1106 /
gpt-4o / gemini-1.5-flash / dall-e-3 underlying models; rename
gpt-3.5-turbo-end-user-test alias to gpt-5-mini-end-user-test; bump
text-embedding-ada-002 underlying to text-embedding-3-small. User-
facing aliases (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, etc.)
preserved for backward compatibility with tests.
- simple_config.yaml, otel_test_config.yaml, spend_tracking_config.yaml:
bump gpt-3.5-turbo underlying to gpt-5-mini.
- pass_through_config.yaml: claude-3-5-sonnet / claude-3-7-sonnet /
claude-3-haiku entries replaced with claude-sonnet-4-5 / claude-
haiku-4-5 / claude-opus-4-7.
- oai_misc_config.yaml: align alias name with the gpt-5-mini rename.
Test changes (proactive: claude-sonnet-4-20250514 / claude-opus-4-
20250514 retire 2026-06-15):
- tests/llm_translation/test_anthropic_completion.py: bump 3 references
+ paired Vertex AI ID to claude-sonnet-4-5.
- tests/llm_translation/test_optional_params.py: bump 2 references.
- tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py
and test_bedrock_anthropic_messages_test.py: bump router fixtures
using the deprecated model IDs.
- tests/pass_through_unit_tests/base_anthropic_messages_tool_search_test.py:
modernize docstring examples.
- tests/test_end_users.py: update references to renamed alias.
* test: modernize placeholder model literals in router_unit_tests
Mass replace_all on fixture/placeholder model literals across the
router_unit_tests/ suite (model name is a routing key / label, not the
test subject). Sub-agent sweep so far — additional commits will follow
for logging_callback_tests/, enterprise/, top-level tests/test_*.py,
and other CI-mounted dirs.
Mappings applied:
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 / claude-3-opus-20240229 /
claude-3-haiku-20240307 / claude-3-5-sonnet-20240620 ->
claude-sonnet-4-5-20250929 / claude-opus-4-7 /
claude-haiku-4-5-20251001 as appropriate
Explicitly preserved:
- gpt-4o-mini-* variants (transcribe, tts, etc.) where they're current
- gpt-4-turbo / gpt-4-vision-preview / gpt-4-0613 (subject literals)
- JSONL batch body literals
- Mock LLM response model fields (must match upstream)
- Fake/mock identifiers
* test: modernize placeholder model literals across remaining CI suites
Sub-agent sweep across logging_callback_tests/, guardrails_tests/,
enterprise/, pass_through_unit_tests/, otel_tests/,
llm_responses_api_testing/, batches_tests/, spend_tracking_tests/,
litellm_utils_tests/, unified_google_tests/, and a few top-level
tests/test_*.py files where the model literal is a fixture or
placeholder (router model_list, mock standard logging payload, mock
callback data) rather than the test's subject.
Mappings applied (see scope notes below):
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5.5 (corrected from initial gpt-5 — bare gpt-5
is not a valid OpenAI alias; only gpt-5.5 / gpt-5.4 / gpt-5.2-codex
/ gpt-5-mini exist)
- gpt-4o-mini (bare) -> gpt-5-mini
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 -> claude-sonnet-4-5-20250929
- claude-3-opus-20240229 -> claude-opus-4-7
- claude-3-haiku-20240307 -> claude-haiku-4-5-20251001
- claude-3-5-sonnet-20240620/20241022 -> claude-sonnet-4-5-20250929
- claude-3-7-sonnet-20250219 -> claude-sonnet-4-6
- gemini-1.5-flash -> gemini-2.5-flash
- gemini-1.5-pro -> gemini-2.5-pro
Explicitly preserved (not modernized):
- llm_translation/ tests where model is the SUBJECT (provider-specific
translation/transformation logic). Only the deprecated 20250514
references were already bumped in a prior commit.
- Cost-calc / tokenizer subject tests in test_utils.py (skip-ranges
documented by the sub-agent).
- Bedrock model IDs in test_health_check.py path-stripping tests.
- JSONL batch request bodies and mock LLM response bodies (must match
upstream literal).
- Langfuse expected-request-body JSON fixtures (cost values are exact-
match-asserted; changing the model would shift response_cost).
- gpt-3.5-turbo-instruct (text-completion endpoint; no modern OpenAI
equivalent).
- Top-level tests calling the proxy through user-facing aliases
(gpt-3.5-turbo, gpt-4, text-embedding-ada-002, dall-e-3) — aliases
in proxy_server_config.yaml stay; only the underlying model was
bumped.
- tests/test_gpt5_azure_temperature_support.py (the test's whole point
is model-name handling).
- Fake / mock / openai/fake identifiers.
Notable side fixes:
- test_spend_accuracy_tests.py: UPSTREAM_MODEL now matches what
spend_tracking_config.yaml's proxy actually routes to (gpt-5-mini),
resolving a latent inconsistency.
- proxy_server_config.yaml: bare `gpt-5` alias renamed to `gpt-5.5`
(bare gpt-5 is not a valid OpenAI alias).
- test_batches_logging_unit_tests.py: explicit_models list entries
kept distinct (gpt-5-mini + gpt-5.5) after bulk rename.
* test: fix CI failures from model modernization sweep
CI surfaced 4 categories of regression from the bulk modernization:
1. Azure deployment names are customer-specific. Reverted:
- tests/litellm_utils_tests/test_health_check.py: azure/text-
embedding-3-small -> azure/text-embedding-ada-002 (the CI Azure
account does not have a text-embedding-3-small deployment).
- tests/logging_callback_tests/test_custom_callback_router.py:
same revert for two router fixtures driving aembedding.
2. gpt-5 family does not accept temperature != 1. Tests that pass a
custom temperature swapped from gpt-5-mini to gpt-4.1-mini (modern
non-reasoning OpenAI mini that still accepts temperature/logprobs):
- tests/logging_callback_tests/test_datadog.py
- tests/logging_callback_tests/test_langsmith_unit_test.py
- tests/logging_callback_tests/test_otel_logging.py
3. proxy_server_config.yaml's gpt-3.5-turbo-large alias was routing to
gpt-5.5 (a reasoning model that rejects logprobs). The proxy test
tests/test_openai_endpoints.py::test_chat_completion_streaming
exercises logprobs/top_logprobs through that alias. Bumped the
underlying model to gpt-4.1 (non-reasoning, still modern).
4. tests/logging_callback_tests/test_gcs_pub_sub.py asserts against a
pinned JSON fixture (gcs_pub_sub_body/spend_logs_payload.json) with
hardcoded model="gpt-4o" and a model-specific spend value. Reverted
the litellm.acompletion calls in the test to model="gpt-4o" so the
fixture's exact-match assertions still hold.
5. tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py:
anthropic.messages.create routing to openai/gpt-5-mini returned an
empty content[0] with max_tokens=100 (reasoning-token consumption).
Swapped to openai/gpt-4.1-mini.
* test: fix Assistants API model + 2 cursor[bot] review nits
1. pass_through_unit_tests/test_custom_logger_passthrough.py: gpt-5.5
isn't accepted by the /v1/assistants endpoint
("unsupported_model"). Switch to gpt-4.1-mini (modern, Assistants-
API-supported, non-reasoning).
2. example_config_yaml/pass_through_config.yaml: the previous sweep
bumped the claude-3-7-sonnet alias to claude-opus-4-7, which is a
tier change (Sonnet -> Opus). Map to claude-sonnet-4-6 to keep the
Sonnet tier intact. (Cursor bugbot review.)
3. example_config_yaml/simple_config.yaml: model_name was left as
gpt-3.5-turbo while the underlying was bumped to gpt-5-mini, which
muddles the "simple" example. Make both sides gpt-5-mini so the
most basic example is a straight 1:1 mapping again. (Cursor bugbot
review.)
* fix: revert gpt-4/gpt-3.5-turbo alias underlying to non-reasoning models
tests/test_openai_endpoints.py::test_completion calls the proxy alias
"gpt-4" with temperature=0, and other tests call gpt-3.5-turbo with
custom temperature / logprobs / the legacy /v1/completions endpoint.
The earlier modernization mapped both aliases to gpt-5.5 / gpt-5-mini,
which are reasoning models that reject temperature != 1 and don't
expose /v1/completions. Map the aliases to gpt-4.1 / gpt-4.1-mini
(modern non-reasoning OpenAI models) instead — keeps user-facing
aliases preserved while picking a current underlying that still
supports the parameters/endpoints the tests exercise.
|
||
|
Cursor AgentandMateo Wang
|
3390fc1972 |
test(vcr): mark Bedrock prompt-caching cross-call tests VCR-incompatible
The pass_through prompt-caching tests
(test_prompt_caching_returns_cache_read_tokens_on_second_call,
test_prompt_caching_streaming_second_call_returns_cache_read) make a
warm-up call and then assert the *second* call sees a non-zero
cache_read_input_tokens count from the upstream's prompt-cache. VCR
replay can't model cross-call provider state — both calls match the
same cassette episode, so the second call returns the first call's
pre-warmup response and the assertion fails:
AssertionError: Expected cache_read_input_tokens > 0 on second call,
but got 0. Full usage: {'input_tokens': 4986,
'cache_creation_input_tokens': 4974, 'cache_read_input_tokens': 0}
This started biting after the AWS SigV4 fingerprint stabilization
(
|
||
|
Cursor AgentandMateo Wang
|
b637d9f64a |
test(vcr): classify cache verdicts, detect live calls, surface cost leaks
Convert the per-test VCR verdict line from a single 'NOOP / HIT / MISS /
PARTIAL' tag into a classified outcome that distinguishes the cases that
silently bill the live API on every CI run from the ones that don't:
HIT pure replay
PARTIAL mixed replay + new recordings
MISS:RECORDED new cassette saved to Redis (cached next run)
MISS:OVERFLOW cassette > MAX_EPISODES_PER_CASSETTE; persister
refused to save; re-bills every run
MISS:NOT_PERSISTED test failed; save_cassette skipped; re-bills
NOOP VCR-marked but no HTTP traffic (mocked elsewhere)
UNMARKED:LIVE_CALL test bypassed VCR AND opened a TCP connection
to a known LLM provider host -> wasted spend
UNMARKED:NO_TRAFFIC test bypassed VCR but didn't call out
The UNMARKED:LIVE_CALL signal is what converts 'this test probably hits
live' into 'this test connected to api.openai.com'. We install a
socket.connect / socket.create_connection wrapper for the duration of
each non-VCR-marked test and record any outbound TCP to a known LLM
provider hostname. The probe sits below the httpx layer so vcrpy and
respx (which both patch above the socket) are unaffected.
Replace the file-level _RESPX_CONFLICTING_FILES blacklists in the
llm_translation and local_testing conftests with per-item respx
detection in apply_vcr_auto_marker_to_items. A test now skips VCR when
it actually carries @pytest.mark.respx or has respx_mock in its fixture
chain - not just because some other test in the same file imports
MockRouter. Items skipped by skip_files are split into respx_conflict
(real conflict, the module wires up respx) vs file_opt_out (dead skip-
list entry whose module never touches respx) so the session summary
makes pruning obvious.
Stabilize the AWS SigV4 fingerprint: the Authorization header on
Bedrock requests rotates its Credential date and Signature on every
call, which previously pushed every Bedrock test past the 50-episode
overflow threshold. Extract the access-key id only
('aws-sigv4:AKIA...') so two requests with the same identity match.
Always emit verdict logging when VCR is active (set
LITELLM_VCR_VERBOSE=0 to opt back into the legacy quiet mode). Add a
session-end classification summary that lists overflow tests, unmarked
live-call tests, and the skip-reason breakdown.
Wire the live-call probe + summary hook into every test directory that
already uses the Redis-backed VCR cache (audio_tests, guardrails_tests,
image_gen_tests, litellm_utils_tests, llm_responses_api_testing,
llm_translation, local_testing, logging_callback_tests, ocr_tests,
pass_through_unit_tests, router_unit_tests, search_tests,
unified_google_tests).
Add tests/llm_translation/test_vcr_classification.py covering the
verdict classifier, skip-reason tagging, AWS SigV4 fingerprint stability,
live-host classification, and session summary rendering.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
|
||
|
|
7e13256fee |
test: add 24hr Redis-backed VCR cache to additional test suites (#27159)
* test: add 24hr Redis-backed VCR cache to additional test suites Extracts the existing llm_translation VCR plumbing into a reusable helper (tests/_vcr_conftest_common.py) and wires it into the conftest.py files of the test directories listed in LIT-2787: audio_tests, batches_tests, guardrails_tests, image_gen_tests, litellm_utils_tests, local_testing, logging_callback_tests, pass_through_unit_tests, router_unit_tests, unified_google_tests The same helper is also adopted by the pre-existing llm_translation and llm_responses_api_testing conftests to remove the copy-pasted VCR setup. Each consuming conftest: - registers the Redis persister via pytest_recording_configure - auto-marks collected tests with pytest.mark.vcr (skipping respx-using files where applicable, since respx and vcrpy both patch httpx) - gates cassette writes on test success via _vcr_outcome_gate The cache is opt-in via CASSETTE_REDIS_URL; when unset, VCR is disabled and tests hit live providers as before. LITELLM_VCR_DISABLE=1 still forces a bypass for ad-hoc local runs. Test directories that run LiteLLM proxy in Docker (build_and_test, proxy_logging_guardrails_model_info_tests, proxy_store_model_in_db_tests) are intentionally not included: VCR.py patches the in-process httpx transport and cannot intercept calls made from inside a Docker container. The installing_litellm_on_python* jobs make no LLM calls and don't benefit from caching. https://linear.app/litellm-ai/issue/LIT-2787/add-24hr-caching-to-additional-test-suites * test(vcr): add safe-body matcher to handle JSONL and binary request bodies vcrpy's stock body matcher inspects Content-Type and unconditionally runs json.loads on application/json bodies. JSON Lines payloads (used by the Bedrock batch S3 PUT and other upload paths) crash that with json.JSONDecodeError: Extra data, before the matcher can return 'not a match'. This was the root cause of the batches_testing CI job failing on test_async_create_file once VCR auto-marking was applied to the batches_tests directory. Add a conservative byte-equality body matcher and use it in place of 'body' in the shared match_on tuple. The matcher is strictly more conservative than vcrpy's default — the only thing it gives up is 'different JSON key order is treated as the same body', which doesn't apply to deterministic litellm-built request payloads. It can never produce a false positive that the default would have rejected, so there is no cross-contamination risk. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(vcr): exclude tests that VCR replay actively breaks A few tests are incompatible with cassette replay and were failing on the latest CI run after VCR auto-marking was extended to local_testing and logging_callback_tests: - test_amazing_s3_logs.py (logging_callback_tests): the test asserts on a per-run response_id that should round-trip through a real S3 PUT/LIST. vcrpy's boto3 stub intercepts the PUT and the LIST replays stale keys, so the freshly-generated id is never found. - test_async_embedding_azure (logging_callback_tests) and test_amazing_sync_embedding (local_testing): the failure branches deliberately pass api_key='my-bad-key' to assert that the failure callback fires. We scrub auth headers from cassettes (so the bad-key request matches the prior good-key request), and vcrpy replays the recorded 200 — the failure callback never fires. - test_assistants.py (local_testing): the OpenAI Assistants polling APIs mint fresh thread/run IDs every recording session and then poll until status=='completed'. Replays of those polled GETs can never match a freshly-generated run id, so every CI run effectively re-records and the suite blows past the 15m no_output_timeout. Skip these from VCR auto-marking so they continue to hit live providers as they did before this change. The remaining tests in each directory still get cached. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(vcr): expand skip lists for second batch of incompatible tests Followup to the previous commit. After re-running CI on the rebuilt branch, three more tests surfaced as VCR-replay-incompatible: - litellm_utils_testing :: test_get_valid_models_from_dynamic_api_key Calls GET /v1/models with api_key='123' to assert the result is empty. We scrub auth headers, so the bad-key request matches the prior good-key cassette and replays the recorded model list. - litellm_utils_testing :: test_litellm_overhead.py Measures litellm_overhead_time_ms as a percentage of total wall-clock time. With cached responses the upstream 'network' time collapses to microseconds, blowing past the 40%% threshold the test asserts on. Skip the whole file (every parametrization is at risk). - local_testing_part1 :: test_async_custom_handler_completion and test_async_custom_handler_embedding Same bad-key failure-callback pattern as the already-skipped test_amazing_sync_embedding. - litellm_router_testing :: test_router_caching.py Asserts on litellm's own router-level response cache by comparing response1.id to response2.id across repeat upstream calls (test bypasses litellm cache via ttl=0 and expects upstream to return a *new* id). With VCR replay both upstream calls return the same cassette body, so the ids are identical. Skip the whole file. - logging_callback_tests :: test_async_chat_azure (preemptive) Same shape as already-skipped test_async_embedding_azure; was masked by upstream OpenAI rate-limit failures on baseline. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(vcr): use item.path and tighten matcher docstring - Replace pytest's deprecated item.fspath with item.path in apply_vcr_auto_marker_to_items so we don't emit deprecation warnings under pytest 8. - Clarify _safe_body_matcher docstring to reflect actual behavior (direct == first, then UTF-8 bytes comparison, no repr fallback). Addresses Greptile review feedback on PR #27159. * test(vcr): swallow all RedisError on cassette save/load Cassette persistence is strictly best-effort: any Redis-side failure (connection blip, timeout, OutOfMemoryError when the maxmemory cap is hit, READONLY replicas, etc.) should degrade to 'test passed but cassette not cached' rather than fail the test on teardown. Previously the persister only caught ConnectionError and TimeoutError, so OutOfMemoryError — which Redis Cloud raises when the cassette cache hits its memory cap and there are no evictable keys — propagated out of vcrpy's autouse fixture and ERRORed otherwise-passing tests on teardown. This caused the litellm_utils_testing CircleCI job to fail on the latest commit's run, even though the underlying test was a unit test that used mock_response and produced no real upstream traffic (the cassette was dirtied by a background langfuse callback). The rerun only succeeded because Redis evictions happened to free enough room before the SET — i.e. it was timing-dependent flakiness. Catch redis.exceptions.RedisError (the common base of all server- and client-side Redis exceptions) on both save and load, and parametrize the regression tests across ConnectionError, TimeoutError, and OutOfMemoryError to pin the new behavior. * test(vcr): surface cassette-cache failures with warnings + session banner When the persister silently swallows a Redis OOM (or any RedisError) on save/load there is otherwise no visible signal that the cache is degraded — tests pass, the cassette just isn't persisted, and the next session still hits the same Redis at the same near-cap memory. Add three layers of observability so that failure mode is loud: 1. Per-process health counters ("save_failures", "load_failures", and the last error string for each), exposed via cassette_cache_health() and reset via reset_cassette_cache_health(). The persister increments these in addition to logging. 2. VCRCassetteCacheWarning (UserWarning subclass) emitted via warnings.warn() inside the persister's except block. Pytest's built-in warnings summary at session end automatically lists every such warning, so the failure is visible in CI logs without any conftest-level wiring. 3. Session-end banner via emit_cassette_cache_session_banner() and a stderr-fallback atexit handler registered from register_persister_if_enabled(). Two states: - red "VCR CASSETTE CACHE DEGRADED" when save_failures or load_failures > 0 - yellow "VCR CASSETTE CACHE NEAR CAPACITY" (no failures, but used_memory >= 85% of maxmemory) so the next session knows the Redis is approaching OOM before any SET actually fails Capacity comes from a best-effort INFO memory probe (cassette_cache_capacity_snapshot) that returns None on any failure or when maxmemory is uncapped. The atexit handler skips xdist workers so only the controller emits. Tests: parametrize the existing save/load swallow-error tests across ConnectionError/TimeoutError/OutOfMemoryError, add direct tests for the health counters and warning emission, and a new test_vcr_conftest_common_banner.py covering banner output for every state (silent/red/yellow/disabled/xdist-worker). * test(vcr): bucket cassettes by API key fingerprint, drop bad-key skips Tests that deliberately call an LLM API with a bad key (e.g. to assert that the failure callback fires, or that check_valid_key returns False) were being silently served the prior good-key cassette: we scrub the real Authorization / x-api-key header from the cassette before storing it, so a follow-up bad-key call is byte-identical to the good-key call under the existing match_on tuple. Add a 'key_fingerprint' custom matcher that distinguishes requests by the SHA-256 of their API-key headers. The fingerprint is stamped into a synthetic 'x-litellm-key-fp' header by a new before_record_request hook, which then strips the real auth headers (we have to do the scrubbing here instead of via vcrpy's filter_headers knob, because filter_headers runs *first* and would erase the value we want to hash). Bad-key requests now get a different cassette bucket than good-key requests, so vcrpy will not replay a recorded 200 in place of the expected 401. The fingerprint is a one-way hash of the secret, so cassettes never contain the key. This permanently removes the 'bad-key' category of skips: - tests/local_testing: dropped ::test_amazing_sync_embedding, ::test_async_custom_handler_completion, ::test_async_custom_handler_embedding - tests/logging_callback_tests: dropped ::test_async_chat_azure, ::test_async_embedding_azure - tests/litellm_utils_tests: dropped ::test_get_valid_models_from_dynamic_api_key Coverage: 7 new unit tests in tests/test_litellm/test_vcr_safe_body_matcher.py covering header stripping, fingerprint determinism, no-auth bucketing, good-vs-bad key discrimination, x-api-key (Anthropic/Azure) discrimination, and idempotence under replay. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(vcr): drop redundant comments and docstrings Trim narration of code that is already self-evident from function and variable names. Keep the two genuinely non-obvious bits: - ordering constraint between filter_headers and before_record_request, which would invite a maintainer to re-introduce the bug if removed - the per-directory _VCR_INCOMPATIBLE_FILES rationale, since 'why exactly is this skipped' is not knowable from the test name alone Also drop the 40-line commented-out drop-in conftest snippet at the bottom of _vcr_conftest_common.py — the consuming conftests are the canonical reference. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(vcr): make _before_record_request idempotent vcrpy invokes before_record_request more than once per request: can_play_response_for calls it, then __contains__ / _responses (reached via play_response) call it again on the result. The second invocation sees a request whose auth headers we already stripped, so a naive recompute yields "no-key" and overwrites the real fingerprint stored in the header. This makes can_play_response_for and play_response disagree on matchability — the former says "yes, we have a stored response for this" (matching no-key to no-key) and the latter throws UnhandledHTTPRequestError because it computes a fresh real fingerprint that doesn't match the stored no-key. In CI this manifested as ~30 failing tests across guardrails_testing, audio_testing, batches_testing, image_gen_testing, llm_responses_api, litellm_router_unit_testing, etc. Skip the recompute when the header is already set, so re-applying the hook is a no-op. Adds a regression test that fires the hook twice on the same dict and asserts the fingerprint stays put. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(vcr): drop more redundant docstrings and headers * test(vcr): enable 24hr cache for ocr_tests and search_tests These two directories were the only non-dockerized test suites in the build_and_test workflow that make live LLM/provider API calls but were not VCR-enabled by this PR. Together they account for 96 tests: - tests/ocr_tests/ (31): Mistral OCR, Azure AI OCR, Azure Document Intelligence, Vertex AI OCR. Pure-unit tests inside the same files (e.g. TestAzureDocumentIntelligencePagesParam) make no HTTP calls and become benign VCR NOOPs. - tests/search_tests/ (65): Brave, DataForSEO, DuckDuckGo, Exa, Firecrawl, Google PSE, Linkup, Parallel.ai, Perplexity, SearchAPI, Searxng, Serper, Tavily. Both directories use the canonical minimal conftest pattern from tests/audio_tests/conftest.py with no skip lists. None of the test files use respx, none assert on per-call upstream non-determinism (no response1.id != response2.id, no overhead-as-fraction-of-total, no live polling), so the default match_on tuple should cache cleanly. If a flake surfaces during the first cassette-recording CI run, we can add a targeted skip the same way we did for the other dirs. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
ishaan-berriandGitHub
|
fdd9f3d129 |
fix: block path traversal SSRF in BitBucket, Arize Phoenix, and AssemblyAI clients (#26943)
* fix: sanitize BitBucket file path to block path traversal SSRF * fix: sanitize Arize Phoenix prompt_version_id to block SSRF * fix: sanitize AssemblyAI transcript_id to block SSRF * test: add path traversal SSRF security tests for BitBucket client * test: add SSRF security tests for Arize Phoenix client * style: black format arize_phoenix_client.py * style: black format assembly_passthrough_logging_handler.py * test: add SSRF security tests for AssemblyAI transcript_id validation * fix: move AssemblyAI transcript_id validation before try/except so ValueError propagates |
||
|
Ryan Crabbe
|
b1a0a3fc17 |
fix(tests): use Sonnet 4.5 for Bedrock invoke prompt-caching tests
Claude 3.5 Sonnet v2 reached EOL on Bedrock 2026-03-01, returning the same 404 EOL error as 3.7 Sonnet. Sonnet 4.5 supports both InvokeModel and Converse APIs on Bedrock, so use the same model for both routes. |
||
|
Ryan Crabbe
|
dc46467235 |
fix(tests): replace deprecated Bedrock Claude 3.7 Sonnet model ID
AWS Bedrock has reached end-of-life for `claude-3-7-sonnet-20250219-v1:0`, returning 404s with "This model version has reached the end of its life." Update test references to `claude-sonnet-4-5-20250929-v1:0` (same capability surface: thinking, tools, prompt caching, PDF input, vision, computer use). The bedrock/invoke pass-through tests stay on Sonnet 3.5 since Sonnet 4.5 is converse-only on Bedrock. |
||
|
Ryan Crabbe
|
3745ba2ea7 |
replace retired claude-3-haiku-20240307 with claude-haiku-4-5-20251001 in anthropic messages passthrough test
Anthropic retired claude-3-haiku-20240307 on 2026-04-20, causing the test_anthropic_messages_litellm_router_non_streaming_with_logging test to 404. Update the model references in this file to the current pinned haiku version. |
||
|
Ishaan Jaffer
|
e8461b5b97 | style: run black formatter on files from main merge | ||
|
David ChenandGitHub
|
d1df4e838b |
Litellm fix update bedrock models (#24947)
* update bedrock models in tests * updated more tests and model_prices_and_context_window * fix model id and pricing * replace more sonnet models * update tests * git push * update pricing * flaky total cost * monkey patch * relax the cost change * fix and revert some changes * revert the pricing * chore: move cost/pricing changes to bedrock-cost-fixes branch * chore: split Bedrock file-api beta stripping to separate branch Removes strip_unsupported_file_api_betas_for_bedrock_invoke from this branch; see litellm_bedrock_invoke_strip_file_api_betas for that fix. Made-with: Cursor |
||
|
|
e4442a4d98 |
test fix us.anthropic.claude-haiku-4-5-20251001-v1:0 (#24931)
* test fix us.anthropic.claude-haiku-4-5-20251001-v1:0 * ignore mypy cache files --------- Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> Co-authored-by: David Chen <clfhhc@gmail.com> |
||
|
Ishaan Jaffer
|
bae3dcde13 | test_get_credentials_from_env | ||
|
Krrish Dholakia
|
bc829d51f2 | test: test | ||
|
Krrish Dholakia
|
a41ba7bb6a | test: update test | ||
|
|
2ea9e207bd |
Litellm ishaan march 20 (#24303)
* feat(redis): add circuit breaker to RedisCache to fast-fail when Redis is down (#24181) * feat(redis): add circuit breaker env var constants * feat(redis): add RedisCircuitBreaker and apply guard decorator to all async ops * fix(dual_cache): fall back to L1 instead of re-raising on Redis increment failures * test(caching): add circuit breaker unit tests * fix(redis): fast-fail concurrent HALF_OPEN probes — only one probe at a time * fix(dual_cache): return None fallback when in_memory_cache is absent and Redis fails * test(caching): add regression tests for HALF_OPEN concurrency and None fallback * Fix blocking sync next in __anext__ (#24177) * Fix blocking sync next * Update tests/test_litellm/litellm_core_utils/test_streaming_handler.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix PEP 479 regression in __anext__ sync iterator exhaustion asyncio.to_thread re-raises thread exceptions inside a coroutine, where PEP 479 converts StopIteration to RuntimeError before any except clause can catch it. Add _next_sync_or_exhausted() module-level helper that catches StopIteration in the thread and returns a sentinel instead, then raise StopAsyncIteration in the coroutine. Also rewrites the non-blocking test to use asyncio.gather() instead of asyncio.create_task() (which returned None on Python 3.9 / pytest-asyncio in CI), and adds an exhaustion regression test that drains the wrapper fully and asserts no RuntimeError leaks out. --------- Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * feat: add git-subdir source type to claude-code/plugins API (#24223) Support a third plugin source type `git-subdir` alongside the existing `github` and `url` types, as documented in the official Claude Code plugin marketplaces spec. New format: {"source": "git-subdir", "url": "...", "path": "subdir/path"} - Validates url and path fields are present and non-empty - Rejects absolute paths, '..' segments, backslashes, and percent-encoded traversal sequences (including double-encoded variants via regex check) - Extracts path validation into _validate_git_subdir_path() helper - Updates Pydantic field description to document all three source types - Adds isValidUrl() check for url/git-subdir source types in the UI form - Adds "Git Subdir" option to the UI form with a required Path field - Adds unit tests covering success, update, missing/empty fields, path traversal variants, and unknown source type Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * [FEAT] add extract_header and extract_footer to Mistral OCR supported params (#24213) * docs: add git-subdir source type to claude-code plugin marketplace docs (#24289) * fix(ui): swap J/K keyboard navigation in log details drawer (#24279) (#24286) J should navigate down (next) and K should navigate up (previous), matching vim/standard conventions. * fix: use async_set_cache in user_api_key_auth hot path (#24302) * fix: use async_set_cache in auth hot path to avoid blocking event loop * test: assert no blocking set_cache call in _user_api_key_auth_builder * test: broaden blocking call check to all sync DualCache methods * test: fix regression test to actually catch blocking cache calls * fix: ruff lint unused variable + UI build MessageManager error - litellm/caching/redis_cache.py: remove unused variable 'e' in circuit breaker exception handler (F841) - add_plugin_form.tsx: use MessageManager.error() instead of undefined message.error() for git URL validation Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * docs: add REDIS_CIRCUIT_BREAKER env vars to config_settings reference Add REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD and REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT to the environment variables reference table so test_env_keys.py passes. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> --------- Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Vincenzo Barrea <manamana88@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Robert Kirscht <rkirscht242@gmail.com> Co-authored-by: Imgyu Kim <kimimgo@gmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> |
||
|
|
29e3fd5d79 |
[Release Fix] (#22411)
* fix(lint): suppress PLR0915 for 3 complex methods that exceed 50-statement limit - streaming_iterator.py: _process_event (84 statements) - transformation.py: translate_messages_to_responses_input (51 statements) - transformation.py: transform_realtime_response (54 statements) Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(mypy): resolve type errors in public_endpoints, user_api_key_auth, common_utils, transformation - public_endpoints.py: fix _cached_endpoints type annotation - user_api_key_auth.py: accept Optional[str] for end_user_id parameter - common_utils.py: add NewProjectRequest/UpdateProjectRequest to Union type - transformation.py: add ChatCompletionRedactedThinkingBlock and list[Any] to content type Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(proxy-extras): bump version to 0.4.50 and sync schema - Bump litellm-proxy-extras from 0.4.49 to 0.4.50 - Sync schema.prisma with main proxy schema - Includes new LiteLLM_ClaudeCodePluginTable model - Includes new @@index([startTime, request_id]) on SpendLogs - Update version references in requirements.txt and pyproject.toml Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(router): use string id in test_add_deployment and add defensive str() in register_model - Change test to use string '100' instead of int 100 for model_info.id - Add str() conversion in register_model to prevent AttributeError on non-string keys Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(security): update minimatch to 10.2.4 to fix CVE-2026-27903 and CVE-2026-27904 - Run npm audit fix in docs/my-website - Updates minimatch from 10.2.1 to 10.2.4 (fixes HIGH severity ReDoS vulnerabilities) Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(test): update realtime guardrail test assertions to match actual guardrail behavior - test_text_message_blocked_by_guardrail_no_ai_response: allow guardrail's own block message text in response.done (previously expected empty content) - test_voice_transcript_blocked_by_guardrail: allow guardrail to send response.cancel + block message + response.create flow (previously expected no response.create) Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix: revert proxy-extras version in requirements.txt and pyproject.toml The litellm-proxy-extras 0.4.50 is not published to PyPI yet, so consumer references must stay at 0.4.49. Only the source package pyproject.toml should be bumped to 0.4.50 for the publish_proxy_extras CI job. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix: make transcript delta check optional in voice guardrail test The guardrail sends an error event (guardrail_violation) when blocking voice transcripts; it does not always produce transcript deltas. Remove the assertion requiring response.audio_transcript.delta since the error event is the primary signal that blocked content was handled. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * Add missing env keys to documentation: LITELLM_MAX_STREAMING_DURATION_SECONDS and LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES These two environment variables were used in code but not documented in the environment variables reference section of config_settings.md, causing the test_env_keys.py CI test to fail. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * Fix 13 mypy type errors across 6 files - in_flight_requests_middleware.py: Fix type: ignore error codes from [union-attr] to [attr-defined], add [arg-type] for Gauge **kwargs - transformation.py: Add [assignment] ignore for output_format reassignment, add fallback empty string for tool use id to fix arg-type - responses/main.py: Remove redundant type annotation on second secret_fields assignment to fix no-redef - streaming_iterator.py: Add [assignment] ignores for intermediate cache token assignments - handler.py: Add [typeddict-item] ignore for AnthropicMessagesRequest construction from dict - public_endpoints.py: Add [arg-type] ignore for _load_endpoints() return type mismatch with SupportedEndpoint model Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix: add auth overrides to spend tracking tests, fix realtime guardrail assertion, update UI minimatch - Add app.dependency_overrides for user_api_key_auth in 4 spend tracking tests that were returning 401 Unauthorized (error_code, error_message, error_code_and_key_alias, key_hash) - Fix realtime guardrail test to check ANY error event for guardrail_violation instead of just the first (OpenAI may send its own errors first) - Update ui/litellm-dashboard/package-lock.json to fix minimatch vulnerability Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * Fix failing MCP e2e and create_mcp_server UI tests Test 1 (test_independent_clients_no_shared_session): - Add allow_all_keys: true to MCP servers in test config. With master_key and no DB, get_allowed_mcp_servers returned empty, causing 0 tools and 403 on tool calls. allow_all_keys bypasses per-key restrictions. - Add asyncio.sleep(0.5) between client connections to allow MCP SDK TaskGroup cleanup and avoid ExceptionGroup on connection close (MCP #915). Test 2 (create_mcp_server 'auth value is provided'): - Use userEvent.setup({ delay: null }) for instant keystrokes to avoid timeout from default typing delay on CI. - Increase per-test timeout to 15000ms for CI environments. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix: stabilize proxy unit tests for parallel execution - test_response_polling_handler: add xdist_group to prevent heavy import OOM - test_db_schema_migration: use temp dir for worker isolation, sync schema.prisma index - test_custom_tokenizer_bug: use lighter tokenizer to prevent OOM in parallel Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix: add auth overrides to more spend tracking and model info tests - Fix test_ui_view_spend_logs_pagination missing auth override (401) - Fix test_view_spend_tags missing auth override (401) - Fix test_view_spend_tags_no_database missing auth override (401) - Fix test_empty_model_list.py to use app.dependency_overrides instead of patch() for FastAPI dependency injection auth Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(test): use patch.object for aiohttp transport test to work in parallel execution The @patch decorator was not intercepting the static method call in parallel xdist workers. Using patch.object on the directly-imported class is more reliable. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(security): update minimatch from 10.2.1 to 10.2.4 in Dockerfile The Docker image was explicitly pinning minimatch@10.2.1 which has HIGH severity ReDoS vulnerabilities (GHSA-7r86-cg39-jmmj, GHSA-23c5-xmqv-rm74). Update to 10.2.4 which includes fixes for both CVEs. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(ui): prevent MCP and TeamInfo test timeouts on CI - Add userEvent.setup({ delay: null }) to all tests using userEvent in both files - Add timeout: 15000 to tests with significant user interaction (typing, multiple clicks) - Fixes: create_mcp_server Bearer Token test, TeamInfo cancel button test Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix: stabilize parallel test execution and aiohttp transport test - test_aiohttp_handler: rewrite transport test to not rely on static method mock (consistently fails in parallel xdist workers) - test_proxy_cli: add xdist_group to prevent timeout during heavy imports - test_swagger_chat_completions: add xdist_group to prevent timeout Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(security): add serialize-javascript override to fix GHSA-5c6j-r48x-rmvq Add npm override for serialize-javascript>=7.0.3 in docs/my-website to fix HIGH severity RCE vulnerability via RegExp.flags. Also bump minimatch override to >=10.2.4. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * Fix flaky tests: remove broken Vertex model, add retries for Anthropic - Remove vertex_ai/meta/llama-4-scout-17b-16e-instruct-maas from test_partner_models_httpx_streaming - consistently returns 400 BadRequest - Add @pytest.mark.flaky(retries=6, delay=10) to test_function_call_parsing for transient Anthropic API overload errors - Add @pytest.mark.flaky(retries=6, delay=10) to test_openai_stream_options_call for transient Anthropic InternalServerError Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(ci): add xdist_group(proxy_heavy) to prevent OOM in parallel proxy tests - Add pytestmark = pytest.mark.xdist_group('proxy_heavy') to test_proxy_utils.py - Change test_db_schema_migration.py from schema_migration to proxy_heavy group - Add @pytest.mark.xdist_group('proxy_heavy') to test_proxy_server.py::test_health Groups heavy proxy tests to run on same worker, avoiding worker OOM crashes. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * Fix vertex AI qwen global endpoint test to mock vertexai module import The test_vertex_ai_qwen_global_endpoint_url test was failing because the VertexAIPartnerModels.completion() method tries to 'import vertexai' before any of the mocked code runs. In environments without google-cloud-aiplatform installed, this import fails with a VertexAIError(status_code=400). Fix by: - Adding patch.dict('sys.modules', {'vertexai': MagicMock()}) to mock the vertexai module import - Adding vertex_ai_location parameter to the acompletion call for completeness Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(ci): add xdist_group to health endpoint and watsonx tests for parallel stability - test_health_liveliness_endpoint: add xdist_group('proxy_health') to prevent timeout - test_watsonx_gpt_oss tests: add xdist_group('watsonx_heavy') to prevent mock interference Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(test): pre-populate WatsonX IAM token cache to prevent parallel test interference The watsonx prompt transformation test was failing in parallel execution because litellm.module_level_client.post mock was being interfered with by other tests. Pre-populating the IAM token cache avoids the HTTP call entirely. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(test): add spend data polling with retries for e2e pass-through tests - test_vertex_with_spend.test.js: Replace 15s fixed wait with polling loop (up to 6 attempts, 10s apart) for spend data to appear in DB - Increase test timeout from 25s to 90s to accommodate polling - base_anthropic_messages_tool_search_test.py: Add flaky(retries=3) for streaming test that depends on live Anthropic API Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(ci): reduce parallel workers from 8 to 4 for proxy tests to prevent OOM - litellm_proxy_unit_testing_part2: -n 8 -> -n 4 - litellm_mapped_tests_proxy_part2: -n 8 -> -n 4, timeout 60 -> 120 - Worker crashes consistently caused by too many parallel proxy tests each loading the full FastAPI app and heavy dependency tree Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(db): add migration for SpendLogs composite index (startTime, request_id) The @@index([startTime, request_id]) was added to schema.prisma but had no corresponding migration. This caused test_aaaasschema_migration_check to fail because prisma migrate diff detected the missing index. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(db): add migration for MCP available_on_public_internet default change to true The schema.prisma changed the default for available_on_public_internet from false to true, but no migration was created. This caused the schema migration test to detect drift. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(test): increase server wait time and add retry to flaky external API tests - test_basic_python_version.py: increase server startup wait from 60s to 90s for slower CI environments (fixes installing_litellm_on_python_3_13) - test_a2a_agent.py: add flaky(retries=3, delay=5) for non-streaming test that depends on live A2A agent endpoint Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(test): add flaky retries to all intermittent external API tests for 0-fail CI Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(test): add auth overrides to file endpoint tests that return 500 The test_target_storage tests were getting 500 because the FastAPI auth dependency wasn't overridden. Added app.dependency_overrides for proper auth bypass in test environment. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> |
||
|
Sameer Kankute
|
2349b51d80 | Fix test_pass_through_request_logging_failure | ||
|
Sameer Kankuteandyuneng-jiang
|
61eaf96046 | Fix passthrough tests | ||
|
Sameer Kankute
|
5399dbd1c1 | Fix beta header old tests | ||
|
Sameer Kankute
|
b92bf3756a | Fix beta header old tests | ||
|
Sameer Kankute
|
53bc1c8b79 | Fix test_bedrock_messages_api_header_forwarding | ||
|
|
2a3843aa57 |
[Fix] inconsistent response format in anthropic.messages.acreate() when using non anthropic providers (#20442)
* _translate_openai_content_to_anthropic * test_response_format_consistency * test fixes unit tests * test fix * fix: use dict access for anthropic content blocks in tests (#20447) The translate_openai_response_to_anthropic method returns dicts, not objects. Changed .type/.text/.thinking attribute access to dict ['key'] access. --------- Co-authored-by: shin-bot-litellm <shin-bot-litellm@berri.ai> |
||
|
Ishaan Jaffer
|
53d3868ff2 | TestBedrockInvokeToolSearch | ||
|
Sameer KankuteandGitHub
|
12463809bd |
Merge pull request #19638 from BerriAI/main
merge main in stagin 1 22 26 |
||
|
Harshit JainandGitHub
|
69c8698e62 |
fix: pass through endpoints update registry (#19420)
* fix: pass through endpoints update registry * add test case, fix lint error and comment to avoid confusion * fix pass through endpoints test case |
||
|
Ishaan JaffandGitHub
|
ab606c9a73 |
[Feat] Add Structured output for /v1/messages with Anthropic API, Azure Anthropic API, Bedrock Converse (#19545)
* fix: add AnthropicMessagesRequestOptionalParams * add _update_headers_with_anthropic_beta * fix output format tests * test_structured_output_e2e * TestAnthropicAPIStructuredOutput * test_structured_output_e2e * fix BASE * TestAzureAnthropicStructuredOutput * fix: Bedrock Converse * add nthropic Messages Pass-Through Architecture * fix: bedrock invoke output_format * fix: transform_anthropic_messages_request for vertex anthropic * TestBedrockInvokeStructuredOutput * docs anthropic vertex * docs fix * docs fix |
||
|
yuneng-jiang
|
1a9a7df437 | use mock db for cluade code marketplace | ||
|
Ishaan JaffandGitHub
|
a82467d679 |
[Feat] - Add self hosted Claude Code Plugin Marketplace (#19378)
* init schema * init endpoints * fix: claude_code_marketplace_router * refactor * fix: claude_code_marketplace_router * claude_code_marketplace_router |
||
|
Ishaan JaffandGitHub
|
e817aa713e |
[Fix] Claude Code x Bedrock Invoke fails with advanced-tool-use-2025-11-20 (#19373)
* _filter_unsupported_beta_headers_for_bedrock * test_bedrock_sonnet_4_5_with_advanced_tool_use_beta_header |
||
|
Ishaan JaffandGitHub
|
1417b002a3 |
[Feat] Claude Code x LiteLLM WebSearch - QA Fixes to work with Claude Code (#19294)
* fix websearch_interception_converted_stream * test_websearch_interception_no_tool_call_streaming * FakeAnthropicMessagesStreamIterator * LITELLM_WEB_SEARCH_TOOL_NAME * fixes tools def for litellm web search * fixes FakeAnthropicMessagesStreamIterator * test_litellm_standard_websearch_tool * use new hook for modfying before any transfroms from litellm * init WebSearchInterceptionLogger + ARCHITECTURE * fix config.yaml * init doc for claude code web search * docs fix * doc fix * fix mypy linting |
||
|
Ishaan JaffandGitHub
|
104283ae8f |
[Feat] Claude Code - Add Websearch support using LiteLLM /search (using web search interception hook) (#19263)
* init WebSearchInterceptionLogger * test_websearch_interception_real_call * init async_should_run_agentic_completion * async_should_run_agentic_loop * async_run_agentic_loop * refactor folder * fix organization * WebSearchTransformation * WebSearchInterceptionLogger * _call_agentic_completion_hooks * WebSearch Interception Architecture * test_websearch_interception_real_call * add streaming * add transform_request for streaming * get_llm_provider * test fix * fix info * init from config.yaml * fixes * test handler * fix _is_streaming_response * async_run_agentic_loop * mypy fix |
||
|
Ishaan JaffandGitHub
|
362b1a1577 |
[Feat] Add support for Tool Search on /messages API - Azure, Bedrock, Anthropic API (#19165)
* fix _update_headers_with_anthropic_beta * init ANTHROPIC_BETA_HEADER_VALUES * fix ANTHROPIC_BETA_HEADER_VALUES * fix: _update_headers_with_anthropic_beta - anthropic API * init _update_headers_with_anthropic_beta - azure AI support * init VertexAIPartnerModelsAnthropicMessagesConfig * fix _get_total_tokens_from_usage * working TestBedrockInvokeToolSearch * fix get_extra_headers * TestBedrockInvokeToolSearch * _get_tool_search_beta_header_for_bedrock * fix mypy linting |
||
|
Ishaan JaffandGitHub
|
458f773861 |
[Feat] Claude Code - Add support for Prompt Caching with Bedrock Converse (#19123)
* init BaseAnthropicMessagesPromptCachingTest * fix UsageDelta * fix: _create_initial_usage_delta * TestBedrockInvokePromptCaching * translate_anthropic_messages_to_openai wiht cache control * fix translate_anthropic_messages_to_openai |
||
|
Ishaan JaffandGitHub
|
06ded8750e |
[Fix] Claude Code (/messages) - Litellm fix claude code Bedrock Invoke usage, request signing (#19111)
* test_should_not_fail_with_forwarded_headers_bedrock_invoke_messages * use common get_request_headers for BaseAWS * fix get_request_headers * test_should_not_fail_with_forwarded_headers_bedrock_invoke_messages |
||
|
Ishaan JaffandGitHub
|
747829dadb |
[Fix] Claude Code + Bedrock Converse Usage - ensure budget tokens are passed to converse api correctly (#19107)
* test_bedrock_converse_budget_tokens_preserved * test_openai_model_with_thinking_converts_to_reasoning_effort * fix translate_anthropic_thinking_to_reasoning_effort * test_bedrock_converse_budget_tokens_preserved * test_anthropic_messages_bedrock_converse_with_thinking |
||
|
Alexsander HamirandGitHub
|
5534038e93 | Fix CI: Revert security scan changes and add GitGuardian ignore rules (#18358) | ||
|
Ishaan Jaffer
|
6112160a16 |
Revert "[Fix] Security - Remove example API keys with high entropy (#18255)"
This reverts commit
|
||
|
Alexsander HamirandGitHub
|
24edbccf5c | [Fix] Security - Remove example API keys with high entropy (#18255) | ||
|
Sameer Kankute
|
8fd0c81e5b | Add cost tracking for streaming in vertex ai | ||
|
Ishaan Jaffer
|
159db27d5c | fix test claude-sonnet-4-5-20250929 | ||
|
Sameer KankuteandGitHub
|
c1369a07ba | Add Add per model group header forwarding for Bedrock Invoke API (#16042) | ||
|
Sameer Kankute
|
85d4142845 | Fix litellm_param based costing | ||
|
Krish DholakiaandGitHub
|
64083111d3 |
(Feat) Add Vertex AI Live API WebSocket Passthrough with Cost Tracking
(Feat) Add Vertex AI Live API WebSocket Passthrough with Cost Tracking |