* fix(gemini): normalize response_schema on native generateContent
The /v1beta/models/{model}:generateContent passthrough forwarded
generationConfig.response_schema verbatim, so schemas containing $defs,
$ref, anyOf-with-null, default, or title were rejected by Gemini even
though /chat/completions already handles them.
GoogleGenAIConfig.transform_generate_content_request now calls a new
_normalize_response_schema helper that mirrors the chat/completions
path: Gemini 2.0+ models get the schema promoted to responseJsonSchema
via _build_json_schema (preserving $defs/$ref natively), older models
keep responseSchema but the schema is flattened with
_build_vertex_schema. VertexAIGoogleGenAIConfig (which overrides the
transform entirely) calls the same helper before building the request.
* fix(gemini): preserve caller-supplied responseJsonSchema when responseSchema co-present
Previously, when both responseJsonSchema and responseSchema were present
on Gemini 2.0+, _normalize_response_schema processed responseJsonSchema
first (no-op normalization) then unconditionally promoted responseSchema
to responseJsonSchema, clobbering the caller-supplied value.
Now skip the promotion (and drop the redundant responseSchema) when the
caller already supplied responseJsonSchema.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* chore: strip restating comments from response-schema normalize
Drop the docstring on _normalize_response_schema and the two inline
comments that just restated what the surrounding code/asserts already
say. Function name + variable names carry the intent; PR description
covers the why-it-exists context.
* perf(gemini): drop redundant deepcopy on responseJsonSchema normalize
_build_json_schema is a no-op (returns its argument unchanged), so the
deepcopy + round-trip on the responseJsonSchema branch allocated a full
schema copy on every request with no observable effect. Forward the
caller's value as-is, and just move the popped responseSchema value when
promoting on Gemini 2.0+.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* style: remove unneeded comment
* fix(gemini): drop unsupported responseJsonSchema for older models
* test(gemini): add parity test between native and chat schema normalization
Per @Sameerlite review: lock the two Gemini schema-normalization paths
together. If either GoogleGenAIConfig._normalize_response_schema (native
generateContent) or VertexGeminiConfig.apply_response_schema_transformation
(/chat/completions) drifts, the parity test fails — forcing both to be
updated together.
* fix(google_genai): preserve key naming convention in _normalize_response_schema
When the input schema key is snake_case (response_schema), the promoted
JSON schema key should also be snake_case (response_json_schema) instead
of mixing in camelCase (responseJsonSchema). This matters for the Vertex
AI google_genai path which converts all keys to snake_case before
calling _normalize_response_schema.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
OpenAI returns 'The model dall-e-3 does not exist' for the test account,
breaking test_openai_img_gen_health_check and test_image_generation.
Switch to gpt-image-1, matching the existing TestOpenAIGPTImage1 pattern.
SERVER_ROOT_PATH is a process-startup env var. Read it once in
__init__ instead of calling get_server_root_path() + rstrip on every
request that arrives before all lazy features have loaded.
LazyFeatureMiddleware compared the raw scope path against registered
prefixes (e.g. /policies), so requests under a server root path like
/api/v1/policies/... never matched, the feature never loaded, and the
endpoint returned 404. Strip the configured root path before matching,
normalizing trailing slashes and enforcing a component boundary so
/api does not falsely match /apiv2.
1. Missing litellm_request child span when proxy parent in metadata:
_get_span_context now returns (ctx, None) for the metadata-injected
proxy parent so the primary span is always emitted as a child of ctx.
Proxy span lifecycle managed by new _end_proxy_span_from_kwargs.
2. open_telemetry_logger overwrite by later handlers:
_init_otel_logger_on_litellm_proxy now uses first-registered-wins —
only assigns proxy_server.open_telemetry_logger when currently None.
3. Duplicate litellm_request success spans in streaming paths:
Added _mark_success_span_once with per-handler dedupe key stored in
kwargs metadata, suppressing the second span when both sync and async
success callbacks fire for the same request.
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`apply_client_tag_policy_pre_auth` overwrote string-typed metadata
with `{}` before merging header tags, dropping any tags inside. A
caller could send `metadata='{"tags":["over-budget"]}'` plus
`x-litellm-tags: within-budget` and bypass `_tag_max_budget_check`
on the body tag. Parse the string via `safe_json_loads` first so
existing tags survive the merge.
Also drop the empty `tests/test_litellm/proxy/credential_endpoints/`
directory — the cascade-rename tests it held imported a function
that was never implemented (out of scope for this PR).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
``model_dump(exclude_unset=True)`` in ``prepare_key_update_data``
includes any field the caller explicitly set, even when the value is
``None``. The previous guard short-circuited on ``getattr(data,
'user_id', None) is None``, which conflated "field omitted" (safe)
with "field explicitly set to null" (writes NULL to the token row,
detaching the key from its user and bypassing user-row role
checks).
Switch the omitted-vs-set distinction to ``data.model_fields_set``;
treat explicit-null and explicit-empty-string identically as a
removal attempt, both 403-rejected for non-admin callers.
Parametrized regression adds ``explicit_null_blocked`` alongside the
existing ``rebind_blocked`` / ``empty_blocked`` / ``same_user_id_allowed``
cases.
Removes the allow_client_tags metadata check from apply_client_tag_policy_pre_auth so
x-litellm-tags headers are always merged into request metadata, matching the post-auth
behavior in add_litellm_data_to_request. Updates pre-call tests accordingly and adds a
new test suite covering cascading credential renames into model rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous wording read "User=<new_owner> is not allowed to update the
key to belong to user=<current_owner>" — easy to misread as "caller
wants to keep the key on its current owner". Reframe as
"Non-admin caller is not allowed to rebind the key from
user=<existing> to user=<incoming>" so the direction of the failed
operation is unambiguous.
Same shape preserved (HTTPException 403); only the ``detail`` string
changes. Regression test substring updated.
A non-admin caller could rebind their own key's ``user_id`` via
``/key/regenerate``. ``_execute_virtual_key_regeneration`` had org/team
guards but no ``user_id`` guard, and ``prepare_key_update_data`` did not
strip the field — it survived ``model_dump(exclude_unset=True)`` into
the Prisma update. On the next request,
``_return_user_api_key_auth_obj`` resolved the rebound ``user_id``
against ``litellm_usertable`` and returned ``PROXY_ADMIN`` whenever
the target row's ``user_role`` was admin (e.g. the default
``user_id="default_user_id"`` created on first password-UI login).
``/key/update`` had the equivalent guard inline at
``_validate_update_key_data``; extract it to a shared helper
``_validate_caller_can_change_key_ownership`` and call from both
``/key/update`` and ``_execute_virtual_key_regeneration``. Future
regenerate-style endpoints inherit the guard for free.
Also tighten the premium gate that allowed the master-key rotation
branch to skip the enterprise check. The previous predicate was
``data.new_master_key is not None`` — a field-presence test, not an
identity check. Any non-premium caller could send any value in that
field and the premium check would no-op. Verify the caller actually
holds the master key via ``_is_master_key`` before allowing the
non-premium path.
Tests:
- ``test_regenerate_user_id_rebind_guard`` — parametrized table over
cross-user rebind (blocked), empty-string removal (blocked), and
same-user no-op rebind (allowed).
- ``test_regenerate_premium_gate_requires_actual_master_key`` /
``test_regenerate_premium_gate_allows_actual_master_key_holder`` —
ensure the premium check requires the caller actually present the
master key, and that legitimate master-key rotation still works.
* fix(proxy): resolve cache handling issues in _lookup_deprecated_key
- Updated the in-memory cache for deprecated key lookups to store a 3-tuple (active_token_id, cache_expires_at_ts, revoke_at_ts) instead of a 2-tuple, ensuring proper unpacking and backward compatibility.
- Removed duplicate cache reads and added logic to handle legacy cache entries gracefully.
- Enhanced unit tests to cover scenarios for cache hits, DB misses, and respect for revoke_at timestamps, ensuring robust handling of the grace-period key-rotation feature.
* refactor(proxy): streamline cache handling in _lookup_deprecated_key
- Simplified the cache retrieval logic by directly unpacking the 3-tuple cache entries, removing the need for backward compatibility checks for 2-tuple entries.
- Updated unit tests to ensure that pre-warmed 3-tuple cache entries are served correctly without unnecessary database lookups.
* chore(ci): add new unit test for deprecated key grace period
- Included `test_deprecated_key_grace_period.py` in the CI workflow to enhance coverage for deprecated key handling scenarios.
* fix(proxy): remove unnecessary check for revoke_at in _lookup_deprecated_key
- Eliminated the redundant check for None on revoke_at, streamlining the logic for handling deprecated keys in the cache. This change enhances the efficiency of the key lookup process.
* test(proxy): add end-to-end tests for deprecated key lookup behavior
- Introduced a new test class `TestDeprecatedKeyLookupDbE2E` to validate the behavior of deprecated key lookups against a real Prisma-backed database.
- The test ensures that old key hashes resolve correctly and that repeated lookups utilize the in-memory cache without errors.
- Cleaned up the `_lookup_deprecated_key` function by removing an unnecessary check for `revoke_at`, enhancing the efficiency of the key lookup process.
* feat(ui): add Vertex AI Search as vector store provider
Adds a "Vertex AI Search" entry to the provider dropdown
(custom_llm_provider=vertex_ai/search_api) with fields for project,
location (global/us/eu select), and optional collection ID. Extends
VectorStoreFieldConfig with `options` so select fields can be
data-driven instead of falling through to the embedding-model list.
* fix(ui): clarify vertex_collection_id placeholder copy
Placeholder previously displayed "default_collection" — the literal
fallback value — which invited users to type it instead of leaving the
field blank. Switch to an example placeholder and tighten the tooltip.
The tag-strip block was removed in the parent commit but two surrounding
comments still referenced "tags without opt-in" and "runs AFTER the
strip". Update them to describe the remaining user_api_key_* and
_pipeline_managed_guardrails strip that the snapshot/merge ordering
actually protects against.
Caller-supplied tags (`x-litellm-tags` header, body `tags`, `metadata.tags`)
were silently dropped unless the key/team had
`metadata.allow_client_tags: true` set. Restore the documented behavior:
tags from the request always flow into `metadata.tags` and union with any
admin-configured static tags from key/team/project metadata.
Removes the `allow_client_tags` opt-in flag from the pre-call pipeline.
The flag was only ever read here; it has no schema or endpoint footprint,
so leftover values in existing key metadata are inert.
Test cleanup mirrors the simplification: drop the three tests that
verified the strip-when-not-opted-in path, drop the `allow_client_tags`
fixture lines from the merge/union tests.
Second wave of failures from the 2026-05-12 DALL-E shutdown:
- tests/image_gen_tests/test_image_edits.py::TestOpenAIImageEditDallE2
and tests/image_gen_tests/test_image_generation.py::TestOpenAIDalle3
are explicitly named for the deprecated models and can't pass; remove.
gpt-image-1 coverage already exists in sibling classes.
- tests/local_testing/test_router.py image gen tests use dall-e-3 only
as a routing example; swap to gpt-image-1.
- tests/local_testing/test_custom_callback_input.py image_generation
success/failure paths swapped to gpt-image-1.
DALL-E 2 and DALL-E 3 were removed from the OpenAI API on 2026-05-12,
causing e2e image-generation tests to fail with "model does not exist".
Swap all live-API DALL-E references in proxy-backed tests to gpt-image-1
and update the dall-e-2 alias in proxy_server_config.yaml to point at
openai/gpt-image-1 (preserves any historical dall-e-2 callers).
* chore: reject bare str at file-input sinks to prevent local-file read (#27667)
Squash-merged by litellm-agent from stuxf's PR.
* fix: use os.PathLike in ocr sink and check truthy reasoningSummary for bridge
- ocr/main.py: widen Path check to os.PathLike for consistency with other sinks
- main.py: bridge condition checks truthiness of reasoning_summary, not just None
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: remove unused pathlib.Path import in ocr/main.py
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Second wave of failures from the 2026-05-12 DALL-E shutdown:
- tests/image_gen_tests/test_image_edits.py::TestOpenAIImageEditDallE2
and tests/image_gen_tests/test_image_generation.py::TestOpenAIDalle3
are explicitly named for the deprecated models and can't pass; remove.
gpt-image-1 coverage already exists in sibling classes.
- tests/local_testing/test_router.py image gen tests use dall-e-3 only
as a routing example; swap to gpt-image-1.
- tests/local_testing/test_custom_callback_input.py image_generation
success/failure paths swapped to gpt-image-1.
DALL-E 2 and DALL-E 3 were removed from the OpenAI API on 2026-05-12,
causing e2e image-generation tests to fail with "model does not exist".
Swap all live-API DALL-E references in proxy-backed tests to gpt-image-1
and update the dall-e-2 alias in proxy_server_config.yaml to point at
openai/gpt-image-1 (preserves any historical dall-e-2 callers).
The tag-strip block was removed in the parent commit but two surrounding
comments still referenced "tags without opt-in" and "runs AFTER the
strip". Update them to describe the remaining user_api_key_* and
_pipeline_managed_guardrails strip that the snapshot/merge ordering
actually protects against.
Caller-supplied tags (`x-litellm-tags` header, body `tags`, `metadata.tags`)
were silently dropped unless the key/team had
`metadata.allow_client_tags: true` set. Restore the documented behavior:
tags from the request always flow into `metadata.tags` and union with any
admin-configured static tags from key/team/project metadata.
Removes the `allow_client_tags` opt-in flag from the pre-call pipeline.
The flag was only ever read here; it has no schema or endpoint footprint,
so leftover values in existing key metadata are inert.
Test cleanup mirrors the simplification: drop the three tests that
verified the strip-when-not-opted-in path, drop the `allow_client_tags`
fixture lines from the merge/union tests.
The enterprise package is installed as `litellm_enterprise` (per
enterprise/pyproject.toml), but several tests imported it as
`enterprise.litellm_enterprise.*` — a path that only resolves
because the repo root happens to sit on sys.path, letting Python's
implicit namespace package machinery discover `enterprise/` as a
directory.
This breaks any test runner that relocates source (e.g. the
mutation-testing workflow, which copies tests under `mutants/`) and
also caused two `patch()` strings to target a module path that does
not match what production code imports — meaning those mocks were
never actually patching the production module's attribute.
Replace `from enterprise.litellm_enterprise.` with the canonical
`from litellm_enterprise.` across 6 test files, and fix two
`patch()` target strings (and one `sys.modules` patch key in the
SSO test) to match.
* feat(ui): search teams by team ID alongside name
The Teams page search box only matched team_alias, so pasting a team UUID
returned zero results. Detect when the input is a full UUID and route it
to the team_id filter instead; otherwise keep the existing alias substring
search. Placeholder now reads "Search teams by name or ID...".
Resolves LIT-2648
* refactor: search teams via backend OR clause, drop client-side UUID detection
Adds a `search` query param to /v2/team/list that ORs across team_id
(exact) and team_alias (case-insensitive contains), so the search box
sends one param regardless of input format. Removes the isLikelyTeamId
helper and the client-side branching it fed.
* fix(ui): resolve created_by to human-readable name for team keys
The team's Virtual Keys table rendered a raw UUID under Created By
for keys created on behalf of a real user, and the key details page
header showed "-" because it was reading the wrong field
(user_email/user_id instead of created_by_user).
- TeamVirtualKeysTable Created By column now prefers
created_by_user.user_alias > user_email > UUID, with a Popover
on hover that surfaces all three fields with copy icons
(matches the existing AllKeys table pattern in VirtualKeysTable.tsx)
- key_info_view passes the resolved created_by_user value to the
details-page header so it renders the readable name
Resolves LIT-2517
* fix(ui): add created_by to KeyResponse type
The backend returns created_by on every key row, but the frontend
type omitted it. The Created By cell already reads the field via
info.row.original, which trips the production typecheck.
* feat(ui): add Expires to key Overview header; merge User into one field
The key details page header omitted Expires (only Settings tab had it)
and showed User Email + User ID as two separate rows. This PR:
- adds Expires below Created At, reusing the Settings tab's
formatTimestamp(...) ?? "Never" formatting so both views agree
- merges User Email / User ID into a single "User" field that
displays alias / email / user_id (in that fallback order) with a
Popover on hover exposing all three with copy icons — mirrors the
Created By pattern in TeamVirtualKeysTable.tsx and VirtualKeysTable.tsx
Refs LIT-2517
* chore(ui): User field icon + alias-primary test
Address review feedback on #27696:
- Swap MailOutlined → UserOutlined on the merged User cell; the
envelope icon implied "this is an email" but the cell can render
alias / email / user_id depending on what's available.
- Add a test asserting userAlias displays as primary and overrides
userEmail — closes the gap where a fallback-order regression
would have silently passed.
* fix(ui): truncate long User values and reshuffle header layout
- Swap column groupings so Created By stays paired with Created At
(matches the pre-merge layout); User and Expires now share col 1.
- Ellipsis-truncate the visible User cell at maxWidth 200 so a raw
UUID fallback doesn't sprawl. Full identity still revealed via the
hover Popover.
- Cap each Popover row at maxWidth 220 so a UUID truncates inside the
panel too; antd's built-in ellipsis tooltip surfaces the full value.
* fix: invalidate cached tag object on tag budget reset (#27481) (#27572)
Squash-merged by litellm-agent from oss-agent-shin's PR.
* chore(mcp): tighten stdio server registration paths (#27570)
Squash-merged by litellm-agent from stuxf's PR.
* fix(proxy): clear MCP OpenAPI mappings on server eviction; widen budget cache invalidation
Evict OpenAPI tools from global_mcp_tool_registry and strip tool_name_to_mcp_server_name_mapping entries when a server leaves the runtime registry (remove_server and approval-status eviction). Invalidate user_api_key_cache for keys, orgs, and team members on budget-tier spend resets alongside tags.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): align update_server eviction with remove_server name fallback
Document budget-reset test assertion flip (cross-pod cache staleness).
Greptile: eviction now pops by server_id then server_name like remove_server;
test docstring explains assert_not_awaited -> assert_any_await change.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix org budget cache invalidation
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Greptile flagged that the strict isinstance(dict) guard in
_encryption_boundary_key would silently return None for any non-dict input,
including a LiteLLM_Params Pydantic instance, which exposes a custom .get()
method and is intended to be used dict-style in some router paths. If such
an instance ever flowed into healthy_deployments, the guard would drop every
candidate from boundary matching and fall through to the full deployment
pool, i.e. trigger the exact invalid_encrypted_content failure this check
exists to prevent.
Loosen the guard to accept any object exposing a callable .get(): plain
dicts (the common case) and LiteLLM_Params-style Pydantic instances. The
function still returns None for non-dict-like values (None, lists, strings,
ints, bare objects).
Adds regression tests covering:
- LiteLLM_Params Pydantic instance resolves to the same boundary tuple as
an equivalent plain dict
- non-dict-like values and dicts missing required fields still return None
When a Responses API follow-up switches model_name (e.g. gpt-5.3-codex ->
gpt-5.4, or to a LiteLLM-side alias of the same Azure deployment), the
router has already filtered healthy_deployments to the new group, so the
originating model_id is no longer present. The encrypted_content_affinity
check would log "decoded deployment not found" and fall back to the full
deployment pool, where simple-shuffle could land on a different Azure
resource and trip a 400 invalid_encrypted_content.
Fall back to pinning by the originating deployment's encryption boundary
(api_base + api_key) when the model_id miss is across model groups. The
encrypted_content travels with the Azure resource, not the model_name,
so any deployment on the same resource accepts it.
LIT-2531
* fix(tests): drop deprecated OpenAI-Beta realtime header
OpenAI deprecated the 'OpenAI-Beta: realtime=v1' header; the live
service now returns code 4000 invalid_beta with
"Unknown beta requested: 'realtime'.". Two integration tests in
tests/llm_translation/realtime/test_realtime_guardrails_openai.py
hardcoded the header and started failing across all PRs.
Library code is unaffected: the OpenAI realtime handler only
forwards 'OpenAI-Beta: realtime=v1' upstream when the proxy *client*
sends it (litellm/llms/openai/realtime/handler.py). Default proxy
behavior uses the GA protocol.
Connect to OpenAI without the deprecated header, and accept the GA
event name 'response.output_audio_transcript.delta' alongside the
beta-protocol name 'response.audio_transcript.delta' for the
transcript-delta assertion.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(logging): post_call tolerates non-JSON-serializable values
post_call() did json.dumps(original_response) without default=str, so
any provider passing a dict containing datetime/Decimal/etc. would
raise TypeError. Bedrock batch retrieval hits this with
get_model_invocation_job() responses that include datetime fields
(submitTime, lastModifiedTime, endTime), failing
tests/batches_tests/test_bedrock_files_and_batches.py::test_async_file_and_batch
across all PRs.
Pass default=str so non-serializable values fall back to str().
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(tests): mock boto3 in bedrock retrieve batch test
The test patched AsyncHTTPHandler.get, but the bedrock retrieve
handler uses boto3.client('bedrock').get_model_invocation_job
directly, so the real AWS call was being made on every run, failing
with AccessDeniedException because the hardcoded test ARN belongs to
a different AWS account.
- Mock boto3.client and BedrockBatchesConfig.get_credentials so the
test never touches AWS.
- Use status=Completed in the mock response so output_file_id is
populated (the handler intentionally leaves it None for
non-completed jobs).
- Assert the predicted per-job output object URI (matches what the
handler actually returns) instead of the bare output prefix.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* docs(tests): include GA event name in guardrail-block test docstring
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* ci: add manually-triggered mutation testing smoke workflow
Adds a workflow_dispatch-only GitHub Actions workflow that runs mutmut
against a single source/test pair (router_settings_endpoints) to validate
the tooling end-to-end before scaling.
The workflow reinstalls litellm non-editable so the mutants/ sandbox is
not shadowed by the editable .pth on sys.path, and sets PYTHONPATH so
the trampolined sandbox copy wins over site-packages.
mutmut itself is pulled in via uv run --with so it does not appear in
uv.lock or affect the shared dev environment.
Includes a temporary push: trigger scoped to this branch so we can
iterate before the workflow file lands on the default branch — to be
removed before merging (workflow_dispatch only requires the file on the
default branch to surface the manual trigger button).
* ci(mutation): disable rerun and xdist plugins for mutmut runs
mutmut's in-process pytest.main() call hits
`INTERNALERROR: no option named 'filtered_exceptions'` from
pytest-retry's pytest_configure hook. Reruns are also wrong for
mutation testing — a "failed" mutant test that gets retried would
mask which mutants are killed vs. survive. Disable retry,
rerunfailures, and xdist via pytest_add_cli_args in [tool.mutmut].
* ci(mutation): uninstall pytest-retry before mutmut runs
`-p no:retry` (and similar names) didn't match pytest-retry's
entry-point name, so the plugin still loaded and crashed during
mutmut's "Running clean tests" phase. Uninstalling the package is
surgical and doesn't depend on guessing the entry-point name.
* ci(mutation): emit per-survivor diffs to run-page summary + artifact
The previous artifact only contained `mutmut results` text (which in
mutmut 3.x lists survivor names but not the actual mutations). Adds:
- `mutmut export-cicd-stats` to produce mutmut-cicd-stats.json with the
killed/survived/total scoreboard.
- `mutmut show <name>` per surviving mutant to capture each mutation as
a unified diff.
- A `mutmut-report.md` that combines summary + run-progress tail +
per-survivor diffs, written to both the artifact and
$GITHUB_STEP_SUMMARY (visible on the run page, no download needed).
- Corrected artifact paths: stats files live under mutants/, not the
project root.
- The trampolined source file from the sandbox so survivors can be
inspected even outside `mutmut show`.
* ci(mutation): document intended manual weekly cadence in trigger comment
* ci(mutation): generate ACH-style report with embedded function bodies
Replaces the inline bash markdown generation with a Python script that:
- Groups survivors by function (one section per function, function body
shown once per section, surviving mutants nested as subsections)
- Embeds each enclosing function's source via Python AST (so the agent
has full context, not just a 3-line `mutmut show` diff)
- Inlines the existing test file(s) listed in [tool.mutmut].tests_dir
- Writes an ACH-style task description at the bottom following the
prompt template from arXiv 2501.12862
Output goes to mutation-report.md (artifact) and the head of the file
is appended to $GITHUB_STEP_SUMMARY for at-a-glance visibility.
* fix(mutation report): correctly parse function names with leading underscores
mutmut's mutant-name prefix is x_ (single underscore), so a function
named _foo produces mutants x__foo__mutmut_N. The previous regex
\.x__(.+)__mutmut_ ate the function's leading underscore as part of
the prefix. Changed to \.x_(.+)__mutmut_ so leading underscores are
preserved in the captured function name; verified for normal, leading-
underscore, and dunder-method names.
* feat(mutation report): full Meta ACH-style rendering with MUTANT delimiters
For each surviving mutant, parse the mutmut sandbox trampoline file and
render the mutated function as it appears in the source — with the
differing lines wrapped in `# MUTANT START` / `# MUTANT END` comments,
matching the format from Meta's ACH paper (arXiv 2501.12862, Table 1).
Renames the function header back to its original name so the agent sees
the function as it would appear in the file. Falls back to the unified
diff if the trampoline lookup fails.
Handles replace, insert, and delete diff ops; uses difflib's
SequenceMatcher to find the differing line ranges.
The unified diff is preserved in a collapsible <details> block as
secondary context.
* ci(mutation): scope to whole management_endpoints folder, drop temp push trigger
Final scope before merge:
- paths_to_mutate / tests_dir broadened from one file to the entire
management_endpoints source/test folders
- Trigger is now `workflow_dispatch` only — the temporary push: block
used during workflow iteration is removed
- timeout-minutes bumped from 60 to 350 (just under the GH-hosted job
cap of 360); whole-folder mutation against ~15 files / ~7.5k LOC can
take a few hours
- Artifact path for the trampoline files glob-expanded to cover all
files under mutants/litellm/proxy/management_endpoints/
* fix(mutation report): warn when multiple functions in a file share a name
Addresses the Greptile review concern: ast.walk's first-match-wins
behavior could embed the wrong function body when a file defines the
same name in multiple places (e.g., a module-level helper and a class
method). mutmut's mutant identifier does not carry class context, so
we can't always determine which definition was mutated.
find_function_in_file now returns the start line of every matching
definition; render() surfaces a "Note: N functions named X" warning
in the report when there is more than one match. The first match is
still embedded as the body — the warning tells the reader to verify
manually instead of silently using the wrong context.
Smoke-tested against the existing artifact: single-match files render
unchanged.
* Fix mutation report anchors
* Fix mutation report TOC anchors
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(bedrock): support retrieve for model-invocation-job batch ARNs
`bedrock.retrieve_batch` previously only handled `:async-invoke/` ARNs
(Twelve Labs Marengo embeddings). The `:model-invocation-job/` ARNs
returned by `CreateModelInvocationJob` (the bulk batch inference API
behind `bedrock.create_batch`) fell through and returned a misleading
data-plane error, leaving created jobs unretrievable through the
LiteLLM batches API.
The two ARN families live on different AWS service endpoints
(`bedrock-runtime` data plane vs `bedrock` control plane), so they need
distinct handlers. This adds:
* `BedrockBatchesHandler._handle_model_invocation_job_status` — calls
the control plane via boto3 (`bedrock:GetModelInvocationJob`),
reusing `BaseAWSLLM.get_credentials` for credential resolution so
model_list / env / role-assumption configs continue to apply. The
response is reshaped into a `LiteLLMBatch` with the same status
mapping `transform_create_batch_response` already uses.
* Output-file-URI prediction. Bedrock surfaces the user-supplied
`s3OutputDataConfig.s3Uri` *prefix* in `GetModelInvocationJob`, but
results actually land at `<prefix>/<job-id>/<basename(input)>.out`.
We compute that single-file URI client-side and surface it as
`output_file_id`, so OpenAI-style `client.files.content(...)` works
without an extra `ListObjectsV2` round-trip. The bare prefix stays
in metadata for callers that want the manifest.
* Dispatch in `litellm/batches/main.py` for the new ARN family,
alongside the existing async-invoke branch.
* Unit tests covering ARN parsing, output-URI prediction (incl. edge
cases), the full status mapping, region resolution precedence, and
failure-message propagation.
Note: `request_counts` is intentionally `(0, 0, 0)` —
`GetModelInvocationJob` does not report per-record counts; getting
accurate numbers requires parsing `manifest.json.out` from the output
S3 prefix, which is left to callers.
Made-with: Cursor
* fix(bedrock): address PR feedback on model-invocation-job retrieve
Addresses Greptile P2 findings on #26834:
1. Use the bare job id (not the full ARN) when constructing the
`api_base` URL for `pre_call` logging. Passing the full ARN double-
counts the `model-invocation-job/` segment and embeds colons in the
path, producing misleading log lines.
2. Drop the `or output_prefix` fallback when `_predict_output_file_uri`
returns None. A bare prefix is not a downloadable object and surfacing
it as `output_file_id` re-creates the very NoSuchKey bug this handler
exists to fix. The bare prefix is still preserved in
`metadata["output_s3_uri"]` for callers that want to do their own S3
listing or read `manifest.json.out`.
`metadata["output_file_uri"]` uses "" rather than None to satisfy the
OpenAI Batch metadata schema (`dict[str, str]`); callers should branch
on the typed `output_file_id` field instead.
Also expands test coverage on the new code path:
- new "stay None" regression test for the prediction-fail case
- pre_call/post_call logging hook assertions (incl. the bare-id URL)
- explicit cancelled_at / expired_at coverage
- _to_epoch type-handling matrix and the boto3 ImportError branch
- defensive _extract_region_from_bedrock_arn exception path
- empty-basename case for _predict_output_file_uri
Patch coverage on the changed lines is now 100% (the only remaining
uncovered lines in the file belong to the pre-existing
`_handle_async_invoke_status` method, which this PR does not touch).
Made-with: Cursor
* test(bedrock): cover retrieve_batch dispatch for both ARN families
Codecov flagged 8 uncovered lines on `litellm/batches/main.py` after
this PR refactored the Bedrock dispatch into a single guard with two
sub-branches (`async-invoke` + `model-invocation-job`). Existing tests
exercised the handlers directly but not the dispatch in `main.py`.
Adds `tests/test_litellm/batches/test_retrieve_batch_bedrock_dispatch.py`
with 6 mocked tests that exercise `litellm.retrieve_batch` end-to-end
for the dispatch logic:
- async-invoke ARN routes to `_handle_async_invoke_status`
- async-invoke ARN with no region falls back to "us-east-1" (preserves
prior behavior on this branch)
- model-invocation-job ARN routes to the new
`_handle_model_invocation_job_status` handler
- model-invocation-job ARN with no region forwards None (so the new
handler can sniff region from the ARN itself, rather than getting
silently routed to us-east-1)
- unrelated bedrock ARN family falls through to the generic
provider-config retrieve path (neither special handler invoked)
- non-bedrock batch ids skip the bedrock dispatch entirely
Both handlers are mocked at the import site so the tests don't hit
AWS — the focus here is purely the new dispatch logic in main.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(bedrock): move retrieve_batch dispatch test to tests/test_litellm/
The dispatch test landed under `tests/test_litellm/batches/`, a new
directory that no upstream `test-unit-*.yml` workflow's `test-path`
allow-list includes. As a result, the test was never executed in CI
and codecov reported `litellm/batches/main.py` patch coverage at
11.11% (8 lines uncovered) — the lines belonging to this PR's
dispatch refactor itself.
Move the file up one level so it matches the
`tests/test_litellm/test_*.py` glob that `test-unit-misc.yml`
already runs, and adjust `sys.path.insert` for the new depth.
The companion handler tests under
`tests/test_litellm/llms/bedrock/batches/test_handler.py` are
unaffected — they're picked up by the `llms` directory in
`test-unit-llm-providers.yml`.
Made-with: Cursor
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(responses): map chat tool_choice to Responses API when bridging from completions
OpenAI /v1/responses rejects tool_choice.function. Normalize forced-function
choice from chat shape to {type, name} in LiteLLMResponsesTransformationHandler.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(responses): strip tool_choice.function when top-level name is set
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
