tiennm99/litellm
1
0
Fork 0
mirror of https://github.com/tiennm99/litellm.git synced 2026-08-03 02:20:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

Litellm oss 090626 (#30021)

Browse Source 
* fix(mcp): report scoped server name during initialize (#29865)

* fix mcp scoped server name

* Update litellm/proxy/_experimental/mcp_server/mcp_context.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* test(mcp): cover scoped server name in the SSE initialize handler

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(ui): show all session logs in the drawer, not just the first 50 (#29795)

* fix(ui): show newest session logs first

* test(ui): keep session log pagination coverage

* fix(ui): show all session logs in the drawer, not just the first page

The session detail drawer fetched session logs via sessionSpendLogsCall
without page/page_size, so it only ever received the backend default of one
page (50 rows). Sessions with more than 50 calls had the rest unreachable in
the UI (#29153).

sessionSpendLogsCall now takes page/page_size, and the drawer fetches the
first page, reads total_pages, then fetches the remaining pages and
accumulates them before the existing client-side sort. This keeps the single
continuous list (and the selected-log lookup and keyboard navigation, which
all assume the full session) correct. Fetching is bounded by a page cap, and
the sidebar shows a "showing most recent N" note if a session exceeds it.

The rows are lightweight metadata (the endpoint excludes messages/response),
so the full set is small; request/response bodies are still loaded per log on
demand.

* fix(ui): default session drawer to most recent log, newest first

Open a session with its most recent log selected, and order the sidebar
newest-first to match the all-sessions logs overview. MCP calls stay
grouped last. The latest log by time is computed explicitly, since the
MCP grouping means it is not always the first row.

* Apply fetching pages in batches suggestion from @greptile-apps[bot]

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(ui): derive session total from accumulated rows when backend omits it

Compute the session total after all pages are fetched, falling back to the
accumulated row count rather than the first page's. Guards the truncation
note against a backend response that omits total but spans multiple pages.

---------

Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(proxy): handle Mistral multipart passthrough (#29927)

* fix(proxy): handle Mistral multipart passthrough

* chore: satisfy passthrough ci formatting

* test(proxy): cover Mistral passthrough in CI shard

* fix(vertex_ai): use REP host for context caching on eu/us multi-region endpoints (#29573)

Context caching built the cachedContents URL as
https://{location}-aiplatform.googleapis.com, which is an invalid host for the
eu/us multi-region endpoints and returns 404. The inference path already
resolves these to the REP host (https://aiplatform.{geo}.rep.googleapis.com)
via get_vertex_base_url(); reuse that helper in
_get_token_and_url_context_caching so caching uses the same host as inference.

Adds tests covering the eu/us multi-region cachedContents URLs (v1 and
v1beta1).

Fixes #29571

* Support per-model encrypted content affinity config (#29760)

Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>

* fix: propagate upstream status code in proxy API exception handler (#29402)

* fix: propagate upstream status code in proxy API exception handler

When Google GenAI / Vertex returns a 404 for deprecated or missing
models via streamGenerateContent, the exception was falling through to
a generic handler that defaulted to 500. Now provider exceptions
carrying a valid HTTP status_code correctly propagate it through to
the ProxyException.

* fix: apply black formatting to common_request_processing.py

* fix: tighten status code range to 400-599 and deduplicate ProxyException raise

* fix(tests): use valid vertex_location in context caching tests

Replace "test_location" (contains underscore) with "us-central1" so tests
pass the regex validation added in get_vertex_base_url().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(sdk): add xAI OAuth provider (#29866)

* Add xAI OAuth provider

* Update oauth.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Fix xAI OAuth CI failures

* Add xAI OAuth coverage tests

* Move xAI OAuth coverage tests to core utils

* Address xAI OAuth review comments

* Prevent xAI OAuth api_base token exfiltration

* Treat blank xAI OAuth api keys as absent

* Wrap invalid xAI OAuth JSON responses

* Use xAI OAuth behind explicit flag

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(proxy) #27734 allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update (#27751)

* fix(proxy): allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update

Fixes #27734

Sending null for budget_duration, team_member_budget,
team_member_budget_duration, team_member_rpm_limit, or
team_member_tpm_limit via /key/update or /team/update returned 200 OK
but silently ignored the null value. The fields remained unchanged in
the database.

Root causes:
- /key/update: prepare_key_update_data() popped budget_duration from the
  update dict but never re-added it (or budget_reset_at) when the value
  was None.
- /team/update: _set_budget_reset_at() only acted when budget_duration
  was non-None, leaving a stale budget_reset_at in the DB.
- /team/update: team_member_* null values bypassed the budget table
  update entirely because should_create_budget() requires at least one
  non-None field.

* test(proxy): cover no-budget-row path in clear_team_member_budget_fields

* fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes (#30028)

* fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes

When output_parse_pii=true on the Anthropic native path (anthropic/claude-*),
response chunks arrive as raw bytes in SSE format. _stream_pii_unmasking was
yielding those bytes unchanged, so <PERSON_1> tokens were never replaced with
the original values before reaching the caller.

Add _unmask_sse_bytes_chunk to parse each data: line, find content_block_delta
/ text_delta events, and apply _unmask_pii_text before re-encoding. Wire it
into _stream_pii_unmasking so bytes chunks are unmasked when pii_tokens exist.

* fix(presidio): handle CRLF line endings and non-ASCII PII in SSE unmask

Strip trailing \r before the [DONE] guard so CRLF-terminated SSE chunks
don't bypass it and silently swallow a JSONDecodeError. Add
ensure_ascii=False to json.dumps so non-ASCII replacement values like
accented names are preserved as UTF-8 on the wire rather than being
\uXXXX-escaped. Add regression tests for both cases.

* feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses) (#29925)

* feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses)

Bedrock Mantle serves the Responses API on two upstream paths:
  - gpt frontier models (gpt-5.5 / gpt-5.4) on /openai/v1/responses
  - every other Responses-capable model (e.g. gpt-oss) on the standard /v1/responses

BedrockMantleResponsesAPIConfig gains a `use_openai_path` flag; the provider gate in
utils.py picks the path per model: openai.gpt-* (non gpt-oss) -> /openai/v1/responses;
any model declared mode=responses (price-map entry or user model_info) -> /v1/responses;
everything else returns None and keeps the existing chat-completions emulation.

Adds gpt-5.5 / gpt-5.4 price-map entries, registry wiring, and the routing-matrix tests.

* feat(bedrock_mantle): data-driven frontier routing via use_openai_responses_path

Addresses the Greptile review point that frontier detection should be a
price-map field rather than a hardcoded name match. The gate now routes a
model to /openai/v1/responses when its price-map entry declares
use_openai_responses_path, so a frontier model whose name does not follow the
openai.gpt- convention can be onboarded by JSON alone. The name-convention
check is kept as a fallback that needs no price-map entry, which preserves
zero-change routing for a future gpt-6 before its entry loads. gpt-5.5 / gpt-5.4
get the flag in both price maps. Adds tests for the data-driven flag path and
for the flag presence on the gpt-5.x entries; both branches are mutation-tested.

* test(model_prices): allow use_openai_responses_path in price-map schema

The model_prices_and_context_window.json schema validator
(test_aaamodel_prices_and_context_window_json_is_valid) enforces
additionalProperties: false, so the new use_openai_responses_path flag on the
gpt-5.5 / gpt-5.4 entries failed validation. Add it to the schema as a boolean,
alongside the other supports_* / capability flags.

* Add Tensormesh serverless models to the model cost map (#30037)

* Add Tensormesh serverless models to the model cost map

* Flag reasoning support on the Tensormesh models that expose thinking mode

* fix(proxy): invalidate stale key spend counter after budget reset or manual spend update (#30001)

* fix(proxy): reconcile stale key spend counter after budget reset

* fix(proxy): invalidate stale key spend counter after budget reset or manual spend update

* fix(proxy): remove read-time stale counter reconciliation to prevent budget bypass

* revert: undo unrelated formatting changes in enterprise directory

* test(proxy): add unit test for key spend update invalidating counter

* test(proxy): fix mocked update_data and hash token expectations in unit test

* fix(proxy): use Responses-API transformer in pass-through cost tracking (#29728)

The `elif is_responses:` branch of `openai_passthrough_handler` was
calling the chat-completions `transform_response` on a Responses API
payload. The chat-completions transformer expects `choices: [...]`
in the raw response; the Responses API uses `output: [...]` and
`usage.input_tokens` / `usage.output_tokens` (not
`prompt_tokens` / `completion_tokens`). The result was a
KeyError 'choices' deep inside `convert_to_model_response_object`,
swallowed by the surrounding `except Exception` in the handler, and
the SpendLogs row was written by the fallback path with zeroed-out
tokens, spend, and model.

This bug silently undercounts cost for every successful pass-through
call to either OpenAI's `/v1/responses` or Azure's
`/openai/v1/responses` (deployments configured for the Responses
API). Reproduced 2026-06-04 against a real Azure OpenAI Responses
API deployment proxied through LiteLLM v1.88.0.

Fix: use the dedicated
`OpenAIResponsesAPIConfig.transform_response_api_response` for the
Responses branch. This transformer already exists in LiteLLM
(`litellm/llms/openai/responses/transformation.py`) and knows the
Responses-API on-the-wire shape. `litellm.completion_cost` already
handles `ResponsesAPIResponse` natively with `call_type="responses"`,
so no downstream changes are needed.

Tests:

  test_responses_api_uses_responses_transformer_not_chat_completions
    NEW. Real regression test — exercises the openai_passthrough_handler
    with a real-shaped Responses payload (no `choices`, has `output`
    and Responses-API `usage` keys) and NO mocked `get_provider_config`.
    Pre-fix: raises KeyError 'choices' inside the chat-completions
    transformer (the bug). Post-fix: returns a ResponsesAPIResponse,
    completion_cost is called with call_type="responses" and a
    ResponsesAPIResponse instance (asserted).
    Verified to fail on un-fixed handler + pass on fixed handler
    before commit.

  test_responses_api_cost_tracking
    UPDATED. Old test mocked `get_provider_config` (no longer called
    in the responses branch post-fix). Now mocks the Responses
    transformer directly (`OpenAIResponsesAPIConfig.transform_response_api_response`)
    to test the downstream cost-calc contract.

Out of scope for this PR (separate followup):
  - Recognizing *.cognitiveservices.azure.com (the newer Azure
    OpenAI hostname) in the is_openai_*_route checks. Separate PR.

Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>

* fix(skills): execute DB skills by matching the litellm_skill_ tool name prefix (#30116)

Skill IDs are generated as litellm_skill_<uuid> and the model-facing
tool name is the sanitized skill ID, but the post-call execution gates
in SkillsInjectionHook only ran tools whose name starts with "skill_",
so DB skills were silently returned to the client as raw tool calls.

Fixes #28122.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(anthropic): synthesize content_block_start when Responses stream omits output_item.added (#30115)

* fix(team): reserve team budget raises for proxy admins on /team/update (#30030)

The caller's PERSONAL max_budget was the wrong yardstick for /team/update: a
team's spend ceiling has nothing to do with the admin's own key budget. That
comparison was an unintended side effect of reusing _check_user_team_limits()
(which exists for the /team/new path) and broke the UI, which re-sends the
unchanged budget on every save.

New behavior on /team/update for standalone teams:
- A team admin (already authorized via _verify_team_access) may freely KEEP or
  LOWER the team budget, and change models/tpm/rpm, without being gated by their
  personal limits.
- GROWING a team's spend ceiling is a budget-authority action reserved for proxy
  admins -> 403 for team admins. "Growing" covers both raising max_budget above
  the team's current finite value and removing the cap entirely (max_budget=null,
  detected via model_fields_set so an explicit null is distinguished from an
  omitted field). For a team that currently has no cap, setting a finite value is
  a restriction and is allowed.
- Org-scoped teams remain governed by _check_org_team_limits() (capped by the
  org budget).

Also reverts the #29525 existing_team_max_budget workaround in
_check_user_team_limits() back to the create-only form; /team/new still enforces
the creator's personal caps.

docs(access_control): resolve the contradiction in the team-admin section —
team admins can keep/lower the budget and manage rate limits/models, but cannot
raise the team budget (proxy-admin only).

tests: unit + behavior coverage for raise-blocked, cap-removal-blocked (team
admin), raise/removal allowed (proxy admin), uncapped-team restriction allowed,
keep/lower/resend allowed, and unchanged create-path guards.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(ui): data-driven App Router migration E2E smoke (default + server-root-path) (#29974)

* test(ui): add a data-driven App Router migration E2E smoke

Add a growing Playwright smoke for migrated pages: for each segment it deep-links
to the path route, asserts the URL and that the dashboard shell rendered, then
clicks off to a legacy page and asserts navigation still works. Driven by
e2e_tests/fixtures/migratedPages.ts, so adding a page is one line.

Runs in two situations against the same proxy: the default mount (npm run
e2e:migration) and a non-root SERVER_ROOT_PATH mount (npm run e2e:migration:root).
globalSetup now logs in at `${SERVER_ROOT_PATH}/ui/login` so the admin storage
state is valid under a prefix. Seeded with api-reference; append the rest as their
migrations merge.

* test(ui): support headed slow-motion + watch pauses in the migration smoke

Honor SLOWMO in the server-root-path config (the default config already did),
and add an env-gated E2E_WATCH_MS pause so a headed run lingers on each state.
Both are no-ops by default, so CI behavior is unchanged.

* test(ui): make the migration smoke a sidebar-click user journey

Rework the smoke from deep-linking to a real navigation journey: start at the
landing page, click the migrated page in the sidebar (expanding submenus for
nested items), assert the path route rendered, reload it (the check a wrong
server_root_path breaks), bounce to a legacy page and back, and — once two pages
are migrated — navigate directly between two migrated pages. Verifies via URL +
shell render, driven by the same fixture list.

* test(ui): address review on the migration smoke

Escape ROOT and segment before interpolating them into RegExp URL matchers so a
future segment containing regex metacharacters can't silently widen the match.
Make the server-root-path config fail fast when SERVER_ROOT_PATH is unset instead
of silently re-running the default mount and passing without exercising the prefix.

* test(ui): drop unused watch helper and fix stale smoke README

* test(ui): run the migration smoke under a server root path in CI

* test(ui): harden + instrument the server-root-path proxy reboot in CI

* test(ui): run the server-root-path migration smoke as its own CI job

Replace the in-place proxy reboot in e2e_ui_testing with a dedicated
e2e_ui_testing_server_root_path job that boots the proxy once with
SERVER_ROOT_PATH=/litellm, matching how every other proxy variant in the
config gets its own job rather than killing and relaunching the live proxy.

The reboot was failing deterministically: after pkill -9 and relaunch the
prefixed proxy never came back up on :4000 (connection refused), so the smoke
never ran. The readiness step that was supposed to surface the cause could
never reach its boot-log tail because CircleCI runs steps under bash -eo
pipefail and the preceding `curl -sv ... | tail` aborted the step with curl's
exit 7. Booting the proxy as the job's own background step lets any boot crash
land in that step's log instead of being swallowed.

The default e2e_ui_testing job is unchanged aside from dropping the reboot,
prefixed-readiness, and prefixed-smoke steps; the migration smoke still runs at
the root mount there via the default Playwright config.

* fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232)

* fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through

* test: mock post_call_response_headers_hook in audio speech route tests

* chore(ui): remove dead App Router route stubs under (dashboard) (#30045)

models-and-endpoints, organizations, and virtual-keys each had a page.tsx
route under (dashboard)/ that is not in MIGRATED_PAGES, so the sidebar and
deep links never resolve to it and the route is unreachable. Each was a thin
wrapper that handed the shared view empty or no-op props (empty modelData with
a no-op setModelData, hardcoded empty organizations, no-op
setUserRole/setUserEmail), so reaching one would render a degraded page in any
case. The real wrapper belongs in the PR that flips each page into
MIGRATED_PAGES, written with eyes on it and a test

This continues the dead-scaffolding cleanup from #28891. The shared components
these wrappers rendered (ModelsAndEndpointsView, OrganizationFilters) stay,
since the legacy ?page= switch in app/page.tsx and src/components still import
them

* fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session (#30000)

* fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session

* fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss

* fix(mcp): allow team access-group grants in OAuth authorize/token access check (#30041)

* fix(mcp): honor team access-group grants in OAuth authorize/token access check

* test(mcp): mock build_effective_auth_contexts in non-admin authorize tests for isolation

* docs(security): require a reproduction video for vulnerability reports (#30048) (#30063)

With AI models capable of automated vulnerability discovery now publicly
available, we expect a large increase in report volume, much of it
unverified. Requiring a video of the exploit running against a live
instance raises the bar for submissions and keeps triage focused on
reproducible issues. Reports without a video will be closed and reopened
if one is added later.

Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>

* feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796)

* feat(ui): add admin flag to disable in-product UI nudges for everyone

Admins can now suppress the survey and Claude Code feedback popups for
all users via a single disable_ui_nudges UI setting, instead of relying
on each user dismissing them individually.

* fix(ui): suppress nudges while ui settings are loading

Gate nudgesDisabled on the ui-settings loading state so an admin with
disable_ui_nudges on doesn't see the survey prompt flash, and the
getInProductNudgesCall fetch doesn't fire, on a cold page load before
the flag resolves. Falls back to showing nudges if the fetch errors.

* test(ui): wrap CreateKeyPage test in QueryClientProvider

page.tsx now calls useUISettings (react-query), which needs a
QueryClient that layout.tsx supplies in production but the test did
not. Add the provider and mock getUiSettings so the query resolves.

* chore(ui): remove dead dashboard files and unused dependencies (#30047)

* chore(ui): remove dead dashboard files and unused dependencies

knip flagged seven orphaned source/config files with no importers and
five declared dependencies that nothing in the tree uses. Removing them
shrinks the dashboard bundle's source surface and keeps the manifest
honest; vite stays installed transitively via vitest, so test tooling is
unaffected.

* fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow

The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec
(tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml
workflow step still depend on it, so the redirect e2e job failed to load a
config that no longer existed.

* fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009)

* fix(proxy): authorize batch files using upload target_model_names (LIT-3593)

After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)

Restores the reverse-lookup for the JSONL body.model fallback path so that
legacy/pre-target_model_names managed files still map stripped provider IDs
back to proxy aliases before auth. Also cleans up redundant `or None`.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)"

This reverts commit 30d2e96f77ef521ccaaf2193fe554980380eb669.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI (#30064)

* Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI

Adds cost map entries for claude-fable-5 ($10/$50 per MTok, 1M context,
128K output, adaptive thinking only) on the Anthropic API, Bedrock
converse (base, global, and us/eu geo inference profiles at the 10%
regional premium), Vertex AI, and Azure AI (Microsoft Foundry, which
serves Fable 5 with the full 1M context window unlike Opus 4.8).

Registers anthropic.claude-fable-5 in BEDROCK_CONVERSE_MODELS, lists the
model in the setup wizard, and extends the reasoning effort e2e grid.
The Bedrock, Vertex, and Azure grid cells carry fail_reason markers
until the CI accounts are provisioned: Bedrock needs the provider data
sharing opt-in Fable 5 requires, and the Foundry resource needs a
claude-fable-5 deployment.

The first-party entry carries provider_specific_entry {us: 1.1} for the
inference_geo premium and deliberately no fast multiplier since Fable 5
has no fast mode.

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* Drop removed sampling params for Claude 4.7+ when drop_params is set

Fable 5, Opus 4.7, and Opus 4.8 removed sampling params: the API rejects
top_p, top_k, and any temperature other than 1 with a 400. LiteLLM was
forwarding them even with drop_params enabled because the Anthropic and
Bedrock converse transformations passed temperature/top_p through
unconditionally.

Mirror the GPT-5/o-series handling: temperature=1 still passes through,
other values and any top_p are dropped when drop_params is set, and
without drop_params a clean client-side UnsupportedParamsError tells the
caller how to opt in, instead of surfacing the raw provider error.

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* Drive sampling param gating from the cost map and cover top_k

Greptile review follow-ups on the sampling param fix: the restriction for
Fable 5 / Opus 4.7 / 4.8 is now declared as supports_sampling_params: false
on every affected cost map entry (perplexity excluded; that route is
OpenAI-compatible and maps sampling params upstream) and read back through
a tri-state map lookup, keeping the name check only as a fallback for
provider-routed ids whose hosted map entries predate the flag, the same
layering supports_adaptive_thinking uses. top_k bypasses map_openai_params
as a provider-specific kwarg, so it is gated at the shared
AnthropicConfig.transform_request boundary (direct, Bedrock invoke, Vertex,
Azure) and in the Bedrock converse _handle_top_k_value path, with
drop_params threaded through the converse transform helpers.

Also updates the reasoning effort grid cell count assertion for the four
Fable 5 rows added on this branch (29 x 11 cells).

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* Declare supports_sampling_params in the cost map schema

The model map validation schema uses additionalProperties: false, so the
new flag must be declared for the 28 entries that carry it; this was the
one failing job (misc / Run tests) on the previous commit.

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* fix(bedrock): gate top_k=0 on converse to match Anthropic boundary

Truthiness check let top_k=0 silently disappear on models that removed
sampling params, while AnthropicConfig.transform_request treats 0 as
present and raises UnsupportedParamsError (or drops when drop_params is
set). Switch to 'is not None' so converse, direct Anthropic, invoke,
Vertex, and Azure all behave the same for top_k=0.

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* fix(anthropic): avoid index -1 content_block_delta in messages stream

When a /v1/messages request is routed through the Responses API
adapter, AnthropicResponsesStreamWrapper only emits content_block_start
on response.output_item.added. Some upstreams (LMStudio for example)
never send that event, so the text delta handler fell back to
_current_block_index, which starts at -1, and clients received
content_block_delta events with index -1 and no preceding
content_block_start. Anthropic SDKs then fail with "text part -1 not
found"

The text delta handler now synthesizes a content_block_start with a
fresh block index whenever the delta references an unregistered item_id
or no block is open yet, and registers the item_id so follow-up deltas
reuse the same index

Addresses the /v1/messages defect in #27442

* Make test sys.path shim resolve relative to the file, not the CWD

os.path.abspath("../../../../../../..") depends on where pytest is
invoked from; anchoring on os.path.dirname(__file__) makes the import
work from any working directory. Also corrects the depth: the repo root
is six levels above this file, not seven.

---------

Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: tin-berri <tin@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>

* fix: enable compact-2026-01-12 beta header for vertex_ai provider (#30114)

* fix(team): reserve team budget raises for proxy admins on /team/update (#30030)

The caller's PERSONAL max_budget was the wrong yardstick for /team/update: a
team's spend ceiling has nothing to do with the admin's own key budget. That
comparison was an unintended side effect of reusing _check_user_team_limits()
(which exists for the /team/new path) and broke the UI, which re-sends the
unchanged budget on every save.

New behavior on /team/update for standalone teams:
- A team admin (already authorized via _verify_team_access) may freely KEEP or
  LOWER the team budget, and change models/tpm/rpm, without being gated by their
  personal limits.
- GROWING a team's spend ceiling is a budget-authority action reserved for proxy
  admins -> 403 for team admins. "Growing" covers both raising max_budget above
  the team's current finite value and removing the cap entirely (max_budget=null,
  detected via model_fields_set so an explicit null is distinguished from an
  omitted field). For a team that currently has no cap, setting a finite value is
  a restriction and is allowed.
- Org-scoped teams remain governed by _check_org_team_limits() (capped by the
  org budget).

Also reverts the #29525 existing_team_max_budget workaround in
_check_user_team_limits() back to the create-only form; /team/new still enforces
the creator's personal caps.

docs(access_control): resolve the contradiction in the team-admin section —
team admins can keep/lower the budget and manage rate limits/models, but cannot
raise the team budget (proxy-admin only).

tests: unit + behavior coverage for raise-blocked, cap-removal-blocked (team
admin), raise/removal allowed (proxy admin), uncapped-team restriction allowed,
keep/lower/resend allowed, and unchanged create-path guards.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(ui): data-driven App Router migration E2E smoke (default + server-root-path) (#29974)

* test(ui): add a data-driven App Router migration E2E smoke

Add a growing Playwright smoke for migrated pages: for each segment it deep-links
to the path route, asserts the URL and that the dashboard shell rendered, then
clicks off to a legacy page and asserts navigation still works. Driven by
e2e_tests/fixtures/migratedPages.ts, so adding a page is one line.

Runs in two situations against the same proxy: the default mount (npm run
e2e:migration) and a non-root SERVER_ROOT_PATH mount (npm run e2e:migration:root).
globalSetup now logs in at `${SERVER_ROOT_PATH}/ui/login` so the admin storage
state is valid under a prefix. Seeded with api-reference; append the rest as their
migrations merge.

* test(ui): support headed slow-motion + watch pauses in the migration smoke

Honor SLOWMO in the server-root-path config (the default config already did),
and add an env-gated E2E_WATCH_MS pause so a headed run lingers on each state.
Both are no-ops by default, so CI behavior is unchanged.

* test(ui): make the migration smoke a sidebar-click user journey

Rework the smoke from deep-linking to a real navigation journey: start at the
landing page, click the migrated page in the sidebar (expanding submenus for
nested items), assert the path route rendered, reload it (the check a wrong
server_root_path breaks), bounce to a legacy page and back, and — once two pages
are migrated — navigate directly between two migrated pages. Verifies via URL +
shell render, driven by the same fixture list.

* test(ui): address review on the migration smoke

Escape ROOT and segment before interpolating them into RegExp URL matchers so a
future segment containing regex metacharacters can't silently widen the match.
Make the server-root-path config fail fast when SERVER_ROOT_PATH is unset instead
of silently re-running the default mount and passing without exercising the prefix.

* test(ui): drop unused watch helper and fix stale smoke README

* test(ui): run the migration smoke under a server root path in CI

* test(ui): harden + instrument the server-root-path proxy reboot in CI

* test(ui): run the server-root-path migration smoke as its own CI job

Replace the in-place proxy reboot in e2e_ui_testing with a dedicated
e2e_ui_testing_server_root_path job that boots the proxy once with
SERVER_ROOT_PATH=/litellm, matching how every other proxy variant in the
config gets its own job rather than killing and relaunching the live proxy.

The reboot was failing deterministically: after pkill -9 and relaunch the
prefixed proxy never came back up on :4000 (connection refused), so the smoke
never ran. The readiness step that was supposed to surface the cause could
never reach its boot-log tail because CircleCI runs steps under bash -eo
pipefail and the preceding `curl -sv ... | tail` aborted the step with curl's
exit 7. Booting the proxy as the job's own background step lets any boot crash
land in that step's log instead of being swallowed.

The default e2e_ui_testing job is unchanged aside from dropping the reboot,
prefixed-readiness, and prefixed-smoke steps; the migration smoke still runs at
the root mount there via the default Playwright config.

* fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232)

* fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through

* test: mock post_call_response_headers_hook in audio speech route tests

* chore(ui): remove dead App Router route stubs under (dashboard) (#30045)

models-and-endpoints, organizations, and virtual-keys each had a page.tsx
route under (dashboard)/ that is not in MIGRATED_PAGES, so the sidebar and
deep links never resolve to it and the route is unreachable. Each was a thin
wrapper that handed the shared view empty or no-op props (empty modelData with
a no-op setModelData, hardcoded empty organizations, no-op
setUserRole/setUserEmail), so reaching one would render a degraded page in any
case. The real wrapper belongs in the PR that flips each page into
MIGRATED_PAGES, written with eyes on it and a test

This continues the dead-scaffolding cleanup from #28891. The shared components
these wrappers rendered (ModelsAndEndpointsView, OrganizationFilters) stay,
since the legacy ?page= switch in app/page.tsx and src/components still import
them

* fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session (#30000)

* fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session

* fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss

* fix(mcp): allow team access-group grants in OAuth authorize/token access check (#30041)

* fix(mcp): honor team access-group grants in OAuth authorize/token access check

* test(mcp): mock build_effective_auth_contexts in non-admin authorize tests for isolation

* docs(security): require a reproduction video for vulnerability reports (#30048) (#30063)

With AI models capable of automated vulnerability discovery now publicly
available, we expect a large increase in report volume, much of it
unverified. Requiring a video of the exploit running against a live
instance raises the bar for submissions and keeps triage focused on
reproducible issues. Reports without a video will be closed and reopened
if one is added later.

Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>

* feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796)

* feat(ui): add admin flag to disable in-product UI nudges for everyone

Admins can now suppress the survey and Claude Code feedback popups for
all users via a single disable_ui_nudges UI setting, instead of relying
on each user dismissing them individually.

* fix(ui): suppress nudges while ui settings are loading

Gate nudgesDisabled on the ui-settings loading state so an admin with
disable_ui_nudges on doesn't see the survey prompt flash, and the
getInProductNudgesCall fetch doesn't fire, on a cold page load before
the flag resolves. Falls back to showing nudges if the fetch errors.

* test(ui): wrap CreateKeyPage test in QueryClientProvider

page.tsx now calls useUISettings (react-query), which needs a
QueryClient that layout.tsx supplies in production but the test did
not. Add the provider and mock getUiSettings so the query resolves.

* chore(ui): remove dead dashboard files and unused dependencies (#30047)

* chore(ui): remove dead dashboard files and unused dependencies

knip flagged seven orphaned source/config files with no importers and
five declared dependencies that nothing in the tree uses. Removing them
shrinks the dashboard bundle's source surface and keeps the manifest
honest; vite stays installed transitively via vitest, so test tooling is
unaffected.

* fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow

The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec
(tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml
workflow step still depend on it, so the redirect e2e job failed to load a
config that no longer existed.

* fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009)

* fix(proxy): authorize batch files using upload target_model_names (LIT-3593)

After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)

Restores the reverse-lookup for the JSONL body.model fallback path so that
legacy/pre-target_model_names managed files still map stripped provider IDs
back to proxy aliases before auth. Also cleans up redundant `or None`.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)"

This reverts commit 30d2e96f77ef521ccaaf2193fe554980380eb669.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI (#30064)

* Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI

Adds cost map entries for claude-fable-5 ($10/$50 per MTok, 1M context,
128K output, adaptive thinking only) on the Anthropic API, Bedrock
converse (base, global, and us/eu geo inference profiles at the 10%
regional premium), Vertex AI, and Azure AI (Microsoft Foundry, which
serves Fable 5 with the full 1M context window unlike Opus 4.8).

Registers anthropic.claude-fable-5 in BEDROCK_CONVERSE_MODELS, lists the
model in the setup wizard, and extends the reasoning effort e2e grid.
The Bedrock, Vertex, and Azure grid cells carry fail_reason markers
until the CI accounts are provisioned: Bedrock needs the provider data
sharing opt-in Fable 5 requires, and the Foundry resource needs a
claude-fable-5 deployment.

The first-party entry carries provider_specific_entry {us: 1.1} for the
inference_geo premium and deliberately no fast multiplier since Fable 5
has no fast mode.

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* Drop removed sampling params for Claude 4.7+ when drop_params is set

Fable 5, Opus 4.7, and Opus 4.8 removed sampling params: the API rejects
top_p, top_k, and any temperature other than 1 with a 400. LiteLLM was
forwarding them even with drop_params enabled because the Anthropic and
Bedrock converse transformations passed temperature/top_p through
unconditionally.

Mirror the GPT-5/o-series handling: temperature=1 still passes through,
other values and any top_p are dropped when drop_params is set, and
without drop_params a clean client-side UnsupportedParamsError tells the
caller how to opt in, instead of surfacing the raw provider error.

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* Drive sampling param gating from the cost map and cover top_k

Greptile review follow-ups on the sampling param fix: the restriction for
Fable 5 / Opus 4.7 / 4.8 is now declared as supports_sampling_params: false
on every affected cost map entry (perplexity excluded; that route is
OpenAI-compatible and maps sampling params upstream) and read back through
a tri-state map lookup, keeping the name check only as a fallback for
provider-routed ids whose hosted map entries predate the flag, the same
layering supports_adaptive_thinking uses. top_k bypasses map_openai_params
as a provider-specific kwarg, so it is gated at the shared
AnthropicConfig.transform_request boundary (direct, Bedrock invoke, Vertex,
Azure) and in the Bedrock converse _handle_top_k_value path, with
drop_params threaded through the converse transform helpers.

Also updates the reasoning effort grid cell count assertion for the four
Fable 5 rows added on this branch (29 x 11 cells).

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* Declare supports_sampling_params in the cost map schema

The model map validation schema uses additionalProperties: false, so the
new flag must be declared for the 28 entries that carry it; this was the
one failing job (misc / Run tests) on the previous commit.

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* fix(bedrock): gate top_k=0 on converse to match Anthropic boundary

Truthiness check let top_k=0 silently disappear on models that removed
sampling params, while AnthropicConfig.transform_request treats 0 as
present and raises UnsupportedParamsError (or drops when drop_params is
set). Switch to 'is not None' so converse, direct Anthropic, invoke,
Vertex, and Azure all behave the same for top_k=0.

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* fix: enable compact-2026-01-12 beta header for vertex_ai provider

The vertex_ai block in anthropic_beta_headers_config.json mapped
compact-2026-01-12 to null, so update_headers_with_filtered_beta
stripped the header before the request reached Vertex while the
compact_20260112 context edit stayed in the body, and Vertex rejected
the request with HTTP 400. Vertex rawPredict accepts the header, and
the bedrock and databricks blocks already forward it. Mirrors #21867,
which enabled context-1m-2025-08-07 for vertex_ai the same way.

Fixes #27290.

---------

Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: tin-berri <tin@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>

* fix(proxy): coerce litellm_settings.max_budget env var to float (#30113)

* fix(team): reserve team budget raises for proxy admins on /team/update (#30030)

The caller's PERSONAL max_budget was the wrong yardstick for /team/update: a
team's spend ceiling has nothing to do with the admin's own key budget. That
comparison was an unintended side effect of reusing _check_user_team_limits()
(which exists for the /team/new path) and broke the UI, which re-sends the
unchanged budget on every save.

New behavior on /team/update for standalone teams:
- A team admin (already authorized via _verify_team_access) may freely KEEP or
  LOWER the team budget, and change models/tpm/rpm, without being gated by their
  personal limits.
- GROWING a team's spend ceiling is a budget-authority action reserved for proxy
  admins -> 403 for team admins. "Growing" covers both raising max_budget above
  the team's current finite value and removing the cap entirely (max_budget=null,
  detected via model_fields_set so an explicit null is distinguished from an
  omitted field). For a team that currently has no cap, setting a finite value is
  a restriction and is allowed.
- Org-scoped teams remain governed by _check_org_team_limits() (capped by the
  org budget).

Also reverts the #29525 existing_team_max_budget workaround in
_check_user_team_limits() back to the create-only form; /team/new still enforces
the creator's personal caps.

docs(access_control): resolve the contradiction in the team-admin section —
team admins can keep/lower the budget and manage rate limits/models, but cannot
raise the team budget (proxy-admin only).

tests: unit + behavior coverage for raise-blocked, cap-removal-blocked (team
admin), raise/removal allowed (proxy admin), uncapped-team restriction allowed,
keep/lower/resend allowed, and unchanged create-path guards.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(ui): data-driven App Router migration E2E smoke (default + server-root-path) (#29974)

* test(ui): add a data-driven App Router migration E2E smoke

Add a growing Playwright smoke for migrated pages: for each segment it deep-links
to the path route, asserts the URL and that the dashboard shell rendered, then
clicks off to a legacy page and asserts navigation still works. Driven by
e2e_tests/fixtures/migratedPages.ts, so adding a page is one line.

Runs in two situations against the same proxy: the default mount (npm run
e2e:migration) and a non-root SERVER_ROOT_PATH mount (npm run e2e:migration:root).
globalSetup now logs in at `${SERVER_ROOT_PATH}/ui/login` so the admin storage
state is valid under a prefix. Seeded with api-reference; append the rest as their
migrations merge.

* test(ui): support headed slow-motion + watch pauses in the migration smoke

Honor SLOWMO in the server-root-path config (the default config already did),
and add an env-gated E2E_WATCH_MS pause so a headed run lingers on each state.
Both are no-ops by default, so CI behavior is unchanged.

* test(ui): make the migration smoke a sidebar-click user journey

Rework the smoke from deep-linking to a real navigation journey: start at the
landing page, click the migrated page in the sidebar (expanding submenus for
nested items), assert the path route rendered, reload it (the check a wrong
server_root_path breaks), bounce to a legacy page and back, and — once two pages
are migrated — navigate directly between two migrated pages. Verifies via URL +
shell render, driven by the same fixture list.

* test(ui): address review on the migration smoke

Escape ROOT and segment before interpolating them into RegExp URL matchers so a
future segment containing regex metacharacters can't silently widen the match.
Make the server-root-path config fail fast when SERVER_ROOT_PATH is unset instead
of silently re-running the default mount and passing without exercising the prefix.

* test(ui): drop unused watch helper and fix stale smoke README

* test(ui): run the migration smoke under a server root path in CI

* test(ui): harden + instrument the server-root-path proxy reboot in CI

* test(ui): run the server-root-path migration smoke as its own CI job

Replace the in-place proxy reboot in e2e_ui_testing with a dedicated
e2e_ui_testing_server_root_path job that boots the proxy once with
SERVER_ROOT_PATH=/litellm, matching how every other proxy variant in the
config gets its own job rather than killing and relaunching the live proxy.

The reboot was failing deterministically: after pkill -9 and relaunch the
prefixed proxy never came back up on :4000 (connection refused), so the smoke
never ran. The readiness step that was supposed to surface the cause could
never reach its boot-log tail because CircleCI runs steps under bash -eo
pipefail and the preceding `curl -sv ... | tail` aborted the step with curl's
exit 7. Booting the proxy as the job's own background step lets any boot crash
land in that step's log instead of being swallowed.

The default e2e_ui_testing job is unchanged aside from dropping the reboot,
prefixed-readiness, and prefixed-smoke steps; the migration smoke still runs at
the root mount there via the default Playwright config.

* fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232)

* fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through

* test: mock post_call_response_headers_hook in audio speech route tests

* chore(ui): remove dead App Router route stubs under (dashboard) (#30045)

models-and-endpoints, organizations, and virtual-keys each had a page.tsx
route under (dashboard)/ that is not in MIGRATED_PAGES, so the sidebar and
deep links never resolve to it and the route is unreachable. Each was a thin
wrapper that handed the shared view empty or no-op props (empty modelData with
a no-op setModelData, hardcoded empty organizations, no-op
setUserRole/setUserEmail), so reaching one would render a degraded page in any
case. The real wrapper belongs in the PR that flips each page into
MIGRATED_PAGES, written with eyes on it and a test

This continues the dead-scaffolding cleanup from #28891. The shared components
these wrappers rendered (ModelsAndEndpointsView, OrganizationFilters) stay,
since the legacy ?page= switch in app/page.tsx and src/components still import
them

* fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session (#30000)

* fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session

* fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss

* fix(mcp): allow team access-group grants in OAuth authorize/token access check (#30041)

* fix(mcp): honor team access-group grants in OAuth authorize/token access check

* test(mcp): mock build_effective_auth_contexts in non-admin authorize tests for isolation

* docs(security): require a reproduction video for vulnerability reports (#30048) (#30063)

With AI models capable of automated vulnerability discovery now publicly
available, we expect a large increase in report volume, much of it
unverified. Requiring a video of the exploit running against a live
instance raises the bar for submissions and keeps triage focused on
reproducible issues. Reports without a video will be closed and reopened
if one is added later.

Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>

* feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796)

* feat(ui): add admin flag to disable in-product UI nudges for everyone

Admins can now suppress the survey and Claude Code feedback popups for
all users via a single disable_ui_nudges UI setting, instead of relying
on each user dismissing them individually.

* fix(ui): suppress nudges while ui settings are loading

Gate nudgesDisabled on the ui-settings loading state so an admin with
disable_ui_nudges on doesn't see the survey prompt flash, and the
getInProductNudgesCall fetch doesn't fire, on a cold page load before
the flag resolves. Falls back to showing nudges if the fetch errors.

* test(ui): wrap CreateKeyPage test in QueryClientProvider

page.tsx now calls useUISettings (react-query), which needs a
QueryClient that layout.tsx supplies in production but the test did
not. Add the provider and mock getUiSettings so the query resolves.

* chore(ui): remove dead dashboard files and unused dependencies (#30047)

* chore(ui): remove dead dashboard files and unused dependencies

knip flagged seven orphaned source/config files with no importers and
five declared dependencies that nothing in the tree uses. Removing them
shrinks the dashboard bundle's source surface and keeps the manifest
honest; vite stays installed transitively via vitest, so test tooling is
unaffected.

* fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow

The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec
(tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml
workflow step still depend on it, so the redirect e2e job failed to load a
config that no longer existed.

* fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009)

* fix(proxy): authorize batch files using upload target_model_names (LIT-3593)

After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)

Restores the reverse-lookup for the JSONL body.model fallback path so that
legacy/pre-target_model_names managed files still map stripped provider IDs
back to proxy aliases before auth. Also cleans up redundant `or None`.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)"

This reverts commit 30d2e96f77ef521ccaaf2193fe554980380eb669.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI (#30064)

* Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI

Adds cost map entries for claude-fable-5 ($10/$50 per MTok, 1M context,
128K output, adaptive thinking only) on the Anthropic API, Bedrock
converse (base, global, and us/eu geo inference profiles at the 10%
regional premium), Vertex AI, and Azure AI (Microsoft Foundry, which
serves Fable 5 with the full 1M context window unlike Opus 4.8).

Registers anthropic.claude-fable-5 in BEDROCK_CONVERSE_MODELS, lists the
model in the setup wizard, and extends the reasoning effort e2e grid.
The Bedrock, Vertex, and Azure grid cells carry fail_reason markers
until the CI accounts are provisioned: Bedrock needs the provider data
sharing opt-in Fable 5 requires, and the Foundry resource needs a
claude-fable-5 deployment.

The first-party entry carries provider_specific_entry {us: 1.1} for the
inference_geo premium and deliberately no fast multiplier since Fable 5
has no fast mode.

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* Drop removed sampling params for Claude 4.7+ when drop_params is set

Fable 5, Opus 4.7, and Opus 4.8 removed sampling params: the API rejects
top_p, top_k, and any temperature other than 1 with a 400. LiteLLM was
forwarding them even with drop_params enabled because the Anthropic and
Bedrock converse transformations passed temperature/top_p through
unconditionally.

Mirror the GPT-5/o-series handling: temperature=1 still passes through,
other values and any top_p are dropped when drop_params is set, and
without drop_params a clean client-side UnsupportedParamsError tells the
caller how to opt in, instead of surfacing the raw provider error.

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* Drive sampling param gating from the cost map and cover top_k

Greptile review follow-ups on the sampling param fix: the restriction for
Fable 5 / Opus 4.7 / 4.8 is now declared as supports_sampling_params: false
on every affected cost map entry (perplexity excluded; that route is
OpenAI-compatible and maps sampling params upstream) and read back through
a tri-state map lookup, keeping the name check only as a fallback for
provider-routed ids whose hosted map entries predate the flag, the same
layering supports_adaptive_thinking uses. top_k bypasses map_openai_params
as a provider-specific kwarg, so it is gated at the shared
AnthropicConfig.transform_request boundary (direct, Bedrock invoke, Vertex,
Azure) and in the Bedrock converse _handle_top_k_value path, with
drop_params threaded through the converse transform helpers.

Also updates the reasoning effort grid cell count assertion for the four
Fable 5 rows added on this branch (29 x 11 cells).

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* Declare supports_sampling_params in the cost map schema

The model map validation schema uses additionalProperties: false, so the
new flag must be declared for the 28 entries that carry it; this was the
one failing job (misc / Run tests) on the previous commit.

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* fix(bedrock): gate top_k=0 on converse to match Anthropic boundary

Truthiness check let top_k=0 silently disappear on models that removed
sampling params, while AnthropicConfig.transform_request treats 0 as
present and raises UnsupportedParamsError (or drops when drop_params is
set). Switch to 'is not None' so converse, direct Anthropic, invoke,
Vertex, and Azure all behave the same for top_k=0.

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* fix(proxy): coerce litellm_settings.max_budget env var to float

When max_budget is set in litellm_settings via os.environ/MAX_BUDGET,
the env var resolves to a string and the generic setattr branch in
ProxyConfig.load_config stored it as-is, so the startup check
litellm.max_budget > 0 raised TypeError. The earlier fix (#23855) only
covered the CLI initialize() path. Coerce the value to float in the
settings loop, matching the existing max_internal_user_budget handling.

Fixes #26696.

---------

Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: tin-berri <tin@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>

* fix(router): don't drop bedrock pass-through deployments using IAM credentials (#30111)

* Fix Bedrock passthrough deployment dropped when using IAM credentials

Bedrock deployments with use_in_pass_through enabled and IAM/OIDC auth
(aws_role_name, no api_key) hit the generic pass-through branch in
Router._initialize_deployment_for_pass_through, which calls
set_pass_through_credentials and raises "api_key is required". The
exception drops the deployment from the router entirely, breaking both
passthrough and normal routing for that model.

Skip the credential store write when no api_key is set; the bedrock
passthrough route resolves AWS credentials at request time via
BedrockConverseLLM.get_credentials(), not the passthrough credential
store, so there is nothing to register here.

Fixes #27728.

* Reset passthrough credentials singleton before api_key credential test

The test reads the module-level passthrough_endpoint_router singleton,
so a stale "openai" entry written by an earlier test in the same
process could make the assertion pass without exercising the code path.
Clearing the credentials dict up front makes the test order-independent.

* fix(sdk): stop mirroring reasoning_content in provider_specific_fields (#30110)

The dict-to-response conversion path mirrored reasoning_content into
provider_specific_fields, while live provider transforms (Anthropic's
_build_provider_specific_fields) only set it top-level on the Message.
Cache-replayed messages therefore serialized differently from live
ones, breaking disk cache key stability for multi-turn conversations
with extended thinking.

The mirror was added for DeepSeek before Message.reasoning_content
existed as a top-level attribute. The top-level field is still set by
the converter, so DeepSeek's request-side promotion is unaffected.

Fixes #27337.

* fix(mcp): coerce mcp_server_cost_info values to float at ingest (#30109)

* fix(mcp): coerce mcp_server_cost_info values to float at ingest

YAML 1.1 parses scientific notation without a decimal point
(e.g. 7e-05) as a string, and MCPServerCostInfo is a TypedDict with no
runtime validation, so a string-typed default_cost_per_query from
config.yaml flowed through the proxy untouched and crashed the MCP
server settings page with '.toFixed is not a function'. Normalize
mcp_server_cost_info on both the config and DB load paths, dropping
non-numeric values with a warning instead of failing the server load.

Fixes #27097.

* fix(mcp): drop non-numeric default_cost_per_query instead of nulling it

Keeping the key with a None value still exposes a null to the UI,
which can crash .toFixed formatting when the consumer checks key
existence rather than truthiness. Delete the key on coercion failure,
matching how non-numeric per-tool cost entries are already omitted.

* fix(proxy): count embedding and text completion tokens toward TPM limits (#30105)

* fix(proxy): count embedding and text completion tokens toward TPM limits

The parallel request limiters only read token usage off ModelResponse,
so EmbeddingResponse and TextCompletionResponse objects left
total_tokens at 0 and the per key, user, team, and end user TPM
counters never incremented. Requests to /v1/embeddings and
/v1/completions were effectively free against any tpm_limit. In the v3
limiter this was worse: the post-call reconciliation computed actual
usage as 0 and refunded the pre-call reservation made at request time.

Broaden the isinstance checks to accept EmbeddingResponse and
TextCompletionResponse, which both expose a Usage object, at the four
per-scope sites in parallel_request_limiter.py and at the usage
extraction in parallel_request_limiter_v3.py. ResponsesAPIResponse was
already covered in v3 via BaseLiteLLMOpenAIResponseObject.

Fixes #27738.

* test(proxy): cover v1 limiter TPM counting for embedding and text completion responses

Exercise the broadened isinstance sites in parallel_request_limiter.py
by asserting that async_log_success_event adds total_tokens to the per
key, user, team, and end user TPM counters for EmbeddingResponse and
TextCompletionResponse objects. The counters are pre-seeded at zero so
the assertion is exactly the increment; on the pre-fix code these
responses left total_tokens at 0 and the test fails.

* fix(openai): forward client headers on the text completion path (#30103)

* fix(openai): forward client headers on the text completion path

litellm.completion() merges caller headers with extra_headers, but the
text-completion-openai branch never passed the merged dict to
openai_text_completions.completion(), and the handler only used its
headers argument for logging. Pass the merged headers through the call
site and set them as extra_headers on the outgoing request, mirroring
the chat completion handler, so x-* client headers forwarded by the
proxy reach the provider on /v1/completions.

Fixes #27410.

* Drop redundant extra_headers assignment and fix test module collision

completion() merges extra_headers into headers before the
text-completion-openai branch, and the handler now sets the merged
headers as extra_headers on the request, so the branch-local
optional_params["extra_headers"] assignment was a dead duplicate.
Removing it keeps the assignment in one place while both entry paths
(litellm.text_completion and direct handler callers) still forward
headers; a new regression test pins the extra_headers kwarg path.

Also rename the test module to test_completion_handler.py since its
basename collided with tests/test_litellm/llms/bedrock/batches/
test_handler.py and broke pytest collection.

* fix(bedrock): route Anthropic-shape count_tokens to InvokeModel and base64-encode the body (#30102)

* fix(bedrock): route Anthropic-shape count_tokens to InvokeModel

POST /v1/messages/count_tokens with Anthropic content blocks
({"type": "text"|"tool_use"|...}) was routed to the Converse input of
the Bedrock CountTokens API. The Converse transform copies list content
through verbatim, so Bedrock rejected the request with a 400 and the
caller silently fell back to the local tokenizer, returning counts that
can be off by ~50% on tool-heavy payloads.

_detect_input_type now routes messages whose content blocks carry a
"type" key (Anthropic shape) to the invokeModel input, which forwards
the body verbatim. The invokeModel body is now base64-encoded as the
CountTokens API requires (InvokeModelTokensRequest.body is a
base64-encoded blob), and Anthropic Messages bodies get the
anthropic_version and max_tokens fields Bedrock validates against.

Fixes #27632.

* refactor(bedrock): name the CountTokens max_tokens placeholder

Replace the magic 1024 with a module-level
DEFAULT_ANTHROPIC_INVOKE_MODEL_MAX_TOKENS constant so the intent is
explicit and there is a single place to update if Bedrock's InvokeModel
schema ever changes. Module-local rather than litellm/constants.py
because the value is only a schema-validation placeholder for token
counting, not a user-tunable generation default.

* Add above-512k pricing tier for MiniMax-M3 and correct its base rates (#30095)

* Add above-512k pricing tier support for MiniMax-M3

MiniMax-M3 doubles its per-token rates once a prompt exceeds 512k
input tokens. The tiered cost parser already handles arbitrary
thresholds, but get_model_info only copies whitelisted keys from
ModelInfoBase, which had no 512k variants, so above_512k keys were
silently dropped and long-context requests were priced at the flat
rate.

Add the input, output, and cache-read above_512k_tokens fields to
ModelInfoBase and pass them through in get_model_info. Update the
minimax/MiniMax-M3 entry with the tiered rates and correct the base
rates, which matched the above-512k tier instead of the published
base tier (https://platform.minimax.io/docs/guides/pricing-paygo).

Fixes #29663.

* Add above-512k keys to pricing schema, set MiniMax-M3 context to 1M

Register the three new above_512k_tokens cost keys in the INTENDED_SCHEMA
of test_aaamodel_prices_and_context_window_json_is_valid, declared the same
way as the existing above_200k/above_272k tier keys, so the schema check
accepts the MiniMax-M3 tiered pricing entry.

Also raise MiniMax-M3 max_input_tokens from 512000 to 1000000 in both
pricing JSONs. The MiniMax API docs
(https://platform.minimax.io/docs/guides/text-generation) state the model
supports a 1,000,000-token context window, and the pay-as-you-go pricing
page (https://platform.minimax.io/docs/guides/pricing-paygo) prices input
above 512k tokens, which only makes sense if inputs beyond 512k are
accepted. This makes the above-512k pricing tier reachable.

* fix(bedrock): make document names unique across conversation turns (#30093)

* fix(bedrock): make document names unique across conversation turns

PR #16275 derived Bedrock document names purely from a content hash so
that names stay deterministic for prompt caching. When the same PDF or
document appears in more than one conversation turn, every occurrence
gets the identical name and Bedrock rejects the request with "Messages
can not contain duplicate document names".

Add _rename_duplicate_bedrock_document_names, a post-pass over the
assembled message blocks that keeps the first occurrence's hash-based
name and appends a positional suffix (_2, _3, ...) to later
occurrences. Apply it in both _bedrock_converse_messages_pt and
_bedrock_converse_messages_pt_async. Names remain deterministic across
requests and the first occurrence is unchanged, so prompt cache
prefixes stay stable.

Fixes #29418.

* fix(bedrock): avoid suffix collisions with organic document names

A renamed duplicate could collide with a document whose hash-derived
name already ends in the same positional suffix (e.g. an organic
report_2 next to two documents named report). Collect every document
name up front and bump the suffix until the candidate is unused, so
renames can collide neither with organic names nor with each other.

* fix(_types): remove ResponsesAPIResponse from PassThroughEndpointLoggingResultValues

The import of ResponsesAPIResponse was removed from the file but a usage
was left in the Union type, causing a NameError on import and breaking
all CI tests. Remove the stale reference to match the cleanup intent.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(_types): restore ResponsesAPIResponse import and add use_xai_oauth to filter list

Two related fixes:
1. Re-add ResponsesAPIResponse import in _types.py — it was removed but still
   needed in PassThroughEndpointLoggingResultValues (used in
   openai_passthrough_logging_handler.py).
2. Add use_xai_oauth to all_litellm_params so it is filtered before forwarding
   kwargs to providers like OpenAI that do not recognize it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Hari <kancharla.ha@northeastern.edu>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Ceder Dens <ceder.dens@uantwerpen.be>
Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: 冯基魁 <56265583+fengjikui@users.noreply.github.com>
Co-authored-by: victoruce <161634297+victoruce@users.noreply.github.com>
Co-authored-by: kejunleng <33445544+silencedoctor@users.noreply.github.com>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Tyson Cung <45380903+tysoncung@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com>
Co-authored-by: Daan <255322319+daanhendrio@users.noreply.github.com>
Co-authored-by: Avani Prajapati <143805019+Avani-prajapati@users.noreply.github.com>
Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com>
Co-authored-by: daitran-tensormesh <dai@tensormesh.ai>
Co-authored-by: Dimitris Spachos <dspachos@gmail.com>
Co-authored-by: Liam Scott <liam@uilliam.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: tin-berri <tin@berri.ai>
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
This commit is contained in:
Sameer Kankute
2026-06-10 10:34:07 -07:00
committed by GitHub
co-authored by Hari greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Ceder Dens Yufeng He 冯基魁 victoruce kejunleng shin-berri yuneng-jiang Tyson Cung Claude Sonnet 4.6 Jeremy Chapeau Daan Avani Prajapati Kent daitran-tensormesh Dimitris Spachos Liam Scott Cursor Filippo Menghi milan-berri ryan-crabbe-berri michelligabriele tin-berri stuxf Mateo Wang
parent 2fe9feda71
commit 3b40ac987f
76 changed files with 5277 additions and 232 deletions
Download Patch File Download Diff File Expand all files Collapse all files
litellm/anthropic_beta_headers_config.json
+1 -1
View File
@@ -129,7 +129,7 @@
"bash_20241022": null,
"bash_20250124": null,
"code-execution-2025-08-25": null,
"compact-2026-01-12": null,
"compact-2026-01-12": "compact-2026-01-12",
"computer-use-2025-01-24": "computer-use-2025-01-24",
"computer-use-2025-11-24": "computer-use-2025-11-24",
"context-1m-2025-08-07": "context-1m-2025-08-07",
litellm/litellm_core_utils/get_litellm_params.py
+1
View File
@@ -34,6 +34,7 @@ _OPTIONAL_KWARGS_KEYS = frozenset(
"aws_bedrock_runtime_endpoint",
"tpm",
"rpm",
"use_xai_oauth",
}
)
litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py
-5
View File
@@ -633,11 +633,6 @@ def convert_to_model_response_object( # noqa: PLR0915
thinking_blocks = choice["message"]["thinking_blocks"]
provider_specific_fields["thinking_blocks"] = thinking_blocks
if reasoning_content:
provider_specific_fields["reasoning_content"] = (
reasoning_content
)
message = Message(
content=content,
role=choice["message"]["role"] or "assistant",
litellm/litellm_core_utils/prompt_templates/factory.py
+45 -2
View File
@@ -4290,6 +4290,49 @@ def _deduplicate_bedrock_tool_content(
return _deduplicate_bedrock_content_blocks(tool_content, "toolResult")
def _rename_duplicate_bedrock_document_names(
contents: List[BedrockMessageBlock],
) -> List[BedrockMessageBlock]:
"""
Rename duplicate document names across all messages in a Bedrock request.
Document names are derived from a content hash, so the same file appearing
in multiple conversation turns produces identical names and Bedrock rejects
the request with "Messages can not contain duplicate document names". The
first occurrence keeps its original name so prompt-cache prefixes stay
stable; later occurrences get a deterministic positional suffix
(``_2``, ``_3``, ...), bumped further if the suffixed name already
belongs to another document (e.g. an organic name ending in ``_2``).
"""
used_names: Set[str] = set()
for message in contents:
for block in message.get("content") or []:
document = block.get("document")
if isinstance(document, dict) and document.get("name"):
used_names.add(document["name"])
name_counts: Dict[str, int] = {}
for message in contents:
for block in message.get("content") or []:
document = block.get("document")
if not isinstance(document, dict):
continue
name = document.get("name")
if not name:
continue
count = name_counts.get(name, 0) + 1
name_counts[name] = count
if count > 1:
suffix = count
new_name = f"{name}_{suffix}"
while new_name in used_names:
suffix += 1
new_name = f"{name}_{suffix}"
used_names.add(new_name)
document["name"] = new_name
return contents
def _sort_bedrock_assistant_content_blocks(
blocks: List[BedrockContentBlock],
) -> List[BedrockContentBlock]:
@@ -4938,7 +4981,7 @@ class BedrockConverseMessagesProcessor:
llm_provider=llm_provider,
)
return contents
return _rename_duplicate_bedrock_document_names(contents)
@staticmethod
def translate_thinking_blocks_to_reasoning_content_blocks(
@@ -5360,7 +5403,7 @@ def _bedrock_converse_messages_pt( # noqa: PLR0915
llm_provider=llm_provider,
)
return contents
return _rename_duplicate_bedrock_document_names(contents)
def make_valid_bedrock_tool_name(input_tool_name: str) -> str:
litellm/llms/anthropic/experimental_pass_through/responses_adapters/streaming_iterator.py
+15 -1
View File
@@ -155,10 +155,24 @@ class AnthropicResponsesStreamWrapper:
event.get("delta", "") if isinstance(event, dict) else ""
)
block_idx = (
self._item_id_to_block_index.get(item_id, self._current_block_index)
self._item_id_to_block_index.get(item_id, -1)
if item_id
else self._current_block_index
)
if block_idx < 0:
# Some providers (e.g. LMStudio) skip response.output_item.added,
# so no text block is open yet; synthesize content_block_start
# instead of emitting a delta with index -1
block_idx = self._next_block_index()
if item_id:
self._item_id_to_block_index[item_id] = block_idx
self._chunk_queue.append(
{
"type": "content_block_start",
"index": block_idx,
"content_block": {"type": "text", "text": ""},
}
)
self._chunk_queue.append(
{
"type": "content_block_delta",
litellm/llms/bedrock/count_tokens/transformation.py
+32 -4
View File
@@ -11,6 +11,11 @@ from typing import Any, Dict, List, Optional
from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM
from litellm.llms.bedrock.common_utils import get_bedrock_base_model
# Placeholder satisfying the Anthropic InvokeModel schema's required
# max_tokens field; CountTokens only counts input, so it has no effect
# on any generation.
DEFAULT_ANTHROPIC_INVOKE_MODEL_MAX_TOKENS = 1024
class BedrockCountTokensConfig(BaseAWSLLM):
"""
@@ -32,8 +37,20 @@ class BedrockCountTokensConfig(BaseAWSLLM):
Returns:
'converse' or 'invokeModel'
"""
# If the request has messages in the expected Anthropic format, use converse
if "messages" in request_data and isinstance(request_data["messages"], list):
messages = request_data.get("messages")
if isinstance(messages, list):
# Anthropic content blocks carry a "type" key ({"type": "text", ...});
# Converse blocks don't ({"text": ...}, {"toolUse": ...}). Converse
# rejects Anthropic-shape blocks, so route those to invokeModel,
# which forwards the body verbatim.
for message in messages:
if not isinstance(message, dict):
continue
content = message.get("content")
if isinstance(content, list) and any(
isinstance(block, dict) and "type" in block for block in content
):
return "invokeModel"
return "converse"
# For raw text or other formats, use invokeModel
@@ -68,7 +85,7 @@ class BedrockCountTokensConfig(BaseAWSLLM):
{
"input": {
"invokeModel": {
"body": "{...raw model input...}"
"body": "<base64-encoded raw model input>"
}
}
}
@@ -168,13 +185,24 @@ class BedrockCountTokensConfig(BaseAWSLLM):
self, request_data: Dict[str, Any]
) -> Dict[str, Any]:
"""Transform to InvokeModel input format."""
import base64
import json
# For InvokeModel, we need to provide the raw body that would be sent to the model
# Remove the 'model' field from the body as it's not part of the model input
body_data = {k: v for k, v in request_data.items() if k != "model"}
return {"input": {"invokeModel": {"body": json.dumps(body_data)}}}
if "messages" in body_data:
# Bedrock validates the body against the model's InvokeModel schema;
# Anthropic Messages bodies require these fields.
body_data.setdefault("anthropic_version", "bedrock-2023-05-31")
body_data.setdefault(
"max_tokens", DEFAULT_ANTHROPIC_INVOKE_MODEL_MAX_TOKENS
)
# The CountTokens API expects invokeModel.body as a base64-encoded blob
encoded_body = base64.b64encode(json.dumps(body_data).encode()).decode()
return {"input": {"invokeModel": {"body": encoded_body}}}
def get_bedrock_count_tokens_endpoint(
self,
litellm/llms/bedrock_mantle/responses/transformation.py
+12 -4
View File
@@ -1,8 +1,10 @@
"""
Amazon Bedrock Mantle - Responses API backend.
gpt-5.5 / gpt-5.4 on Mantle are exposed ONLY on the `/openai/v1/responses`
path (not the standard `/v1/responses`). Payloads and SSE follow the OpenAI
Mantle serves Responses on two upstream paths: gpt frontier models (gpt-5.5 /
gpt-5.4) on `/openai/v1/responses`, and everything else that supports Responses
(e.g. gpt-oss) on the standard `/v1/responses`. The gate picks the path per
model and injects it via `use_openai_path`. Payloads and SSE follow the OpenAI
Responses spec, so this config inherits OpenAIResponsesAPIConfig and overrides
only the endpoint URL and authentication.
@@ -48,9 +50,14 @@ _MANTLE_HOST_RE = re.compile(
class BedrockMantleResponsesAPIConfig(OpenAIResponsesAPIConfig):
def __init__(self, aws_signer: Optional[BaseAWSLLM] = None):
def __init__(
self,
aws_signer: Optional[BaseAWSLLM] = None,
use_openai_path: bool = True,
):
super().__init__()
self._aws_signer = aws_signer or BaseAWSLLM()
self.use_openai_path = use_openai_path
@property
def custom_llm_provider(self) -> LlmProviders:
@@ -94,7 +101,8 @@ class BedrockMantleResponsesAPIConfig(OpenAIResponsesAPIConfig):
# single resolved region so aws_region_name wins; preserve custom proxy hosts.
if _MANTLE_HOST_RE.match(base):
base = f"https://bedrock-mantle.{region}.api.aws"
return f"{base}/openai/v1/responses"
path = "/openai/v1/responses" if self.use_openai_path else "/v1/responses"
return f"{base}{path}"
def validate_environment(
self, headers: dict, model: str, litellm_params: Optional[GenericLiteLLMParams]
litellm/llms/litellm_proxy/skills/README.md
+8 -8
View File
@@ -18,7 +18,7 @@ flowchart TB
F[Request with container.skills] --> G[SkillsInjectionHook]
G --> H{skill_id prefix?}
H -->|"litellm:skill_abc"| I[Fetch from LiteLLM DB]
H -->|"litellm_skill_abc"| I[Fetch from LiteLLM DB]
H -->|"skill_xyz" no prefix| J[Pass to Anthropic as native skill]
I --> K{Model provider?}
@@ -57,7 +57,7 @@ sequenceDiagram
Note over LiteLLM,PreHook: PRE-CALL HOOK
LiteLLM->>PreHook: Intercept request
PreHook->>PreHook: Fetch skill from DB (litellm:skill_id)
PreHook->>PreHook: Fetch skill from DB (litellm_skill_id)
PreHook->>PreHook: Extract SKILL.md from ZIP
PreHook->>PreHook: Inject SKILL.md into system prompt
PreHook->>PreHook: Add litellm_code_execution tool
@@ -105,7 +105,7 @@ response = await litellm.acompletion(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Create a bouncing ball GIF"}],
container={
"skills": [{"type": "custom", "skill_id": "litellm:skill_abc123"}]
"skills": [{"type": "custom", "skill_id": "litellm_skill_abc123"}]
},
)
@@ -261,7 +261,7 @@ response = litellm.completion(
messages=[{"role": "user", "content": "Analyze this data..."}],
container={
"skills": [
{"type": "custom", "skill_id": "litellm:skill_abc123"} # litellm: prefix
{"type": "custom", "skill_id": "litellm_skill_abc123"} # litellm_skill_ prefix
]
}
)
@@ -277,7 +277,7 @@ response = litellm.completion(
"messages": [{"role": "user", "content": "Help me analyze data"}],
"container": {
"skills": [
{"type": "custom", "skill_id": "litellm:skill_abc123"}
{"type": "custom", "skill_id": "litellm_skill_abc123"}
]
}
}
@@ -287,7 +287,7 @@ response = litellm.completion(
The hook (`litellm/proxy/hooks/litellm_skills/main.py`) intercepts the request:
1. **Detects `litellm:` prefix** → Fetches skill from database
1. **Detects `litellm_skill_` prefix** → Fetches skill from database
2. **Checks model provider** → Bedrock is not Anthropic
3. **Extracts SKILL.md** from stored ZIP file
4. **Converts skill to tool** + **Injects content into system prompt**
@@ -361,8 +361,8 @@ model LiteLLM_SkillsTable {
| Create skill on Anthropic | `anthropic` | N/A | Forward to Anthropic API |
| Create skill in LiteLLM DB | `litellm_proxy` | N/A | Store in database |
| Use Anthropic native skill | N/A | `skill_xyz` | Pass to Anthropic container.skills |
| Use LiteLLM skill on Anthropic | N/A | `litellm:skill_abc` | Convert to tools |
| Use LiteLLM skill on Bedrock/OpenAI | N/A | `litellm:skill_abc` | Convert to tools + inject SKILL.md |
| Use LiteLLM skill on Anthropic | N/A | `litellm_skill_abc` | Convert to tools |
| Use LiteLLM skill on Bedrock/OpenAI | N/A | `litellm_skill_abc` | Convert to tools + inject SKILL.md |
## Testing
litellm/llms/litellm_proxy/skills/constants.py
+4
View File
@@ -4,6 +4,10 @@ Constants for LiteLLM Skills
Centralized constants for skills processing, code execution, and sandbox configuration.
"""
LITELLM_SKILL_ID_PREFIX: str = "litellm_skill_"
"""Prefix for DB-backed skill IDs. The model-facing tool name is the skill ID
with hyphens/spaces replaced by underscores, which leaves this prefix intact."""
# Code execution loop settings
DEFAULT_MAX_ITERATIONS: int = 10
"""Maximum number of iterations for the automatic code execution loop."""
litellm/llms/litellm_proxy/skills/handler.py
+2 -1
View File
@@ -10,6 +10,7 @@ from typing import Any, Dict, List, Optional
from litellm._logging import verbose_logger
from litellm.caching.in_memory_cache import InMemoryCache
from litellm.llms.litellm_proxy.skills.constants import LITELLM_SKILL_ID_PREFIX
from litellm.proxy._types import LiteLLM_SkillsTable, NewSkillRequest, UserAPIKeyAuth
from litellm.proxy.common_utils.resource_ownership import (
get_primary_resource_owner_scope,
@@ -68,7 +69,7 @@ class LiteLLMSkillsHandler:
) -> LiteLLM_SkillsTable:
prisma_client = await LiteLLMSkillsHandler._get_prisma_client()
skill_id = f"litellm_skill_{uuid.uuid4()}"
skill_id = f"{LITELLM_SKILL_ID_PREFIX}{uuid.uuid4()}"
owner = get_primary_resource_owner_scope(user_api_key_dict) or user_id
if owner is None:
# Identity-less callers (no user_id / team_id / org_id /
litellm/llms/openai/completion/handler.py
+2
View File
@@ -49,6 +49,8 @@ class OpenAITextCompletion(BaseLLM):
headers: Optional[dict] = None,
):
try:
if headers:
optional_params = {**optional_params, "extra_headers": headers}
if headers is None:
headers = self.validate_environment(api_key=api_key)
if model is None or messages is None:
litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py
+5 -9
View File
@@ -19,7 +19,7 @@ from litellm.types.llms.vertex_ai import (
VertexAICachedContentResponseObject,
)
from ..common_utils import VertexAIError
from ..common_utils import VertexAIError, get_vertex_base_url
from ..vertex_llm_base import VertexBase
from .transformation import (
separate_cached_messages,
@@ -69,17 +69,13 @@ class ContextCachingEndpoints(VertexBase):
elif custom_llm_provider == "vertex_ai":
auth_header = vertex_auth_header
endpoint = "cachedContents"
if vertex_location == "global":
url = f"https://aiplatform.googleapis.com/v1/projects/{vertex_project}/locations/{vertex_location}/{endpoint}"
else:
url = f"https://{vertex_location}-aiplatform.googleapis.com/v1/projects/{vertex_project}/locations/{vertex_location}/{endpoint}"
base_url = get_vertex_base_url(vertex_location)
url = f"{base_url}/v1/projects/{vertex_project}/locations/{vertex_location}/{endpoint}"
else:
auth_header = vertex_auth_header
endpoint = "cachedContents"
if vertex_location == "global":
url = f"https://aiplatform.googleapis.com/v1beta1/projects/{vertex_project}/locations/{vertex_location}/{endpoint}"
else:
url = f"https://{vertex_location}-aiplatform.googleapis.com/v1beta1/projects/{vertex_project}/locations/{vertex_location}/{endpoint}"
base_url = get_vertex_base_url(vertex_location)
url = f"{base_url}/v1beta1/projects/{vertex_project}/locations/{vertex_location}/{endpoint}"
return self._check_custom_proxy(
api_base=api_base,
litellm/llms/xai/chat/transformation.py
+67
View File
@@ -5,6 +5,7 @@ import httpx
import litellm
from litellm._logging import verbose_logger
from litellm.constants import XAI_API_BASE
from litellm.exceptions import AuthenticationError
from litellm.litellm_core_utils.prompt_templates.common_utils import (
filter_value_from_dict,
strip_name_from_messages,
@@ -39,6 +40,72 @@ class XAIChatConfig(OpenAIGPTConfig):
dynamic_api_key = XAIModelInfo.get_api_key(api_key)
return api_base, dynamic_api_key
def validate_environment(
self,
headers: dict,
model: str,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> dict:
from litellm.llms.xai.oauth import (
XAIOAuthAuthenticator,
XAIOAuthError,
should_use_xai_oauth,
)
dynamic_api_key = XAIModelInfo.get_api_key(api_key)
if should_use_xai_oauth(litellm_params) and not dynamic_api_key:
try:
headers["Authorization"] = (
f"Bearer {XAIOAuthAuthenticator().get_access_token()}"
)
except XAIOAuthError as exc:
raise AuthenticationError(
model=model,
llm_provider=self.custom_llm_provider or "xai",
message=str(exc),
) from exc
if "content-type" not in headers and "Content-Type" not in headers:
headers["Content-Type"] = "application/json"
return headers
return super().validate_environment(
headers=headers,
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
api_key=dynamic_api_key,
api_base=api_base,
)
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
from litellm.llms.xai.oauth import XAIOAuthAuthenticator, should_use_xai_oauth
dynamic_api_key = XAIModelInfo.get_api_key(api_key)
if should_use_xai_oauth(litellm_params) and not dynamic_api_key:
api_base = XAIOAuthAuthenticator().get_api_base()
return super().get_complete_url(
api_base=api_base,
api_key=dynamic_api_key,
model=model,
optional_params=optional_params,
litellm_params=litellm_params,
stream=stream,
)
def get_supported_openai_params(self, model: str) -> list:
base_openai_params = [
"logit_bias",
litellm/llms/xai/oauth.py
+421
View File
@@ -0,0 +1,421 @@
import base64
import hashlib
import json
import os
import secrets
import sys
import threading
import time
import uuid
import webbrowser
from http.server import BaseHTTPRequestHandler, HTTPServer
from typing import Any, Dict, Optional, Tuple, Union
from urllib.parse import parse_qs, urlencode, urlparse
import httpx
from litellm._logging import verbose_logger
from litellm.constants import XAI_API_BASE
from litellm.llms.custom_httpx.http_handler import HTTPHandler, _get_httpx_client
from litellm.secret_managers.main import get_secret_str
XAI_OAUTH_ISSUER = "https://auth.x.ai"
XAI_OAUTH_DISCOVERY_URL = f"{XAI_OAUTH_ISSUER}/.well-known/openid-configuration"
XAI_OAUTH_CLIENT_ID = "b1a00492-073a-47ea-816f-4c329264a828"
XAI_OAUTH_SCOPE = "openid profile email offline_access grok-cli:access api:access"
XAI_OAUTH_REDIRECT_HOST = "127.0.0.1"
XAI_OAUTH_REDIRECT_PORT = 56121
XAI_OAUTH_REDIRECT_PATH = "/callback"
XAI_OAUTH_EXPIRY_SKEW_SECONDS = 120
XAI_OAUTH_CALLBACK_TIMEOUT_SECONDS = 180
_XAI_OAUTH_REFRESH_LOCK = threading.Lock()
class XAIOAuthError(Exception):
pass
class XAIOAuthLoginRequiredError(XAIOAuthError):
pass
class _CallbackHandler(BaseHTTPRequestHandler):
server: "_CallbackServer"
def do_GET(self) -> None:
parsed = urlparse(self.path)
if parsed.path != XAI_OAUTH_REDIRECT_PATH:
self.send_response(404)
self.end_headers()
return
params = parse_qs(parsed.query)
result = {
"code": params.get("code", [None])[0],
"state": params.get("state", [None])[0],
"error": params.get("error", [None])[0],
"error_description": params.get("error_description", [None])[0],
}
self.server.callback_result = result
if result["state"] != self.server.expected_state:
self.send_response(400)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.end_headers()
self.wfile.write(
b"<html><body><h1>xAI authorization state mismatch.</h1></body></html>"
)
return
self.send_response(200)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.end_headers()
body = (
b"<html><body><h1>xAI authorization failed.</h1>You can close this tab.</body></html>"
if result["error"]
else b"<html><body><h1>xAI authorization received.</h1>You can close this tab.</body></html>"
)
self.wfile.write(body)
def log_message(self, format: str, *args: Any) -> None:
return
class _CallbackServer(HTTPServer):
expected_state: str
callback_result: Optional[Dict[str, Optional[str]]]
class XAIOAuthAuthenticator:
def __init__(
self, http_client: Optional[Union[httpx.Client, HTTPHandler]] = None
) -> None:
self.token_dir = get_secret_str("XAI_OAUTH_TOKEN_DIR") or os.path.expanduser(
"~/.config/litellm/xai_oauth"
)
self.auth_file = os.path.join(
self.token_dir, get_secret_str("XAI_OAUTH_AUTH_FILE") or "auth.json"
)
self.http_client = http_client
def get_api_base(self) -> str:
return (
get_secret_str("XAI_OAUTH_API_BASE")
or get_secret_str("XAI_API_BASE")
or XAI_API_BASE
)
def get_access_token(self) -> str:
auth_data = self._read_auth_file()
if not auth_data:
raise XAIOAuthLoginRequiredError(
"xAI OAuth login required. Run `litellm xai-oauth login`."
)
access_token = auth_data.get("access_token")
if access_token and not self._is_expired(auth_data):
return access_token
refresh_token = auth_data.get("refresh_token")
if not refresh_token:
raise XAIOAuthLoginRequiredError(
"xAI OAuth refresh token missing. Run `litellm xai-oauth login`."
)
with _XAI_OAUTH_REFRESH_LOCK:
locked_auth_data = self._read_auth_file() or auth_data
access_token = locked_auth_data.get("access_token")
if access_token and not self._is_expired(locked_auth_data):
return access_token
refreshed = self._refresh_tokens(locked_auth_data)
return refreshed["access_token"]
def login(self, force: bool = False, no_browser: bool = False) -> Dict[str, Any]:
existing = self._read_auth_file()
if existing and not force and existing.get("access_token"):
if not self._is_expired(existing):
return existing
if existing.get("refresh_token"):
try:
return self._refresh_tokens(existing)
except XAIOAuthError:
pass
discovery = self._discover()
verifier, challenge = self._pkce_pair()
state = uuid.uuid4().hex
nonce = uuid.uuid4().hex
server, redirect_uri = self._start_callback_server(state)
authorize_url = self._build_authorize_url(
authorization_endpoint=discovery["authorization_endpoint"],
redirect_uri=redirect_uri,
challenge=challenge,
state=state,
nonce=nonce,
)
if no_browser or not webbrowser.open(authorize_url):
sys.stdout.write(
f"Open this URL to authenticate with xAI:\n{authorize_url}\n"
)
sys.stdout.flush()
result = self._wait_for_callback(server)
if result.get("state") != state:
raise XAIOAuthError("xAI OAuth state mismatch")
if result.get("error"):
description = result.get("error_description") or result["error"]
raise XAIOAuthError(f"xAI authorization failed: {description}")
code = result.get("code")
if not code:
raise XAIOAuthError("xAI authorization failed: no code returned")
token_payload = self._exchange_token(
discovery["token_endpoint"],
{
"grant_type": "authorization_code",
"code": code,
"redirect_uri": redirect_uri,
"client_id": XAI_OAUTH_CLIENT_ID,
"code_verifier": verifier,
},
)
auth_data = self._build_auth_record(token_payload, discovery["token_endpoint"])
self._write_auth_file(auth_data)
return auth_data
def _client(self) -> Union[httpx.Client, HTTPHandler]:
return self.http_client or _get_httpx_client()
def _ensure_token_dir(self) -> None:
os.makedirs(self.token_dir, mode=0o700, exist_ok=True)
try:
os.chmod(self.token_dir, 0o700)
except OSError:
verbose_logger.debug("Could not chmod xAI OAuth token directory")
def _read_auth_file(self) -> Optional[Dict[str, Any]]:
try:
with open(self.auth_file, "r") as f:
data = json.load(f)
return data if isinstance(data, dict) else None
except (IOError, json.JSONDecodeError):
return None
def _write_auth_file(self, data: Dict[str, Any]) -> None:
self._ensure_token_dir()
tmp_file = os.path.join(
self.token_dir,
f".{os.path.basename(self.auth_file)}.{uuid.uuid4().hex}.tmp",
)
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL
if hasattr(os, "O_NOFOLLOW"):
flags |= os.O_NOFOLLOW
fd = os.open(tmp_file, flags, 0o600)
try:
with os.fdopen(fd, "w") as f:
json.dump(data, f)
f.flush()
os.fsync(f.fileno())
os.replace(tmp_file, self.auth_file)
try:
os.chmod(self.auth_file, 0o600)
except OSError:
verbose_logger.debug("Could not chmod xAI OAuth auth file")
except Exception:
try:
os.close(fd)
except OSError:
pass
try:
os.unlink(tmp_file)
except OSError:
pass
raise
def _is_expired(self, auth_data: Dict[str, Any]) -> bool:
expires_at = auth_data.get("expires_at")
if expires_at is None:
return True
try:
return time.time() >= float(expires_at) - XAI_OAUTH_EXPIRY_SKEW_SECONDS
except (TypeError, ValueError):
return True
def _discover(self) -> Dict[str, str]:
try:
response = self._client().get(
XAI_OAUTH_DISCOVERY_URL, headers={"Accept": "application/json"}
)
response.raise_for_status()
except httpx.HTTPStatusError as exc:
raise XAIOAuthError(
f"xAI OAuth discovery request failed: {exc.response.status_code} {exc.response.text}"
) from exc
try:
data = response.json()
except ValueError as exc:
raise XAIOAuthError(
"xAI OAuth discovery response was not valid JSON"
) from exc
authorization_endpoint = data.get("authorization_endpoint")
token_endpoint = data.get("token_endpoint")
if not authorization_endpoint or not token_endpoint:
raise XAIOAuthError("xAI OAuth discovery missing endpoints")
return {
"authorization_endpoint": self._validate_xai_endpoint(
authorization_endpoint
),
"token_endpoint": self._validate_xai_endpoint(token_endpoint),
}
def _validate_xai_endpoint(self, url: str) -> str:
parsed = urlparse(url)
host = (parsed.hostname or "").lower()
if parsed.scheme != "https" or (host != "x.ai" and not host.endswith(".x.ai")):
raise XAIOAuthError(
f"xAI OAuth discovery returned unexpected endpoint: {url}"
)
return url
def _pkce_pair(self) -> Tuple[str, str]:
verifier = (
base64.urlsafe_b64encode(secrets.token_bytes(32)).rstrip(b"=").decode()
)
challenge = (
base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest())
.rstrip(b"=")
.decode()
)
return verifier, challenge
def _start_callback_server(self, state: str) -> Tuple[_CallbackServer, str]:
last_error: Optional[OSError] = None
for port in (XAI_OAUTH_REDIRECT_PORT, 0):
try:
server = _CallbackServer(
(XAI_OAUTH_REDIRECT_HOST, port), _CallbackHandler
)
server.expected_state = state
server.callback_result = None
actual_port = server.server_address[1]
redirect_uri = f"http://{XAI_OAUTH_REDIRECT_HOST}:{actual_port}{XAI_OAUTH_REDIRECT_PATH}"
return server, redirect_uri
except OSError as exc:
last_error = exc
raise XAIOAuthError(f"Could not start xAI OAuth callback server: {last_error}")
def _build_authorize_url(
self,
authorization_endpoint: str,
redirect_uri: str,
challenge: str,
state: str,
nonce: str,
) -> str:
params = {
"response_type": "code",
"client_id": XAI_OAUTH_CLIENT_ID,
"redirect_uri": redirect_uri,
"scope": XAI_OAUTH_SCOPE,
"code_challenge": challenge,
"code_challenge_method": "S256",
"state": state,
"nonce": nonce,
}
return f"{authorization_endpoint}?{urlencode(params)}"
def _wait_for_callback(self, server: _CallbackServer) -> Dict[str, Optional[str]]:
server.timeout = 1
deadline = time.time() + XAI_OAUTH_CALLBACK_TIMEOUT_SECONDS
try:
while time.time() < deadline:
server.handle_request()
if server.callback_result is not None:
return server.callback_result
finally:
server.server_close()
raise XAIOAuthError("Timed out waiting for xAI OAuth callback")
def _exchange_token(
self, token_endpoint: str, data: Dict[str, str]
) -> Dict[str, Any]:
try:
response = self._client().post(
token_endpoint,
headers={
"Accept": "application/json",
"Content-Type": "application/x-www-form-urlencoded",
},
data=data,
)
response.raise_for_status()
except httpx.HTTPStatusError as exc:
raise XAIOAuthError(
f"xAI OAuth token request failed: {exc.response.status_code} {exc.response.text}"
) from exc
try:
body = response.json()
except ValueError as exc:
raise XAIOAuthError("xAI OAuth token response was not valid JSON") from exc
if not isinstance(body, dict):
raise XAIOAuthError("xAI OAuth token response was not an object")
return body
def _build_auth_record(
self,
token_payload: Dict[str, Any],
token_endpoint: str,
fallback_refresh_token: Optional[str] = None,
) -> Dict[str, Any]:
access_token = token_payload.get("access_token")
refresh_token = token_payload.get("refresh_token") or fallback_refresh_token
if not access_token:
raise XAIOAuthError("xAI OAuth token response missing access_token")
if not refresh_token:
raise XAIOAuthError("xAI OAuth token response missing refresh_token")
expires_in = token_payload.get("expires_in") or 3600
try:
expires_at = int(time.time() + int(expires_in))
except (TypeError, ValueError):
expires_at = int(time.time() + 3600)
return {
"access_token": access_token,
"refresh_token": refresh_token,
"id_token": token_payload.get("id_token"),
"token_type": token_payload.get("token_type") or "Bearer",
"token_endpoint": token_endpoint,
"expires_at": expires_at,
}
def _refresh_tokens(self, auth_data: Dict[str, Any]) -> Dict[str, Any]:
token_endpoint = auth_data.get("token_endpoint")
if not token_endpoint:
token_endpoint = self._discover()["token_endpoint"]
token_endpoint = self._validate_xai_endpoint(token_endpoint)
refresh_token = auth_data.get("refresh_token")
if not refresh_token:
raise XAIOAuthLoginRequiredError(
"xAI OAuth refresh token missing. Run `litellm xai-oauth login`."
)
token_payload = self._exchange_token(
token_endpoint,
{
"grant_type": "refresh_token",
"refresh_token": refresh_token,
"client_id": XAI_OAUTH_CLIENT_ID,
},
)
refreshed = self._build_auth_record(
token_payload,
token_endpoint,
fallback_refresh_token=refresh_token,
)
self._write_auth_file(refreshed)
return refreshed
def should_use_xai_oauth(litellm_params: Optional[Dict[str, Any]]) -> bool:
return bool((litellm_params or {}).get("use_xai_oauth"))
litellm/llms/xai/responses/transformation.py
+32 -6
View File
@@ -3,6 +3,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union
import litellm
from litellm._logging import verbose_logger
from litellm.constants import XAI_API_BASE
from litellm.exceptions import AuthenticationError
from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
from litellm.llms.xai.common_utils import XAIModelInfo
from litellm.secret_managers.main import get_secret_str
@@ -220,10 +221,27 @@ class XAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
litellm_params.api_key, legacy_generic_before_env=True
)
if not api_key:
from litellm.llms.xai.oauth import (
XAIOAuthAuthenticator,
XAIOAuthError,
should_use_xai_oauth,
)
if should_use_xai_oauth(litellm_params.model_dump()):
try:
api_key = XAIOAuthAuthenticator().get_access_token()
except XAIOAuthError as exc:
raise AuthenticationError(
model=model,
llm_provider=self.custom_llm_provider.value,
message=str(exc),
) from exc
if not api_key:
raise ValueError(
"XAI API key is required. Set api_key, litellm.xai_key, "
"litellm.api_key, or XAI_API_KEY."
"litellm.api_key, XAI_API_KEY, or use_xai_oauth=True."
)
headers.update(
@@ -244,12 +262,20 @@ class XAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
Returns:
str: The full URL for the XAI /responses endpoint
"""
api_base = (
api_base
or litellm.api_base
or get_secret_str("XAI_API_BASE")
or XAI_API_BASE
from litellm.llms.xai.oauth import XAIOAuthAuthenticator, should_use_xai_oauth
api_key = XAIModelInfo.get_api_key(
litellm_params.get("api_key"), legacy_generic_before_env=True
)
if should_use_xai_oauth(litellm_params) and not api_key:
api_base = XAIOAuthAuthenticator().get_api_base()
else:
api_base = (
api_base
or litellm.api_base
or get_secret_str("XAI_API_BASE")
or XAI_API_BASE
)
# Remove trailing slashes
api_base = api_base.rstrip("/")
litellm/main.py
+2 -3
View File
@@ -1638,6 +1638,7 @@ def completion( # type: ignore # noqa: PLR0915
litellm_request_debug=kwargs.get("litellm_request_debug", False),
tpm=kwargs.get("tpm"),
rpm=kwargs.get("rpm"),
use_xai_oauth=kwargs.get("use_xai_oauth", False),
)
cast(LiteLLMLoggingObj, logging).update_environment_variables(
model=model,
@@ -2134,9 +2135,6 @@ def completion( # type: ignore # noqa: PLR0915
headers = headers or litellm.headers
if extra_headers is not None:
optional_params["extra_headers"] = extra_headers
## LOAD CONFIG - if set
config = litellm.OpenAITextCompletionConfig.get_config()
for k, v in config.items():
@@ -2162,6 +2160,7 @@ def completion( # type: ignore # noqa: PLR0915
_response = openai_text_completions.completion(
model=model,
messages=messages,
headers=headers,
model_response=model_response,
print_verbose=print_verbose,
api_key=api_key,
litellm/model_prices_and_context_window_backup.json
+168 -4
View File
@@ -24392,9 +24392,12 @@
"max_output_tokens": 8192
},
"minimax/MiniMax-M3": {
"input_cost_per_token": 6e-07,
"output_cost_per_token": 2.4e-06,
"cache_read_input_token_cost": 1.2e-07,
"input_cost_per_token": 3e-07,
"input_cost_per_token_above_512k_tokens": 6e-07,
"output_cost_per_token": 1.2e-06,
"output_cost_per_token_above_512k_tokens": 2.4e-06,
"cache_read_input_token_cost": 6e-08,
"cache_read_input_token_cost_above_512k_tokens": 1.2e-07,
"litellm_provider": "minimax",
"mode": "chat",
"supports_function_calling": true,
@@ -24403,7 +24406,7 @@
"supports_reasoning": true,
"supports_system_messages": true,
"supports_vision": true,
"max_input_tokens": 512000,
"max_input_tokens": 1000000,
"max_output_tokens": 128000
},
"mistral.devstral-2-123b": {
@@ -41646,6 +41649,7 @@
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
"use_openai_responses_path": true,
"supported_endpoints": ["/v1/responses"],
"supported_modalities": ["text", "image"],
"supported_output_modalities": ["text"],
@@ -41665,6 +41669,7 @@
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
"use_openai_responses_path": true,
"supported_endpoints": ["/v1/responses"],
"supported_modalities": ["text", "image"],
"supported_output_modalities": ["text"],
@@ -41966,5 +41971,164 @@
"/v1/audio/transcriptions"
],
"supports_audio_input": true
},
"tensormesh/Qwen/Qwen3.5-397B-A17B-FP8": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 6e-07,
"output_cost_per_token": 3.6e-06,
"cache_read_input_token_cost": 0,
"max_input_tokens": 262144,
"max_output_tokens": 262144,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 4.5e-07,
"output_cost_per_token": 1.8e-06,
"cache_read_input_token_cost": 0,
"max_input_tokens": 262144,
"max_output_tokens": 262144,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/Qwen/Qwen3.6-27B-FP8": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 3.2e-07,
"output_cost_per_token": 3.2e-06,
"cache_read_input_token_cost": 0,
"max_input_tokens": 262144,
"max_output_tokens": 262144,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/lukealonso/GLM-5.1-NVFP4-MTP": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 1.4e-06,
"output_cost_per_token": 4.4e-06,
"cache_read_input_token_cost": 0,
"max_input_tokens": 202752,
"max_output_tokens": 202752,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/deepseek-ai/DeepSeek-V4-Flash": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 1.4e-07,
"output_cost_per_token": 2.8e-07,
"cache_read_input_token_cost": 0,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/moonshotai/Kimi-K2.6": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 9.6e-07,
"output_cost_per_token": 4e-06,
"cache_read_input_token_cost": 0,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/MiniMaxAI/MiniMax-M2.5": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 3e-07,
"output_cost_per_token": 1.2e-06,
"cache_read_input_token_cost": 0,
"max_input_tokens": 196608,
"max_output_tokens": 196608,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/google/gemma-4-31B-it": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 1.4e-07,
"output_cost_per_token": 5.6e-07,
"cache_read_input_token_cost": 0,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/openai/gpt-oss-120b": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 1.5e-07,
"output_cost_per_token": 6e-07,
"cache_read_input_token_cost": 0,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/openai/gpt-oss-20b": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 7e-08,
"output_cost_per_token": 2.8e-07,
"cache_read_input_token_cost": 0,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
}
}
litellm/proxy/_experimental/mcp_server/mcp_context.py
+6
View File
@@ -19,3 +19,9 @@ _mcp_active_toolset_id: ContextVar[Optional[str]] = ContextVar(
_mcp_gateway_initialize_instructions: ContextVar[Optional[str]] = ContextVar(
"_mcp_gateway_initialize_instructions", default=None
)
# Per-request scoped server name; set in MCP HTTP/SSE handlers when the path
# identifies exactly one upstream server. Never populated from client-supplied headers.
_mcp_gateway_server_name: ContextVar[Optional[str]] = ContextVar(
"_mcp_gateway_server_name", default=None
)
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
+48
View File
@@ -354,6 +354,52 @@ def _deserialize_json_list(data: Any) -> Optional[List[Dict[str, Any]]]:
]
def _normalize_mcp_server_cost_info(mcp_info: MCPInfo) -> None:
"""Coerce ``mcp_server_cost_info`` numeric fields to ``float`` at ingest.
YAML 1.1 parses scientific notation without a decimal point (e.g.
``7e-05``) as a string, and ``MCPServerCostInfo`` is a TypedDict with no
runtime validation, so string-typed costs flow through to the UI and
crash its ``.toFixed`` formatting. Values that cannot be coerced are
dropped with a warning instead of failing the server load.
"""
cost_info = mcp_info.get("mcp_server_cost_info")
if not isinstance(cost_info, dict):
return
server_name = mcp_info.get("server_name")
normalized = dict(cost_info)
default_cost = normalized.get("default_cost_per_query")
if default_cost is not None:
try:
normalized["default_cost_per_query"] = float(default_cost)
except (TypeError, ValueError):
verbose_logger.warning(
"MCP server '%s' has non-numeric default_cost_per_query %r; ignoring it",
server_name,
default_cost,
)
del normalized["default_cost_per_query"]
tool_costs = normalized.get("tool_name_to_cost_per_query")
if isinstance(tool_costs, dict):
normalized_tool_costs = {}
for tool_name, cost in tool_costs.items():
try:
normalized_tool_costs[tool_name] = float(cost)
except (TypeError, ValueError):
verbose_logger.warning(
"MCP server '%s' has non-numeric cost %r for tool '%s'; ignoring it",
server_name,
cost,
tool_name,
)
normalized["tool_name_to_cost_per_query"] = normalized_tool_costs
mcp_info["mcp_server_cost_info"] = normalized
def _create_sampling_callback(user_api_key_auth: Optional[Any] = None):
"""
Create a sampling callback for MCP ClientSession.
@@ -621,6 +667,7 @@ class MCPServerManager:
mcp_info["server_name"] = server_name
if "description" not in mcp_info and server_config.get("description"):
mcp_info["description"] = server_config.get("description")
_normalize_mcp_server_cost_info(mcp_info)
# Use alias for name if present, else server_name
alias = server_config.get("alias", None)
@@ -1091,6 +1138,7 @@ class MCPServerManager:
mcp_info["server_name"] = mcp_server.server_name or mcp_server.server_id
if "description" not in mcp_info and mcp_server.description:
mcp_info["description"] = mcp_server.description
_normalize_mcp_server_cost_info(mcp_info)
auth_type = cast(MCPAuthType, mcp_server.auth_type)
server_url = mcp_server.url
litellm/proxy/_experimental/mcp_server/server.py
+25 -4
View File
@@ -47,6 +47,7 @@ from litellm.proxy._experimental.mcp_server.discoverable_endpoints import (
from litellm.proxy._experimental.mcp_server.mcp_context import (
_mcp_active_toolset_id,
_mcp_gateway_initialize_instructions,
_mcp_gateway_server_name,
)
from litellm.proxy._experimental.mcp_server.mcp_debug import MCPDebug
from litellm.proxy._experimental.mcp_server.utils import (
@@ -323,10 +324,14 @@ if MCP_AVAILABLE:
notification_options=notification_options,
experimental_capabilities=experimental_capabilities or {},
)
updates: Dict[str, Any] = {}
merged = _mcp_gateway_initialize_instructions.get()
if merged is not None:
return opts.model_copy(update={"instructions": merged})
return opts
updates["instructions"] = merged
scoped_server_name = _mcp_gateway_server_name.get()
if scoped_server_name is not None:
updates["server_name"] = scoped_server_name
return opts.model_copy(update=updates) if updates else opts
########################################################
############ Initialize the MCP Server #################
@@ -1544,6 +1549,7 @@ if MCP_AVAILABLE:
user_api_key_auth: Optional[UserAPIKeyAuth],
mcp_servers: Optional[List[str]],
client_ip: Optional[str],
scoped_server_endpoint: bool = False,
) -> AsyncIterator[None]:
allowed = await _get_allowed_mcp_servers(
user_api_key_auth=user_api_key_auth,
@@ -1565,11 +1571,22 @@ if MCP_AVAILABLE:
return_exceptions=True,
)
merged = _merge_gateway_initialize_instructions(allowed_mcp_servers=allowed)
tok = _mcp_gateway_initialize_instructions.set(merged)
scoped_server_name = None
if scoped_server_endpoint and len(allowed) == 1:
scoped_server = allowed[0]
scoped_server_name = (
scoped_server.alias
or scoped_server.server_name
or scoped_server.name
or scoped_server.server_id
)
instructions_token = _mcp_gateway_initialize_instructions.set(merged)
server_name_token = _mcp_gateway_server_name.set(scoped_server_name)
try:
yield
finally:
_mcp_gateway_initialize_instructions.reset(tok)
_mcp_gateway_initialize_instructions.reset(instructions_token)
_mcp_gateway_server_name.reset(server_name_token)
async def _get_tools_from_mcp_servers( # noqa: PLR0915
user_api_key_auth: Optional[UserAPIKeyAuth],
@@ -3620,6 +3637,7 @@ if MCP_AVAILABLE:
oauth2_headers,
raw_headers,
) = await extract_mcp_auth_context(scope, path)
scoped_server_endpoint = len(_get_mcp_servers_in_path(path) or []) == 1
# Extract client IP for MCP access control
_client_ip = IPAddressUtils.get_mcp_client_ip(StarletteRequest(scope))
@@ -3896,6 +3914,7 @@ if MCP_AVAILABLE:
user_api_key_auth,
mcp_servers,
_client_ip,
scoped_server_endpoint=scoped_server_endpoint,
):
await target_manager.handle_request(scope, receive, local_send)
if use_stateful and session_id and scope.get("method") == "DELETE":
@@ -3980,6 +3999,7 @@ if MCP_AVAILABLE:
oauth2_headers,
raw_headers,
) = await extract_mcp_auth_context(scope, path)
scoped_server_endpoint = len(_get_mcp_servers_in_path(path) or []) == 1
# Extract client IP for MCP access control
_sse_client_ip = IPAddressUtils.get_mcp_client_ip(StarletteRequest(scope))
@@ -4052,6 +4072,7 @@ if MCP_AVAILABLE:
user_api_key_auth,
mcp_servers,
_sse_client_ip,
scoped_server_endpoint=scoped_server_endpoint,
):
await sse_session_manager.handle_request(scope, receive, send)
except MCPUpstreamAuthError as e:
litellm/proxy/_types.py
+2
View File
@@ -23,6 +23,7 @@ from litellm.litellm_core_utils.initialize_dynamic_callback_params import (
from litellm.types.integrations.slack_alerting import AlertType
from litellm.types.llms.openai import (
AllMessageValues,
ResponsesAPIResponse,
)
from litellm.types.mcp import (
MCPAuthType,
@@ -3834,6 +3835,7 @@ PassThroughEndpointLoggingResultValues = Union[
EmbeddingResponse,
VideoObject,
StandardPassThroughResponseObject,
ResponsesAPIResponse,
]
litellm/proxy/auth/auth_checks.py
+5 -2
View File
@@ -3516,10 +3516,13 @@ async def _virtual_key_max_budget_check(
if valid_token.max_budget is not None:
from litellm.proxy.proxy_server import get_current_spend
fallback_spend = valid_token.spend or 0.0
counter_key = f"spend:key:{valid_token.token}"
# Read spend from cross-pod counter (Redis-first) or cached object (fallback)
spend = await get_current_spend(
counter_key=f"spend:key:{valid_token.token}",
fallback_spend=valid_token.spend or 0.0,
counter_key=counter_key,
fallback_spend=fallback_spend,
)
####################################
litellm/proxy/common_request_processing.py
+15 -1
View File
@@ -1949,12 +1949,26 @@ class ProxyBaseLLMRequestProcessing:
code=status.HTTP_400_BAD_REQUEST,
headers=headers,
)
# Extract status_code from the exception if it carries one.
# Provider exceptions (NotFoundError, BadRequestError, GeminiError,
# VertexAIError, etc.) all have a status_code attribute reflecting
# the upstream API response. Use it to return the correct HTTP code
# instead of defaulting to 500.
_exc_status_code = getattr(e, "status_code", None)
if (
_exc_status_code is not None
and isinstance(_exc_status_code, int)
and 400 <= _exc_status_code <= 599
):
_code = _exc_status_code
else:
_code = status.HTTP_500_INTERNAL_SERVER_ERROR
raise ProxyException(
message=getattr(e, "message", error_msg),
type=getattr(e, "type", "None"),
param=getattr(e, "param", "None"),
openai_code=getattr(e, "code", None),
code=getattr(e, "status_code", 500),
code=_code,
provider_specific_fields=getattr(e, "provider_specific_fields", None),
headers=headers,
)
litellm/proxy/guardrails/guardrail_hooks/presidio.py
+40 -1
View File
@@ -1225,6 +1225,39 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
for chunk in all_chunks:
yield chunk
@staticmethod
def _unmask_sse_bytes_chunk(chunk: bytes, pii_tokens: Dict[str, str]) -> bytes:
try:
text = chunk.decode("utf-8")
except UnicodeDecodeError:
return chunk
result_lines: List[str] = []
for line in text.split("\n"):
line = line.rstrip("\r")
if line.startswith("data: ") and line != "data: [DONE]":
raw_json = line[6:]
try:
event = json.loads(raw_json)
delta = event.get("delta") if isinstance(event, dict) else None
if (
isinstance(delta, dict)
and event.get("type") == "content_block_delta"
and delta.get("type") == "text_delta"
and isinstance(delta.get("text"), str)
):
unmasked = _OPTIONAL_PresidioPIIMasking._unmask_pii_text(
delta["text"], pii_tokens
)
if unmasked != delta["text"]:
event["delta"]["text"] = unmasked
line = "data: " + json.dumps(event, ensure_ascii=False)
except (json.JSONDecodeError, KeyError, TypeError):
pass
result_lines.append(line)
return "\n".join(result_lines).encode("utf-8")
async def _stream_pii_unmasking(
self,
response: Any,
@@ -1237,13 +1270,19 @@ class _OPTIONAL_PresidioPIIMasking(CustomGuardrail):
from litellm.main import stream_chunk_builder
from litellm.types.utils import ModelResponse
metadata = (request_data.get("metadata") or {}) if request_data else {}
pii_tokens: Dict[str, str] = metadata.get("pii_tokens", {})
remaining_chunks: List[ModelResponseStream] = []
try:
async for chunk in response:
if isinstance(chunk, ModelResponseStream):
remaining_chunks.append(chunk)
elif isinstance(chunk, bytes):
yield chunk # type: ignore[misc]
if pii_tokens:
yield self._unmask_sse_bytes_chunk(chunk, pii_tokens) # type: ignore[misc]
else:
yield chunk # type: ignore[misc]
continue
if not remaining_chunks:
litellm/proxy/hooks/litellm_skills/main.py
+9 -8
View File
@@ -19,7 +19,7 @@ Usage:
response = await litellm.acompletion(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Create a bouncing ball GIF"}],
container={"skills": [{"skill_id": "litellm:skill_abc123"}]},
container={"skills": [{"skill_id": "litellm_skill_abc123"}]},
)
# Response includes file_ids for generated files
"""
@@ -31,6 +31,7 @@ from typing import Any, Dict, List, Optional, Union
from litellm._logging import verbose_proxy_logger
from litellm.caching.caching import DualCache
from litellm.integrations.custom_logger import CustomLogger
from litellm.llms.litellm_proxy.skills.constants import LITELLM_SKILL_ID_PREFIX
from litellm.llms.litellm_proxy.skills.prompt_injection import (
SkillPromptInjectionHandler,
)
@@ -43,7 +44,7 @@ class SkillsInjectionHook(CustomLogger):
Pre/Post-call hook that processes skills from container.skills parameter.
Pre-call (async_pre_call_hook):
- Skills with 'litellm:' prefix are fetched from LiteLLM DB
- Skills with 'litellm_skill_' prefix are fetched from LiteLLM DB
- For Anthropic models: native skills pass through, LiteLLM skills converted to tools
- For non-Anthropic models: LiteLLM skills are converted to tools + execute_code tool
@@ -78,7 +79,7 @@ class SkillsInjectionHook(CustomLogger):
Process skills from container.skills before the LLM call.
1. Check if container.skills exists in request
2. Separate skills by prefix (litellm: vs native)
2. Separate skills by prefix (litellm_skill_ vs native)
3. Fetch LiteLLM skills from database
4. For Anthropic: keep native skills in container
5. For non-Anthropic: convert LiteLLM skills to tools, inject content, add execute_code
@@ -108,7 +109,7 @@ class SkillsInjectionHook(CustomLogger):
continue
skill_id = skill.get("skill_id", "")
if skill_id.startswith("litellm_"):
if skill_id.startswith(LITELLM_SKILL_ID_PREFIX):
# Fetch from LiteLLM DB
db_skill = await self._fetch_skill_from_db(
skill_id,
@@ -287,7 +288,7 @@ class SkillsInjectionHook(CustomLogger):
Fetch a skill from the LiteLLM database.
Args:
skill_id: The skill ID (without 'litellm:' prefix)
skill_id: The skill ID (including the 'litellm_skill_' prefix)
Returns:
LiteLLM_SkillsTable or None if not found
@@ -382,10 +383,10 @@ class SkillsInjectionHook(CustomLogger):
has_executable_tool = False
for tc in tool_calls:
tool_name = tc.get("name", "")
# Execute if it's litellm_code_execution OR a skill tool (skill_xxx)
# Execute if it's litellm_code_execution OR a skill tool (litellm_skill_xxx)
if (
tool_name == LiteLLMInternalTools.CODE_EXECUTION.value
or tool_name.startswith("skill_")
or tool_name.startswith(LITELLM_SKILL_ID_PREFIX)
):
has_executable_tool = True
break
@@ -543,7 +544,7 @@ class SkillsInjectionHook(CustomLogger):
result = await self._execute_code(
code, skill_files, executor, generated_files
)
elif tool_name.startswith("skill_"):
elif tool_name.startswith(LITELLM_SKILL_ID_PREFIX):
# Skill tool - execute the skill's code
result = await self._execute_skill_tool(
tool_name, tool_input, skill_files, executor, generated_files
litellm/proxy/hooks/parallel_request_limiter.py
+16 -5
View File
@@ -7,7 +7,7 @@ from pydantic import BaseModel
from typing_extensions import TypedDict
import litellm
from litellm import DualCache, ModelResponse
from litellm import DualCache, EmbeddingResponse, ModelResponse, TextCompletionResponse
from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.core_helpers import _get_parent_otel_span_from_kwargs
@@ -570,7 +570,9 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
total_tokens = 0
if isinstance(response_obj, ModelResponse):
if isinstance(
response_obj, (ModelResponse, EmbeddingResponse, TextCompletionResponse)
):
total_tokens = response_obj.usage.total_tokens # type: ignore
# ------------
@@ -659,7 +661,10 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
if user_api_key_user_id is not None:
total_tokens = 0
if isinstance(response_obj, ModelResponse):
if isinstance(
response_obj,
(ModelResponse, EmbeddingResponse, TextCompletionResponse),
):
total_tokens = response_obj.usage.total_tokens # type: ignore
request_count_api_key = (
@@ -692,7 +697,10 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
if user_api_key_team_id is not None:
total_tokens = 0
if isinstance(response_obj, ModelResponse):
if isinstance(
response_obj,
(ModelResponse, EmbeddingResponse, TextCompletionResponse),
):
total_tokens = response_obj.usage.total_tokens # type: ignore
request_count_api_key = (
@@ -725,7 +733,10 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
if user_api_key_end_user_id is not None:
total_tokens = 0
if isinstance(response_obj, ModelResponse):
if isinstance(
response_obj,
(ModelResponse, EmbeddingResponse, TextCompletionResponse),
):
total_tokens = response_obj.usage.total_tokens # type: ignore
request_count_api_key = (
litellm/proxy/hooks/parallel_request_limiter_v3.py
+15 -4
View File
@@ -39,7 +39,13 @@ from litellm.proxy.common_utils.proxy_rate_limit_error import (
from litellm.proxy.hooks.rate_limiter_utils import resolve_llm_provider_for_rate_limit
from litellm.types.caching import RedisPipelineIncrementOperation
from litellm.types.llms.openai import BaseLiteLLMOpenAIResponseObject
from litellm.types.utils import CallTypes, ModelResponse, Usage
from litellm.types.utils import (
CallTypes,
EmbeddingResponse,
ModelResponse,
TextCompletionResponse,
Usage,
)
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
@@ -2736,9 +2742,14 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
# Get total tokens from response
total_tokens = 0
# spot fix for /responses api
if isinstance(response_obj, ModelResponse) or isinstance(
response_obj, BaseLiteLLMOpenAIResponseObject
if isinstance(
response_obj,
(
ModelResponse,
EmbeddingResponse,
TextCompletionResponse,
BaseLiteLLMOpenAIResponseObject,
),
):
_usage = getattr(response_obj, "usage", None)
total_tokens = self._get_total_tokens_from_usage(
litellm/proxy/management_endpoints/key_management_endpoints.py
+23 -6
View File
@@ -1850,11 +1850,10 @@ async def prepare_key_update_data(
if "budget_duration" in non_default_values:
budget_duration = non_default_values.pop("budget_duration")
if (
budget_duration
and (isinstance(budget_duration, str))
and len(budget_duration) > 0
):
if budget_duration is None:
non_default_values["budget_duration"] = None
non_default_values["budget_reset_at"] = None
elif isinstance(budget_duration, str) and len(budget_duration) > 0:
from litellm.proxy.common_utils.timezone_utils import get_budget_reset_time
key_reset_at = get_budget_reset_time(budget_duration=budget_duration)
@@ -2518,7 +2517,7 @@ async def update_key_fn( # noqa: PLR0915
},
)
data_json: dict = data.model_dump(exclude_unset=True, exclude_none=True)
data_json: dict = data.model_dump(exclude_unset=True)
key = data_json.pop("key")
# get the row from db
@@ -2588,6 +2587,17 @@ async def update_key_fn( # noqa: PLR0915
proxy_logging_obj=proxy_logging_obj,
)
if data.spend is not None:
try:
from litellm.proxy.proxy_server import _invalidate_spend_counter
token_to_invalidate = _hash_token_if_needed(key)
await _invalidate_spend_counter(
counter_key=f"spend:key:{token_to_invalidate}"
)
except Exception:
pass
asyncio.create_task(
KeyManagementEventHooks.async_key_updated_hook(
data=data,
@@ -4775,6 +4785,13 @@ async def reset_key_spend_fn(
proxy_logging_obj=proxy_logging_obj,
)
try:
from litellm.proxy.proxy_server import _invalidate_spend_counter
await _invalidate_spend_counter(counter_key=f"spend:key:{hashed_api_key}")
except Exception:
pass
max_budget = updated_key.max_budget
budget_reset_at = updated_key.budget_reset_at
litellm/proxy/management_endpoints/team_endpoints.py
+64 -5
View File
@@ -357,6 +357,42 @@ class TeamMemberBudgetHandler:
data_dict.pop("team_member_rpm_limit", None)
data_dict.pop("team_member_tpm_limit", None)
@staticmethod
async def clear_team_member_budget_fields(
team_table: LiteLLM_TeamTable,
user_api_key_dict: "UserAPIKeyAuth",
updated_kv: dict,
explicitly_set_fields: set,
) -> dict:
"""Clear explicitly-nulled fields on the team member budget row."""
from litellm.proxy._types import BudgetNewRequest
from litellm.proxy.management_endpoints.budget_management_endpoints import (
update_budget,
)
if team_table.metadata is None:
team_table.metadata = {}
team_member_budget_id = team_table.metadata.get("team_member_budget_id")
if team_member_budget_id is not None and isinstance(team_member_budget_id, str):
budget_request = BudgetNewRequest(budget_id=team_member_budget_id)
if "team_member_budget" in explicitly_set_fields:
budget_request.max_budget = None
if "team_member_budget_duration" in explicitly_set_fields:
budget_request.budget_duration = None
budget_request.budget_reset_at = None
if "team_member_rpm_limit" in explicitly_set_fields:
budget_request.rpm_limit = None
if "team_member_tpm_limit" in explicitly_set_fields:
budget_request.tpm_limit = None
await update_budget(
budget_obj=budget_request,
user_api_key_dict=user_api_key_dict,
)
TeamMemberBudgetHandler._clean_team_member_fields(updated_kv)
return updated_kv
@staticmethod
async def backfill_team_member_budget_entries(
team_id: str,
@@ -1872,11 +1908,25 @@ async def update_team( # noqa: PLR0915
# Check budget_duration and budget_reset_at
_set_budget_reset_at(data, updated_kv)
if TeamMemberBudgetHandler.should_create_budget(
team_member_budget=data.team_member_budget,
team_member_rpm_limit=data.team_member_rpm_limit,
team_member_tpm_limit=data.team_member_tpm_limit,
team_member_budget_duration=data.team_member_budget_duration,
_team_member_fields_in_request = {
field
for field in [
"team_member_budget",
"team_member_rpm_limit",
"team_member_tpm_limit",
"team_member_budget_duration",
]
if field in updated_kv
}
if (
_team_member_fields_in_request
and TeamMemberBudgetHandler.should_create_budget(
team_member_budget=data.team_member_budget,
team_member_rpm_limit=data.team_member_rpm_limit,
team_member_tpm_limit=data.team_member_tpm_limit,
team_member_budget_duration=data.team_member_budget_duration,
)
):
updated_kv = await TeamMemberBudgetHandler.upsert_team_member_budget_table(
team_table=existing_team_row,
@@ -1899,6 +1949,13 @@ async def update_team( # noqa: PLR0915
team_member_budget_id=_backfill_budget_id,
prisma_client=prisma_client,
)
elif _team_member_fields_in_request:
updated_kv = await TeamMemberBudgetHandler.clear_team_member_budget_fields(
team_table=existing_team_row,
user_api_key_dict=user_api_key_dict,
updated_kv=updated_kv,
explicitly_set_fields=_team_member_fields_in_request,
)
else:
TeamMemberBudgetHandler._clean_team_member_fields(updated_kv)
@@ -1987,6 +2044,8 @@ def _set_budget_reset_at(data: UpdateTeamRequest, updated_kv: dict) -> None:
reset_at = get_budget_reset_time(budget_duration=data.budget_duration)
updated_kv["budget_reset_at"] = reset_at
elif "budget_duration" in updated_kv and updated_kv["budget_duration"] is None:
updated_kv["budget_reset_at"] = None
if data.budget_limits is not None and len(data.budget_limits) > 0:
from litellm.proxy.common_utils.timezone_utils import get_budget_reset_time
litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py
+1 -6
View File
@@ -426,12 +426,7 @@ async def mistral_proxy_route(
)
## check for streaming
is_streaming_request = False
# anthropic is streaming when 'stream' = True is in the body
if request.method == "POST":
_request_body = await request.json()
if _request_body.get("stream"):
is_streaming_request = True
is_streaming_request = await is_streaming_request_fn(request)
## CREATE PASS-THROUGH
endpoint_func = create_pass_through_route(
litellm/proxy/pass_through_endpoints/llm_provider_handlers/openai_passthrough_logging_handler.py
+54 -22
View File
@@ -5,7 +5,7 @@ Handles cost tracking and logging for OpenAI passthrough endpoints, specifically
"""
from datetime import datetime
from typing import List, Optional, Union
from typing import List, Optional, Tuple, Union
from urllib.parse import urlparse
import httpx
@@ -18,6 +18,7 @@ from litellm.litellm_core_utils.litellm_logging import (
)
from litellm.llms.openai.openai import OpenAIConfig
from litellm.llms.openai.openai import OpenAIConfig as OpenAIConfigType
from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
from litellm.proxy._types import PassThroughEndpointLoggingTypedDict
from litellm.proxy.pass_through_endpoints.llm_provider_handlers.base_passthrough_logging_handler import (
BasePassthroughLoggingHandler,
@@ -29,6 +30,7 @@ from litellm.types.passthrough_endpoints.pass_through_endpoints import (
EndpointType,
PassthroughStandardLoggingPayload,
)
from litellm.types.llms.openai import ResponsesAPIResponse
from litellm.types.utils import ImageResponse, LlmProviders, PassthroughCallTypes
from litellm.utils import ModelResponse, TextCompletionResponse
@@ -236,6 +238,42 @@ class OpenAIPassthroughLoggingHandler(BasePassthroughLoggingHandler):
)
return 0.0
@staticmethod
def _build_responses_api_response_and_cost(
model: str,
httpx_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: str,
) -> Tuple[ResponsesAPIResponse, float]:
"""Transform a Responses API raw response into a ResponsesAPIResponse
and compute its cost.
The Responses API has a different on-the-wire shape from chat
completions (`output: [...]` instead of `choices: [...]`), so the
chat-completions `transform_response` raises KeyError 'choices' on
a Responses payload. Use the dedicated Responses-API transformer
(`OpenAIResponsesAPIConfig.transform_response_api_response`) here.
Returns (litellm_model_response, response_cost) — symmetric with the
chat-completions branch which produces the same two values inline,
and analogous to the image branches' `_calculate_image_*_cost` helpers
(which return cost only because the image-response object is trivial
to build inline; the Responses payload needs a real transformer).
"""
responses_config = OpenAIResponsesAPIConfig()
litellm_model_response = responses_config.transform_response_api_response(
model=model,
raw_response=httpx_response,
logging_obj=logging_obj,
)
response_cost = litellm.completion_cost(
completion_response=litellm_model_response,
model=model,
custom_llm_provider=custom_llm_provider,
call_type="responses",
)
return litellm_model_response, response_cost
@staticmethod
def openai_passthrough_handler( # noqa: PLR0915
httpx_response: httpx.Response,
@@ -301,7 +339,12 @@ class OpenAIPassthroughLoggingHandler(BasePassthroughLoggingHandler):
try:
response_cost = 0.0
litellm_model_response: Optional[
Union[ModelResponse, TextCompletionResponse, ImageResponse]
Union[
ModelResponse,
TextCompletionResponse,
ImageResponse,
ResponsesAPIResponse,
]
] = None
handler_instance = OpenAIPassthroughLoggingHandler()
@@ -384,29 +427,18 @@ class OpenAIPassthroughLoggingHandler(BasePassthroughLoggingHandler):
litellm_model_response._hidden_params = {}
litellm_model_response._hidden_params["response_cost"] = response_cost
elif is_responses:
# Handle responses API cost calculation
provider_config = handler_instance.get_provider_config(model=model)
existing_litellm_params = kwargs.get("litellm_params", {}) or {}
litellm_model_response = provider_config.transform_response(
raw_response=httpx_response,
model_response=litellm.ModelResponse(),
# Responses-API cost tracking — see
# `_build_responses_api_response_and_cost` for why this needs
# a dedicated transformer (the chat-completions transform
# crashes on the Responses payload shape).
(
litellm_model_response,
response_cost,
) = OpenAIPassthroughLoggingHandler._build_responses_api_response_and_cost(
model=model,
messages=request_body.get("messages", []),
httpx_response=httpx_response,
logging_obj=logging_obj,
optional_params=request_body.get("optional_params", {}),
api_key="",
request_data=request_body,
encoding=litellm.encoding,
json_mode=False,
litellm_params=existing_litellm_params,
)
# Calculate cost using LiteLLM's cost calculator with responses call type
response_cost = litellm.completion_cost(
completion_response=litellm_model_response,
model=model,
custom_llm_provider=custom_llm_provider,
call_type="responses",
)
# Update kwargs with cost information
litellm/proxy/pass_through_endpoints/success_handler.py
+11 -1
View File
@@ -458,7 +458,16 @@ class PassThroughEndpointLogging:
return False
def _is_supported_openai_endpoint(self, url_route: str) -> bool:
"""Check if the OpenAI endpoint is supported by the passthrough logging handler."""
"""Check if the OpenAI endpoint is supported by the passthrough logging handler.
The Responses API route is included because
`openai_passthrough_handler` has a dedicated `elif is_responses:`
branch that knows how to extract usage + cost from the
Responses-API on-the-wire shape. Without including it here, the
outer dispatch filters Responses calls out before reaching the
handler — the inner branch is then unreachable and Responses
calls land in `LiteLLM_SpendLogs` with zero tokens / zero spend.
"""
from .llm_provider_handlers.openai_passthrough_logging_handler import (
OpenAIPassthroughLoggingHandler,
)
@@ -469,6 +478,7 @@ class PassThroughEndpointLogging:
url_route
)
or OpenAIPassthroughLoggingHandler.is_openai_image_editing_route(url_route)
or OpenAIPassthroughLoggingHandler.is_openai_responses_route(url_route)
)
def _set_cost_per_request(
litellm/proxy/proxy_cli.py
+16
View File
@@ -555,6 +555,7 @@ class ProxyInitializationHelpers:
@click.command()
@click.argument("cli_args", nargs=-1)
@click.option(
"--host", default="0.0.0.0", help="Host for the server to listen on.", envvar="HOST"
)
@@ -808,6 +809,7 @@ class ProxyInitializationHelpers:
help="Enable uvicorn hot reload (dev only). Also reloads when the --config YAML file changes. Incompatible with --num_workers>1, --run_gunicorn, and --run_hypercorn.",
)
def run_server( # noqa: PLR0915
cli_args,
host,
port,
api_base,
@@ -854,6 +856,20 @@ def run_server( # noqa: PLR0915
use_v2_migration_resolver: bool,
reload: bool,
):
if cli_args:
if cli_args == ("xai-oauth", "login"):
from litellm.llms.xai.oauth import XAIOAuthAuthenticator
authenticator = XAIOAuthAuthenticator()
auth_data = authenticator.login()
click.echo(
f"xAI OAuth login successful. Credentials saved to {authenticator.auth_file}."
)
if auth_data.get("expires_at"):
click.echo(f"Access token expires at {auth_data['expires_at']}.")
return
raise click.UsageError(f"Unknown command: {' '.join(cli_args)}")
if setup:
from litellm.setup_wizard import run_setup_wizard
litellm/proxy/proxy_server.py
+2
View File
@@ -4089,6 +4089,8 @@ class ProxyConfig:
verbose_proxy_logger.debug(
f"litellm.post_call_rules: {litellm.post_call_rules}"
)
elif key == "max_budget":
litellm.max_budget = float(value)
elif key == "max_internal_user_budget":
litellm.max_internal_user_budget = float(value) # type: ignore
elif key == "default_max_internal_user_budget":
litellm/proxy/spend_tracking/spend_management_endpoints.py
+1 -1
View File
@@ -3405,7 +3405,7 @@ async def ui_view_session_spend_logs(
session_id, status, mcp_namespaced_tool_name, agent_id
FROM "LiteLLM_SpendLogs"
WHERE session_id = $1
ORDER BY "startTime" ASC
ORDER BY "startTime" DESC
LIMIT $2 OFFSET $3
"""
result = await prisma_client.db.query_raw(
litellm/router.py
+72 -15
View File
@@ -1658,6 +1658,67 @@ class Router:
f"Dictionary '{fallback_dict}' must have exactly one key, but has {len(fallback_dict)} keys."
)
def _add_encrypted_content_affinity_check(
self, enable_global_affinity: bool
) -> None:
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
EncryptedContentAffinityCheck,
)
def _move_before_deployment_affinity(
callback_list: List[Any],
callback_to_move: EncryptedContentAffinityCheck,
) -> None:
if callback_to_move not in callback_list:
return
callback_list.remove(callback_to_move)
insert_index = next(
(
idx
for idx, callback in enumerate(callback_list)
if isinstance(callback, DeploymentAffinityCheck)
),
len(callback_list),
)
callback_list.insert(insert_index, callback_to_move)
if (
enable_global_affinity
or EncryptedContentAffinityCheck.has_model_group_affinity_enabled(
self.model_group_affinity_config
)
):
if self.optional_callbacks is None:
self.optional_callbacks = []
existing_ec_callback: Optional[EncryptedContentAffinityCheck] = None
for cb in self.optional_callbacks:
if isinstance(cb, EncryptedContentAffinityCheck):
existing_ec_callback = cb
break
if existing_ec_callback is not None:
existing_ec_callback.router = self
existing_ec_callback.enable_global_affinity = (
existing_ec_callback.enable_global_affinity
or enable_global_affinity
)
existing_ec_callback.model_group_affinity_config = (
self.model_group_affinity_config or {}
)
ec_callback = existing_ec_callback
else:
ec_callback = EncryptedContentAffinityCheck(
router=self,
enable_global_affinity=enable_global_affinity,
model_group_affinity_config=self.model_group_affinity_config,
)
self.optional_callbacks.append(ec_callback)
litellm.logging_callback_manager.add_litellm_callback(ec_callback)
_move_before_deployment_affinity(self.optional_callbacks, ec_callback)
_move_before_deployment_affinity(litellm.callbacks, ec_callback)
def add_optional_pre_call_checks(
self, optional_pre_call_checks: Optional[OptionalPreCallChecks]
):
@@ -1721,22 +1782,11 @@ class Router:
# ---------------------------------------------------------------------
# Encrypted content affinity
# ---------------------------------------------------------------------
if "encrypted_content_affinity" in optional_pre_call_checks:
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
EncryptedContentAffinityCheck,
self._add_encrypted_content_affinity_check(
enable_global_affinity=(
"encrypted_content_affinity" in optional_pre_call_checks
)
if self.optional_callbacks is None:
self.optional_callbacks = []
already_registered = any(
isinstance(cb, EncryptedContentAffinityCheck)
for cb in self.optional_callbacks
)
if not already_registered:
ec_callback = EncryptedContentAffinityCheck(router=self)
self.optional_callbacks.append(ec_callback)
litellm.logging_callback_manager.add_litellm_callback(ec_callback)
)
# ---------------------------------------------------------------------
# Remaining optional pre-call checks
@@ -8471,6 +8521,13 @@ class Router:
credential_values.get("api_key")
or deployment.litellm_params.api_key
)
if api_key is None:
verbose_router_logger.debug(
"Skipping pass-through credential setup for deployment model=%s, custom_llm_provider=%s; no api_key set. Providers like bedrock resolve credentials at request time.",
model,
custom_llm_provider,
)
return
passthrough_endpoint_router.set_pass_through_credentials(
custom_llm_provider=custom_llm_provider,
api_base=api_base,
litellm/router_utils/pre_call_checks/deployment_affinity_check.py
+6 -1
View File
@@ -39,7 +39,12 @@ class DeploymentAffinityCheck(CustomLogger):
CACHE_KEY_PREFIX = "deployment_affinity:v1"
VALID_FLAGS = frozenset(
{"deployment_affinity", "responses_api_deployment_check", "session_affinity"}
{
"deployment_affinity",
"responses_api_deployment_check",
"session_affinity",
"encrypted_content_affinity",
}
)
def __init__(
litellm/router_utils/pre_call_checks/encrypted_content_affinity_check.py
+33 -3
View File
@@ -37,7 +37,7 @@ Safe to enable globally:
"""
import time
from typing import TYPE_CHECKING, Any, List, Optional, cast
from typing import TYPE_CHECKING, Any, Dict, List, Optional, cast
import httpx
@@ -64,17 +64,45 @@ class EncryptedContentAffinityCheck(CustomLogger):
The ``model_id`` is decoded directly from the litellm-encoded item IDs –
no caching or TTL management needed.
Wired via ``Router(optional_pre_call_checks=["encrypted_content_affinity"])``.
Wired via ``Router(optional_pre_call_checks=["encrypted_content_affinity"])`` or
per-model group ``model_group_affinity_config``.
"""
def __init__(self, router: Optional["Router"] = None) -> None:
def __init__(
self,
router: Optional["Router"] = None,
enable_global_affinity: bool = True,
model_group_affinity_config: Optional[Dict[str, List[str]]] = None,
) -> None:
super().__init__()
self.router = router
self.enable_global_affinity = enable_global_affinity
self.model_group_affinity_config: Dict[str, List[str]] = (
model_group_affinity_config or {}
)
# ------------------------------------------------------------------
# Helpers
# ------------------------------------------------------------------
@staticmethod
def has_model_group_affinity_enabled(
model_group_affinity_config: Optional[Dict[str, List[str]]],
) -> bool:
if not model_group_affinity_config:
return False
return any(
"encrypted_content_affinity" in checks
for checks in model_group_affinity_config.values()
)
def _is_enabled_for_model_group(self, model_group: str) -> bool:
group_checks = self.model_group_affinity_config.get(model_group)
return self.enable_global_affinity or (
group_checks is not None and "encrypted_content_affinity" in group_checks
)
@staticmethod
def _extract_model_id_from_input(request_input: Any) -> Optional[str]:
"""
@@ -213,6 +241,8 @@ class EncryptedContentAffinityCheck(CustomLogger):
"""
request_kwargs = request_kwargs or {}
typed_healthy_deployments = cast(List[dict], healthy_deployments)
if not self._is_enabled_for_model_group(model):
return typed_healthy_deployments
# Signal to the response post-processor that encrypted item IDs should be
# encoded in the output of this request. Only set the flag when
litellm/types/router.py
+4
View File
@@ -220,6 +220,10 @@ class GenericLiteLLMParams(CredentialLiteLLMParams, CustomPricingLiteLLMParams):
use_in_pass_through: Optional[bool] = False
use_litellm_proxy: Optional[bool] = False
use_chat_completions_api: Optional[bool] = None
use_xai_oauth: Optional[bool] = Field(
default=False,
description="Use stored xAI OAuth credentials when no xAI API key is configured.",
)
model_config = ConfigDict(extra="allow", arbitrary_types_allowed=True)
merge_reasoning_content_in_choices: Optional[bool] = False
model_info: Optional[Dict] = None
litellm/types/utils.py
+8
View File
@@ -197,6 +197,7 @@ class ModelInfoBase(ProviderSpecificModelInfo, total=False):
] # OpenAI priority service tier pricing
cache_read_input_token_cost_above_200k_tokens: Optional[float]
cache_read_input_token_cost_above_272k_tokens: Optional[float]
cache_read_input_token_cost_above_512k_tokens: Optional[float]
input_cost_per_character: Optional[float] # only for vertex ai models
input_cost_per_audio_token: Optional[float]
input_cost_per_token_above_128k_tokens: Optional[float] # only for vertex ai models
@@ -206,6 +207,9 @@ class ModelInfoBase(ProviderSpecificModelInfo, total=False):
input_cost_per_token_above_272k_tokens: Optional[
float
] # GPT-5.4/5.4-pro: prompts >272K priced at 2x input
input_cost_per_token_above_512k_tokens: Optional[
float
] # MiniMax-M3: prompts >512K priced at 2x input
input_cost_per_character_above_128k_tokens: Optional[
float
] # only for vertex ai models
@@ -239,6 +243,9 @@ class ModelInfoBase(ProviderSpecificModelInfo, total=False):
output_cost_per_token_above_272k_tokens: Optional[
float
] # GPT-5.4/5.4-pro: prompts >272K priced at 1.5x output
output_cost_per_token_above_512k_tokens: Optional[
float
] # MiniMax-M3: prompts >512K priced at 2x output
output_cost_per_character_above_128k_tokens: Optional[
float
] # only for vertex ai models
@@ -3217,6 +3224,7 @@ all_litellm_params = (
"search_tool_name",
"order",
"enable_json_schema_validation",
"use_xai_oauth",
]
+ list(StandardCallbackDynamicParams.__annotations__.keys())
+ list(CustomPricingLiteLLMParams.model_fields.keys())
litellm/utils.py
+47 -14
View File
@@ -5775,6 +5775,7 @@ def _get_model_info_helper( # noqa: PLR0915
]
split_model = potential_model_names["split_model"]
custom_llm_provider = potential_model_names["custom_llm_provider"]
model_cost_custom_llm_provider = custom_llm_provider
#########################
provider_config: Optional[BaseLLMModelInfo] = None
if custom_llm_provider and custom_llm_provider in LlmProvidersSet:
@@ -5840,7 +5841,8 @@ def _get_model_info_helper( # noqa: PLR0915
key = _matched_key
_model_info = _get_model_info_from_model_cost(key=cast(str, key))
if not _check_provider_match(
model_info=_model_info, custom_llm_provider=custom_llm_provider
model_info=_model_info,
custom_llm_provider=model_cost_custom_llm_provider,
):
_model_info = None
if _model_info is None:
@@ -5849,7 +5851,8 @@ def _get_model_info_helper( # noqa: PLR0915
key = _matched_key
_model_info = _get_model_info_from_model_cost(key=cast(str, key))
if not _check_provider_match(
model_info=_model_info, custom_llm_provider=custom_llm_provider
model_info=_model_info,
custom_llm_provider=model_cost_custom_llm_provider,
):
_model_info = None
if _model_info is None:
@@ -5858,7 +5861,8 @@ def _get_model_info_helper( # noqa: PLR0915
key = _matched_key
_model_info = _get_model_info_from_model_cost(key=cast(str, key))
if not _check_provider_match(
model_info=_model_info, custom_llm_provider=custom_llm_provider
model_info=_model_info,
custom_llm_provider=model_cost_custom_llm_provider,
):
_model_info = None
if _model_info is None:
@@ -5867,7 +5871,8 @@ def _get_model_info_helper( # noqa: PLR0915
key = _matched_key
_model_info = _get_model_info_from_model_cost(key=cast(str, key))
if not _check_provider_match(
model_info=_model_info, custom_llm_provider=custom_llm_provider
model_info=_model_info,
custom_llm_provider=model_cost_custom_llm_provider,
):
_model_info = None
if _model_info is None:
@@ -5876,7 +5881,8 @@ def _get_model_info_helper( # noqa: PLR0915
key = _matched_key
_model_info = _get_model_info_from_model_cost(key=cast(str, key))
if not _check_provider_match(
model_info=_model_info, custom_llm_provider=custom_llm_provider
model_info=_model_info,
custom_llm_provider=model_cost_custom_llm_provider,
):
_model_info = None
@@ -5884,7 +5890,6 @@ def _get_model_info_helper( # noqa: PLR0915
raise ValueError(
"This model isn't mapped yet. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json"
)
_input_cost_per_token: Optional[float] = _model_info.get(
"input_cost_per_token"
)
@@ -5936,6 +5941,9 @@ def _get_model_info_helper( # noqa: PLR0915
cache_read_input_token_cost_above_272k_tokens=_model_info.get(
"cache_read_input_token_cost_above_272k_tokens", None
),
cache_read_input_token_cost_above_512k_tokens=_model_info.get(
"cache_read_input_token_cost_above_512k_tokens", None
),
cache_read_input_token_cost_flex=_model_info.get(
"cache_read_input_token_cost_flex", None
),
@@ -5957,6 +5965,9 @@ def _get_model_info_helper( # noqa: PLR0915
input_cost_per_token_above_272k_tokens=_model_info.get(
"input_cost_per_token_above_272k_tokens", None
),
input_cost_per_token_above_512k_tokens=_model_info.get(
"input_cost_per_token_above_512k_tokens", None
),
input_cost_per_query=_model_info.get("input_cost_per_query", None),
input_cost_per_second=_model_info.get("input_cost_per_second", None),
input_cost_per_audio_token=_model_info.get(
@@ -6012,6 +6023,9 @@ def _get_model_info_helper( # noqa: PLR0915
output_cost_per_token_above_272k_tokens=_model_info.get(
"output_cost_per_token_above_272k_tokens", None
),
output_cost_per_token_above_512k_tokens=_model_info.get(
"output_cost_per_token_above_512k_tokens", None
),
output_cost_per_second=_model_info.get("output_cost_per_second", None),
output_cost_per_second_1080p=_model_info.get(
"output_cost_per_second_1080p", None
@@ -8922,14 +8936,33 @@ class ProviderConfigManager:
elif litellm.LlmProviders.HOSTED_VLLM == provider:
return litellm.HostedVLLMResponsesAPIConfig()
elif litellm.LlmProviders.BEDROCK_MANTLE == provider:
# Only OpenAI gpt frontier models (gpt-5.x, and future gpt-6 etc.) are
# served on the /openai/v1/responses path. gpt-oss and every non-OpenAI
# model on Mantle (nvidia, mistral, google, zai, ...) are chat-completions
# only and 400 on that path, so they fall through to None to keep the
# chat-completions emulation (see litellm/responses/main.py "config is None").
model_lower = model.lower() if model else ""
if "openai.gpt-" in model_lower and "gpt-oss" not in model_lower:
return litellm.BedrockMantleResponsesAPIConfig()
# Mantle serves Responses on two upstream paths. A model takes the
# /openai/v1/responses path when its price-map entry declares
# use_openai_responses_path (data-driven, so a non-gpt-named frontier
# model can be onboarded by JSON alone), or, as a fallback needing no
# price-map entry, when its name matches the openai.gpt- frontier
# convention (minus gpt-oss) -- this keeps a future gpt-6 routing
# correctly before its entry loads. Any other model declared
# mode=responses takes the standard /v1/responses path. Everything
# else returns None and keeps the chat-completions emulation (see
# responses/main.py "config is None").
if not model:
return None
model_lower = model.lower()
entry = litellm.model_cost.get(f"bedrock_mantle/{model}", {})
on_openai_path = entry.get("use_openai_responses_path") is True
name_is_frontier = (
"openai.gpt-" in model_lower and "gpt-oss" not in model_lower
)
if on_openai_path or name_is_frontier:
return litellm.BedrockMantleResponsesAPIConfig(use_openai_path=True)
try:
if get_model_info(model, "bedrock_mantle").get("mode") == "responses":
return litellm.BedrockMantleResponsesAPIConfig(
use_openai_path=False
)
except Exception:
pass
return None
return None
model_prices_and_context_window.json
+168 -4
View File
@@ -24392,9 +24392,12 @@
"max_output_tokens": 8192
},
"minimax/MiniMax-M3": {
"input_cost_per_token": 6e-07,
"output_cost_per_token": 2.4e-06,
"cache_read_input_token_cost": 1.2e-07,
"input_cost_per_token": 3e-07,
"input_cost_per_token_above_512k_tokens": 6e-07,
"output_cost_per_token": 1.2e-06,
"output_cost_per_token_above_512k_tokens": 2.4e-06,
"cache_read_input_token_cost": 6e-08,
"cache_read_input_token_cost_above_512k_tokens": 1.2e-07,
"litellm_provider": "minimax",
"mode": "chat",
"supports_function_calling": true,
@@ -24403,7 +24406,7 @@
"supports_reasoning": true,
"supports_system_messages": true,
"supports_vision": true,
"max_input_tokens": 512000,
"max_input_tokens": 1000000,
"max_output_tokens": 128000
},
"mistral.devstral-2-123b": {
@@ -41686,6 +41689,7 @@
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
"use_openai_responses_path": true,
"supported_endpoints": ["/v1/responses"],
"supported_modalities": ["text", "image"],
"supported_output_modalities": ["text"],
@@ -41705,6 +41709,7 @@
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
"use_openai_responses_path": true,
"supported_endpoints": ["/v1/responses"],
"supported_modalities": ["text", "image"],
"supported_output_modalities": ["text"],
@@ -42178,5 +42183,164 @@
"source": "https://soniox.com/pricing",
"supported_endpoints": ["/v1/audio/transcriptions"],
"supports_audio_input": true
},
"tensormesh/Qwen/Qwen3.5-397B-A17B-FP8": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 6e-07,
"output_cost_per_token": 3.6e-06,
"cache_read_input_token_cost": 0,
"max_input_tokens": 262144,
"max_output_tokens": 262144,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 4.5e-07,
"output_cost_per_token": 1.8e-06,
"cache_read_input_token_cost": 0,
"max_input_tokens": 262144,
"max_output_tokens": 262144,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/Qwen/Qwen3.6-27B-FP8": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 3.2e-07,
"output_cost_per_token": 3.2e-06,
"cache_read_input_token_cost": 0,
"max_input_tokens": 262144,
"max_output_tokens": 262144,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/lukealonso/GLM-5.1-NVFP4-MTP": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 1.4e-06,
"output_cost_per_token": 4.4e-06,
"cache_read_input_token_cost": 0,
"max_input_tokens": 202752,
"max_output_tokens": 202752,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/deepseek-ai/DeepSeek-V4-Flash": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 1.4e-07,
"output_cost_per_token": 2.8e-07,
"cache_read_input_token_cost": 0,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/moonshotai/Kimi-K2.6": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 9.6e-07,
"output_cost_per_token": 4e-06,
"cache_read_input_token_cost": 0,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/MiniMaxAI/MiniMax-M2.5": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 3e-07,
"output_cost_per_token": 1.2e-06,
"cache_read_input_token_cost": 0,
"max_input_tokens": 196608,
"max_output_tokens": 196608,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/google/gemma-4-31B-it": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 1.4e-07,
"output_cost_per_token": 5.6e-07,
"cache_read_input_token_cost": 0,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/openai/gpt-oss-120b": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 1.5e-07,
"output_cost_per_token": 6e-07,
"cache_read_input_token_cost": 0,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
},
"tensormesh/openai/gpt-oss-20b": {
"litellm_provider": "tensormesh",
"mode": "chat",
"input_cost_per_token": 7e-08,
"output_cost_per_token": 2.8e-07,
"cache_read_input_token_cost": 0,
"max_input_tokens": 131072,
"max_output_tokens": 131072,
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"supports_system_messages": true,
"supports_reasoning": true,
"source": "https://serverless.tensormesh.ai/v1/models/openrouter"
}
}
tests/llm_translation/test_llm_response_utils/test_convert_dict_to_chat_completion.py
+36
View File
@@ -2425,6 +2425,42 @@ class TestConvertToModelResponseObjectCompletion:
assert result.choices[0].message.content == "The answer is 4."
assert result.choices[0].message.reasoning_content == "2+2=4"
def test_reasoning_content_not_mirrored_into_provider_specific_fields(self):
"""Mirroring reasoning_content into provider_specific_fields made
cache-replayed messages diverge from live Anthropic messages, which
only set it top-level, breaking cache key stability (issue #27337)."""
response_object = {
"id": "chatcmpl-5",
"model": "claude-sonnet-4-5",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "The answer is 4.",
"role": "assistant",
"reasoning_content": "2+2=4",
"thinking_blocks": [
{
"type": "thinking",
"thinking": "2+2=4",
"signature": "sig",
}
],
},
}
],
"usage": {"prompt_tokens": 5, "completion_tokens": 10, "total_tokens": 15},
}
result = convert_to_model_response_object(
response_object=response_object,
model_response_object=ModelResponse(),
)
message = result.choices[0].message
assert message.reasoning_content == "2+2=4"
assert "reasoning_content" not in (message.provider_specific_fields or {})
def test_response_none_raises(self):
with pytest.raises(Exception):
convert_to_model_response_object(
tests/router_unit_tests/test_router_endpoints.py
+57
View File
@@ -1236,3 +1236,60 @@ async def test_init_containers_api_endpoints_managed_id_without_model_id_applies
assert call_kw["container_id"] == "cfile_upstream_abc"
assert call_kw["file_id"] == "cfile_xyz"
assert call_kw["custom_llm_provider"] == "azure"
def test_router_model_group_encrypted_content_affinity_callback_registration():
from litellm.router_utils.pre_call_checks.deployment_affinity_check import (
DeploymentAffinityCheck,
)
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
EncryptedContentAffinityCheck,
)
model_group = "openai.gpt-5.1-codex"
model_group_affinity_config = {
model_group: ["encrypted_content_affinity"],
}
router = Router(
model_list=[
{
"model_name": model_group,
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key",
},
}
],
model_group_affinity_config=model_group_affinity_config,
num_retries=0,
)
try:
callbacks = router.optional_callbacks or []
encrypted_content_callbacks = [
cb for cb in callbacks if isinstance(cb, EncryptedContentAffinityCheck)
]
deployment_callback = next(
cb for cb in callbacks if isinstance(cb, DeploymentAffinityCheck)
)
assert len(encrypted_content_callbacks) == 1
assert encrypted_content_callbacks[0].enable_global_affinity is False
assert (
encrypted_content_callbacks[0].model_group_affinity_config
== model_group_affinity_config
)
assert callbacks.index(encrypted_content_callbacks[0]) < callbacks.index(
deployment_callback
)
router._add_encrypted_content_affinity_check(enable_global_affinity=True)
callbacks = router.optional_callbacks or []
encrypted_content_callbacks = [
cb for cb in callbacks if isinstance(cb, EncryptedContentAffinityCheck)
]
assert len(encrypted_content_callbacks) == 1
assert encrypted_content_callbacks[0].enable_global_affinity is True
assert encrypted_content_callbacks[0].router is router
finally:
router.discard()
tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py
+35
View File
@@ -328,6 +328,41 @@ def test_generic_cost_per_token_gpt54_above_272k_tokens():
assert round(completion_cost, 10) == round(expected_completion, 10)
def test_generic_cost_per_token_minimax_m3_above_512k_tokens():
"""MiniMax-M3: prompts >512K input tokens priced at 2x input, output, and cache read."""
model = "minimax/MiniMax-M3"
custom_llm_provider = "minimax"
os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
litellm.model_cost = litellm.get_model_cost_map(url="")
model_cost_map = litellm.model_cost[model]
prompt_tokens = 600000
cached_tokens = 100000
completion_tokens = 1000
usage = Usage(
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
total_tokens=prompt_tokens + completion_tokens,
prompt_tokens_details=PromptTokensDetailsWrapper(cached_tokens=cached_tokens),
)
prompt_cost, completion_cost = generic_cost_per_token(
model=model,
usage=usage,
custom_llm_provider=custom_llm_provider,
)
expected_prompt = (
model_cost_map["input_cost_per_token_above_512k_tokens"]
* (prompt_tokens - cached_tokens)
+ model_cost_map["cache_read_input_token_cost_above_512k_tokens"]
* cached_tokens
)
expected_completion = (
model_cost_map["output_cost_per_token_above_512k_tokens"] * completion_tokens
)
assert round(prompt_cost, 10) == round(expected_prompt, 10)
assert round(completion_cost, 10) == round(expected_completion, 10)
def test_generic_cost_per_token_gpt55():
"""gpt-5.5: base pricing — $5/1M input, $30/1M output, $0.50/1M cached input."""
model = "gpt-5.5"
tests/test_litellm/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_factory.py
+88
View File
@@ -11,6 +11,7 @@ from litellm.litellm_core_utils.prompt_templates.factory import (
BedrockImageProcessor,
_bedrock_converse_messages_pt,
_bedrock_tools_pt,
_rename_duplicate_bedrock_document_names,
_convert_to_bedrock_tool_call_invoke,
_convert_to_bedrock_tool_call_result,
anthropic_messages_pt,
@@ -2809,6 +2810,93 @@ def test_bedrock_converse_messages_pt_document_deterministic_name():
assert name1 == name2
def test_bedrock_converse_messages_pt_renames_duplicate_document_names():
"""
The same document in multiple turns must not produce duplicate names;
Bedrock rejects requests with "Messages can not contain duplicate
document names". The first occurrence keeps its hash-based name and
later occurrences get a deterministic positional suffix.
"""
document_block = {
"type": "document",
"source": {
"type": "base64",
"media_type": "application/pdf",
"data": "dGVzdA==",
},
}
messages = [
{
"role": "user",
"content": [document_block, {"type": "text", "text": "summarize this"}],
},
{"role": "assistant", "content": "It says test."},
{
"role": "user",
"content": [document_block, {"type": "text", "text": "summarize again"}],
},
]
result1 = _bedrock_converse_messages_pt(
messages, "anthropic.claude-sonnet-4-6", "bedrock"
)
result2 = _bedrock_converse_messages_pt(
messages, "anthropic.claude-sonnet-4-6", "bedrock"
)
names1 = [
block["document"]["name"]
for message in result1
for block in message["content"]
if "document" in block
]
names2 = [
block["document"]["name"]
for message in result2
for block in message["content"]
if "document" in block
]
assert len(names1) == 2
assert len(set(names1)) == 2
assert names1[1] == f"{names1[0]}_2"
assert names1 == names2
single_turn = _bedrock_converse_messages_pt(
[messages[0]], "anthropic.claude-sonnet-4-6", "bedrock"
)
assert names1[0] == single_turn[0]["content"][0]["document"]["name"]
def test_rename_duplicate_bedrock_document_names_skips_organic_suffixes():
"""
A renamed duplicate must not collide with a document whose organic name
already carries the would-be suffix (e.g. an existing ``report_2``),
regardless of whether that document appears before or after the rename.
"""
def _contents(names):
return [
{
"role": "user",
"content": [{"document": {"name": name}} for name in names],
}
]
def _names(contents):
return [block["document"]["name"] for block in contents[0]["content"]]
organic_first = _rename_duplicate_bedrock_document_names(
_contents(["report", "report_2", "report"])
)
assert _names(organic_first) == ["report", "report_2", "report_3"]
organic_last = _rename_duplicate_bedrock_document_names(
_contents(["report", "report", "report_2"])
)
assert _names(organic_last) == ["report", "report_3", "report_2"]
def test_bedrock_converse_messages_pt_document_rejects_url_source():
"""Test that a URL-type document source raises a clear error instead of KeyError."""
messages = [
tests/test_litellm/litellm_core_utils/test_xai_oauth_routing.py
+81
View File
@@ -0,0 +1,81 @@
import os
import sys
sys.path.insert(0, os.path.abspath("../../.."))
import litellm
from litellm import LlmProviders
from litellm.litellm_core_utils.get_litellm_params import get_litellm_params
from litellm.litellm_core_utils.get_llm_provider_logic import (
_get_openai_compatible_provider_info,
)
from litellm.llms.xai.chat.transformation import XAIChatConfig
from litellm.llms.xai.responses.transformation import XAIResponsesAPIConfig
from litellm.types.router import GenericLiteLLMParams
from litellm.utils import (
ProviderConfigManager,
get_optional_params,
validate_environment,
)
def test_xai_provider_config_routing():
chat_config = ProviderConfigManager.get_provider_chat_config(
model="grok-3-mini",
provider=LlmProviders.XAI,
)
responses_config = ProviderConfigManager.get_provider_responses_api_config(
model="grok-3-mini",
provider=LlmProviders.XAI,
)
assert isinstance(chat_config, XAIChatConfig)
assert isinstance(responses_config, XAIResponsesAPIConfig)
def test_xai_openai_compatible_provider_info():
model, custom_llm_provider, dynamic_api_key, api_base = (
_get_openai_compatible_provider_info(
model="xai/grok-3-mini",
api_base="https://api.x.ai/v1",
api_key="api-key",
dynamic_api_key=None,
)
)
assert model == "grok-3-mini"
assert custom_llm_provider == "xai"
assert api_base == "https://api.x.ai/v1"
assert dynamic_api_key == "api-key"
def test_xai_get_model_info_uses_xai_pricing_metadata():
model_info = litellm.get_model_info("xai/grok-3-mini")
assert model_info["litellm_provider"] == "xai"
assert model_info["key"] == "xai/grok-3-mini"
assert model_info["mode"] == "chat"
def test_xai_validate_environment_reads_api_key(monkeypatch):
monkeypatch.setenv("XAI_API_KEY", "api-key")
result = validate_environment(model="xai/grok-3-mini")
assert result == {"keys_in_environment": True, "missing_keys": []}
def test_xai_oauth_flag_is_generic_litellm_param():
litellm_params = GenericLiteLLMParams(use_xai_oauth=True)
runtime_params = get_litellm_params(use_xai_oauth=True)
result = get_optional_params(
model="grok-3-mini",
custom_llm_provider="xai",
temperature=0.2,
drop_params=True,
)
assert result["temperature"] == 0.2
assert litellm_params.use_xai_oauth is True
assert runtime_params["use_xai_oauth"] is True
assert "use_xai_oauth" not in result
tests/test_litellm/llms/anthropic/experimental_pass_through/responses_adapters/test_responses_adapters_streaming_iterator.py
+79
View File
@@ -0,0 +1,79 @@
"""
Tests for AnthropicResponsesStreamWrapper
(litellm/llms/anthropic/experimental_pass_through/responses_adapters/streaming_iterator.py)
"""
import os
import sys
sys.path.insert(
0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../../../../../.."))
)
from litellm.llms.anthropic.experimental_pass_through.responses_adapters.streaming_iterator import (
AnthropicResponsesStreamWrapper,
)
def _process_all(events: list) -> list:
wrapper = AnthropicResponsesStreamWrapper(responses_stream=None, model="m")
for event in events:
wrapper._process_event(event)
return list(wrapper._chunk_queue)
class TestProcessEventTextDeltaWithoutOutputItemAdded:
"""Streams that skip response.output_item.added (e.g. LMStudio) must still
open a text block before any delta and never emit index -1."""
def test_process_event_synthesizes_content_block_start_before_delta(self):
chunks = _process_all(
[
{"type": "response.output_text.delta", "item_id": "i1", "delta": "Hel"},
{"type": "response.output_text.delta", "item_id": "i1", "delta": "lo"},
]
)
assert [c["type"] for c in chunks] == [
"content_block_start",
"content_block_delta",
"content_block_delta",
]
assert chunks[0]["content_block"] == {"type": "text", "text": ""}
assert [c["index"] for c in chunks] == [0, 0, 0]
assert chunks[1]["delta"] == {"type": "text_delta", "text": "Hel"}
def test_process_event_delta_without_item_id_never_yields_negative_index(self):
chunks = _process_all([{"type": "response.output_text.delta", "delta": "Hi"}])
assert [(c["type"], c["index"]) for c in chunks] == [
("content_block_start", 0),
("content_block_delta", 0),
]
def test_process_event_unregistered_item_id_opens_new_text_block(self):
chunks = _process_all(
[
{
"type": "response.output_item.added",
"item": {"type": "reasoning", "id": "rs_1"},
},
{"type": "response.output_text.delta", "item_id": "m1", "delta": "Hi"},
]
)
assert chunks[1]["type"] == "content_block_start"
assert chunks[1]["content_block"] == {"type": "text", "text": ""}
assert [c["index"] for c in chunks[1:]] == [1, 1]
def test_process_event_registered_item_id_does_not_synthesize_start(self):
chunks = _process_all(
[
{
"type": "response.output_item.added",
"item": {"type": "message", "id": "m1"},
},
{"type": "response.output_text.delta", "item_id": "m1", "delta": "Hi"},
]
)
assert [(c["type"], c["index"]) for c in chunks] == [
("content_block_start", 0),
("content_block_delta", 0),
]
tests/test_litellm/llms/bedrock/count_tokens/test_bedrock_count_tokens_transformation.py
+71 -1
View File
@@ -1,10 +1,15 @@
import base64
import json
import os
import sys
sys.path.insert(
0, os.path.abspath("../../../../..")
) # Adds the parent directory to the system path
from litellm.llms.bedrock.count_tokens.transformation import BedrockCountTokensConfig
from litellm.llms.bedrock.count_tokens.transformation import (
DEFAULT_ANTHROPIC_INVOKE_MODEL_MAX_TOKENS,
BedrockCountTokensConfig,
)
def test_detect_input_type():
@@ -20,6 +25,71 @@ def test_detect_input_type():
assert config._detect_input_type(request_with_text) == "invokeModel"
def test_detect_input_type_anthropic_blocks_route_to_invoke_model():
"""Anthropic-shape content blocks must not go through the Converse path,
which Bedrock rejects with a 400 (and the caller then silently falls back
to the local tokenizer)."""
config = BedrockCountTokensConfig()
request = {
"messages": [
{
"role": "assistant",
"content": [
{"type": "text", "text": "Reading the file."},
{
"type": "tool_use",
"id": "toolu_01",
"name": "read_file",
"input": {"path": "main.py"},
},
],
},
],
}
assert config._detect_input_type(request) == "invokeModel"
def test_detect_input_type_converse_blocks_route_to_converse():
"""Converse-shape blocks (no "type" key) keep using the converse input."""
config = BedrockCountTokensConfig()
request = {"messages": [{"role": "user", "content": [{"text": "hi"}]}]}
assert config._detect_input_type(request) == "converse"
def test_transform_to_invoke_model_format_base64_encodes_body():
"""The CountTokens API expects invokeModel.body as a base64-encoded blob;
Anthropic Messages bodies additionally need anthropic_version/max_tokens
to pass Bedrock's InvokeModel schema validation."""
config = BedrockCountTokensConfig()
request = {
"model": "anthropic.claude-3-sonnet-20240229-v1:0",
"messages": [{"role": "user", "content": [{"type": "text", "text": "Hello"}]}],
}
result = config.transform_anthropic_to_bedrock_count_tokens(request)
body = json.loads(base64.b64decode(result["input"]["invokeModel"]["body"]))
assert body["messages"] == request["messages"]
assert "model" not in body
assert body["anthropic_version"] == "bedrock-2023-05-31"
assert body["max_tokens"] == DEFAULT_ANTHROPIC_INVOKE_MODEL_MAX_TOKENS
def test_transform_to_invoke_model_format_raw_body_unchanged():
"""Non-messages bodies (e.g. Titan inputText) must not get Anthropic fields."""
config = BedrockCountTokensConfig()
result = config.transform_anthropic_to_bedrock_count_tokens(
{"model": "amazon.titan-text-express-v1", "inputText": "hello"}
)
body = json.loads(base64.b64decode(result["input"]["invokeModel"]["body"]))
assert body == {"inputText": "hello"}
def test_transform_anthropic_to_bedrock_request():
"""Test basic request transformation"""
config = BedrockCountTokensConfig()
tests/test_litellm/llms/bedrock_mantle/test_bedrock_mantle_responses_transformation.py
+242 -6
View File
@@ -1,11 +1,13 @@
"""
Unit tests for Amazon Bedrock Mantle Responses API configuration.
Mantle's gpt-5.5 / gpt-5.4 are served ONLY on the non-standard
`/openai/v1/responses` path. These tests lock the URL construction and
Bearer auth that make that routing work.
Mantle serves Responses on two paths: gpt frontier models on
`/openai/v1/responses` and other Responses-capable models (e.g. gpt-oss) on the
standard `/v1/responses`. These tests lock the per-model path selection in the
gate, the URL construction for both paths, and the shared Bearer auth.
"""
import copy
import os
import sys
@@ -89,6 +91,42 @@ class TestBedrockMantleResponsesURL:
url = cfg.get_complete_url(api_base=None, litellm_params={})
assert url == "https://bedrock-mantle.us-east-1.api.aws/openai/v1/responses"
def test_standard_path_uses_region_from_env(self, monkeypatch):
monkeypatch.setenv("BEDROCK_MANTLE_REGION", "us-east-2")
monkeypatch.delenv("BEDROCK_MANTLE_API_BASE", raising=False)
cfg = BedrockMantleResponsesAPIConfig(use_openai_path=False)
url = cfg.get_complete_url(api_base=None, litellm_params={})
assert url == "https://bedrock-mantle.us-east-2.api.aws/v1/responses"
assert "/openai/v1/responses" not in url
def test_standard_path_normalizes_v1_base(self, monkeypatch):
monkeypatch.delenv("BEDROCK_MANTLE_API_BASE", raising=False)
cfg = BedrockMantleResponsesAPIConfig(use_openai_path=False)
url = cfg.get_complete_url(
api_base="https://bedrock-mantle.us-east-2.api.aws/v1",
litellm_params={},
)
assert url == "https://bedrock-mantle.us-east-2.api.aws/v1/responses"
assert url.count("/responses") == 1
assert "/v1/v1/responses" not in url
def test_standard_path_full_endpoint_base_not_doubled(self, monkeypatch):
monkeypatch.delenv("BEDROCK_MANTLE_API_BASE", raising=False)
cfg = BedrockMantleResponsesAPIConfig(use_openai_path=False)
url = cfg.get_complete_url(
api_base="https://bedrock-mantle.us-east-2.api.aws/v1/responses",
litellm_params={},
)
assert url == "https://bedrock-mantle.us-east-2.api.aws/v1/responses"
assert url.count("/responses") == 1
def test_default_construction_keeps_openai_path(self, monkeypatch):
monkeypatch.setenv("BEDROCK_MANTLE_REGION", "us-east-2")
monkeypatch.delenv("BEDROCK_MANTLE_API_BASE", raising=False)
cfg = BedrockMantleResponsesAPIConfig()
url = cfg.get_complete_url(api_base=None, litellm_params={})
assert url == "https://bedrock-mantle.us-east-2.api.aws/openai/v1/responses"
class TestBedrockMantleResponsesAuth:
def test_config_api_key_takes_priority(self, monkeypatch):
@@ -158,6 +196,36 @@ class TestBedrockMantleResponsesAuth:
is True
)
def test_standard_path_still_uses_bearer_auth(self, monkeypatch):
monkeypatch.setenv("BEDROCK_MANTLE_API_KEY", "env-key")
monkeypatch.delenv("AWS_BEARER_TOKEN_BEDROCK", raising=False)
cfg = BedrockMantleResponsesAPIConfig(use_openai_path=False)
headers = cfg.validate_environment(
headers={},
model="openai.gpt-oss-120b",
litellm_params=GenericLiteLLMParams(),
)
assert headers["Authorization"] == "Bearer env-key"
def test_standard_path_opts_out_of_native_features(self):
cfg = BedrockMantleResponsesAPIConfig(use_openai_path=False)
assert cfg.supports_native_file_search() is False
assert cfg.supports_native_websocket() is False
class TestBedrockMantleResponsesRequestBody:
def test_standard_path_outbound_body_carries_bare_model(self):
cfg = BedrockMantleResponsesAPIConfig(use_openai_path=False)
body = cfg.transform_responses_api_request(
model="openai.gpt-oss-120b",
input="hello",
response_api_optional_request_params={},
litellm_params=GenericLiteLLMParams(),
headers={},
)
assert body["model"] == "openai.gpt-oss-120b"
assert "input" in body
class TestBedrockMantleResponsesRegistry:
def test_registry_returns_config_for_gpt_5_5(self):
@@ -168,6 +236,7 @@ class TestBedrockMantleResponsesRegistry:
model="openai.gpt-5.5",
)
assert isinstance(cfg, BedrockMantleResponsesAPIConfig)
assert cfg.use_openai_path is True
def test_registry_returns_config_for_gpt_5_4_enum(self):
from litellm.utils import ProviderConfigManager
@@ -177,6 +246,7 @@ class TestBedrockMantleResponsesRegistry:
model="openai.gpt-5.4",
)
assert isinstance(cfg, BedrockMantleResponsesAPIConfig)
assert cfg.use_openai_path is True
def test_registry_returns_none_for_gpt_oss(self):
# Regression guard: gpt-oss must NOT get the native Responses config; it
@@ -199,9 +269,10 @@ class TestBedrockMantleResponsesRegistry:
assert cfg is None
def test_registry_returns_config_for_future_frontier_model(self):
# Forward-compatibility: an unseen OpenAI gpt frontier model (e.g. gpt-6) must
# get the native Responses config without a code change. The gate allow-lists
# the openai.gpt- family (minus gpt-oss), so gpt-6 matches automatically.
# Forward-compatibility: an unseen OpenAI gpt frontier model (e.g. gpt-6),
# not yet in the price map, must get the openai-path Responses config with
# no code or JSON change. The name-convention fallback (openai.gpt- minus
# gpt-oss) catches it before any price-map entry exists.
from litellm.utils import ProviderConfigManager
cfg = ProviderConfigManager.get_provider_responses_api_config(
@@ -209,6 +280,48 @@ class TestBedrockMantleResponsesRegistry:
model="openai.gpt-6",
)
assert isinstance(cfg, BedrockMantleResponsesAPIConfig)
assert cfg.use_openai_path is True
def test_price_map_flag_routes_non_gpt_name_to_openai_path(
self, restore_model_cost
):
# Data-driven onboarding: a frontier model whose name does NOT match the
# openai.gpt- convention can still be routed to /openai/v1/responses by
# declaring use_openai_responses_path in its price-map entry, with no code
# change. The string fallback alone could never catch this name.
from litellm.utils import ProviderConfigManager, register_model
register_model(
{
"bedrock_mantle/somelab.frontier-x": {
"litellm_provider": "bedrock_mantle",
"mode": "responses",
"use_openai_responses_path": True,
}
}
)
cfg = ProviderConfigManager.get_provider_responses_api_config(
provider="bedrock_mantle",
model="somelab.frontier-x",
)
assert isinstance(cfg, BedrockMantleResponsesAPIConfig)
assert cfg.use_openai_path is True
def test_gpt_5_5_price_map_declares_openai_responses_path(self, local_cost_map):
# The gpt-5.x entries must carry the data-driven flag so frontier routing
# does not rely on the name-string fallback alone.
assert (
litellm.model_cost["bedrock_mantle/openai.gpt-5.5"].get(
"use_openai_responses_path"
)
is True
)
assert (
litellm.model_cost["bedrock_mantle/openai.gpt-5.4"].get(
"use_openai_responses_path"
)
is True
)
@pytest.mark.parametrize(
"model",
@@ -243,6 +356,129 @@ class TestBedrockMantleResponsesRegistry:
)
assert cfg is None
def test_declared_responses_non_openai_routes_to_standard_path(
self, restore_model_cost
):
# New feature: a non-OpenAI model declared mode=responses (e.g. via a
# user's proxy model_info block) must route to the STANDARD /v1/responses
# path, not the frontier /openai/v1/responses path. Fails before the
# path-aware gate exists (old gate returned None for non-gpt models).
from litellm.utils import ProviderConfigManager, register_model
register_model(
{
"bedrock_mantle/somelab.future-model": {
"litellm_provider": "bedrock_mantle",
"mode": "responses",
}
}
)
cfg = ProviderConfigManager.get_provider_responses_api_config(
provider="bedrock_mantle",
model="somelab.future-model",
)
assert isinstance(cfg, BedrockMantleResponsesAPIConfig)
assert cfg.use_openai_path is False
def test_gpt_oss_opt_in_routes_to_standard_path(self, restore_model_cost):
# When a user opts gpt-oss into native Responses via model_info mode,
# it must take the STANDARD /v1/responses path (gpt-oss Responses is on
# /v1/responses, NOT the frontier /openai/v1/responses path).
from litellm.utils import ProviderConfigManager, register_model
register_model(
{
"bedrock_mantle/openai.gpt-oss-120b": {
"litellm_provider": "bedrock_mantle",
"mode": "responses",
}
}
)
cfg = ProviderConfigManager.get_provider_responses_api_config(
provider="bedrock_mantle",
model="openai.gpt-oss-120b",
)
assert isinstance(cfg, BedrockMantleResponsesAPIConfig)
assert cfg.use_openai_path is False
def test_unmapped_model_degrades_to_none_without_crashing(self, restore_model_cost):
# A non-frontier model that is not in model_cost makes get_model_info
# raise; the gate must swallow it and return None rather than crash.
from litellm.utils import ProviderConfigManager
litellm.model_cost.pop("bedrock_mantle/somelab.unmapped-model", None)
litellm.get_model_info.cache_clear()
cfg = ProviderConfigManager.get_provider_responses_api_config(
provider="bedrock_mantle",
model="somelab.unmapped-model",
)
assert cfg is None
def test_register_model_restore_undoes_existing_key_overwrite(self):
# Self-contained guard for the deepcopy requirement of restore_model_cost.
# register_model overwrites an existing key by mutating its nested dict in
# place, so the snapshot must be a deepcopy: a shallow dict() copy would
# share that nested dict and leave mode=responses after restore, making
# the final assertion fail. The in-place clear+update mirrors the fixture.
from litellm.utils import ProviderConfigManager, register_model
snapshot = copy.deepcopy(litellm.model_cost)
litellm.get_model_info.cache_clear()
try:
register_model(
{
"bedrock_mantle/openai.gpt-oss-120b": {
"litellm_provider": "bedrock_mantle",
"mode": "responses",
}
}
)
during = ProviderConfigManager.get_provider_responses_api_config(
provider="bedrock_mantle", model="openai.gpt-oss-120b"
)
assert isinstance(during, BedrockMantleResponsesAPIConfig)
finally:
litellm.model_cost.clear()
litellm.model_cost.update(snapshot)
litellm.get_model_info.cache_clear()
after = ProviderConfigManager.get_provider_responses_api_config(
provider="bedrock_mantle", model="openai.gpt-oss-120b"
)
assert after is None
@pytest.fixture
def restore_model_cost():
"""Snapshot litellm.model_cost so register_model edits don't leak across tests.
register_model mutates the global litellm.model_cost, and get_model_info is
lru_cached, so without restore + cache_clear a registered model would bleed
into sibling tests in the same process.
Two subtleties make this fixture non-obvious:
1. The snapshot must be a deepcopy. register_model overwrites an existing key
via `litellm.model_cost.setdefault(key, {}).update(...)`, mutating the
nested dict in place; a shallow copy would share those nested dicts and
could not capture the pre-mutation values of an existing entry.
2. The restore must be in place (clear + update the SAME dict object), not a
reassignment. The conftest autouse `isolate_litellm_state` fixture
snapshots `litellm.model_cost` by reference and restores that reference on
its teardown, which runs after this one. Reassigning `litellm.model_cost`
to a fresh dict here is undone when conftest reinstalls its (in-place
mutated) reference, so the registered mode would leak and poison
TestBedrockMantleResponsesPricing. Mutating the original object in place
restores the contents conftest's reference points at.
"""
original_model_cost = copy.deepcopy(litellm.model_cost)
litellm.get_model_info.cache_clear()
try:
yield
finally:
litellm.model_cost.clear()
litellm.model_cost.update(original_model_cost)
litellm.get_model_info.cache_clear()
@pytest.fixture
def local_cost_map(monkeypatch):
tests/test_litellm/llms/openai/completion/test_completion_handler.py
+93
View File
@@ -0,0 +1,93 @@
"""
Tests that client headers are forwarded to the provider on the OpenAI
text completion path.
Regression tests for https://github.com/BerriAI/litellm/issues/27410
"""
import os
import sys
import pytest
import respx
from httpx import Response
sys.path.insert(0, os.path.abspath("../../../../.."))
import litellm
from litellm import atext_completion, text_completion
@pytest.fixture(autouse=True)
def setup_env(monkeypatch):
monkeypatch.setenv("OPENAI_API_KEY", "sk-test-fake-key")
@pytest.fixture
def mock_completions_endpoint():
return respx.post("https://api.openai.com/v1/completions").mock(
return_value=Response(
200,
json={
"id": "cmpl-test123",
"object": "text_completion",
"created": 1677652288,
"model": "gpt-3.5-turbo-instruct",
"choices": [
{
"text": "hi",
"index": 0,
"logprobs": None,
"finish_reason": "stop",
}
],
"usage": {
"prompt_tokens": 1,
"completion_tokens": 1,
"total_tokens": 2,
},
},
)
)
@respx.mock
def test_completion_forwards_client_headers_to_provider(mock_completions_endpoint):
text_completion(
model="gpt-3.5-turbo-instruct",
prompt="hello",
max_tokens=5,
headers={"x-mycorp-llmcall-id": "abc-123"},
)
request_headers = mock_completions_endpoint.calls.last.request.headers
assert request_headers["x-mycorp-llmcall-id"] == "abc-123"
@respx.mock
def test_completion_forwards_extra_headers_to_provider(mock_completions_endpoint):
text_completion(
model="gpt-3.5-turbo-instruct",
prompt="hello",
max_tokens=5,
extra_headers={"x-mycorp-llmcall-id": "abc-123"},
)
request_headers = mock_completions_endpoint.calls.last.request.headers
assert request_headers["x-mycorp-llmcall-id"] == "abc-123"
@respx.mock
async def test_acompletion_forwards_client_headers_to_provider(
mock_completions_endpoint, monkeypatch
):
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
await atext_completion(
model="gpt-3.5-turbo-instruct",
prompt="hello",
max_tokens=5,
headers={"x-mycorp-llmcall-id": "abc-123"},
)
request_headers = mock_completions_endpoint.calls.last.request.headers
assert request_headers["x-mycorp-llmcall-id"] == "abc-123"
tests/test_litellm/llms/openai_like/test_tensormesh_provider.py
+72
View File
@@ -2,8 +2,23 @@
Tests for Tensormesh provider configuration and integration.
"""
import pytest
import litellm
TENSORMESH_MODELS = [
"tensormesh/Qwen/Qwen3.5-397B-A17B-FP8",
"tensormesh/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8",
"tensormesh/Qwen/Qwen3.6-27B-FP8",
"tensormesh/lukealonso/GLM-5.1-NVFP4-MTP",
"tensormesh/deepseek-ai/DeepSeek-V4-Flash",
"tensormesh/moonshotai/Kimi-K2.6",
"tensormesh/MiniMaxAI/MiniMax-M2.5",
"tensormesh/google/gemma-4-31B-it",
"tensormesh/openai/gpt-oss-120b",
"tensormesh/openai/gpt-oss-20b",
]
class TestTensormeshProviderConfig:
"""Test Tensormesh provider configuration"""
@@ -82,3 +97,60 @@ class TestTensormeshProviderConfig:
assert len(router.model_list) == 1
assert router.model_list[0]["model_name"] == "tensormesh-chat"
class TestTensormeshCostMap:
"""The serverless models are registered in the cost map so LiteLLM can
price requests and unblock tool-calling params on the JSON provider path."""
@pytest.fixture(autouse=True)
def _use_local_model_cost_map(self, monkeypatch):
original_model_cost = litellm.model_cost
monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "True")
litellm.model_cost = litellm.get_model_cost_map(url="")
litellm.get_model_info.cache_clear()
try:
yield
finally:
litellm.model_cost = original_model_cost
litellm.get_model_info.cache_clear()
def test_models_registered_with_capabilities(self):
for model in TENSORMESH_MODELS:
info = litellm.get_model_info(model)
assert info["litellm_provider"] == "tensormesh"
assert info["mode"] == "chat"
assert litellm.supports_function_calling(model) is True, model
assert litellm.supports_response_schema(model) is True, model
assert litellm.model_cost[model]["supports_tool_choice"] is True, model
assert litellm.model_cost[model]["supports_prompt_caching"] is True, model
def test_reasoning_flag_matches_expected_set(self):
reasoning_models = {
"tensormesh/deepseek-ai/DeepSeek-V4-Flash",
"tensormesh/Qwen/Qwen3.5-397B-A17B-FP8",
"tensormesh/Qwen/Qwen3.6-27B-FP8",
"tensormesh/lukealonso/GLM-5.1-NVFP4-MTP",
"tensormesh/MiniMaxAI/MiniMax-M2.5",
"tensormesh/moonshotai/Kimi-K2.6",
"tensormesh/openai/gpt-oss-120b",
"tensormesh/openai/gpt-oss-20b",
"tensormesh/google/gemma-4-31B-it",
}
for model in TENSORMESH_MODELS:
assert litellm.supports_reasoning(model) is (model in reasoning_models), model
def test_cost_is_wired_and_cache_reads_are_free(self):
prompt_cost, completion_cost = litellm.cost_per_token(
model="tensormesh/openai/gpt-oss-120b",
prompt_tokens=1_000_000,
completion_tokens=1_000_000,
)
assert prompt_cost == pytest.approx(0.15)
assert completion_cost == pytest.approx(0.60)
assert (
litellm.model_cost["tensormesh/openai/gpt-oss-120b"][
"cache_read_input_token_cost"
]
== 0
)
tests/test_litellm/llms/vertex_ai/context_caching/test_vertex_ai_context_caching.py
+26 -26
View File
@@ -86,7 +86,7 @@ class TestContextCachingEndpoints:
cached_content = "cached_content_123"
optional_params = self.sample_optional_params.copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Execute
result = self.context_caching.check_and_create_cache(
@@ -129,7 +129,7 @@ class TestContextCachingEndpoints:
mock_separate.return_value = ([], self.sample_messages) # No cached messages
optional_params = self.sample_optional_params.copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Execute
result = self.context_caching.check_and_create_cache(
@@ -177,7 +177,7 @@ class TestContextCachingEndpoints:
optional_params = self.sample_optional_params.copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Execute
result = self.context_caching.check_and_create_cache(
@@ -254,7 +254,7 @@ class TestContextCachingEndpoints:
optional_params = self.sample_optional_params.copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Execute
result = self.context_caching.check_and_create_cache(
@@ -324,7 +324,7 @@ class TestContextCachingEndpoints:
optional_params = self.sample_optional_params.copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Execute and Assert
with pytest.raises(VertexAIError) as exc_info:
@@ -364,7 +364,7 @@ class TestContextCachingEndpoints:
cached_content = "cached_content_123"
optional_params = self.sample_optional_params.copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Execute
result = await self.context_caching.async_check_and_create_cache(
@@ -404,7 +404,7 @@ class TestContextCachingEndpoints:
mock_separate.return_value = ([], self.sample_messages)
optional_params = self.sample_optional_params.copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Execute
result = await self.context_caching.async_check_and_create_cache(
@@ -453,7 +453,7 @@ class TestContextCachingEndpoints:
optional_params = self.sample_optional_params.copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Execute
result = await self.context_caching.async_check_and_create_cache(
@@ -535,7 +535,7 @@ class TestContextCachingEndpoints:
optional_params = self.sample_optional_params.copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Execute
result = await self.context_caching.async_check_and_create_cache(
@@ -606,7 +606,7 @@ class TestContextCachingEndpoints:
optional_params = self.sample_optional_params.copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Execute and Assert
with pytest.raises(VertexAIError) as exc_info:
@@ -648,7 +648,7 @@ class TestContextCachingEndpoints:
optional_params = self.sample_optional_params.copy()
original_tools = optional_params["tools"].copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Mock the check_cache to return existing cache so we don't make HTTP calls
with patch.object(
@@ -694,7 +694,7 @@ class TestContextCachingEndpoints:
optional_params = self.sample_optional_params.copy()
original_tools = optional_params["tools"].copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Execute
result = self.context_caching.check_and_create_cache(
@@ -735,7 +735,7 @@ class TestContextCachingEndpoints:
optional_params = self.sample_optional_params.copy()
original_tools = optional_params["tools"].copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Execute
result = await self.context_caching.async_check_and_create_cache(
@@ -778,7 +778,7 @@ class TestContextCachingEndpoints:
optional_params = self.sample_optional_params.copy()
original_tools = optional_params["tools"].copy()
test_project = "test_project"
test_location = "test_location"
test_location = "us-central1"
# Mock the async_check_cache to return existing cache so we don't make HTTP calls
with patch.object(
@@ -837,7 +837,7 @@ class TestContextCachingEndpoints:
logging_obj=self.mock_logging,
custom_llm_provider=custom_llm_provider,
vertex_project="test_project",
vertex_location="test_location",
vertex_location="us-central1",
vertex_auth_header="vertext_test_token",
)
@@ -870,7 +870,7 @@ class TestContextCachingEndpoints:
logging_obj=self.mock_logging,
custom_llm_provider=custom_llm_provider,
vertex_project="test_project",
vertex_location="test_location",
vertex_location="us-central1",
vertex_auth_header="vertext_test_token",
)
@@ -908,7 +908,7 @@ class TestContextCachingEndpoints:
logging_obj=self.mock_logging,
custom_llm_provider=custom_llm_provider,
vertex_project="test_project",
vertex_location="test_location",
vertex_location="us-central1",
vertex_auth_header="vertext_test_token",
)
@@ -942,7 +942,7 @@ class TestContextCachingEndpoints:
logging_obj=self.mock_logging,
custom_llm_provider=custom_llm_provider,
vertex_project="test_project",
vertex_location="test_location",
vertex_location="us-central1",
vertex_auth_header="vertext_test_token",
)
@@ -1002,7 +1002,7 @@ class TestContextCachingEndpoints:
logging_obj=self.mock_logging,
custom_llm_provider=custom_llm_provider,
vertex_project="test_project",
vertex_location="test_location",
vertex_location="us-central1",
vertex_auth_header="vertext_test_token",
)
@@ -1072,7 +1072,7 @@ class TestContextCachingEndpoints:
logging_obj=self.mock_logging,
custom_llm_provider=custom_llm_provider,
vertex_project="test_project",
vertex_location="test_location",
vertex_location="us-central1",
vertex_auth_header="vertext_test_token",
)
@@ -1138,7 +1138,7 @@ class TestContextCachingEndpoints:
logging_obj=self.mock_logging,
custom_llm_provider=custom_llm_provider,
vertex_project="test_project",
vertex_location="test_location",
vertex_location="us-central1",
vertex_auth_header="vertext_test_token",
)
@@ -1205,7 +1205,7 @@ class TestContextCachingEndpoints:
logging_obj=self.mock_logging,
custom_llm_provider=custom_llm_provider,
vertex_project="test_project",
vertex_location="test_location",
vertex_location="us-central1",
vertex_auth_header="vertext_test_token",
)
@@ -1280,7 +1280,7 @@ class TestContextCachingEndpoints:
logging_obj=self.mock_logging,
custom_llm_provider=custom_llm_provider,
vertex_project="test_project",
vertex_location="test_location",
vertex_location="us-central1",
vertex_auth_header="vertext_test_token",
)
@@ -1336,7 +1336,7 @@ class TestContextCachingEndpoints:
logging_obj=self.mock_logging,
custom_llm_provider=custom_llm_provider,
vertex_project="test_project",
vertex_location="test_location",
vertex_location="us-central1",
vertex_auth_header="vertext_test_token",
)
@@ -1390,7 +1390,7 @@ class TestContextCachingEndpoints:
cached_content=None,
custom_llm_provider=custom_llm_provider,
vertex_project="test_project",
vertex_location="test_location",
vertex_location="us-central1",
vertex_auth_header="test_token",
)
@@ -1441,7 +1441,7 @@ class TestContextCachingEndpoints:
cached_content=None,
custom_llm_provider=custom_llm_provider,
vertex_project="test_project",
vertex_location="test_location",
vertex_location="us-central1",
vertex_auth_header="test_token",
)
tests/test_litellm/llms/xai/test_xai_oauth.py
+801
View File
@@ -0,0 +1,801 @@
import base64
import hashlib
import json
import os
import threading
import time
from urllib.parse import parse_qs, urlparse
from unittest.mock import MagicMock
import httpx
import litellm
import pytest
from click.testing import CliRunner
import litellm.llms.xai.oauth as xai_oauth_module
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.llms.xai.oauth import (
XAI_OAUTH_CLIENT_ID,
XAI_OAUTH_SCOPE,
XAIOAuthError,
XAIOAuthAuthenticator,
XAIOAuthLoginRequiredError,
)
from litellm.llms.xai.chat.transformation import XAIChatConfig
from litellm.llms.xai.responses.transformation import XAIResponsesAPIConfig
from litellm.types.router import GenericLiteLLMParams
from litellm.utils import get_optional_params, validate_environment
def _write_auth_file(tmp_path, payload):
token_dir = tmp_path / "xai_oauth"
token_dir.mkdir()
auth_file = token_dir / "auth.json"
auth_file.write_text(json.dumps(payload))
return token_dir, auth_file
def test_get_access_token_uses_fresh_local_token(tmp_path, monkeypatch):
token_dir, _ = _write_auth_file(
tmp_path,
{
"access_token": "fresh-token",
"refresh_token": "refresh-token",
"expires_at": time.time() + 3600,
},
)
monkeypatch.setenv("XAI_OAUTH_TOKEN_DIR", str(token_dir))
assert XAIOAuthAuthenticator().get_access_token() == "fresh-token"
def test_get_access_token_refreshes_and_preserves_refresh_token(tmp_path, monkeypatch):
token_dir, auth_file = _write_auth_file(
tmp_path,
{
"access_token": "expired-token",
"refresh_token": "refresh-token",
"token_endpoint": "https://auth.x.ai/oauth/token",
"expires_at": time.time() - 1,
},
)
monkeypatch.setenv("XAI_OAUTH_TOKEN_DIR", str(token_dir))
def handler(request: httpx.Request) -> httpx.Response:
body = dict(item.split("=") for item in request.content.decode().split("&"))
assert body["grant_type"] == "refresh_token"
assert body["refresh_token"] == "refresh-token"
assert body["client_id"] == XAI_OAUTH_CLIENT_ID
return httpx.Response(
200,
json={
"access_token": "new-token",
"expires_in": 3600,
"token_type": "Bearer",
},
)
client = httpx.Client(transport=httpx.MockTransport(handler))
assert XAIOAuthAuthenticator(http_client=client).get_access_token() == "new-token"
stored = json.loads(auth_file.read_text())
assert stored["access_token"] == "new-token"
assert stored["refresh_token"] == "refresh-token"
def test_get_access_token_reuses_token_refreshed_by_parallel_request():
expired_auth_data = {
"access_token": "expired-token",
"refresh_token": "refresh-token",
"token_endpoint": "https://auth.x.ai/oauth/token",
"expires_at": time.time() - 1,
}
refreshed_auth_data = {
"access_token": "already-refreshed-token",
"refresh_token": "rotated-refresh-token",
"token_endpoint": "https://auth.x.ai/oauth/token",
"expires_at": time.time() + 3600,
}
authenticator = XAIOAuthAuthenticator()
authenticator._read_auth_file = MagicMock(
side_effect=[expired_auth_data, refreshed_auth_data]
)
authenticator._refresh_tokens = MagicMock()
assert authenticator.get_access_token() == "already-refreshed-token"
authenticator._refresh_tokens.assert_not_called()
def test_get_access_token_requires_login_without_auth_file(tmp_path, monkeypatch):
monkeypatch.setenv("XAI_OAUTH_TOKEN_DIR", str(tmp_path / "missing"))
with pytest.raises(XAIOAuthLoginRequiredError):
XAIOAuthAuthenticator().get_access_token()
def test_get_access_token_ignores_invalid_auth_file(tmp_path, monkeypatch):
token_dir = tmp_path / "xai_oauth"
token_dir.mkdir()
(token_dir / "auth.json").write_text("{not-json")
monkeypatch.setenv("XAI_OAUTH_TOKEN_DIR", str(token_dir))
with pytest.raises(XAIOAuthLoginRequiredError):
XAIOAuthAuthenticator().get_access_token()
def test_refresh_failure_surfaces_oauth_error(tmp_path, monkeypatch):
token_dir, _ = _write_auth_file(
tmp_path,
{
"access_token": "expired-token",
"refresh_token": "refresh-token",
"token_endpoint": "https://auth.x.ai/oauth/token",
"expires_at": time.time() - 1,
},
)
monkeypatch.setenv("XAI_OAUTH_TOKEN_DIR", str(token_dir))
client = httpx.Client(
transport=httpx.MockTransport(
lambda request: httpx.Response(401, text="invalid_grant", request=request)
)
)
with pytest.raises(XAIOAuthError) as exc_info:
XAIOAuthAuthenticator(http_client=client).get_access_token()
assert "401 invalid_grant" in str(exc_info.value)
def test_build_auth_record_requires_access_and_refresh_tokens():
authenticator = XAIOAuthAuthenticator()
with pytest.raises(XAIOAuthError, match="access_token"):
authenticator._build_auth_record(
{"refresh_token": "refresh-token"},
"https://auth.x.ai/oauth/token",
)
with pytest.raises(XAIOAuthError, match="refresh_token"):
authenticator._build_auth_record(
{"access_token": "access-token"},
"https://auth.x.ai/oauth/token",
)
def test_build_auth_record_defaults_expiry_and_token_type():
authenticator = XAIOAuthAuthenticator()
auth_data = authenticator._build_auth_record(
{
"access_token": "access-token",
"refresh_token": "refresh-token",
"expires_in": "not-a-number",
},
"https://auth.x.ai/oauth/token",
)
assert auth_data["token_type"] == "Bearer"
assert auth_data["expires_at"] > time.time()
def test_is_expired_treats_missing_or_invalid_expiry_as_expired():
authenticator = XAIOAuthAuthenticator()
assert authenticator._is_expired({}) is True
assert authenticator._is_expired({"expires_at": "not-a-number"}) is True
def test_write_auth_file_creates_private_file(tmp_path, monkeypatch):
token_dir = tmp_path / "xai_oauth"
monkeypatch.setenv("XAI_OAUTH_TOKEN_DIR", str(token_dir))
authenticator = XAIOAuthAuthenticator()
old_umask = os.umask(0o022)
replace_calls = []
real_replace = os.replace
def assert_private_temp_file(src, dst):
replace_calls.append((src, dst))
assert oct(os.stat(src).st_mode & 0o777) == "0o600"
with open(src) as f:
assert json.load(f)["refresh_token"] == "refresh-token"
real_replace(src, dst)
monkeypatch.setattr(os, "replace", assert_private_temp_file)
try:
authenticator._write_auth_file(
{
"access_token": "access-token",
"refresh_token": "refresh-token",
"expires_at": time.time() + 3600,
}
)
finally:
os.umask(old_umask)
stored = json.loads((token_dir / "auth.json").read_text())
assert stored["access_token"] == "access-token"
assert replace_calls
assert oct(os.stat(token_dir).st_mode & 0o777) == "0o700"
assert oct(os.stat(token_dir / "auth.json").st_mode & 0o777) == "0o600"
def test_discovery_rejects_unexpected_endpoint():
authenticator = XAIOAuthAuthenticator()
with pytest.raises(XAIOAuthError, match="unexpected endpoint"):
authenticator._validate_xai_endpoint("https://evil.example.com/oauth/token")
with pytest.raises(XAIOAuthError, match="unexpected endpoint"):
authenticator._validate_xai_endpoint("http://auth.x.ai/oauth/token")
def test_discover_returns_validated_xai_endpoints():
def handler(request: httpx.Request) -> httpx.Response:
assert request.url == "https://auth.x.ai/.well-known/openid-configuration"
return httpx.Response(
200,
json={
"authorization_endpoint": "https://auth.x.ai/oauth/authorize",
"token_endpoint": "https://auth.x.ai/oauth/token",
},
)
authenticator = XAIOAuthAuthenticator(
http_client=httpx.Client(transport=httpx.MockTransport(handler))
)
assert authenticator._discover() == {
"authorization_endpoint": "https://auth.x.ai/oauth/authorize",
"token_endpoint": "https://auth.x.ai/oauth/token",
}
def test_discover_requires_authorization_and_token_endpoints():
authenticator = XAIOAuthAuthenticator(
http_client=httpx.Client(
transport=httpx.MockTransport(lambda request: httpx.Response(200, json={}))
)
)
with pytest.raises(XAIOAuthError, match="missing endpoints"):
authenticator._discover()
def test_discover_wraps_http_errors():
authenticator = XAIOAuthAuthenticator(
http_client=httpx.Client(
transport=httpx.MockTransport(
lambda request: httpx.Response(
500, text="discovery failed", request=request
)
)
)
)
with pytest.raises(XAIOAuthError) as exc_info:
authenticator._discover()
assert "xAI OAuth discovery request failed: 500 discovery failed" in str(
exc_info.value
)
def test_discover_wraps_invalid_json_response():
authenticator = XAIOAuthAuthenticator(
http_client=httpx.Client(
transport=httpx.MockTransport(
lambda request: httpx.Response(200, text="<html>not-json</html>")
)
)
)
with pytest.raises(XAIOAuthError, match="discovery response was not valid JSON"):
authenticator._discover()
def test_refresh_discovers_token_endpoint_when_auth_file_is_legacy(
tmp_path, monkeypatch
):
token_dir, auth_file = _write_auth_file(
tmp_path,
{
"access_token": "expired-token",
"refresh_token": "refresh-token",
"expires_at": time.time() - 1,
},
)
monkeypatch.setenv("XAI_OAUTH_TOKEN_DIR", str(token_dir))
def handler(request: httpx.Request) -> httpx.Response:
if request.method == "GET":
return httpx.Response(
200,
json={
"authorization_endpoint": "https://auth.x.ai/oauth/authorize",
"token_endpoint": "https://auth.x.ai/oauth/token",
},
)
return httpx.Response(
200,
json={
"access_token": "discovered-token",
"refresh_token": "new-refresh-token",
"expires_in": 3600,
},
)
authenticator = XAIOAuthAuthenticator(
http_client=httpx.Client(transport=httpx.MockTransport(handler))
)
assert authenticator.get_access_token() == "discovered-token"
stored = json.loads(auth_file.read_text())
assert stored["token_endpoint"] == "https://auth.x.ai/oauth/token"
def test_exchange_token_rejects_non_object_response():
authenticator = XAIOAuthAuthenticator(
http_client=httpx.Client(
transport=httpx.MockTransport(
lambda request: httpx.Response(200, json=["not", "an", "object"])
)
)
)
with pytest.raises(XAIOAuthError, match="was not an object"):
authenticator._exchange_token("https://auth.x.ai/oauth/token", {})
def test_exchange_token_wraps_invalid_json_response():
authenticator = XAIOAuthAuthenticator(
http_client=httpx.Client(
transport=httpx.MockTransport(
lambda request: httpx.Response(200, text="<html>not-json</html>")
)
)
)
with pytest.raises(XAIOAuthError, match="token response was not valid JSON"):
authenticator._exchange_token("https://auth.x.ai/oauth/token", {})
def test_start_callback_server_falls_back_to_ephemeral_port(monkeypatch):
calls = []
real_server = xai_oauth_module._CallbackServer
class FirstPortFailsCallbackServer(real_server):
def __init__(self, server_address, handler_class):
calls.append(server_address[1])
if server_address[1] == xai_oauth_module.XAI_OAUTH_REDIRECT_PORT:
raise OSError("port unavailable")
super().__init__(server_address, handler_class)
monkeypatch.setattr(
xai_oauth_module, "_CallbackServer", FirstPortFailsCallbackServer
)
server, redirect_uri = XAIOAuthAuthenticator()._start_callback_server("state-value")
try:
assert calls == [xai_oauth_module.XAI_OAUTH_REDIRECT_PORT, 0]
assert redirect_uri.startswith("http://127.0.0.1:")
assert redirect_uri.endswith("/callback")
finally:
server.server_close()
def test_wait_for_callback_times_out_and_closes_server(monkeypatch):
server, _ = XAIOAuthAuthenticator()._start_callback_server("state-value")
monkeypatch.setattr(xai_oauth_module, "XAI_OAUTH_CALLBACK_TIMEOUT_SECONDS", 0)
with pytest.raises(XAIOAuthError, match="Timed out"):
XAIOAuthAuthenticator()._wait_for_callback(server)
def test_callback_handler_records_success_and_rejects_state_mismatch():
authenticator = XAIOAuthAuthenticator()
server, redirect_uri = authenticator._start_callback_server("expected-state")
thread = threading.Thread(target=server.handle_request)
thread.start()
response = httpx.get(f"{redirect_uri}?code=auth-code&state=expected-state")
thread.join(timeout=5)
assert response.status_code == 200
assert server.callback_result == {
"code": "auth-code",
"state": "expected-state",
"error": None,
"error_description": None,
}
server, redirect_uri = authenticator._start_callback_server("expected-state")
thread = threading.Thread(target=server.handle_request)
thread.start()
response = httpx.get(f"{redirect_uri}?code=auth-code&state=wrong-state")
thread.join(timeout=5)
assert response.status_code == 400
assert server.callback_result["state"] == "wrong-state"
def test_login_exchanges_authorization_code_and_persists_auth_record(monkeypatch):
authenticator = XAIOAuthAuthenticator()
fake_server = MagicMock()
written_records = []
class FakeUUID:
def __init__(self, value):
self.hex = value
monkeypatch.setattr(
xai_oauth_module.uuid,
"uuid4",
MagicMock(side_effect=[FakeUUID("state-value"), FakeUUID("nonce-value")]),
)
authenticator._read_auth_file = MagicMock(return_value=None)
authenticator._discover = MagicMock(
return_value={
"authorization_endpoint": "https://auth.x.ai/oauth/authorize",
"token_endpoint": "https://auth.x.ai/oauth/token",
}
)
authenticator._pkce_pair = MagicMock(return_value=("verifier", "challenge"))
authenticator._start_callback_server = MagicMock(
return_value=(fake_server, "http://127.0.0.1:56121/callback")
)
authenticator._wait_for_callback = MagicMock(
return_value={"state": "state-value", "code": "auth-code"}
)
authenticator._exchange_token = MagicMock(
return_value={
"access_token": "access-token",
"refresh_token": "refresh-token",
"expires_in": 3600,
}
)
authenticator._write_auth_file = MagicMock(side_effect=written_records.append)
auth_data = authenticator.login(no_browser=True)
authenticator._exchange_token.assert_called_once_with(
"https://auth.x.ai/oauth/token",
{
"grant_type": "authorization_code",
"code": "auth-code",
"redirect_uri": "http://127.0.0.1:56121/callback",
"client_id": XAI_OAUTH_CLIENT_ID,
"code_verifier": "verifier",
},
)
assert auth_data["access_token"] == "access-token"
assert written_records == [auth_data]
def test_login_raises_on_callback_error_or_missing_code(monkeypatch):
authenticator = XAIOAuthAuthenticator()
class FakeUUID:
hex = "state-value"
monkeypatch.setattr(
xai_oauth_module.uuid, "uuid4", MagicMock(return_value=FakeUUID())
)
authenticator._read_auth_file = MagicMock(return_value=None)
authenticator._discover = MagicMock(
return_value={
"authorization_endpoint": "https://auth.x.ai/oauth/authorize",
"token_endpoint": "https://auth.x.ai/oauth/token",
}
)
authenticator._pkce_pair = MagicMock(return_value=("verifier", "challenge"))
authenticator._start_callback_server = MagicMock(
return_value=(MagicMock(), "http://127.0.0.1:56121/callback")
)
authenticator._wait_for_callback = MagicMock(
return_value={
"state": "state-value",
"error": "access_denied",
"error_description": "denied",
}
)
with pytest.raises(XAIOAuthError, match="denied"):
authenticator.login(no_browser=True)
authenticator._wait_for_callback = MagicMock(return_value={"state": "state-value"})
with pytest.raises(XAIOAuthError, match="no code returned"):
authenticator.login(no_browser=True)
def test_pkce_pair_generates_s256_challenge():
verifier, challenge = XAIOAuthAuthenticator()._pkce_pair()
expected = (
base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest())
.rstrip(b"=")
.decode()
)
assert challenge == expected
assert "=" not in verifier
assert "=" not in challenge
def test_build_authorize_url_contains_xai_oauth_parameters():
authorize_url = XAIOAuthAuthenticator()._build_authorize_url(
authorization_endpoint="https://auth.x.ai/oauth/authorize",
redirect_uri="http://127.0.0.1:56121/callback",
challenge="pkce-challenge",
state="state-value",
nonce="nonce-value",
)
parsed = urlparse(authorize_url)
params = parse_qs(parsed.query)
assert parsed.scheme == "https"
assert parsed.netloc == "auth.x.ai"
assert params["response_type"] == ["code"]
assert params["client_id"] == [XAI_OAUTH_CLIENT_ID]
assert params["scope"] == [XAI_OAUTH_SCOPE]
assert params["code_challenge"] == ["pkce-challenge"]
assert params["code_challenge_method"] == ["S256"]
assert params["state"] == ["state-value"]
assert params["nonce"] == ["nonce-value"]
def test_get_llm_provider_uses_single_xai_provider(monkeypatch):
monkeypatch.setenv("XAI_API_KEY", "api-key")
model, provider, api_key, api_base = get_llm_provider("xai/grok-4")
assert model == "grok-4"
assert provider == "xai"
assert api_key == "api-key"
assert api_base == "https://api.x.ai/v1"
def test_xai_oauth_alias_is_not_a_provider():
with pytest.raises(Exception):
get_llm_provider("xai_oauth/grok-4")
def test_chat_config_wraps_flagged_oauth_errors_as_authentication_error(
tmp_path, monkeypatch
):
monkeypatch.setenv("XAI_OAUTH_TOKEN_DIR", str(tmp_path / "missing"))
with pytest.raises(litellm.AuthenticationError) as exc_info:
XAIChatConfig().validate_environment(
headers={},
model="grok-4",
messages=[],
optional_params={},
litellm_params={"use_xai_oauth": True},
api_key=None,
)
assert exc_info.value.llm_provider == "xai"
assert "litellm xai-oauth login" in str(exc_info.value)
def test_chat_config_injects_flagged_oauth_token(tmp_path, monkeypatch):
token_dir, _ = _write_auth_file(
tmp_path,
{
"access_token": "chat-token",
"refresh_token": "refresh-token",
"expires_at": time.time() + 3600,
},
)
monkeypatch.setenv("XAI_OAUTH_TOKEN_DIR", str(token_dir))
headers = XAIChatConfig().validate_environment(
headers={},
model="grok-4",
messages=[],
optional_params={},
litellm_params={"use_xai_oauth": True},
api_key=None,
)
assert headers["Authorization"] == "Bearer chat-token"
def test_chat_config_ignores_api_base_override_for_flagged_oauth(monkeypatch):
monkeypatch.setenv("XAI_OAUTH_API_BASE", "https://api.x.ai/v1")
url = XAIChatConfig().get_complete_url(
api_base="https://attacker.example.com/v1",
api_key=None,
model="grok-4",
optional_params={},
litellm_params={"use_xai_oauth": True},
)
assert url == "https://api.x.ai/v1/chat/completions"
def test_chat_config_treats_blank_api_key_as_absent_for_flagged_oauth(
tmp_path, monkeypatch
):
token_dir, _ = _write_auth_file(
tmp_path,
{
"access_token": "stored-oauth-token",
"refresh_token": "refresh-token",
"expires_at": time.time() + 3600,
},
)
monkeypatch.setenv("XAI_OAUTH_TOKEN_DIR", str(token_dir))
headers = XAIChatConfig().validate_environment(
headers={},
model="grok-4",
messages=[],
optional_params={},
litellm_params={"use_xai_oauth": True},
api_key="",
)
assert headers["Authorization"] == "Bearer stored-oauth-token"
def test_chat_config_allows_api_base_override_with_caller_api_key():
headers = XAIChatConfig().validate_environment(
headers={},
model="grok-4",
messages=[],
optional_params={},
litellm_params={"use_xai_oauth": True},
api_key="caller-api-key",
)
url = XAIChatConfig().get_complete_url(
api_base="https://custom.example.com/v1",
api_key="caller-api-key",
model="grok-4",
optional_params={},
litellm_params={"use_xai_oauth": True},
)
assert headers["Authorization"] == "Bearer caller-api-key"
assert url == "https://custom.example.com/v1/chat/completions"
def test_chat_config_prioritizes_env_api_key_over_oauth_flag(monkeypatch):
monkeypatch.setenv("XAI_API_KEY", "env-api-key")
headers = XAIChatConfig().validate_environment(
headers={},
model="grok-4",
messages=[],
optional_params={},
litellm_params={"use_xai_oauth": True},
api_key=None,
)
url = XAIChatConfig().get_complete_url(
api_base="https://custom.example.com/v1",
api_key=None,
model="grok-4",
optional_params={},
litellm_params={"use_xai_oauth": True},
)
assert headers["Authorization"] == "Bearer env-api-key"
assert url == "https://custom.example.com/v1/chat/completions"
def test_validate_environment_still_reports_xai_api_key(monkeypatch):
monkeypatch.setenv("XAI_API_KEY", "env-api-key")
assert validate_environment("xai/grok-4") == {
"keys_in_environment": True,
"missing_keys": [],
}
def test_xai_oauth_flag_uses_xai_optional_param_mapping():
litellm_params = GenericLiteLLMParams(use_xai_oauth=True)
optional_params = get_optional_params(
model="grok-4",
custom_llm_provider="xai",
temperature=0.2,
max_tokens=8,
)
assert optional_params["temperature"] == 0.2
assert optional_params["max_tokens"] == 8
assert litellm_params.use_xai_oauth is True
assert "use_xai_oauth" not in optional_params
def test_responses_config_injects_flagged_oauth_bearer_token(tmp_path, monkeypatch):
token_dir, _ = _write_auth_file(
tmp_path,
{
"access_token": "responses-token",
"refresh_token": "refresh-token",
"expires_at": time.time() + 3600,
},
)
monkeypatch.setenv("XAI_OAUTH_TOKEN_DIR", str(token_dir))
headers = XAIResponsesAPIConfig().validate_environment(
headers={},
model="grok-4",
litellm_params=GenericLiteLLMParams(use_xai_oauth=True),
)
assert headers["Authorization"] == "Bearer responses-token"
def test_responses_config_endpoint_url_uses_oauth_authenticator(monkeypatch):
monkeypatch.setenv("XAI_OAUTH_API_BASE", "https://xai.example.com/v1/")
config = XAIResponsesAPIConfig()
assert config.get_complete_url(
api_base=None, litellm_params={"use_xai_oauth": True}
) == ("https://xai.example.com/v1/responses")
assert (
config.get_complete_url(
api_base="https://custom.example.com/v1/",
litellm_params={"use_xai_oauth": True},
)
== "https://xai.example.com/v1/responses"
)
assert (
config.get_complete_url(
api_base="https://custom.example.com/v1/",
litellm_params={"api_key": "", "use_xai_oauth": True},
)
== "https://xai.example.com/v1/responses"
)
assert (
config.get_complete_url(
api_base="https://custom.example.com/v1/",
litellm_params={"api_key": "caller-api-key"},
)
== "https://custom.example.com/v1/responses"
)
def test_responses_config_wraps_flagged_oauth_errors_as_authentication_error(
tmp_path, monkeypatch
):
monkeypatch.setenv("XAI_OAUTH_TOKEN_DIR", str(tmp_path / "missing"))
with pytest.raises(litellm.AuthenticationError) as exc_info:
XAIResponsesAPIConfig().validate_environment(
headers={},
model="grok-4",
litellm_params=GenericLiteLLMParams(use_xai_oauth=True),
)
assert XAIResponsesAPIConfig().custom_llm_provider.value == "xai"
assert exc_info.value.llm_provider == "xai"
def test_proxy_cli_xai_oauth_login_uses_single_authenticator(monkeypatch):
from litellm.proxy.proxy_cli import run_server
instances = []
class FakeAuthenticator:
auth_file = "/tmp/xai-oauth-auth.json"
def __init__(self):
instances.append(self)
def login(self):
return {"expires_at": 1234567890}
monkeypatch.setattr(
"litellm.llms.xai.oauth.XAIOAuthAuthenticator", FakeAuthenticator
)
result = CliRunner().invoke(run_server, ["xai-oauth", "login"])
assert result.exit_code == 0
assert len(instances) == 1
assert "Credentials saved to /tmp/xai-oauth-auth.json" in result.output
assert "Access token expires at 1234567890" in result.output
tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server.py
+128 -2
View File
@@ -4970,17 +4970,143 @@ class TestGatewayCreateInitializationOptions:
try:
from litellm.proxy._experimental.mcp_server.mcp_context import (
_mcp_gateway_initialize_instructions,
_mcp_gateway_server_name,
)
from litellm.proxy._experimental.mcp_server.server import server
except ImportError:
pytest.skip("MCP server not available")
tok = _mcp_gateway_initialize_instructions.set(None)
instructions_token = _mcp_gateway_initialize_instructions.set(None)
server_name_token = _mcp_gateway_server_name.set(None)
try:
opts = server.create_initialization_options()
assert getattr(opts, "instructions", None) is None
assert opts.server_name == "litellm-mcp-server"
finally:
_mcp_gateway_initialize_instructions.reset(tok)
_mcp_gateway_initialize_instructions.reset(instructions_token)
_mcp_gateway_server_name.reset(server_name_token)
@pytest.mark.asyncio
async def test_scoped_request_uses_configured_server_alias(self):
try:
from litellm.proxy._experimental.mcp_server.server import (
_gateway_initialize_instructions_request_scope,
global_mcp_server_manager,
server,
)
except ImportError:
pytest.skip("MCP server not available")
scoped_server = MCPServer(
server_id="server-123",
name="upstream-server",
alias="grafana",
transport=MCPTransport.http,
url="https://example.com/mcp",
)
with (
patch(
"litellm.proxy._experimental.mcp_server.server._get_allowed_mcp_servers",
new_callable=AsyncMock,
return_value=[scoped_server],
),
patch.object(
global_mcp_server_manager,
"_ensure_upstream_initialize_instructions_cached",
new_callable=AsyncMock,
),
):
async with _gateway_initialize_instructions_request_scope(
user_api_key_auth=None,
mcp_servers=["grafana"],
client_ip=None,
scoped_server_endpoint=True,
):
assert server.create_initialization_options().server_name == "grafana"
assert (
server.create_initialization_options().server_name == "litellm-mcp-server"
)
@pytest.mark.asyncio
async def test_sse_handler_scopes_server_name_from_single_server_path(self):
try:
from litellm.proxy._experimental.mcp_server import server as mcp_server
from litellm.proxy._experimental.mcp_server.server import (
global_mcp_server_manager,
handle_sse_mcp,
server,
)
except ImportError:
pytest.skip("MCP server not available")
scoped_server = MCPServer(
server_id="server-123",
name="upstream-server",
alias="grafana",
transport=MCPTransport.http,
url="https://example.com/mcp",
)
captured = {}
async def record_request(scope, receive, send):
captured["server_name"] = server.create_initialization_options().server_name
scope = {
"type": "http",
"method": "POST",
"path": "/mcp/grafana",
"headers": [],
}
with (
patch(
"litellm.proxy._experimental.mcp_server.server.extract_mcp_auth_context",
new_callable=AsyncMock,
return_value=(
UserAPIKeyAuth(api_key="sk-test"),
None,
["grafana"],
None,
None,
None,
),
),
patch(
"litellm.proxy._experimental.mcp_server.server._get_allowed_mcp_servers",
new_callable=AsyncMock,
return_value=[scoped_server],
),
patch.object(
global_mcp_server_manager,
"_ensure_upstream_initialize_instructions_cached",
new_callable=AsyncMock,
),
patch(
"litellm.proxy._experimental.mcp_server.server._raise_preemptive_401_for_unauthenticated_servers",
new_callable=AsyncMock,
),
patch(
"litellm.proxy._experimental.mcp_server.server._check_passthrough_upstream_auth",
new_callable=AsyncMock,
),
patch(
"litellm.proxy._experimental.mcp_server.server._SESSION_MANAGERS_INITIALIZED",
True,
),
patch.object(
mcp_server.sse_session_manager,
"handle_request",
side_effect=record_request,
),
):
await handle_sse_mcp(scope, AsyncMock(), AsyncMock())
assert captured["server_name"] == "grafana"
assert (
server.create_initialization_options().server_name == "litellm-mcp-server"
)
def test_contextvar_set_injects_instructions(self):
"""When ContextVar has a value, it appears in InitializationOptions."""
tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server_manager.py
+64
View File
@@ -30,6 +30,7 @@ from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
MCPServerManager,
_deserialize_json_dict,
_deserialize_json_list,
_normalize_mcp_server_cost_info,
)
from litellm.proxy._types import (
LiteLLM_MCPServerTable,
@@ -257,6 +258,69 @@ class TestMCPServerManager:
assert server.alias == "friendly_alias"
assert server.server_name == "validserver"
@pytest.mark.asyncio
async def test_load_servers_from_config_coerces_cost_string_to_float(self):
"""YAML 1.1 parses `7e-05` as a string; ingest must coerce it to float."""
manager = MCPServerManager()
config = {
"google_maps": {
"url": "https://example.com/mcp",
"transport": MCPTransport.http,
"mcp_info": {
"mcp_server_cost_info": {
"default_cost_per_query": "7e-05",
"tool_name_to_cost_per_query": {"geocode": "1e-3"},
}
},
}
}
await manager.load_servers_from_config(config)
server = next(iter(manager.config_mcp_servers.values()))
cost_info = server.mcp_info["mcp_server_cost_info"]
assert cost_info["default_cost_per_query"] == 7e-05
assert isinstance(cost_info["default_cost_per_query"], float)
assert cost_info["tool_name_to_cost_per_query"]["geocode"] == 1e-3
assert isinstance(cost_info["tool_name_to_cost_per_query"]["geocode"], float)
def test_normalize_mcp_server_cost_info_preserves_float_values(self):
mcp_info = {
"server_name": "maps",
"mcp_server_cost_info": {
"default_cost_per_query": 0.01,
"tool_name_to_cost_per_query": {"search": 0.05},
},
}
_normalize_mcp_server_cost_info(mcp_info)
cost_info = mcp_info["mcp_server_cost_info"]
assert cost_info["default_cost_per_query"] == 0.01
assert cost_info["tool_name_to_cost_per_query"] == {"search": 0.05}
def test_normalize_mcp_server_cost_info_drops_non_numeric_values(self):
mcp_info = {
"server_name": "maps",
"mcp_server_cost_info": {
"default_cost_per_query": "not-a-number",
"tool_name_to_cost_per_query": {"search": "free", "geocode": "2e-4"},
},
}
_normalize_mcp_server_cost_info(mcp_info)
cost_info = mcp_info["mcp_server_cost_info"]
assert "default_cost_per_query" not in cost_info
assert cost_info["tool_name_to_cost_per_query"] == {"geocode": 2e-4}
def test_normalize_mcp_server_cost_info_leaves_missing_cost_info_alone(self):
mcp_info = {"server_name": "maps"}
_normalize_mcp_server_cost_info(mcp_info)
assert "mcp_server_cost_info" not in mcp_info
def test_warns_when_custom_separator_invalid(self, monkeypatch, caplog):
"""Invalid MCP_TOOL_PREFIX_SEPARATOR values should log a warning."""
tests/test_litellm/proxy/auth/test_auth_checks.py
+2
View File
@@ -2409,6 +2409,8 @@ async def test_virtual_key_budget_check_fallback_no_counter():
assert exc_info.value.current_cost == 15.0
@pytest.mark.asyncio
async def test_team_budget_check_reads_from_spend_counter():
"""Team budget check should use get_current_spend when counter exists."""
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_presidio.py
+157 -5
View File
@@ -2398,11 +2398,9 @@ async def test_anonymize_text_uses_correct_positions_no_parse_pii():
)
expected = "My name is <PERSON>, my email is <EMAIL_ADDRESS>, phone <PHONE_NUMBER>"
assert result == expected, (
f"anonymize_text produced garbled output with PII remnants.\n"
f"Expected: {expected!r}\n"
f"Got: {result!r}"
)
assert (
result == expected
), f"anonymize_text produced garbled output with PII remnants.\nExpected: {expected!r}\nGot: {result!r}"
assert masked_entity_count == {
"PERSON": 1,
"EMAIL_ADDRESS": 1,
@@ -2495,3 +2493,157 @@ async def test_anonymize_text_uses_correct_positions_with_parse_pii():
assert pii_tokens.get("<PERSON_1>") == "John Smith"
assert pii_tokens.get("<EMAIL_ADDRESS_2>") == "john@example.com"
assert pii_tokens.get("<PHONE_NUMBER_3>") == "555-867-5309"
def test_unmask_sse_bytes_chunk_replaces_text_delta():
import json
pii_tokens = {"<PERSON_1>": "Bobby"}
event = {
"type": "content_block_delta",
"index": 0,
"delta": {"type": "text_delta", "text": "Hello <PERSON_1>, how are you?"},
}
chunk = ("data: " + json.dumps(event) + "\n\n").encode("utf-8")
result = _OPTIONAL_PresidioPIIMasking._unmask_sse_bytes_chunk(chunk, pii_tokens)
decoded = result.decode("utf-8")
parsed = json.loads(decoded.split("data: ", 1)[1].strip())
assert parsed["delta"]["text"] == "Hello Bobby, how are you?"
def test_unmask_sse_bytes_chunk_ignores_non_text_delta():
import json
pii_tokens = {"<PERSON_1>": "Bobby"}
# message_start event — no delta
event = {"type": "message_start", "message": {"id": "msg_01", "role": "assistant"}}
chunk = ("data: " + json.dumps(event) + "\n\n").encode("utf-8")
result = _OPTIONAL_PresidioPIIMasking._unmask_sse_bytes_chunk(chunk, pii_tokens)
assert result == chunk
# input_json_delta — should not be touched
event2 = {
"type": "content_block_delta",
"index": 1,
"delta": {"type": "input_json_delta", "partial_json": '{"name": "<PERSON_1>"}'},
}
chunk2 = ("data: " + json.dumps(event2) + "\n\n").encode("utf-8")
result2 = _OPTIONAL_PresidioPIIMasking._unmask_sse_bytes_chunk(chunk2, pii_tokens)
assert result2 == chunk2
def test_unmask_sse_bytes_chunk_handles_malformed_json():
chunk = b"data: {not valid json}\n\n"
result = _OPTIONAL_PresidioPIIMasking._unmask_sse_bytes_chunk(
chunk, {"<PERSON_1>": "Bobby"}
)
assert result == chunk
def test_unmask_sse_bytes_chunk_handles_unicode_decode_error():
chunk = b"\xff\xfe invalid utf-8"
result = _OPTIONAL_PresidioPIIMasking._unmask_sse_bytes_chunk(
chunk, {"<PERSON_1>": "Bobby"}
)
assert result == chunk
def test_unmask_sse_bytes_chunk_non_ascii_pii_not_escaped():
import json
pii_tokens = {"<PERSON_1>": "José"}
event = {
"type": "content_block_delta",
"index": 0,
"delta": {"type": "text_delta", "text": "Hello <PERSON_1>!"},
}
chunk = ("data: " + json.dumps(event) + "\n\n").encode("utf-8")
result = _OPTIONAL_PresidioPIIMasking._unmask_sse_bytes_chunk(chunk, pii_tokens)
decoded = result.decode("utf-8")
assert "Jos\\u" not in decoded
parsed = json.loads(decoded.split("data: ", 1)[1].strip())
assert parsed["delta"]["text"] == "Hello José!"
def test_unmask_sse_bytes_chunk_handles_crlf_line_endings():
import json
pii_tokens = {"<PERSON_1>": "Bobby"}
event = {
"type": "content_block_delta",
"index": 0,
"delta": {"type": "text_delta", "text": "Hi <PERSON_1>!"},
}
crlf_chunk = ("data: " + json.dumps(event) + "\r\ndata: [DONE]\r\n").encode("utf-8")
result = _OPTIONAL_PresidioPIIMasking._unmask_sse_bytes_chunk(
crlf_chunk, pii_tokens
)
decoded = result.decode("utf-8")
parsed = json.loads(decoded.split("data: ", 1)[1].split("\n")[0].strip())
assert parsed["delta"]["text"] == "Hi Bobby!"
assert "data: [DONE]" in decoded
@pytest.mark.asyncio
async def test_stream_pii_unmasking_unmaskes_bytes_chunks(mock_user_api_key):
import json
guardrail = _OPTIONAL_PresidioPIIMasking(
mock_testing=True,
output_parse_pii=True,
)
pii_tokens = {"<PERSON_1>": "Bobby"}
request_data = {"metadata": {"pii_tokens": pii_tokens}}
def _make_sse_chunk(text: str) -> bytes:
event = {
"type": "content_block_delta",
"index": 0,
"delta": {"type": "text_delta", "text": text},
}
return ("data: " + json.dumps(event) + "\n\n").encode("utf-8")
async def mock_stream():
yield _make_sse_chunk("Hello <PERSON_1>!")
yield _make_sse_chunk(" How can I help?")
chunks = []
async for chunk in guardrail._stream_pii_unmasking(mock_stream(), request_data):
chunks.append(chunk)
assert len(chunks) == 2
first = chunks[0].decode("utf-8")
first_event = json.loads(first.split("data: ", 1)[1].strip())
assert first_event["delta"]["text"] == "Hello Bobby!"
second = chunks[1].decode("utf-8")
second_event = json.loads(second.split("data: ", 1)[1].strip())
assert second_event["delta"]["text"] == " How can I help?"
@pytest.mark.asyncio
async def test_stream_pii_unmasking_passthrough_when_no_tokens(mock_user_api_key):
guardrail = _OPTIONAL_PresidioPIIMasking(
mock_testing=True,
output_parse_pii=True,
)
raw_chunk = b"data: {}\n\n"
request_data: dict = {"metadata": {}}
async def mock_stream():
yield raw_chunk
chunks = []
async for chunk in guardrail._stream_pii_unmasking(mock_stream(), request_data):
chunks.append(chunk)
assert chunks == [raw_chunk]
tests/test_litellm/proxy/hooks/litellm_skills/test_main.py
+67
View File
@@ -0,0 +1,67 @@
from unittest.mock import AsyncMock, patch
import pytest
from litellm.proxy.hooks.litellm_skills.main import SkillsInjectionHook
SKILL_TOOL_NAME = "litellm_skill_e2b8dca8_031a_4481_b034_b9ec7d4eb7bf"
def _request_data():
return {
"model": "claude-sonnet-4-5",
"messages": [{"role": "user", "content": "run the skill"}],
"litellm_metadata": {
"_litellm_code_execution_enabled": True,
"_skill_files": {SKILL_TOOL_NAME: {"main.py": b"print('hi')"}},
},
}
def _tool_use_response(tool_name):
return {
"stop_reason": "tool_use",
"content": [
{"type": "tool_use", "id": "toolu_1", "name": tool_name, "input": {}}
],
}
@pytest.mark.asyncio
async def test_post_call_success_hook_executes_litellm_skill_tool():
"""DB skill tool names carry the litellm_skill_ prefix and must trigger the execution loop."""
hook = SkillsInjectionHook()
response = _tool_use_response(SKILL_TOOL_NAME)
with patch.object(
hook, "_execute_code_loop_messages_api", new=AsyncMock(return_value=response)
) as mock_loop:
result = await hook.async_post_call_success_deployment_hook(
request_data=_request_data(), response=response, call_type=None
)
mock_loop.assert_awaited_once()
assert result is response
@pytest.mark.asyncio
async def test_execute_code_loop_dispatches_litellm_skill_tool():
"""The agentic loop must route litellm_skill_ tool calls to _execute_skill_tool."""
hook = SkillsInjectionHook()
final_response = {"stop_reason": "end_turn", "content": []}
with (
patch.object(
hook, "_execute_skill_tool", new=AsyncMock(return_value="skill ran")
) as mock_exec,
patch("litellm.anthropic.acreate", new=AsyncMock(return_value=final_response)),
):
result = await hook._execute_code_loop_messages_api(
data=_request_data(),
response=_tool_use_response(SKILL_TOOL_NAME),
skill_files={"main.py": b"print('hi')"},
)
mock_exec.assert_awaited_once()
assert mock_exec.await_args.args[0] == SKILL_TOOL_NAME
assert result is final_response
tests/test_litellm/proxy/hooks/test_parallel_request_limiter.py
+86
View File
@@ -0,0 +1,86 @@
"""
Unit Tests for the max parallel request limiter v1 for the proxy
"""
from datetime import datetime
import pytest
from litellm.caching.caching import DualCache
from litellm.proxy.hooks.parallel_request_limiter import (
_PROXY_MaxParallelRequestsHandler,
)
from litellm.proxy.utils import InternalUsageCache, hash_token
from litellm.types.utils import EmbeddingResponse, TextCompletionResponse, Usage
@pytest.mark.parametrize(
"response_obj",
[
EmbeddingResponse(
model="text-embedding-3-small",
usage=Usage(prompt_tokens=50, completion_tokens=0, total_tokens=50),
),
TextCompletionResponse(
model="gpt-3.5-turbo-instruct",
usage=Usage(prompt_tokens=20, completion_tokens=30, total_tokens=50),
),
],
)
@pytest.mark.asyncio
async def test_async_log_success_event_counts_non_chat_response_tokens(response_obj):
"""
Embedding and text completion responses must increment the per key, user,
team, and end user TPM counters, not just chat completion ModelResponse
objects.
"""
_api_key = hash_token("sk-12345")
user_id = "ishaan"
team_id = "litellm-team"
end_user_id = "customer-1"
parallel_request_handler = _PROXY_MaxParallelRequestsHandler(
internal_usage_cache=InternalUsageCache(DualCache())
)
current_date = datetime.now().strftime("%Y-%m-%d")
current_hour = datetime.now().strftime("%H")
current_minute = datetime.now().strftime("%M")
precise_minute = f"{current_date}-{current_hour}-{current_minute}"
scope_ids = [_api_key, user_id, team_id, end_user_id]
for scope_id in scope_ids:
await parallel_request_handler.internal_usage_cache.async_set_cache(
key=f"{scope_id}::{precise_minute}::request_count",
value={"current_requests": 1, "current_tpm": 0, "current_rpm": 1},
litellm_parent_otel_span=None,
)
kwargs = {
"litellm_params": {
"metadata": {
"user_api_key": _api_key,
"user_api_key_user_id": user_id,
"user_api_key_team_id": team_id,
"user_api_key_model_max_budget": {},
}
},
"user": end_user_id,
}
await parallel_request_handler.async_log_success_event(
kwargs=kwargs,
response_obj=response_obj,
start_time=datetime.now(),
end_time=datetime.now(),
)
for scope_id in scope_ids:
current = await parallel_request_handler.internal_usage_cache.async_get_cache(
key=f"{scope_id}::{precise_minute}::request_count",
litellm_parent_otel_span=None,
)
assert current["current_tpm"] == 50, (
f"expected 50 tokens counted for {scope_id}, "
f"got {current['current_tpm']}"
)
tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.py
+68 -1
View File
@@ -20,7 +20,12 @@ from litellm.proxy.hooks.parallel_request_limiter_v3 import (
_PROXY_MaxParallelRequestsHandler_v3 as _PROXY_MaxParallelRequestsHandler,
)
from litellm.proxy.utils import InternalUsageCache, ProxyLogging, hash_token
from litellm.types.utils import ModelResponse, Usage
from litellm.types.utils import (
EmbeddingResponse,
ModelResponse,
TextCompletionResponse,
Usage,
)
class TimeController:
@@ -547,6 +552,68 @@ async def test_token_rate_limit_type_respected_v3(monkeypatch, token_rate_limit_
), f"Expected {expected_tokens[token_rate_limit_type]} tokens for type '{token_rate_limit_type}', got {tpm_operation['increment_value']}"
@pytest.mark.parametrize(
"response_obj",
[
EmbeddingResponse(
model="text-embedding-3-small",
usage=Usage(prompt_tokens=50, completion_tokens=0, total_tokens=50),
),
TextCompletionResponse(
model="gpt-3.5-turbo-instruct",
usage=Usage(prompt_tokens=20, completion_tokens=30, total_tokens=50),
),
],
)
@pytest.mark.asyncio
async def test_async_log_success_event_counts_non_chat_response_tokens(
monkeypatch, response_obj
):
"""
Embedding and text completion responses must increment the TPM counter,
not just chat completion ModelResponse objects.
"""
monkeypatch.setenv("LITELLM_RATE_LIMIT_WINDOW_SIZE", "60")
_api_key = hash_token("sk-12345")
parallel_request_handler = _PROXY_MaxParallelRequestsHandler(
internal_usage_cache=InternalUsageCache(DualCache())
)
monkeypatch.setattr(
parallel_request_handler, "get_rate_limit_type", lambda: "total"
)
mock_kwargs = {
"standard_logging_object": {"metadata": {"user_api_key_hash": _api_key}},
"model": response_obj.model,
}
captured_operations = []
async def mock_increment_pipeline(increment_list, **kwargs):
captured_operations.extend(increment_list)
return True
monkeypatch.setattr(
parallel_request_handler.internal_usage_cache.dual_cache,
"async_increment_cache_pipeline",
mock_increment_pipeline,
)
await parallel_request_handler.async_log_success_event(
kwargs=mock_kwargs,
response_obj=response_obj,
start_time=datetime.now(),
end_time=datetime.now(),
)
tpm_operation = next(
(op for op in captured_operations if op["key"].endswith(":tokens")), None
)
assert tpm_operation is not None, "Should have a TPM increment operation"
assert tpm_operation["increment_value"] == 50
@pytest.mark.asyncio
async def test_async_log_failure_event_v3():
"""
tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py
+154
View File
@@ -6496,6 +6496,9 @@ async def test_reset_key_spend_success(monkeypatch):
patch(
"litellm.proxy.management_endpoints.key_management_endpoints._delete_cache_key_object"
) as mock_delete_cache,
patch(
"litellm.proxy.proxy_server._invalidate_spend_counter"
) as mock_invalidate,
):
mock_hash_token.return_value = hashed_key
mock_check_admin.return_value = None
@@ -6520,6 +6523,76 @@ async def test_reset_key_spend_success(monkeypatch):
assert response["max_budget"] == 200.0
mock_prisma_client.db.litellm_verificationtoken.update.assert_called_once()
mock_delete_cache.assert_awaited_once()
mock_invalidate.assert_awaited_once_with(counter_key=f"spend:key:{hashed_key}")
@pytest.mark.asyncio
async def test_update_key_spend_invalidates_counter(monkeypatch):
"""
Test that updating a key's spend via update_key_fn immediately invalidates the spend counter.
"""
from litellm.proxy.management_endpoints.key_management_endpoints import (
update_key_fn,
)
mock_prisma_client = AsyncMock()
mock_user_api_key_cache = AsyncMock()
mock_proxy_logging_obj = MagicMock()
hashed_key = "0d62f396c1317066f55a96086517047c737087c61eb2bf016b72e6298927b15b"
key_in_db = LiteLLM_VerificationToken(
token=hashed_key,
user_id="test-user",
spend=10.0,
max_budget=200.0,
litellm_budget_table=None,
)
mock_prisma_client.get_data = AsyncMock(return_value=key_in_db)
mock_prisma_client.update_data = AsyncMock(return_value={"data": {"spend": 0.0}})
mock_prisma_client.db.litellm_verificationtoken.find_unique = AsyncMock(
return_value=key_in_db
)
monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", mock_prisma_client)
monkeypatch.setattr(
"litellm.proxy.proxy_server.user_api_key_cache", mock_user_api_key_cache
)
monkeypatch.setattr(
"litellm.proxy.proxy_server.proxy_logging_obj", mock_proxy_logging_obj
)
monkeypatch.setattr("litellm.proxy.proxy_server.llm_router", None)
monkeypatch.setattr("litellm.proxy.proxy_server.premium_user", True)
monkeypatch.setattr("litellm.store_audit_logs", False)
with (
patch(
"litellm.proxy.management_endpoints.key_management_endpoints._delete_cache_key_object"
) as mock_delete_cache,
patch(
"litellm.proxy.proxy_server._invalidate_spend_counter"
) as mock_invalidate,
):
mock_delete_cache.return_value = None
user_api_key_dict = UserAPIKeyAuth(
user_role=LitellmUserRoles.PROXY_ADMIN,
api_key="sk-admin",
user_id="admin-user",
)
mock_request = MagicMock()
mock_request.query_params = {}
await update_key_fn(
request=mock_request,
data=UpdateKeyRequest(key="sk-test-key", spend=0.0),
user_api_key_dict=user_api_key_dict,
litellm_changed_by=None,
)
mock_delete_cache.assert_awaited_once()
mock_invalidate.assert_awaited_once_with(counter_key=f"spend:key:{hashed_key}")
@pytest.mark.asyncio
@@ -11668,3 +11741,84 @@ async def test_ghsa_q775_default_team_id_does_not_grant_session_token_exemption(
msg = str(getattr(err, "detail", "")) + str(getattr(err, "message", ""))
assert str(code) == "400"
assert "cannot exceed" in msg.lower()
@pytest.mark.asyncio
async def test_prepare_key_update_data_budget_duration_null_clears_fields():
"""
When budget_duration is explicitly set to null, prepare_key_update_data
should produce budget_duration=None and budget_reset_at=None so Prisma
clears them in the DB.
"""
existing_key = LiteLLM_VerificationToken(
token="test-token",
key_alias="test-key",
models=[],
user_id="test-user",
team_id=None,
metadata={},
)
update_request = UpdateKeyRequest(key="test-token", budget_duration=None)
result = await prepare_key_update_data(
data=update_request, existing_key_row=existing_key
)
assert "budget_duration" in result
assert result["budget_duration"] is None
assert "budget_reset_at" in result
assert result["budget_reset_at"] is None
@pytest.mark.asyncio
async def test_prepare_key_update_data_budget_duration_not_sent_excluded():
"""
When budget_duration is NOT sent in the request (unset), it should not
appear in the result dict at all — the existing DB value stays unchanged.
"""
existing_key = LiteLLM_VerificationToken(
token="test-token",
key_alias="test-key",
models=[],
user_id="test-user",
team_id=None,
metadata={},
)
update_request = UpdateKeyRequest(key="test-token", models=["gpt-4"])
result = await prepare_key_update_data(
data=update_request, existing_key_row=existing_key
)
assert "budget_duration" not in result
assert "budget_reset_at" not in result
@pytest.mark.asyncio
async def test_prepare_key_update_data_budget_duration_valid_sets_reset():
"""
When budget_duration is set to a valid duration string, both
budget_duration and budget_reset_at should be populated.
"""
existing_key = LiteLLM_VerificationToken(
token="test-token",
key_alias="test-key",
models=[],
user_id="test-user",
team_id=None,
metadata={},
)
update_request = UpdateKeyRequest(key="test-token", budget_duration="30d")
result = await prepare_key_update_data(
data=update_request, existing_key_row=existing_key
)
assert result["budget_duration"] == "30d"
assert result["budget_reset_at"] is not None
tests/test_litellm/proxy/management_endpoints/test_team_endpoints.py
+326
View File
@@ -8886,3 +8886,329 @@ async def test_update_team_blocks_non_admin_passthrough_routes(mock_db_client):
)
assert str(exc.value.code) == "403"
assert "allowed_passthrough_routes" in str(exc.value.message)
def test_set_budget_reset_at_clears_when_budget_duration_null():
"""
When budget_duration is explicitly set to null, _set_budget_reset_at
should set budget_reset_at=None in updated_kv so Prisma clears it in the DB.
"""
from litellm.proxy._types import UpdateTeamRequest
from litellm.proxy.management_endpoints.team_endpoints import _set_budget_reset_at
data = UpdateTeamRequest(team_id="test-team", budget_duration=None)
updated_kv = {"team_id": "test-team", "budget_duration": None}
_set_budget_reset_at(data, updated_kv)
assert "budget_reset_at" in updated_kv
assert updated_kv["budget_reset_at"] is None
def test_set_budget_reset_at_noop_when_budget_duration_not_sent():
"""
When budget_duration is NOT sent (unset), _set_budget_reset_at should
not add budget_reset_at to updated_kv.
"""
from litellm.proxy._types import UpdateTeamRequest
from litellm.proxy.management_endpoints.team_endpoints import _set_budget_reset_at
data = UpdateTeamRequest(team_id="test-team")
updated_kv = {"team_id": "test-team"}
_set_budget_reset_at(data, updated_kv)
assert "budget_reset_at" not in updated_kv
def test_set_budget_reset_at_sets_value_when_budget_duration_provided():
"""
When budget_duration is set to a valid string, _set_budget_reset_at
should compute and set budget_reset_at.
"""
from litellm.proxy._types import UpdateTeamRequest
from litellm.proxy.management_endpoints.team_endpoints import _set_budget_reset_at
data = UpdateTeamRequest(team_id="test-team", budget_duration="30d")
updated_kv = {"team_id": "test-team", "budget_duration": "30d"}
_set_budget_reset_at(data, updated_kv)
assert "budget_reset_at" in updated_kv
assert updated_kv["budget_reset_at"] is not None
@pytest.mark.asyncio
async def test_clear_team_member_budget_duration_calls_update_budget():
"""
When team_member_budget_duration is explicitly null and a budget row
exists, clear_team_member_budget_fields should call update_budget
with budget_duration=None and budget_reset_at=None.
"""
from litellm.proxy.management_endpoints.team_endpoints import (
TeamMemberBudgetHandler,
)
mock_user_api_key_dict = UserAPIKeyAuth(
user_role=LitellmUserRoles.PROXY_ADMIN,
api_key="sk-1234",
user_id="admin-user",
)
team_table = LiteLLM_TeamTable(
team_id="test-team",
metadata={"team_member_budget_id": "budget-123"},
members_with_roles=[],
)
updated_kv = {
"team_id": "test-team",
"team_member_budget_duration": None,
}
with patch(
"litellm.proxy.management_endpoints.budget_management_endpoints.update_budget",
new_callable=AsyncMock,
) as mock_update_budget:
result = await TeamMemberBudgetHandler.clear_team_member_budget_fields(
team_table=team_table,
user_api_key_dict=mock_user_api_key_dict,
updated_kv=updated_kv,
explicitly_set_fields={"team_member_budget_duration"},
)
mock_update_budget.assert_awaited_once()
budget_request = mock_update_budget.call_args.kwargs["budget_obj"]
assert budget_request.budget_id == "budget-123"
assert "budget_duration" in budget_request.model_fields_set
assert budget_request.budget_duration is None
assert "budget_reset_at" in budget_request.model_fields_set
assert budget_request.budget_reset_at is None
assert "team_member_budget_duration" not in result
@pytest.mark.asyncio
async def test_clear_team_member_budget_clears_max_budget():
"""
When team_member_budget is explicitly null, clear_team_member_budget_fields
should call update_budget with max_budget=None.
"""
from litellm.proxy.management_endpoints.team_endpoints import (
TeamMemberBudgetHandler,
)
mock_user_api_key_dict = UserAPIKeyAuth(
user_role=LitellmUserRoles.PROXY_ADMIN,
api_key="sk-1234",
user_id="admin-user",
)
team_table = LiteLLM_TeamTable(
team_id="test-team",
metadata={"team_member_budget_id": "budget-456"},
members_with_roles=[],
)
updated_kv = {
"team_id": "test-team",
"team_member_budget": None,
}
with patch(
"litellm.proxy.management_endpoints.budget_management_endpoints.update_budget",
new_callable=AsyncMock,
) as mock_update_budget:
result = await TeamMemberBudgetHandler.clear_team_member_budget_fields(
team_table=team_table,
user_api_key_dict=mock_user_api_key_dict,
updated_kv=updated_kv,
explicitly_set_fields={"team_member_budget"},
)
mock_update_budget.assert_awaited_once()
budget_request = mock_update_budget.call_args.kwargs["budget_obj"]
assert budget_request.budget_id == "budget-456"
assert "max_budget" in budget_request.model_fields_set
assert budget_request.max_budget is None
assert "team_member_budget" not in result
@pytest.mark.asyncio
async def test_clear_team_member_rpm_tpm_limits():
"""
When team_member_rpm_limit and team_member_tpm_limit are explicitly null,
clear_team_member_budget_fields should clear both on the budget row.
"""
from litellm.proxy.management_endpoints.team_endpoints import (
TeamMemberBudgetHandler,
)
mock_user_api_key_dict = UserAPIKeyAuth(
user_role=LitellmUserRoles.PROXY_ADMIN,
api_key="sk-1234",
user_id="admin-user",
)
team_table = LiteLLM_TeamTable(
team_id="test-team",
metadata={"team_member_budget_id": "budget-789"},
members_with_roles=[],
)
updated_kv = {
"team_id": "test-team",
"team_member_rpm_limit": None,
"team_member_tpm_limit": None,
}
with patch(
"litellm.proxy.management_endpoints.budget_management_endpoints.update_budget",
new_callable=AsyncMock,
) as mock_update_budget:
result = await TeamMemberBudgetHandler.clear_team_member_budget_fields(
team_table=team_table,
user_api_key_dict=mock_user_api_key_dict,
updated_kv=updated_kv,
explicitly_set_fields={"team_member_rpm_limit", "team_member_tpm_limit"},
)
mock_update_budget.assert_awaited_once()
budget_request = mock_update_budget.call_args.kwargs["budget_obj"]
assert budget_request.budget_id == "budget-789"
assert "rpm_limit" in budget_request.model_fields_set
assert budget_request.rpm_limit is None
assert "tpm_limit" in budget_request.model_fields_set
assert budget_request.tpm_limit is None
assert "team_member_rpm_limit" not in result
assert "team_member_tpm_limit" not in result
@pytest.mark.asyncio
async def test_clear_all_team_member_fields_at_once():
"""
When all team_member fields are explicitly null, all corresponding
budget row fields should be cleared in a single update.
"""
from litellm.proxy.management_endpoints.team_endpoints import (
TeamMemberBudgetHandler,
)
mock_user_api_key_dict = UserAPIKeyAuth(
user_role=LitellmUserRoles.PROXY_ADMIN,
api_key="sk-1234",
user_id="admin-user",
)
team_table = LiteLLM_TeamTable(
team_id="test-team",
metadata={"team_member_budget_id": "budget-all"},
members_with_roles=[],
)
updated_kv = {
"team_id": "test-team",
"team_member_budget": None,
"team_member_budget_duration": None,
"team_member_rpm_limit": None,
"team_member_tpm_limit": None,
}
all_fields = {
"team_member_budget",
"team_member_budget_duration",
"team_member_rpm_limit",
"team_member_tpm_limit",
}
with patch(
"litellm.proxy.management_endpoints.budget_management_endpoints.update_budget",
new_callable=AsyncMock,
) as mock_update_budget:
result = await TeamMemberBudgetHandler.clear_team_member_budget_fields(
team_table=team_table,
user_api_key_dict=mock_user_api_key_dict,
updated_kv=updated_kv,
explicitly_set_fields=all_fields,
)
mock_update_budget.assert_awaited_once()
budget_request = mock_update_budget.call_args.kwargs["budget_obj"]
assert budget_request.budget_id == "budget-all"
assert budget_request.max_budget is None
assert budget_request.budget_duration is None
assert budget_request.budget_reset_at is None
assert budget_request.rpm_limit is None
assert budget_request.tpm_limit is None
for field in all_fields:
assert field not in result
@pytest.mark.asyncio
async def test_team_member_budget_duration_not_sent_does_not_update():
"""
When team_member_budget_duration is NOT sent in the request, no budget
update should occur and the field should not appear in updated_kv.
"""
from litellm.proxy.management_endpoints.team_endpoints import (
TeamMemberBudgetHandler,
)
updated_kv = {"team_id": "test-team", "max_budget": 200}
_team_member_fields_in_request = {
field
for field in [
"team_member_budget",
"team_member_rpm_limit",
"team_member_tpm_limit",
"team_member_budget_duration",
]
if field in updated_kv
}
assert len(_team_member_fields_in_request) == 0
TeamMemberBudgetHandler._clean_team_member_fields(updated_kv)
assert "team_member_budget_duration" not in updated_kv
assert "team_member_budget" not in updated_kv
@pytest.mark.asyncio
async def test_clear_team_member_budget_fields_no_budget_row_skips_update():
from litellm.proxy.management_endpoints.team_endpoints import (
TeamMemberBudgetHandler,
)
mock_user_api_key_dict = UserAPIKeyAuth(
user_role=LitellmUserRoles.PROXY_ADMIN,
api_key="sk-1234",
user_id="admin-user",
)
team_table = LiteLLM_TeamTable(
team_id="test-team",
metadata=None,
members_with_roles=[],
)
updated_kv = {
"team_id": "test-team",
"team_member_budget": None,
"team_member_rpm_limit": None,
}
with patch(
"litellm.proxy.management_endpoints.budget_management_endpoints.update_budget",
new_callable=AsyncMock,
) as mock_update_budget:
result = await TeamMemberBudgetHandler.clear_team_member_budget_fields(
team_table=team_table,
user_api_key_dict=mock_user_api_key_dict,
updated_kv=updated_kv,
explicitly_set_fields={"team_member_budget", "team_member_rpm_limit"},
)
mock_update_budget.assert_not_awaited()
assert "team_member_budget" not in result
assert "team_member_rpm_limit" not in result
tests/test_litellm/proxy/pass_through_endpoints/llm_provider_handlers/test_openai_passthrough_logging_handler.py
+249 -19
View File
@@ -683,36 +683,43 @@ class TestOpenAIPassthroughLoggingHandler:
"litellm.litellm_core_utils.litellm_logging.get_standard_logging_object_payload"
)
@patch(
"litellm.proxy.pass_through_endpoints.llm_provider_handlers.openai_passthrough_logging_handler.OpenAIPassthroughLoggingHandler.get_provider_config"
"litellm.llms.openai.responses.transformation.OpenAIResponsesAPIConfig.transform_response_api_response"
)
def test_responses_api_cost_tracking(
self, mock_get_provider_config, mock_get_standard_logging, mock_completion_cost
self,
mock_transform_responses,
mock_get_standard_logging,
mock_completion_cost,
):
"""Test cost tracking for responses API route"""
"""Test cost tracking for responses API route.
Mocks the Responses-API transformer (the dedicated one this branch
of the handler dispatches into post-fix) so we can assert the
downstream cost-calculation contract without depending on the
real transformer's full behavior.
"""
# Arrange
mock_completion_cost.return_value = 0.000050
mock_get_standard_logging.return_value = {"test": "logging_payload"}
# Mock the provider config's transform_response to return a valid ModelResponse
from litellm import ModelResponse
# Mock the Responses transformer's return — a ResponsesAPIResponse
# carrying the usage fields downstream cost-calc expects.
from litellm.types.llms.openai import ResponsesAPIResponse
mock_model_response = ModelResponse(
mock_responses_api_response = ResponsesAPIResponse.model_construct(
id="resp_abc123",
object="response",
created_at=1677652288,
model="gpt-4o-2024-08-06",
choices=[
{
"message": {
"role": "assistant",
"content": "Hello! How can I help you today?",
}
}
],
usage={"prompt_tokens": 20, "completion_tokens": 15, "total_tokens": 35},
status="completed",
output=[],
usage={
"input_tokens": 20,
"output_tokens": 15,
"total_tokens": 35,
},
)
mock_provider_config = MagicMock()
mock_provider_config.transform_response.return_value = mock_model_response
mock_get_provider_config.return_value = mock_provider_config
mock_transform_responses.return_value = mock_responses_api_response
# Mock responses API response
mock_responses_response = {
@@ -768,6 +775,109 @@ class TestOpenAIPassthroughLoggingHandler:
assert mock_logging_obj.model_call_details["model"] == "gpt-4o"
assert mock_logging_obj.model_call_details["custom_llm_provider"] == "openai"
@patch("litellm.completion_cost")
@patch(
"litellm.litellm_core_utils.litellm_logging.get_standard_logging_object_payload"
)
def test_responses_api_uses_responses_transformer_not_chat_completions(
self, mock_get_standard_logging, mock_completion_cost
):
"""Regression test for the Responses-API cost-tracking dispatch bug.
BUG: the `elif is_responses:` branch in `openai_passthrough_handler`
was calling `OpenAIConfig.transform_response` (the chat-completions
transformer) on a Responses API payload. Chat-completions
transform_response expects `choices: [...]` in the raw response;
the Responses API uses `output: [...]` and `usage.input_tokens` /
`usage.output_tokens` (not `prompt_tokens` / `completion_tokens`).
The result was a KeyError 'choices' inside
`convert_to_model_response_object`, swallowed by the surrounding
try/except, and the SpendLogs row was written with zero tokens
and zero spend.
FIX: use the dedicated `OpenAIResponsesAPIConfig.transform_response_api_response`
for the Responses branch.
This test exercises the REAL transformer (no mocked
`get_provider_config`) so that running it against the un-fixed
handler raises and running it against the fixed handler succeeds.
"""
mock_completion_cost.return_value = 0.000050
mock_get_standard_logging.return_value = {"test": "logging_payload"}
# A real-shaped Azure / OpenAI Responses API payload — NO `choices`,
# uses `output` and `usage.input_tokens` / `usage.output_tokens`.
responses_api_body = {
"id": "resp_abc123",
"object": "response",
"created_at": 1677652288,
"model": "gpt-4o-2024-08-06",
"status": "completed",
"output": [
{
"type": "message",
"role": "assistant",
"content": [
{
"type": "output_text",
"text": "Hello!",
}
],
}
],
"usage": {
"input_tokens": 20,
"output_tokens": 15,
"total_tokens": 35,
},
}
mock_httpx_response = self._create_mock_httpx_response(responses_api_body)
mock_logging_obj = self._create_mock_logging_obj()
passthrough_payload = self._create_passthrough_logging_payload()
kwargs = {
"passthrough_logging_payload": passthrough_payload,
"model": "gpt-4o",
"custom_llm_provider": "openai",
}
result = OpenAIPassthroughLoggingHandler.openai_passthrough_handler(
httpx_response=mock_httpx_response,
response_body=responses_api_body,
logging_obj=mock_logging_obj,
url_route="https://api.openai.com/v1/responses",
result="",
start_time=self.start_time,
end_time=self.end_time,
cache_hit=False,
request_body={"model": "gpt-4o", "input": "Tell me about AI"},
**kwargs,
)
# Pre-fix this assertion fails — the handler swallows the
# KeyError raised by the chat-completions transformer and falls
# back to the passthrough_chat_handler which yields a different
# response_cost value. Post-fix, the Responses transformer
# succeeds and we get the mocked 0.000050.
assert result is not None
assert result["kwargs"]["response_cost"] == 0.000050
assert result["kwargs"]["model"] == "gpt-4o"
# `completion_cost` must be called with the responses call type
# and a `ResponsesAPIResponse` (not a `ModelResponse`).
mock_completion_cost.assert_called_once()
call_kwargs = mock_completion_cost.call_args[1]
assert call_kwargs["call_type"] == "responses"
from litellm.types.llms.openai import ResponsesAPIResponse
assert isinstance(call_kwargs["completion_response"], ResponsesAPIResponse), (
"completion_response must be a ResponsesAPIResponse; passing a "
"chat-completions ModelResponse means the Responses transformer "
"isn't being used and we're back in the bug."
)
class TestOpenAIPassthroughIntegration:
"""Integration tests for OpenAI passthrough cost tracking"""
@@ -872,6 +982,126 @@ class TestOpenAIPassthroughIntegration:
)
assert self.handler.is_openai_route("") == False
def test_is_supported_openai_endpoint_includes_responses_api(self):
"""Regression test for the outer dispatch gate.
`_is_supported_openai_endpoint` is the gate that decides whether the
OpenAI handler runs for a given URL. Before this gate accepted the
Responses API, calls to `/v1/responses` would fail the gate and the
handler's `elif is_responses:` branch was unreachable in the live
success-handler pipeline — every Responses-API call landed in
`LiteLLM_SpendLogs` with zero tokens / zero spend even though the
handler had a Responses branch internally.
This test exercises the dispatch decision directly so future
refactors of `_is_supported_openai_endpoint` can't silently
remove Responses from the OR-chain without a test failure.
"""
# Responses must be supported on api.openai.com and openai.azure.com.
assert (
self.handler._is_supported_openai_endpoint(
"https://api.openai.com/v1/responses"
)
is True
)
assert (
self.handler._is_supported_openai_endpoint(
"https://openai.azure.com/v1/responses"
)
is True
)
# The other supported endpoints stay supported (no regression).
assert (
self.handler._is_supported_openai_endpoint(
"https://api.openai.com/v1/chat/completions"
)
is True
)
assert (
self.handler._is_supported_openai_endpoint(
"https://api.openai.com/v1/images/generations"
)
is True
)
assert (
self.handler._is_supported_openai_endpoint(
"https://api.openai.com/v1/images/edits"
)
is True
)
# Unsupported OpenAI endpoints (e.g. /v1/models) still return False.
assert (
self.handler._is_supported_openai_endpoint(
"https://api.openai.com/v1/models"
)
is False
)
@patch(
"litellm.proxy.pass_through_endpoints.llm_provider_handlers.openai_passthrough_logging_handler.OpenAIPassthroughLoggingHandler.openai_passthrough_handler"
)
@pytest.mark.asyncio
async def test_success_handler_dispatches_responses_api_to_openai_handler(
self, mock_openai_handler
):
"""End-to-end dispatch test for the Responses API path.
Pre-fix: `_is_supported_openai_endpoint` returned False for
`/v1/responses` URLs, so the OpenAI handler was never called.
This test would fail (mock never invoked) on the un-fixed
success_handler — passes only when the dispatch gate accepts
Responses URLs.
"""
mock_openai_handler.return_value = {
"result": {"id": "resp_abc123"},
"kwargs": {
"response_cost": 0.0001,
"model": "gpt-4o",
"custom_llm_provider": "openai",
},
}
mock_httpx_response = MagicMock(spec=httpx.Response)
mock_httpx_response.text = (
'{"id": "resp_abc123", "object": "response", '
'"output": [], "usage": {"input_tokens": 5, "output_tokens": 3}}'
)
mock_logging_obj = AsyncMock()
mock_logging_obj.model_call_details = {}
mock_logging_obj.async_success_handler = AsyncMock()
passthrough_payload = PassthroughStandardLoggingPayload(
url="https://api.openai.com/v1/responses",
request_body={"model": "gpt-4o", "input": "Hello"},
request_method="POST",
)
await self.handler.pass_through_async_success_handler(
httpx_response=mock_httpx_response,
response_body={
"id": "resp_abc123",
"object": "response",
"output": [],
"usage": {"input_tokens": 5, "output_tokens": 3},
},
logging_obj=mock_logging_obj,
url_route="https://api.openai.com/v1/responses",
result="",
start_time=datetime.now(),
end_time=datetime.now(),
cache_hit=False,
request_body={"model": "gpt-4o", "input": "Hello"},
passthrough_logging_payload=passthrough_payload,
)
# The OpenAI handler MUST have been invoked. Pre-fix the dispatch
# gate filtered Responses URLs out and the mock was never called.
mock_openai_handler.assert_called_once()
# And we can verify it was dispatched with the Responses URL.
call_kwargs = mock_openai_handler.call_args.kwargs
assert call_kwargs["url_route"] == "https://api.openai.com/v1/responses"
@patch(
"litellm.proxy.pass_through_endpoints.llm_provider_handlers.openai_passthrough_logging_handler.OpenAIPassthroughLoggingHandler.openai_passthrough_handler"
)
tests/test_litellm/proxy/pass_through_endpoints/test_llm_pass_through_endpoints.py
+77 -3
View File
@@ -24,11 +24,13 @@ from litellm.proxy.pass_through_endpoints.llm_passthrough_endpoints import (
get_vertex_base_url,
llm_passthrough_factory_proxy_route,
milvus_proxy_route,
mistral_proxy_route,
openai_proxy_route,
vertex_discovery_proxy_route,
vertex_proxy_route,
vllm_proxy_route,
)
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.passthrough_endpoints.vertex_ai import VertexPassThroughCredentials
@@ -1092,9 +1094,9 @@ class TestVertexAIPassThroughHandler:
assert result is not None
assert result["result"] is not None
assert result["kwargs"].get("custom_llm_provider") == "gemini", (
"Google AI Studio embedContent URLs must set custom_llm_provider=gemini, not vertex_ai"
)
assert (
result["kwargs"].get("custom_llm_provider") == "gemini"
), "Google AI Studio embedContent URLs must set custom_llm_provider=gemini, not vertex_ai"
assert result["kwargs"].get("model") == "gemini-embedding-2-preview"
mock_completion_cost.assert_called_once()
@@ -1261,6 +1263,78 @@ async def test_is_streaming_request_fn():
assert await is_streaming_request_fn(mock_request) is True
@pytest.mark.asyncio
async def test_mistral_passthrough_accepts_multipart_without_json_parsing():
boundary = "----litellm-test-boundary"
body = (
f"--{boundary}\r\n"
'Content-Disposition: form-data; name="purpose"\r\n\r\n'
"ocr\r\n"
f"--{boundary}\r\n"
'Content-Disposition: form-data; name="file"; filename="document.pdf"\r\n'
"Content-Type: application/pdf\r\n\r\n"
"%PDF-1.4 test\r\n"
f"--{boundary}--\r\n"
).encode("utf-8")
async def receive():
return {
"type": "http.request",
"body": body,
"more_body": False,
}
request = Request(
{
"type": "http",
"method": "POST",
"path": "/mistral/v1/files",
"headers": [
(
b"content-type",
f"multipart/form-data; boundary={boundary}".encode("utf-8"),
)
],
"query_string": b"",
},
receive=receive,
)
captured_kwargs = {}
async def fake_endpoint(request, fastapi_response, user_api_key_dict):
return {"ok": True}
def fake_create_pass_through_route(**kwargs):
captured_kwargs.update(kwargs)
return fake_endpoint
user_api_key_dict = UserAPIKeyAuth(token="test-key")
with (
patch(
"litellm.proxy.pass_through_endpoints.llm_passthrough_endpoints.passthrough_endpoint_router.get_credentials",
return_value="mistral-test-key",
),
patch(
"litellm.proxy.pass_through_endpoints.llm_passthrough_endpoints.create_pass_through_route",
side_effect=fake_create_pass_through_route,
),
):
response = await mistral_proxy_route(
endpoint="v1/files",
request=request,
fastapi_response=Response(),
user_api_key_dict=user_api_key_dict,
)
assert response == {"ok": True}
assert captured_kwargs["is_streaming_request"] is False
assert captured_kwargs["custom_headers"] == {
"Authorization": "Bearer mistral-test-key"
}
class TestBedrockLLMProxyRoute:
@pytest.mark.asyncio
async def test_bedrock_llm_proxy_route_application_inference_profile(self):
tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py
+3 -2
View File
@@ -1314,7 +1314,8 @@ async def test_ui_view_session_spend_logs_pagination(client, monkeypatch):
assert session_id == "session-123"
assert page_size == 1
assert skip == 1 # page=2, page_size=1
return [mock_spend_logs[1]]
assert 'ORDER BY "startTime" DESC' in sql_query
return [mock_spend_logs[0]]
class MockPrismaClient:
def __init__(self):
@@ -1337,7 +1338,7 @@ async def test_ui_view_session_spend_logs_pagination(client, monkeypatch):
assert data["page_size"] == 1
assert data["total_pages"] == 2
assert len(data["data"]) == 1
assert data["data"][0]["request_id"] == "req2"
assert data["data"][0]["request_id"] == "req1"
@pytest.mark.asyncio
tests/test_litellm/proxy/test_common_request_processing.py
+30
View File
@@ -2269,6 +2269,36 @@ class TestHandleLLMApiExceptionDictDetail:
assert proxy_exc.message == "Content blocked by guardrail"
assert proxy_exc.provider_specific_fields is None
async def test_not_found_error_preserves_404(self):
"""NotFoundError with status_code=404 should map to ProxyException code=404."""
from litellm.exceptions import NotFoundError
exc = NotFoundError(
message="Model gemini-3.1-flash-lite-preview not found",
model="gemini-3.1-flash-lite-preview",
llm_provider="gemini",
)
proxy_exc = await self._invoke(exc)
assert proxy_exc.code == "404"
assert "NotFoundError" in proxy_exc.message
async def test_exception_with_status_code_propagates(self):
"""Exception with a statically-set status_code should propagate it."""
from litellm.llms.vertex_ai.common_utils import VertexAIError
exc = VertexAIError(
status_code=429,
message="Rate limit exceeded",
)
proxy_exc = await self._invoke(exc)
assert proxy_exc.code == "429"
async def test_exception_without_status_code_defaults_to_500(self):
"""Exception with no status_code attribute defaults to 500."""
exc = ValueError("Something broke")
proxy_exc = await self._invoke(exc)
assert proxy_exc.code == "500"
class TestAsyncStreamingDataGeneratorFastPath:
"""Fast/slow path branching in async_streaming_data_generator."""
tests/test_litellm/proxy/test_proxy_server.py
+30
View File
@@ -2319,6 +2319,36 @@ async def test_custom_ui_sso_sign_in_handler_config_loading():
os.unlink(config_file_path)
@pytest.mark.asyncio
async def test_load_config_max_budget_env_var_coerced_to_float(tmp_path, monkeypatch):
"""
max_budget configured as os.environ/MAX_BUDGET resolves to a string;
load_config must coerce it to float so the startup check
`litellm.max_budget > 0` doesn't raise TypeError.
"""
from litellm.proxy.proxy_server import ProxyConfig
monkeypatch.setenv("MAX_BUDGET", "10")
test_config = {
"model_list": [],
"litellm_settings": {"max_budget": "os.environ/MAX_BUDGET"},
}
config_file = tmp_path / "config.yaml"
config_file.write_text(yaml.dump(test_config))
original_max_budget = litellm.max_budget
try:
proxy_config = ProxyConfig()
await proxy_config.load_config(
router=MagicMock(), config_file_path=str(config_file)
)
assert isinstance(litellm.max_budget, float)
assert litellm.max_budget == 10.0
assert litellm.max_budget > 0
finally:
litellm.max_budget = original_max_budget
@pytest.mark.asyncio
async def test_load_environment_variables_direct_and_os_environ():
"""
tests/test_litellm/router_utils/pre_call_checks/test_encrypted_content_affinity_check.py
+188
View File
@@ -1471,3 +1471,191 @@ async def test_affinity_does_not_raise_when_boundary_peer_available():
assert result == [peer]
assert request_kwargs.get("_encrypted_content_affinity_pinned") is True
@pytest.mark.asyncio
async def test_model_group_affinity_config_enables_encrypted_content_affinity():
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
EncryptedContentAffinityCheck,
)
model_group = "openai.gpt-5.1-codex"
target_deployment = {
"model_name": model_group,
"litellm_params": {"model": "openai/gpt-5.1-codex"},
"model_info": {"id": "deployment-b"},
}
healthy_deployments = [
{
"model_name": model_group,
"litellm_params": {"model": "openai/gpt-5.1-codex"},
"model_info": {"id": "deployment-a"},
},
target_deployment,
]
encoded_id = ResponsesAPIRequestUtils._build_encrypted_item_id(
"deployment-b", "rs_test"
)
request_kwargs = {
"input": [{"type": "reasoning", "id": encoded_id}],
"litellm_metadata": {},
}
check = EncryptedContentAffinityCheck(
enable_global_affinity=False,
model_group_affinity_config={
model_group: ["encrypted_content_affinity"],
},
)
filtered = await check.async_filter_deployments(
model=model_group,
healthy_deployments=healthy_deployments,
messages=None,
request_kwargs=request_kwargs,
)
assert filtered == [target_deployment]
assert request_kwargs["litellm_metadata"]["encrypted_content_affinity_enabled"]
assert request_kwargs.get("_encrypted_content_affinity_pinned") is True
@pytest.mark.asyncio
async def test_model_group_affinity_config_does_not_disable_global_encrypted_content_affinity():
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
EncryptedContentAffinityCheck,
)
model_group = "openai.gpt-5.1-codex"
target_deployment = {
"model_name": model_group,
"litellm_params": {"model": "openai/gpt-5.1-codex"},
"model_info": {"id": "deployment-b"},
}
healthy_deployments = [
{
"model_name": model_group,
"litellm_params": {"model": "openai/gpt-5.1-codex"},
"model_info": {"id": "deployment-a"},
},
target_deployment,
]
encoded_id = ResponsesAPIRequestUtils._build_encrypted_item_id(
"deployment-b", "rs_test"
)
request_kwargs = {
"input": [{"type": "reasoning", "id": encoded_id}],
"litellm_metadata": {},
}
check = EncryptedContentAffinityCheck(
enable_global_affinity=True,
model_group_affinity_config={
model_group: ["deployment_affinity"],
},
)
filtered = await check.async_filter_deployments(
model=model_group,
healthy_deployments=healthy_deployments,
messages=None,
request_kwargs=request_kwargs,
)
assert filtered == [target_deployment]
assert request_kwargs["litellm_metadata"]["encrypted_content_affinity_enabled"]
assert request_kwargs.get("_encrypted_content_affinity_pinned") is True
@pytest.mark.asyncio
async def test_model_group_encrypted_content_affinity_overrides_global_deployment_affinity():
from litellm.router_utils.pre_call_checks.deployment_affinity_check import (
DeploymentAffinityCheck,
)
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
EncryptedContentAffinityCheck,
)
model_group = "openai.gpt-5.1-codex"
user_api_key_hash = "test-user-key"
deployment_a = {
"model_name": model_group,
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key-a",
},
"model_info": {"id": "deployment-a"},
}
deployment_b = {
"model_name": model_group,
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key-b",
},
"model_info": {"id": "deployment-b"},
}
router = litellm.Router(
model_list=[deployment_a, deployment_b],
optional_pre_call_checks=["deployment_affinity"],
model_group_affinity_config={
model_group: ["encrypted_content_affinity"],
},
num_retries=0,
)
try:
callbacks = router.optional_callbacks or []
deployment_callback = next(
cb for cb in callbacks if isinstance(cb, DeploymentAffinityCheck)
)
encrypted_content_callback = next(
cb for cb in callbacks if isinstance(cb, EncryptedContentAffinityCheck)
)
assert callbacks.index(encrypted_content_callback) < callbacks.index(
deployment_callback
)
assert encrypted_content_callback.enable_global_affinity is False
cache_key = DeploymentAffinityCheck.get_affinity_cache_key(
model_group=model_group,
user_key=user_api_key_hash,
)
await deployment_callback.cache.async_set_cache(
key=cache_key,
value={"model_id": "deployment-a"},
ttl=60,
)
encoded_id = ResponsesAPIRequestUtils._build_encrypted_item_id(
"deployment-b", "rs_test"
)
request_kwargs = {
"input": [
{
"type": "reasoning",
"id": encoded_id,
"encrypted_content": "gAAAAABpnW_yEYmSNEyOG...",
}
],
"metadata": {"user_api_key_hash": user_api_key_hash},
"litellm_metadata": {},
}
after_deployment_affinity = await deployment_callback.async_filter_deployments(
model=model_group,
healthy_deployments=[deployment_a, deployment_b],
messages=None,
request_kwargs=request_kwargs,
)
assert after_deployment_affinity == [deployment_a, deployment_b]
after_encrypted_content_affinity = (
await encrypted_content_callback.async_filter_deployments(
model=model_group,
healthy_deployments=after_deployment_affinity,
messages=None,
request_kwargs=request_kwargs,
)
)
assert after_encrypted_content_affinity == [deployment_b]
assert request_kwargs.get("_encrypted_content_affinity_pinned") is True
finally:
router.discard()
tests/test_litellm/test_anthropic_beta_headers_filtering.py
+10
View File
@@ -402,6 +402,16 @@ class TestAnthropicBetaHeadersFiltering:
test_case["expected"] in filtered
), f"Header '{test_case['input']}' should be mapped to '{test_case['expected']}' for {test_case['provider']}, but got: {filtered}"
def test_filter_and_transform_beta_headers_vertex_ai_keeps_compact(self):
"""Vertex AI supports compact context edits, so the compact beta header
must be forwarded instead of stripped (it was previously mapped to null,
which broke compact_20260112 context edits over /v1/messages)."""
filtered = filter_and_transform_beta_headers(
beta_headers=["compact-2026-01-12"], provider="vertex_ai"
)
assert filtered == ["compact-2026-01-12"]
def test_null_value_headers_filtered(self):
"""Test that headers with null values are always filtered out."""
for provider in [
tests/test_litellm/test_router.py
+292
View File
@@ -80,6 +80,256 @@ def test_router_with_model_info_and_model_group():
)
def test_router_model_group_encrypted_content_affinity_callback_registration():
from litellm.router_utils.pre_call_checks.deployment_affinity_check import (
DeploymentAffinityCheck,
)
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
EncryptedContentAffinityCheck,
)
model_group = "openai.gpt-5.1-codex"
model_group_affinity_config = {
model_group: ["encrypted_content_affinity"],
}
original_callbacks = list(litellm.callbacks)
litellm.callbacks = []
router = None
try:
router = litellm.Router(
model_list=[
{
"model_name": model_group,
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key",
},
}
],
model_group_affinity_config=model_group_affinity_config,
num_retries=0,
)
callbacks = router.optional_callbacks or []
encrypted_content_callbacks = [
cb for cb in callbacks if isinstance(cb, EncryptedContentAffinityCheck)
]
deployment_callback = next(
cb for cb in callbacks if isinstance(cb, DeploymentAffinityCheck)
)
assert len(encrypted_content_callbacks) == 1
assert encrypted_content_callbacks[0].enable_global_affinity is False
assert (
encrypted_content_callbacks[0].model_group_affinity_config
== model_group_affinity_config
)
assert callbacks.index(encrypted_content_callbacks[0]) < callbacks.index(
deployment_callback
)
assert litellm.callbacks.index(encrypted_content_callbacks[0]) < (
litellm.callbacks.index(deployment_callback)
)
router._add_encrypted_content_affinity_check(enable_global_affinity=True)
callbacks = router.optional_callbacks or []
encrypted_content_callbacks = [
cb for cb in callbacks if isinstance(cb, EncryptedContentAffinityCheck)
]
assert len(encrypted_content_callbacks) == 1
assert encrypted_content_callbacks[0].enable_global_affinity is True
assert encrypted_content_callbacks[0].router is router
finally:
if router is not None:
router.discard()
litellm.callbacks = original_callbacks
@pytest.mark.asyncio
async def test_encrypted_content_affinity_model_group_config_is_additive():
from litellm.responses.utils import ResponsesAPIRequestUtils
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
EncryptedContentAffinityCheck,
)
model_group = "openai.gpt-5.1-codex"
target_deployment = {
"model_name": model_group,
"litellm_params": {"model": "openai/gpt-5.1-codex"},
"model_info": {"id": "deployment-b"},
}
healthy_deployments = [
{
"model_name": model_group,
"litellm_params": {"model": "openai/gpt-5.1-codex"},
"model_info": {"id": "deployment-a"},
},
target_deployment,
]
encoded_id = ResponsesAPIRequestUtils._build_encrypted_item_id(
"deployment-b", "rs_test"
)
assert EncryptedContentAffinityCheck.has_model_group_affinity_enabled(
{model_group: ["encrypted_content_affinity"]}
)
assert not EncryptedContentAffinityCheck.has_model_group_affinity_enabled(None)
per_group_check = EncryptedContentAffinityCheck(
enable_global_affinity=False,
model_group_affinity_config={
model_group: ["encrypted_content_affinity"],
},
)
request_kwargs = {
"input": [{"type": "reasoning", "id": encoded_id}],
"litellm_metadata": {},
}
filtered = await per_group_check.async_filter_deployments(
model=model_group,
healthy_deployments=healthy_deployments,
messages=None,
request_kwargs=request_kwargs,
)
assert filtered == [target_deployment]
assert request_kwargs["litellm_metadata"]["encrypted_content_affinity_enabled"]
disabled_check = EncryptedContentAffinityCheck(
enable_global_affinity=False,
model_group_affinity_config={
"other-model-group": ["encrypted_content_affinity"],
},
)
disabled_request_kwargs = {
"input": [{"type": "reasoning", "id": encoded_id}],
"litellm_metadata": {},
}
unfiltered = await disabled_check.async_filter_deployments(
model=model_group,
healthy_deployments=healthy_deployments,
messages=None,
request_kwargs=disabled_request_kwargs,
)
assert unfiltered == healthy_deployments
assert "encrypted_content_affinity_enabled" not in disabled_request_kwargs[
"litellm_metadata"
]
global_check = EncryptedContentAffinityCheck(
enable_global_affinity=True,
model_group_affinity_config={
model_group: ["deployment_affinity"],
},
)
global_request_kwargs = {
"input": [{"type": "reasoning", "id": encoded_id}],
"litellm_metadata": {},
}
globally_filtered = await global_check.async_filter_deployments(
model=model_group,
healthy_deployments=healthy_deployments,
messages=None,
request_kwargs=global_request_kwargs,
)
assert globally_filtered == [target_deployment]
assert global_request_kwargs["litellm_metadata"][
"encrypted_content_affinity_enabled"
]
@pytest.mark.asyncio
async def test_encrypted_content_affinity_takes_priority_over_user_key_affinity():
from litellm.responses.utils import ResponsesAPIRequestUtils
from litellm.router_utils.pre_call_checks.deployment_affinity_check import (
DeploymentAffinityCheck,
)
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
EncryptedContentAffinityCheck,
)
model_group = "openai.gpt-5.1-codex"
user_api_key_hash = "test-user-key"
deployment_a = {
"model_name": model_group,
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key-a",
},
"model_info": {"id": "deployment-a"},
}
deployment_b = {
"model_name": model_group,
"litellm_params": {
"model": "openai/gpt-5.1-codex",
"api_key": "mock-api-key-b",
},
"model_info": {"id": "deployment-b"},
}
original_callbacks = list(litellm.callbacks)
litellm.callbacks = []
router = None
try:
router = litellm.Router(
model_list=[deployment_a, deployment_b],
model_group_affinity_config={
model_group: [
"deployment_affinity",
"encrypted_content_affinity",
],
},
num_retries=0,
)
callbacks = router.optional_callbacks or []
deployment_callback = next(
cb for cb in callbacks if isinstance(cb, DeploymentAffinityCheck)
)
encrypted_content_callback = next(
cb for cb in callbacks if isinstance(cb, EncryptedContentAffinityCheck)
)
assert callbacks.index(encrypted_content_callback) < callbacks.index(
deployment_callback
)
assert litellm.callbacks.index(encrypted_content_callback) < (
litellm.callbacks.index(deployment_callback)
)
cache_key = DeploymentAffinityCheck.get_affinity_cache_key(
model_group=model_group,
user_key=user_api_key_hash,
)
await deployment_callback.cache.async_set_cache(
key=cache_key,
value={"model_id": "deployment-a"},
ttl=60,
)
encoded_id = ResponsesAPIRequestUtils._build_encrypted_item_id(
"deployment-b", "rs_test"
)
request_kwargs = {
"input": [{"type": "reasoning", "id": encoded_id}],
"litellm_metadata": {"user_api_key_hash": user_api_key_hash},
}
filtered = await router.async_callback_filter_deployments(
model=model_group,
healthy_deployments=[deployment_a, deployment_b],
messages=None,
parent_otel_span=None,
request_kwargs=request_kwargs,
)
assert filtered == [deployment_b]
assert request_kwargs.get("_encrypted_content_affinity_pinned") is True
finally:
if router is not None:
router.discard()
litellm.callbacks = original_callbacks
@pytest.mark.asyncio
async def test_arouter_with_tags_and_fallbacks():
"""
@@ -4311,6 +4561,48 @@ def test_get_available_deployment_for_pass_through_raises_when_dict_blocked():
)
def test_initialize_deployment_for_pass_through_keeps_bedrock_iam_deployment():
"""
Bedrock deployments using IAM/OIDC auth have no api_key; pass-through
init must not raise and drop them from routing (#27728).
"""
import litellm
router = litellm.Router(
model_list=[
{
"model_name": "bedrock-claude",
"litellm_params": {
"model": "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
"aws_role_name": "arn:aws:iam::123456789012:role/my-role",
"aws_session_name": "my-session",
"use_in_pass_through": True,
},
"model_info": {"id": "bedrock-iam-pt"},
}
]
)
assert [m["model_info"]["id"] for m in router.get_model_list()] == [
"bedrock-iam-pt"
]
def test_initialize_deployment_for_pass_through_sets_credentials_with_api_key():
from litellm.proxy.pass_through_endpoints.llm_passthrough_endpoints import (
passthrough_endpoint_router,
)
passthrough_endpoint_router.credentials.clear()
router = _router_with_two_pass_through_deployments([False, False])
assert len(router.get_model_list()) == 2
assert (
passthrough_endpoint_router.get_credentials(
custom_llm_provider="openai", region_name=None
)
== "sk-fake-for-tests"
)
def test_get_deployment_credentials_returns_none_for_blocked_deployment():
router = _router_with_two_deployments([True, False])
assert router.get_deployment_credentials(model_id="dep-0") is None
tests/test_litellm/test_utils.py
+4
View File
@@ -700,6 +700,7 @@ def test_aaamodel_prices_and_context_window_json_is_valid():
"cache_read_input_token_cost": {"type": "number"},
"cache_read_input_token_cost_above_200k_tokens": {"type": "number"},
"cache_read_input_token_cost_above_272k_tokens": {"type": "number"},
"cache_read_input_token_cost_above_512k_tokens": {"type": "number"},
"cache_read_input_token_cost_batches": {"type": "number"},
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": {
"type": "number"
@@ -721,6 +722,7 @@ def test_aaamodel_prices_and_context_window_json_is_valid():
"input_cost_per_token_above_200k_tokens": {"type": "number"},
"input_cost_per_token_above_256k_tokens": {"type": "number"},
"input_cost_per_token_above_272k_tokens": {"type": "number"},
"input_cost_per_token_above_512k_tokens": {"type": "number"},
"cache_read_input_token_cost_flex": {"type": "number"},
"cache_read_input_token_cost_priority": {"type": "number"},
"cache_read_input_token_cost_above_200k_tokens_priority": {
@@ -811,6 +813,7 @@ def test_aaamodel_prices_and_context_window_json_is_valid():
"output_cost_per_token_above_200k_tokens": {"type": "number"},
"output_cost_per_token_above_256k_tokens": {"type": "number"},
"output_cost_per_token_above_272k_tokens": {"type": "number"},
"output_cost_per_token_above_512k_tokens": {"type": "number"},
"output_cost_per_image_above_1024_and_1024_pixels": {"type": "number"},
"output_cost_per_image_above_1024_and_1024_pixels_and_premium_image": {
"type": "number"
@@ -932,6 +935,7 @@ def test_aaamodel_prices_and_context_window_json_is_valid():
"supports_native_streaming": {"type": "boolean"},
"supports_image_size": {"type": "boolean"},
"supports_native_structured_output": {"type": "boolean"},
"use_openai_responses_path": {"type": "boolean"},
"tiered_pricing": {
"type": "array",
"items": {
ui/litellm-dashboard/src/components/networking.test.ts
+47
View File
@@ -467,3 +467,50 @@ describe("teamInfoCall", () => {
expect(parsed.searchParams.has("team_id")).toBe(false);
});
});
describe("sessionSpendLogsCall", () => {
const originalFetch = global.fetch;
beforeEach(() => {
vi.clearAllMocks();
});
afterEach(() => {
global.fetch = originalFetch;
});
it("should request the first page with defaults so the caller can page through the session", async () => {
const mockFetch = vi.fn().mockResolvedValue({
ok: true,
json: vi.fn().mockResolvedValue({ data: [], total: 0, page: 1, page_size: 100, total_pages: 1 }),
} as any);
global.fetch = mockFetch as any;
await Networking.sessionSpendLogsCall("token", "session-123");
expect(mockFetch).toHaveBeenCalledOnce();
const [url] = mockFetch.mock.calls[0];
const urlStr = typeof url === "string" ? url : (url as Request).url;
const parsed = typeof url === "string" ? new URL(url, "http://example.com") : new URL((url as Request).url);
expect(urlStr).toContain("/spend/logs/session/ui");
expect(parsed.searchParams.get("session_id")).toBe("session-123");
expect(parsed.searchParams.get("page")).toBe("1");
expect(parsed.searchParams.get("page_size")).toBe("100");
});
it("should pass explicit page and page_size query params for later pages", async () => {
const mockFetch = vi.fn().mockResolvedValue({
ok: true,
json: vi.fn().mockResolvedValue({ data: [], total: 250, page: 3, page_size: 100, total_pages: 3 }),
} as any);
global.fetch = mockFetch as any;
await Networking.sessionSpendLogsCall("token", "session-123", 3, 100);
const [url] = mockFetch.mock.calls[0];
const parsed = typeof url === "string" ? new URL(url, "http://example.com") : new URL((url as Request).url);
expect(parsed.searchParams.get("page")).toBe("3");
expect(parsed.searchParams.get("page_size")).toBe("100");
});
});
ui/litellm-dashboard/src/components/networking.tsx
+18 -4
View File
@@ -5616,13 +5616,27 @@ export const teamPermissionsUpdateCall = async (accessToken: string, teamId: str
};
/**
* Get all spend logs for a particular session
* Get a page of spend logs for a particular session.
*
* The backend paginates this endpoint (page / page_size, returning
* { data, total, page, page_size, total_pages }). Callers that need the whole
* session should page through total_pages and accumulate the results.
*/
export const sessionSpendLogsCall = async (accessToken: string, session_id: string) => {
export const sessionSpendLogsCall = async (
accessToken: string,
session_id: string,
page: number = 1,
page_size: number = 100,
) => {
try {
const params = new URLSearchParams({
session_id,
page: String(page),
page_size: String(page_size),
});
let url = proxyBaseUrl
? `${proxyBaseUrl}/spend/logs/session/ui?session_id=${encodeURIComponent(session_id)}`
: `/spend/logs/session/ui?session_id=${encodeURIComponent(session_id)}`;
? `${proxyBaseUrl}/spend/logs/session/ui?${params.toString()}`
: `/spend/logs/session/ui?${params.toString()}`;
const response = await fetch(url, {
method: "GET",
ui/litellm-dashboard/src/components/view_logs/LogDetailsDrawer/LogDetailsDrawer.tsx
+76 -12
View File
@@ -28,6 +28,14 @@ export interface LogDetailsDrawerProps {
const SIDEBAR_WIDTH_PX = 224;
// Session logs are fetched page-by-page from the paginated backend and
// accumulated so the drawer can show the whole session. page_size is the
// backend maximum (le=100); the page cap bounds the fetch and the
// (un-virtualized) sidebar list for pathological sessions, keeping the most
// recent logs since the endpoint returns newest-first.
const SESSION_PAGE_SIZE = 100;
const MAX_SESSION_PAGES = 50;
/* ------------------------------------------------------------------ */
/* TraceEventRow — compact event row used in both session & non- */
/* session sidebar lists. Extracted to avoid JSX duplication. */
@@ -112,13 +120,39 @@ export function LogDetailsDrawer({
const [isSidebarCollapsed, setIsSidebarCollapsed] = useState(false);
const [copiedLeftPanelId, setCopiedLeftPanelId] = useState(false);
const { data: sessionLogs = [] } = useQuery({
const { data: sessionData } = useQuery({
queryKey: ["sessionLogs", sessionId],
queryFn: async () => {
if (!sessionId || !accessToken) return [];
const response = await sessionSpendLogsCall(accessToken, sessionId);
const allSessionLogs: LogEntry[] = response.data || response || [];
return allSessionLogs
if (!sessionId || !accessToken) return { logs: [] as LogEntry[], total: 0 };
// Fetch the first page, then page through the rest so sessions with more
// than one page of logs are shown in full (capped for safety).
const firstPage = await sessionSpendLogsCall(accessToken, sessionId, 1, SESSION_PAGE_SIZE);
let rows: LogEntry[] = firstPage.data || firstPage || [];
const pagesToFetch = Math.min(firstPage.total_pages ?? 1, MAX_SESSION_PAGES);
if (pagesToFetch > 1) {
const BATCH = 5;
const remaining: Awaited<ReturnType<typeof sessionSpendLogsCall>>[] = [];
for (let start = 2; start <= pagesToFetch; start += BATCH) {
const end = Math.min(start + BATCH - 1, pagesToFetch);
const batch = await Promise.all(
Array.from({ length: end - start + 1 }, (_, i) =>
sessionSpendLogsCall(accessToken, sessionId, start + i, SESSION_PAGE_SIZE),
),
);
remaining.push(...batch);
}
for (const page of remaining) {
rows = rows.concat(page.data || []);
}
}
// Fall back to the accumulated row count (not just the first page) when the
// backend omits total, so the truncation note reflects what was fetched.
const total: number = firstPage.total ?? rows.length;
const logs = rows
.map((row) => ({
...row,
request_duration_ms: row.request_duration_ms ?? Date.parse(row.endTime) - Date.parse(row.startTime),
@@ -127,24 +161,49 @@ export function LogDetailsDrawer({
const aIsMcp = MCP_CALL_TYPES.includes(a.call_type) ? 1 : 0;
const bIsMcp = MCP_CALL_TYPES.includes(b.call_type) ? 1 : 0;
if (aIsMcp !== bIsMcp) return aIsMcp - bIsMcp;
return new Date(a.startTime).getTime() - new Date(b.startTime).getTime();
// Newest first, matching the all-sessions logs overview. MCP calls
// stay grouped last (above), newest-first within that group too.
return new Date(b.startTime).getTime() - new Date(a.startTime).getTime();
});
return { logs, total };
},
enabled: Boolean(open && isSessionMode && sessionId && accessToken),
});
const sessionLogs: LogEntry[] = sessionData?.logs ?? [];
// total reported by the backend; when the page cap truncates the fetch this
// exceeds sessionLogs.length, which drives the "showing most recent" note.
const sessionTotalCount = sessionData?.total ?? sessionLogs.length;
const sessionTruncated = sessionTotalCount > sessionLogs.length;
// Default selection for a freshly opened session: the most recent log (latest
// startTime). The list is sorted newest-first, but MCP calls are grouped last,
// so the latest log by time is not necessarily sessionLogs[0]; compute it
// explicitly. A clicked/remembered log still wins over this default.
const mostRecentLog = useMemo<LogEntry | null>(
() =>
sessionLogs.reduce<LogEntry | null>(
(latest, row) =>
!latest || new Date(row.startTime).getTime() > new Date(latest.startTime).getTime() ? row : latest,
null,
),
[sessionLogs],
);
const currentLog = useMemo(() => {
if (!isSessionMode) return logEntry;
if (!sessionLogs.length) return null;
const fallbackLog = mostRecentLog ?? sessionLogs[0];
if (selectedSessionRequestId) {
return sessionLogs.find((row) => row.request_id === selectedSessionRequestId) || sessionLogs[0];
return sessionLogs.find((row) => row.request_id === selectedSessionRequestId) || fallbackLog;
}
if (logEntry?.request_id) {
const clickedLog = sessionLogs.find((row) => row.request_id === logEntry.request_id);
return clickedLog || sessionLogs[0];
return clickedLog || fallbackLog;
}
return sessionLogs[0];
}, [isSessionMode, logEntry, selectedSessionRequestId, sessionLogs]);
return fallbackLog;
}, [isSessionMode, logEntry, selectedSessionRequestId, sessionLogs, mostRecentLog]);
useEffect(() => {
if (!isSessionMode || !sessionLogs.length) return;
@@ -152,10 +211,10 @@ export function LogDetailsDrawer({
const fallbackRequestId =
logEntry?.request_id && sessionLogs.some((row) => row.request_id === logEntry.request_id)
? logEntry.request_id
: sessionLogs[0].request_id;
: (mostRecentLog ?? sessionLogs[0]).request_id;
setSelectedSessionRequestId(fallbackRequestId);
}
}, [isSessionMode, logEntry, selectedSessionRequestId, sessionLogs]);
}, [isSessionMode, logEntry, selectedSessionRequestId, sessionLogs, mostRecentLog]);
// Reset transient UI state when the drawer opens or closes.
useEffect(() => {
@@ -327,6 +386,11 @@ export function LogDetailsDrawer({
</>
)}
</div>
{isSessionMode && sessionTruncated && (
<div className="mt-1 text-[11px] text-amber-600 font-mono">
Showing most recent {logsForList.length} of {sessionTotalCount}
</div>
)}
</div>
<div className="flex-1 overflow-y-auto">
ui/litellm-dashboard/src/lib/http/schema.d.ts
Generated Vendored
+12
View File
@@ -25081,6 +25081,12 @@ export interface components {
* @default false
*/
use_litellm_proxy: boolean | null;
/**
* Use Xai Oauth
* @description Use stored xAI OAuth credentials when no xAI API key is configured.
* @default false
*/
use_xai_oauth: boolean | null;
/** Vector Store Id */
vector_store_id?: string | null;
/** Vertex Credentials */
@@ -32679,6 +32685,12 @@ export interface components {
* @default false
*/
use_litellm_proxy: boolean | null;
/**
* Use Xai Oauth
* @description Use stored xAI OAuth credentials when no xAI API key is configured.
* @default false
*/
use_xai_oauth: boolean | null;
/** Vector Store Id */
vector_store_id?: string | null;
/** Vertex Credentials */