- Log provider token counting failures instead of silently swallowing
- Fall back to local tokenizer when provider returns error response
- Map assistant tool_calls to Responses API function_call items
- Concatenate multiple system messages instead of overwriting
- Hide internal error details from proxy API responses
- Narrow exception catch in handler to network/JSON errors only
- Update test to match new fallback behavior
- Add OpenAITokenCounter using POST /v1/responses/input_tokens endpoint
- Add litellm.acount_tokens() public async API that auto-routes to provider APIs
- Add proxy endpoint POST /v1/responses/input_tokens for OpenAI-compatible counting
- Transform chat tools format to Responses API format for correct token counting
- Fall back to local tiktoken when provider API unavailable
Fixes#22302
Providers like Cerebras return delta.reasoning in streaming responses
for gpt-oss models, but LiteLLM's Delta class expects reasoning_content.
This causes reasoning content to be silently dropped during streaming.
Fixes#13300
Add global media_resolution support for Gemini 2.x models (2.0, 2.5) when
using OpenAI's detail parameter on images. Previously, the detail parameter
was only working for Gemini 3+ models (per-part) and was silently ignored
for older Gemini models.
- Add _get_highest_media_resolution() and _extract_max_media_resolution_from_messages()
to extract highest detail from all images/files in a request
- Update _transform_request_body() to add mediaResolution to generationConfig
for Gemini 2.x models only (not 1.x which doesn't support it, not 3+ which
uses per-part)
- Add mediaResolution field to GenerationConfig TypedDict
- Support detail extraction from both image_url and file content types
- Add comprehensive unit tests and update documentation
Add MistralAudioTranscriptionConfig for Mistral's /v1/audio/transcriptions
endpoint, enabling litellm.transcription() with mistral/voxtral-mini-latest
and other Voxtral models. Supports multipart form-data with OpenAI-compatible
params (language, temperature, response_format, timestamp_granularities)
plus Mistral-specific params like diarize.
* add explicit caching to litellm proxy for gemini models via injection
* fix: add missing `supports_function_calling` for deepinfra models
All 55 deepinfra models that had `supports_tool_choice: true` were
missing the `supports_function_calling` flag, causing
`litellm.supports_function_calling()` to incorrectly return False.
Fixes#22619
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Managed batches - Address PR bot comments from #22464
* feat(togetherai): add support for TogetherAI Qwen3.5-397B-A17B model
* Agent Tracing - support context_id based trace id propogation + nested llm calls (#22626)
* style(ui/): distinguish agent calls from llm calls on ui
* feat: initial grouping working
* feat: set stable contextid for a2a calls - allows for easily passing to downstream llm/mcp calls
* feat(a2a_endpoints.py): fix tracing to avoid recreating logging objects for the same call
allows stable trace id usage
* fix(guardrail_endpoints): handle string ui_type values in _build_field_dict
_build_field_dict unconditionally called .value on ui_type, which crashes
for guardrail configs that use plain strings (e.g. BlockCodeExecutionGuardrailConfigModel
uses "multiselect" and "percentage"). Now checks with hasattr before calling .value.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: propagate trace/session id from headers in MCP server calls
Cherry-picked mcp_server/server.py fixes from 6feb9bab: adds
get_chain_id_from_headers to extract x-litellm-trace-id /
x-litellm-session-id from raw headers, and uses it in call_tool
and list_tools to keep spend logs and tracing consistent with A2A.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [Feat] UI - Add Open in New Tab on leftnav Bar (#22731)
* Add minimal dev_config.yaml for proxy development
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat(ui): wrap left nav items in <a> tags for open-in-new-tab support
Nav items are now rendered as <a> elements with proper href attributes,
enabling right-click → 'Open in new tab', Ctrl/Cmd+click, and
middle-click to open any sidebar page in a new browser tab.
Normal clicks continue to use SPA navigation (no full page reload).
Applied to both leftnav.tsx (query-param routing) and Sidebar2.tsx
(Next.js file-based routing).
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* [Feat] Add Tool Policies for AI Gateway (#22732)
* fix: fix ui render
* fix: fix minor bugs
* refactor: use prisma functions instead of raw sql (safer)
* fix(add-new-tiles-to-tool-policies): allow developer to see what's available
* feat: ensure tool allowlist runs correctly for tool names + mcp's
* refactor: more ui improvements
* feat: working key tool blocking
* feat(tools): show tool logs
* refactor: backend code improvements
* refactor: improve log viewer for tools
* fix: address PR review feedback for tool access control
- Add missing blocked_tools column to root schema.prisma (schema drift)
- Invalidate ToolPolicyRegistry after policy mutations so changes take effect immediately
- Remove dead code: unused get_effective_policies, get_tool_policies_cached, and helpers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: race condition in permission resolution and remove duplicate allowlist check
- Use atomic update_many with object_permission_id=None to prevent concurrent
requests from creating orphaned permission rows and losing tool blocks
- Remove duplicate allowed_tools enforcement from guardrail (already enforced
in auth layer via check_tools_allowlist)
- Move inline uuid import to module level
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* update to account for userAgent
* UI - Add ToolDetails
* input/output policy
* LiteLLM_PolicyAttachmentTable
* LiteLLM_PolicyAttachmentTable
* fix: add _enqueue_tool_registry_upsert
* fix: tool mgmt endpoints
* tool mgmt endpoints
* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix: sync root schema.prisma and fix test_tool_registry_writer for input/output policy
- Migrate root schema.prisma LiteLLM_ToolTable from call_policy to
input_policy/output_policy, add missing user_agent and last_used_at columns
(now consistent with litellm/proxy/schema.prisma and litellm-proxy-extras)
- Fix SpendLogToolIndex comment across all three schema files
- Fix all call_policy references in test_tool_registry_writer.py:
swapped update_tool_policy arguments, wrong get_tools_by_names return type
assertions, _mock_tool_row setting call_policy instead of input_policy
Addresses Greptile review feedback on PR #22732.
Made-with: Cursor
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(proxy): add key_alias, key_hash, requested_model DD APM span tags (#22710)
* feat(proxy): add key_alias, key_hash, requested_model tags to DD APM spans
* refactor(proxy): consolidate DD APM tag helpers into DDSpanTagger class
* refactor(proxy): move DDSpanTagger to its own file litellm/proxy/dd_span_tagger.py
---------
Co-authored-by: liweiguang <codingpunk@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ephrim Stanley <ephrim.stanley@point72.com>
Co-authored-by: Varad Khonde <varadkhonde@gmail.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
The route-level auth check was blocking internal_user role (team admins)
from reaching /key/{key}/reset_spend because KEY_RESET_SPEND was missing
from key_management_routes. Added it so team admins pass the route check
and the endpoint's existing _check_proxy_or_team_admin_for_key enforces
actual authorization.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The _resolve_model_for_cost_lookup function was only checking
litellm_params.model when resolving model names from the router.
For Azure custom deployment names (e.g. azure/openai/gpt-5.3-codex),
this deployment name doesn't exist in the model cost map, so cost
returned /bin/zsh.
Now checks model_info.base_model and litellm_params.base_model first,
falling back to litellm_params.model only if no base_model is set.
This matches how the router resolves base_model everywhere else.
The _safe_get_request_headers caching (commit e7175a52) uses
request.state._cached_headers. With Mock(spec=Request), getattr on
state returns a Mock (truthy), causing RedactedDict to receive a Mock
instead of a dict. Using a real starlette State object fixes this.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Address Greptile review: test_resolve_jwks_url_resolves_oidc_discovery_document
also used the inconsistent patch.object pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Vertex AI / Gemini uses Pydantic's model_json_schema() which omits
additionalProperties: False (Gemini rejects it). The test expected
the same schema for all providers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The patch.object with new_callable=AsyncMock can behave inconsistently
across Python versions, causing mock_response.status_code to return a
MagicMock instead of the assigned value. Direct assignment is simpler
and more reliable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The CompletionTokensDetailsWrapper type now includes video_tokens field,
but this test's expected dict was not updated to include it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Gemini API requires role="user" on function_response content blocks
(added in commit 273cf12afa), but these tests were never updated to match.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two independent fixes for pre-existing test failures on main:
1. Anthropic streaming: The sync __next__ method used a simple
holding_chunk pattern that lost chunks when multiple events needed
to be returned. Refactored to use the same chunk_queue approach as
the async __anext__ method. Also fixed tests that used ModelResponse
(which defaults finish_reason to 'stop') instead of ModelResponseStream.
2. Azure GPT-5.1 logprobs: The base OpenAI class includes logprobs for
gpt-5.1+ models, but Azure hasn't verified support for gpt-5.1.
Added explicit removal of logprobs/top_logprobs for gpt-5.1 (non-5.2)
models in the Azure config.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Only release distributed lock in finally if it was actually acquired;
prevents spurious Redis release_lock calls on early returns
- Treat bare integer maximum_spend_logs_retention_period as days (e.g. 3 → "3d")
instead of silently failing with a ValueError
- Elevate "Skipping cleanup" log from info to error so misconfigured
retention settings are visible without verbose logging
- Add tests for all three fixes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add created_at field to MCPServer type (was missing)
- Map created_at from LiteLLM_MCPServerTable in build_mcp_server_from_table()
- Use server.created_at and server.updated_at instead of datetime.now() in _build_mcp_server_table() and health check table builder
- Add regression tests to verify timestamps are preserved through round-trip conversions
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Custom user-added routes (e.g. /ldap/ngs/ready) used with Depends(user_api_key_auth) were being rejected as admin-only after _run_post_custom_auth_checks was introduced in commit 14badde13c.
The route authorization check in common_checks is designed for LiteLLM's own management routes. Custom auth flows that add their own routes should be trusted since the custom auth function already validated the request. Budget and expiry checks still run.
Add skip_route_check parameter to common_checks() and pass skip_route_check=True from _run_post_custom_auth_checks() to skip route authorization while preserving budget/team/model checks.
Regression test added: test_common_checks_skip_route_check_for_custom_auth
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* feat(guardrails): team-based guardrail registration and approval workflow
Add team-based guardrail submission system where teams can register
Generic Guardrail API guardrails for admin review. Includes:
- POST /guardrails/register endpoint for team-scoped submissions
- Admin review endpoints (list/get/approve/reject submissions)
- Team Guardrails tab in the UI dashboard
- extra_headers support for forwarding client headers to guardrail APIs
- Prisma schema migration for status, submitted_at, reviewed_at fields
- Documentation for team-based guardrails and static/dynamic headers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(guardrails): address review feedback - SSRF, silent failure, redundant query
- Validate api_base URL scheme (http/https only) and hostname in
register_guardrail to prevent SSRF via team submissions
- Return warning field in approve response when in-memory initialization
fails so admins know the guardrail won't work until next sync cycle
- Eliminate redundant DB query in list_guardrail_submissions by fetching
all team guardrails once and deriving both filtered list and summary
counts from the single result set
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(guardrails): add pending_review status guard to reject endpoint
Prevent rejecting already-active or already-rejected guardrails, which
would create a DB/memory inconsistency (active in memory but rejected
in DB). Now mirrors the approve endpoint's status check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Chesars