Document the `summary` optional field in the `thinking` object for the
Anthropic `/v1/messages` adapter, and add a section on summary preservation
when routing to non-Anthropic providers via the adapter.
- Fix F821: add BaseTokenCounter TYPE_CHECKING import in gpt_transformation.py
- Remove duplicate auth invocation in count_response_input_tokens endpoint
- Preserve `strict` field during chat→Responses API tool conversion
- Fix docs tools example to use chat completions format (not Responses API format)
- Return None early for system-only messages to avoid noisy error logs
- Add OpenAITokenCounter using POST /v1/responses/input_tokens endpoint
- Add litellm.acount_tokens() public async API that auto-routes to provider APIs
- Add proxy endpoint POST /v1/responses/input_tokens for OpenAI-compatible counting
- Transform chat tools format to Responses API format for correct token counting
- Fall back to local tiktoken when provider API unavailable
Fixes#22302
Add global media_resolution support for Gemini 2.x models (2.0, 2.5) when
using OpenAI's detail parameter on images. Previously, the detail parameter
was only working for Gemini 3+ models (per-part) and was silently ignored
for older Gemini models.
- Add _get_highest_media_resolution() and _extract_max_media_resolution_from_messages()
to extract highest detail from all images/files in a request
- Update _transform_request_body() to add mediaResolution to generationConfig
for Gemini 2.x models only (not 1.x which doesn't support it, not 3+ which
uses per-part)
- Add mediaResolution field to GenerationConfig TypedDict
- Support detail extraction from both image_url and file content types
- Add comprehensive unit tests and update documentation
Add Mistral to the supported providers list in audio_transcription.md
and add Audio Transcription section to the Mistral provider page with
SDK usage, optional params, diarize support, and proxy configuration.
* azure content enhancement...
* rafactored to increase confidence score
* improvements based on additional feedback
* removed unused import
* Force-split any word longer than max length allowed
* preserve whitespace in text splitting
* moving common initialization to base class
* consolidate enforcement into async_make_request as single point, remove redundant caller-side checks, extract shared init/HTTP logic into base, and fix stale log messages
* clean up
* clean up tests
1. Okta SSO docs (admin_ui_sso.md):
- Rewrite Step 3 to document both Org Auth Server (free) and
Custom Auth Server (paid SKU) as tabbed options
- Add Step 4 for GENERIC_CLIENT_STATE and PKCE configuration
(moved from troubleshooting into the main guide)
- Clarify no_matching_policy error only applies to Custom Auth Server
- Deduplicate troubleshooting section to reference Step 4
2. Custom SSO handler (custom_sso.py + custom_sso.md):
- Replace broken user_info() call with prisma_client.get_data()
- user_info() is a FastAPI route handler requiring Request and
UserAPIKeyAuth params, cannot be called directly
- Keep new_user/add_new_member as commented-out import references
in docs for customers who need them
* add explicit caching to litellm proxy for gemini models via injection
* fix: add missing `supports_function_calling` for deepinfra models
All 55 deepinfra models that had `supports_tool_choice: true` were
missing the `supports_function_calling` flag, causing
`litellm.supports_function_calling()` to incorrectly return False.
Fixes#22619
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Managed batches - Address PR bot comments from #22464
* feat(togetherai): add support for TogetherAI Qwen3.5-397B-A17B model
* Agent Tracing - support context_id based trace id propogation + nested llm calls (#22626)
* style(ui/): distinguish agent calls from llm calls on ui
* feat: initial grouping working
* feat: set stable contextid for a2a calls - allows for easily passing to downstream llm/mcp calls
* feat(a2a_endpoints.py): fix tracing to avoid recreating logging objects for the same call
allows stable trace id usage
* fix(guardrail_endpoints): handle string ui_type values in _build_field_dict
_build_field_dict unconditionally called .value on ui_type, which crashes
for guardrail configs that use plain strings (e.g. BlockCodeExecutionGuardrailConfigModel
uses "multiselect" and "percentage"). Now checks with hasattr before calling .value.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: propagate trace/session id from headers in MCP server calls
Cherry-picked mcp_server/server.py fixes from 6feb9bab: adds
get_chain_id_from_headers to extract x-litellm-trace-id /
x-litellm-session-id from raw headers, and uses it in call_tool
and list_tools to keep spend logs and tracing consistent with A2A.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [Feat] UI - Add Open in New Tab on leftnav Bar (#22731)
* Add minimal dev_config.yaml for proxy development
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat(ui): wrap left nav items in <a> tags for open-in-new-tab support
Nav items are now rendered as <a> elements with proper href attributes,
enabling right-click → 'Open in new tab', Ctrl/Cmd+click, and
middle-click to open any sidebar page in a new browser tab.
Normal clicks continue to use SPA navigation (no full page reload).
Applied to both leftnav.tsx (query-param routing) and Sidebar2.tsx
(Next.js file-based routing).
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* [Feat] Add Tool Policies for AI Gateway (#22732)
* fix: fix ui render
* fix: fix minor bugs
* refactor: use prisma functions instead of raw sql (safer)
* fix(add-new-tiles-to-tool-policies): allow developer to see what's available
* feat: ensure tool allowlist runs correctly for tool names + mcp's
* refactor: more ui improvements
* feat: working key tool blocking
* feat(tools): show tool logs
* refactor: backend code improvements
* refactor: improve log viewer for tools
* fix: address PR review feedback for tool access control
- Add missing blocked_tools column to root schema.prisma (schema drift)
- Invalidate ToolPolicyRegistry after policy mutations so changes take effect immediately
- Remove dead code: unused get_effective_policies, get_tool_policies_cached, and helpers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: race condition in permission resolution and remove duplicate allowlist check
- Use atomic update_many with object_permission_id=None to prevent concurrent
requests from creating orphaned permission rows and losing tool blocks
- Remove duplicate allowed_tools enforcement from guardrail (already enforced
in auth layer via check_tools_allowlist)
- Move inline uuid import to module level
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* update to account for userAgent
* UI - Add ToolDetails
* input/output policy
* LiteLLM_PolicyAttachmentTable
* LiteLLM_PolicyAttachmentTable
* fix: add _enqueue_tool_registry_upsert
* fix: tool mgmt endpoints
* tool mgmt endpoints
* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix: sync root schema.prisma and fix test_tool_registry_writer for input/output policy
- Migrate root schema.prisma LiteLLM_ToolTable from call_policy to
input_policy/output_policy, add missing user_agent and last_used_at columns
(now consistent with litellm/proxy/schema.prisma and litellm-proxy-extras)
- Fix SpendLogToolIndex comment across all three schema files
- Fix all call_policy references in test_tool_registry_writer.py:
swapped update_tool_policy arguments, wrong get_tools_by_names return type
assertions, _mock_tool_row setting call_policy instead of input_policy
Addresses Greptile review feedback on PR #22732.
Made-with: Cursor
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(proxy): add key_alias, key_hash, requested_model DD APM span tags (#22710)
* feat(proxy): add key_alias, key_hash, requested_model tags to DD APM spans
* refactor(proxy): consolidate DD APM tag helpers into DDSpanTagger class
* refactor(proxy): move DDSpanTagger to its own file litellm/proxy/dd_span_tagger.py
---------
Co-authored-by: liweiguang <codingpunk@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ephrim Stanley <ephrim.stanley@point72.com>
Co-authored-by: Varad Khonde <varadkhonde@gmail.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* style(ui/): distinguish agent calls from llm calls on ui
* feat: initial grouping working
* feat: set stable contextid for a2a calls - allows for easily passing to downstream llm/mcp calls
* feat(a2a_endpoints.py): fix tracing to avoid recreating logging objects for the same call
allows stable trace id usage
* fix(guardrail_endpoints): handle string ui_type values in _build_field_dict
_build_field_dict unconditionally called .value on ui_type, which crashes
for guardrail configs that use plain strings (e.g. BlockCodeExecutionGuardrailConfigModel
uses "multiselect" and "percentage"). Now checks with hasattr before calling .value.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: propagate trace/session id from headers in MCP server calls
Cherry-picked mcp_server/server.py fixes from 6feb9bab: adds
get_chain_id_from_headers to extract x-litellm-trace-id /
x-litellm-session-id from raw headers, and uses it in call_tool
and list_tools to keep spend logs and tracing consistent with A2A.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Add OpenRouter image edit docs to both the provider page and the
main image_edits reference page, including supported models, parameter
mappings (size→aspect_ratio, quality→image_size), usage examples,
proxy configuration, and a note about 4K quality model support.
* feat(guardrails): team-based guardrail registration and approval workflow
Add team-based guardrail submission system where teams can register
Generic Guardrail API guardrails for admin review. Includes:
- POST /guardrails/register endpoint for team-scoped submissions
- Admin review endpoints (list/get/approve/reject submissions)
- Team Guardrails tab in the UI dashboard
- extra_headers support for forwarding client headers to guardrail APIs
- Prisma schema migration for status, submitted_at, reviewed_at fields
- Documentation for team-based guardrails and static/dynamic headers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(guardrails): address review feedback - SSRF, silent failure, redundant query
- Validate api_base URL scheme (http/https only) and hostname in
register_guardrail to prevent SSRF via team submissions
- Return warning field in approve response when in-memory initialization
fails so admins know the guardrail won't work until next sync cycle
- Eliminate redundant DB query in list_guardrail_submissions by fetching
all team guardrails once and deriving both filtered list and summary
counts from the single result set
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(guardrails): add pending_review status guard to reject endpoint
Prevent rejecting already-active or already-rejected guardrails, which
would create a DB/memory inconsistency (active in memory but rejected
in DB). Now mirrors the approve endpoint's status check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add CrowdStrike AIDR guardrail hook
* fixup! use apply_guardrail event hook
* fixup! update imports
* fix(guardrails): include AI response in CrowdStrike AIDR output events
Issue:
_build_guard_input_for_response() was:
- Sending only the original user input (messages).
- Not sending the AI provider response.
This fix will:
- Extract response.choices from the ModelResponse object and include them in guard_input payload.
- Thus, ensure AIDR output rules receive the AI-generated content for analysis.
- Fix and update tests.
* fix(guardrails): prevent duplicate input events in CrowdStrike AIDR guardrail
Issue:
The CrowdStrike AIDR guardrail was running on during_call hooks wihtout event_hook configured.
This fix will:
- Set event_hook to ["pre_call", "post_call"] (AIDR admins will control what policy is applied)
This change will:
- Require default_on parameter
- Prevent duplicate API calls to AIDR for the same input
- Avoid unchecked AI provider API calls on during_call hook
* docs: add CrowdStrike AIDR to the list of Guardrails under Integrations
* docs: update CrowdStrike AIDR documentation page
---------
Co-authored-by: Konstantin Lapine <konstantin.lapine@crowdstrike.com>
shivam