Replace per-request `httpx.AsyncClient` with `get_async_httpx_client`
to avoid the +500ms latency penalty from creating new clients per
request. Updates tests to mock the cached client factory.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix genuine regression in responses_api_bridge_check where the second
call assigned to `model_info` instead of `responses_api_model_info`,
preventing gpt-5.4 + tools + reasoning_effort from routing to the
Responses API bridge.
Also update outdated tests:
- Vantage tests: match "csv" file key and use supported column names
- Anthropic caching test: add "type": "custom" to expected tool payload
- Claude Agent SDK test: remove non-deterministic LLM content assertion
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests were outdated after #23472 added pagination (take/order) to find_many
and stale-row cleanup via update_many. Updated assertions to match new call
signatures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- DELETE /vantage/delete now removes the in-memory VantageLogger from
litellm.callbacks via remove_callbacks_by_type, preventing the
scheduler from continuing to fire exports with stale credentials
- Add test_should_cast_decimal_columns_to_float covering the
Decimal→Float64 cast in FocusCsvSerializer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use timedelta(hours=1) instead of replace(hour=hour+1) in _window()
to avoid ValueError when hour=23
- Add test_should_batch_by_row_count covering the >10K rows batching
path (previously only the 2 MB size-limit path was tested)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a pluggable Vantage destination to the existing FOCUS export pipeline,
enabling LiteLLM to export spend data in FOCUS format directly to Vantage's
cost-import API. Supports automatic hourly exports via scheduled background job,
with admin API endpoints for manual control and configuration. Includes CSV
serializer, batching for 10K row / 2MB API limits, and enriched Tags JSON with
team/user/key metadata for Vantage Token Allocation feature.
- Add CSV serializer (FocusCsvSerializer) for FOCUS data
- Add Vantage API destination with automatic batching
- Add VantageLogger that wraps FocusLogger with Vantage defaults
- Add proxy endpoints: /vantage/{init,settings,export,dry-run,delete}
- Register "vantage" callback in logger registry and literal type
- Wire up background job in proxy_server.py startup
- Populate Tags column with JSON metadata (team_id, user_id, user_email, etc.)
- Add 14 unit tests covering serializer, destination, and factory
All tests pass (23 focus tests total, no regressions).
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When guardrails return the full data dict (e.g. guardrails_ai), the
guardrail response logged to spend logs and OTEL traces could contain
data["secret_fields"].raw_headers with plaintext Authorization headers.
This adds a pop("secret_fields") in the guardrail logging path,
matching the existing pattern used by Langfuse and Arize integrations.
Tested: Verified fix removes secret_fields/raw_headers/authorization
from both /spend/logs/ui responses and OTEL trace span attributes.
Addresses 4 critical OpenTelemetry span issues in LiteLLM:
Issue #3: Remove redundant attributes from raw_gen_ai_request spans
- Removed self.set_attributes() call that was duplicating all parent span
attributes (gen_ai.*, metadata.*) onto the raw span
- Raw span now only contains provider-specific llm.{provider}.* attributes
- Reduces storage and eliminates search confusion from duplicate data
Issue #4: Prevent attribute duplication on litellm_proxy_request parent span
- When litellm_request child span exists, removed redundant
set_attributes() call on the parent proxy span
- Child span already carries all attributes; parent duplication doubles
storage and complicates search
Issue #5: Fix orphaned guardrail traces
- Guardrail spans were created with context=None when no parent proxy span
existed, resulting in orphaned root spans (separate trace_id)
- Added _resolve_guardrail_context() helper to ensure guardrails always
have a valid parent (litellm_request or proxy span)
- Applied fix to both _handle_success and _handle_failure paths
Issue #8: Add gen_ai.response.id for embeddings and image generation
- EmbeddingResponse and ImageResponse types don't have provider response IDs
- Added fallback to standard_logging_payload["id"] (litellm call ID) for
correlation across LiteLLM UI, Phoenix traces, and provider logs
- Completions still use provider ID (e.g. "chatcmpl-xxx") when available
Tests added:
- TestRawSpanAttributeIsolation: Verify raw span has no gen_ai/metadata attrs
- TestNoParentSpanDuplication: Verify parent span doesn't get duplicated attrs
- TestGuardrailSpanParenting: Verify guardrails are children (not orphaned)
- TestResponseIdFallback: Verify response ID set for all call types
All existing OTEL tests pass (73 passed, 14 pre-existing protocol failures).
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* add explicit caching to litellm proxy for gemini models via injection
* fix: add missing `supports_function_calling` for deepinfra models
All 55 deepinfra models that had `supports_tool_choice: true` were
missing the `supports_function_calling` flag, causing
`litellm.supports_function_calling()` to incorrectly return False.
Fixes#22619
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Managed batches - Address PR bot comments from #22464
* feat(togetherai): add support for TogetherAI Qwen3.5-397B-A17B model
* Agent Tracing - support context_id based trace id propogation + nested llm calls (#22626)
* style(ui/): distinguish agent calls from llm calls on ui
* feat: initial grouping working
* feat: set stable contextid for a2a calls - allows for easily passing to downstream llm/mcp calls
* feat(a2a_endpoints.py): fix tracing to avoid recreating logging objects for the same call
allows stable trace id usage
* fix(guardrail_endpoints): handle string ui_type values in _build_field_dict
_build_field_dict unconditionally called .value on ui_type, which crashes
for guardrail configs that use plain strings (e.g. BlockCodeExecutionGuardrailConfigModel
uses "multiselect" and "percentage"). Now checks with hasattr before calling .value.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: propagate trace/session id from headers in MCP server calls
Cherry-picked mcp_server/server.py fixes from 6feb9bab: adds
get_chain_id_from_headers to extract x-litellm-trace-id /
x-litellm-session-id from raw headers, and uses it in call_tool
and list_tools to keep spend logs and tracing consistent with A2A.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [Feat] UI - Add Open in New Tab on leftnav Bar (#22731)
* Add minimal dev_config.yaml for proxy development
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat(ui): wrap left nav items in <a> tags for open-in-new-tab support
Nav items are now rendered as <a> elements with proper href attributes,
enabling right-click → 'Open in new tab', Ctrl/Cmd+click, and
middle-click to open any sidebar page in a new browser tab.
Normal clicks continue to use SPA navigation (no full page reload).
Applied to both leftnav.tsx (query-param routing) and Sidebar2.tsx
(Next.js file-based routing).
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* [Feat] Add Tool Policies for AI Gateway (#22732)
* fix: fix ui render
* fix: fix minor bugs
* refactor: use prisma functions instead of raw sql (safer)
* fix(add-new-tiles-to-tool-policies): allow developer to see what's available
* feat: ensure tool allowlist runs correctly for tool names + mcp's
* refactor: more ui improvements
* feat: working key tool blocking
* feat(tools): show tool logs
* refactor: backend code improvements
* refactor: improve log viewer for tools
* fix: address PR review feedback for tool access control
- Add missing blocked_tools column to root schema.prisma (schema drift)
- Invalidate ToolPolicyRegistry after policy mutations so changes take effect immediately
- Remove dead code: unused get_effective_policies, get_tool_policies_cached, and helpers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: race condition in permission resolution and remove duplicate allowlist check
- Use atomic update_many with object_permission_id=None to prevent concurrent
requests from creating orphaned permission rows and losing tool blocks
- Remove duplicate allowed_tools enforcement from guardrail (already enforced
in auth layer via check_tools_allowlist)
- Move inline uuid import to module level
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* update to account for userAgent
* UI - Add ToolDetails
* input/output policy
* LiteLLM_PolicyAttachmentTable
* LiteLLM_PolicyAttachmentTable
* fix: add _enqueue_tool_registry_upsert
* fix: tool mgmt endpoints
* tool mgmt endpoints
* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix: sync root schema.prisma and fix test_tool_registry_writer for input/output policy
- Migrate root schema.prisma LiteLLM_ToolTable from call_policy to
input_policy/output_policy, add missing user_agent and last_used_at columns
(now consistent with litellm/proxy/schema.prisma and litellm-proxy-extras)
- Fix SpendLogToolIndex comment across all three schema files
- Fix all call_policy references in test_tool_registry_writer.py:
swapped update_tool_policy arguments, wrong get_tools_by_names return type
assertions, _mock_tool_row setting call_policy instead of input_policy
Addresses Greptile review feedback on PR #22732.
Made-with: Cursor
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(proxy): add key_alias, key_hash, requested_model DD APM span tags (#22710)
* feat(proxy): add key_alias, key_hash, requested_model tags to DD APM spans
* refactor(proxy): consolidate DD APM tag helpers into DDSpanTagger class
* refactor(proxy): move DDSpanTagger to its own file litellm/proxy/dd_span_tagger.py
---------
Co-authored-by: liweiguang <codingpunk@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ephrim Stanley <ephrim.stanley@point72.com>
Co-authored-by: Varad Khonde <varadkhonde@gmail.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
When standard_logging_object is None (failure case), Langfuse was falling
back to litellm_call_id while the DB used litellm_trace_id as session_id.
This caused the Session ID in LiteLLM logs to not match the trace in
Langfuse. Now Langfuse checks litellm_trace_id first, matching the DB.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The proxy uses async_failure_handler → LangfusePromptManagement.async_log_failure_event(),
which silently returned when standard_logging_object was None. This meant failed LLM calls
never created traces in Langfuse. Remove the early return and fall back to extracting the
error message from kwargs["exception"] when standard_logging_object is unavailable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace simulated test with one that invokes the real
Logging.failure_handler(), mocks LangFuseHandler to capture kwargs,
and asserts original_response is excluded and session_id is preserved.
This ensures the test catches regressions if the production code changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix: The Langfuse failure logging path was passing self.model_call_details
(which includes original_response, potentially a coroutine) instead of the
clean local kwargs copy. This aligns the failure path with the success path
behavior (litellm_logging.py:2956).
Reverted the session_id-as-trace_id approach as it causes trace collisions
in Langfuse (multiple calls in the same session would overwrite each other).
Instead, session_id is correctly used only for Langfuse session grouping via
trace_params["session_id"], while each call retains its own unique trace_id.
Added 4 tests:
- session_id correctly passed as trace session_id (not trace_id)
- session_id preserved for ERROR level (failure) logs
- explicit trace_id takes priority over session_id
- failure path kwargs excludes original_response
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Same bug as team budget: _assemble_user_object fetched user info from DB
but only used budget_reset_at, discarding max_budget. When the key cache
has a stale None for user_max_budget, _safe_get_remaining_budget returns
+Inf. Now falls back to DB max_budget when metadata value is None.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ArizePhoenixLogger now creates spans on its own dedicated TracerProvider
instead of trying to reuse parent spans from the global otel TracerProvider
(which were invisible in Phoenix since they go to a different exporter)
- Auto-initialize ArizePhoenixLogger when otel callback is configured and
Phoenix env vars (PHOENIX_API_KEY, PHOENIX_COLLECTOR_*) are detected
- Use exact type check in get_custom_logger_compatible_class to prevent
ArizePhoenixLogger (subclass) from being returned when looking up otel
- Fix tool_permission guardrail to check non-function tools like
code_interpreter (previously skipped with `type != "function"`)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Set prometheus_emit_stream_label: true in litellm_settings to emit a
stream label (True/False/None) on litellm_proxy_total_requests_metric.
Opt-in to avoid breaking cardinality on existing deployments.
* fix: feat: add litellm_system_prompt support
* feat: support new 'litellm_agent' model provider
* feat: ui/ - new agent builder ui
* fix(anthropic/chat/transformation.py): normalize max_tokens if decimal
* feat(agentbuilderview.tsx): run compliance datasets against litellm agent
Adds per-alert-type digest mode that aggregates duplicate alerts
within a configurable time window and emits a single summary message
with count, start/end timestamps.
Configuration via general_settings.alert_type_config:
alert_type_config:
llm_requests_hanging:
digest: true
digest_interval: 86400
Digest key: (alert_type, request_model, api_base)
Default interval: 24 hours
Window type: fixed interval
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When extended thinking is enabled, the websearch interception agentic loop
builds a follow-up assistant message with only tool_use blocks. Anthropic's
API requires assistant messages to start with thinking/redacted_thinking
blocks when thinking is enabled, causing a 400 Bad Request.
Extract thinking blocks from the model's initial response, thread them
through the agentic loop, and prepend them to the follow-up assistant
message — matching the pattern used by anthropic_messages_pt in factory.py.
Fixes the error: "Expected 'thinking' or 'redacted_thinking', but found
'tool_use'"
Save and restore litellm.initialized_langfuse_clients around
test_max_langfuse_clients_limit to prevent ordering-dependent failures
in other tests that rely on the counter's value.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- test_litellm_pre_call_utils.py: wrap test body in try/finally so
litellm.callbacks is always restored even when an assertion fails,
addressing greptile review comment
- test_langfuse_otel.py: resolve trivial merge conflict in comment
("unpatched" vs "unpatch-ed"), keeping correct spelling
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: guard against None metadata in prometheus metrics
Use get_litellm_metadata_from_kwargs and get_metadata_variable_name_from_kwargs
helpers to properly resolve metadata from both 'metadata' and 'litellm_metadata'
keys, with None safety.
* test: add test for None metadata in prometheus metrics
jyeros