* feat: enforce x-litellm-trace-id in header, if required
* feat: update spend for agent
* refactor: update agent table to follow similar format as other entities - also add a spend column - allows us to see spend of an agent
* fix: cleanup ui
* feat: return spend on agent endpoints
* feat: scope pr
* feat(agents/): support budgets + rate limiting on agents + agent sessions
* fix: address PR review feedback
- Add missing tpm_limit, rpm_limit, session_tpm_limit, session_rpm_limit
columns to root schema.prisma to match proxy and extras schemas
- Add backwards-compatible fallback to key metadata for max_iterations
so existing users don't silently lose enforcement
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: qa'ed RPM limiting on agents
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add LITELLM_WORKER_STARTUP_HOOKS for per-worker initialization (gflags support)
Add support for running user-defined startup hooks in each worker process
during proxy_startup_event. This enables re-initialization of in-process
state (like gflags.FLAGS) that doesn't survive uvicorn worker spawning.
Usage:
export LITELLM_WORKER_STARTUP_HOOKS=mymodule:init_fn,other:setup_fn
Hooks run early in proxy_startup_event (before config/DB loading).
Supports both sync and async callables. Errors propagate to prevent
broken workers from serving traffic. No-op when env var is unset.
Includes 5 tests covering sync/async hooks, multiple hooks, error
propagation, and no-hooks-set scenarios.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs: add Worker Startup Hooks page with gflags usage example
- New docs page: docs/proxy/worker_startup_hooks.md
- Explains the problem (per-process state lost in multi-worker deployments)
- Full gflags example with wrapper module and startup script
- Covers multiple hooks, async hooks, error behavior
- Architecture diagram showing master→worker flow
- Added LITELLM_WORKER_STARTUP_HOOKS to config_settings.md env var table
- Added to sidebar under Setup & Deployment
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Update litellm/proxy/proxy_server.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
- Add forward_llm_provider_auth_headers support from litellm_settings
- When enabled, client x-api-key takes precedence over deployment keys
- Forward x-api-key when x-litellm-api-key or Authorization used for auth
- Fix duplicate patch lines in test_byok_oauth_endpoints.py
- Add Claude Code BYOK documentation with /login and ANTHROPIC_CUSTOM_HEADERS
- Add unit tests for clean_headers x-api-key forwarding logic
- Sync model_prices backup (pre-commit hook)
Made-with: Cursor
Documents exactly how every request and response field gets translated
when LiteLLM routes an Anthropic /v1/messages call through the OpenAI
Responses API path (for OpenAI/Azure targets). Covers messages content
block mapping, tools, tool_choice, thinking→reasoning, context_management,
and the reverse response translation. Wired into the /v1/messages sidebar.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds provider documentation for bedrock_mantle including:
- API key and region configuration
- Supported models with pricing table
- SDK, streaming, and async usage examples
- LiteLLM Proxy config and usage
- Added to Bedrock category in sidebar
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(guardrails): team-based guardrail registration and approval workflow
Add team-based guardrail submission system where teams can register
Generic Guardrail API guardrails for admin review. Includes:
- POST /guardrails/register endpoint for team-scoped submissions
- Admin review endpoints (list/get/approve/reject submissions)
- Team Guardrails tab in the UI dashboard
- extra_headers support for forwarding client headers to guardrail APIs
- Prisma schema migration for status, submitted_at, reviewed_at fields
- Documentation for team-based guardrails and static/dynamic headers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(guardrails): address review feedback - SSRF, silent failure, redundant query
- Validate api_base URL scheme (http/https only) and hostname in
register_guardrail to prevent SSRF via team submissions
- Return warning field in approve response when in-memory initialization
fails so admins know the guardrail won't work until next sync cycle
- Eliminate redundant DB query in list_guardrail_submissions by fetching
all team guardrails once and deriving both filtered list and summary
counts from the single result set
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(guardrails): add pending_review status guard to reject endpoint
Prevent rejecting already-active or already-rejected guardrails, which
would create a DB/memory inconsistency (active in memory but rejected
in DB). Now mirrors the approve endpoint's status check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add CrowdStrike AIDR guardrail hook
* fixup! use apply_guardrail event hook
* fixup! update imports
* fix(guardrails): include AI response in CrowdStrike AIDR output events
Issue:
_build_guard_input_for_response() was:
- Sending only the original user input (messages).
- Not sending the AI provider response.
This fix will:
- Extract response.choices from the ModelResponse object and include them in guard_input payload.
- Thus, ensure AIDR output rules receive the AI-generated content for analysis.
- Fix and update tests.
* fix(guardrails): prevent duplicate input events in CrowdStrike AIDR guardrail
Issue:
The CrowdStrike AIDR guardrail was running on during_call hooks wihtout event_hook configured.
This fix will:
- Set event_hook to ["pre_call", "post_call"] (AIDR admins will control what policy is applied)
This change will:
- Require default_on parameter
- Prevent duplicate API calls to AIDR for the same input
- Avoid unchecked AI provider API calls on during_call hook
* docs: add CrowdStrike AIDR to the list of Guardrails under Integrations
* docs: update CrowdStrike AIDR documentation page
---------
Co-authored-by: Konstantin Lapine <konstantin.lapine@crowdstrike.com>
Add detailed UI walkthrough for Project Management feature including:
- Beta notice with link to API documentation
- Overview of projects and organizational hierarchy
- Prerequisites and setup instructions
- Separate section for enabling projects in UI settings
- Step-by-step guide for creating and managing projects
- Use cases for key organization within teams
- Next steps and related documentation links
- Proper sidebar navigation integration
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* feat(vertex_ai): add Vertex AI Gemini Live support via unified /realtime endpoint
Adds VertexAIRealtimeConfig which translates the OpenAI Realtime WebSocket
protocol to Vertex AI BidiGenerateContent. Supports voice in/voice out
(16 kHz mic → 24 kHz speaker) and text in/text out through the proxy's
/realtime endpoint.
Key changes:
- New litellm/llms/vertex_ai/realtime/transformation.py with VertexAIRealtimeConfig
- Builds correct wss:// URL (regional + global)
- OAuth2 Bearer token auth (not API key)
- Full model path (projects/.../publishers/google/models/...)
- Ignores session.update (Vertex AI only accepts one setup message)
- realtime_api/main.py: vertex_ai branch resolves OAuth token + constructs config
- llm_http_handler.py: auto-sends session setup before bidirectional_forward
- gemini/realtime/transformation.py: fix crashes on empty turnComplete events
- realtime_streaming.py: try/except guard so bad messages don't kill the loop
- proxy_server.py: add missing websockets.exceptions import
* docs: add vertex_realtime to sidebars
* fix: drop unknown event types in Gemini transform; add vertex_ai health check
* fix: propagate UUID fallback IDs from transform_content_done_event to return_additional_content_done_events
* fix: route guardrail backend sends through provider transform; fix str.strip misuse for model prefix
* fix: handle Vertex AI full resource path in session.created; route guardrail block sends through _send_to_backend
* fix: remove unused VertexBase in transformation.py; apply UUID fallback in return_additional_content_done_events
* feat(realtime): add guardrail hook for voice transcription in Realtime API
Adds a new `realtime_input_transcription` guardrail event hook that fires
after Whisper transcription completes, before the LLM generates a response.
When a guardrail blocks, a synthetic warning is sent to the client and
`response.create` is never forwarded — the LLM never responds.
Also rewrites `create_response: true` → `false` in client `session.update`
so the proxy controls when responses are triggered.
* feat(realtime): speak guardrail block message as audio via TTS
Instead of sending synthetic text events when a guardrail blocks,
send response.create with forced instructions so OpenAI's TTS speaks
the warning message — user hears the block instead of just seeing text.
* fix(realtime): speak exact content filter error message via TTS
Extract the human-readable error string from HTTPException.detail
so the spoken warning says e.g. "Content blocked: keyword 'system update'
detected" instead of the raw str(e) repr.
* fix(realtime): reliably enforce create_response=false for guardrails
- Proxy now injects session.update with create_response=false immediately
on session.created (when guardrails are active), instead of rewriting
the client's session.update — works regardless of what the client sends
- Add response.cancel before the warning response.create to kill any
in-flight LLM response that snuck through before the guardrail fired
* refactor(realtime): call apply_guardrail directly, remove dedicated hook method
The async_realtime_input_transcription_hook in CustomGuardrail and
ContentFilterGuardrail was just a thin wrapper that called apply_guardrail —
the same interface used by /chat and /messages. Remove the wrapper and call
apply_guardrail directly from run_realtime_guardrails, keeping the pattern
consistent across all endpoints.
* docs: add Realtime API guardrails tutorial and flow diagram
* fix: address Greptile review comments
- Forward user_api_key_dict through realtime_api/main.py (_arealtime) so
it actually reaches RealTimeStreaming instead of always being None
- Run guardrail interception in provider_config path too (e.g. Gemini),
not only the OpenAI direct path
- Narrow exception catch to HTTPException/ValueError only; re-raise
unexpected errors so programming bugs surface in logs rather than
silently appearing as guardrail blocks
- Update tests: mock apply_guardrail directly (hook method was removed),
replace session.update client-rewrite test with session.created
injection test matching the new server-side approach
* fix: address latest Greptile review comments
- Remove fastapi import from SDK-layer file; check for status_code/detail
attrs instead to identify guardrail-block exceptions vs programming errors
- Add store_message() before continue in transcription interception so
transcription events are logged in the non-provider_config path
- Inject create_response=false on session.created in provider_config path
(Gemini etc.) to match the OpenAI path — prevents LLM auto-responding
before guardrail runs on VAD-detected turns
* Add OpenAI Agents SDK tutorial to docs
* Update OpenAI Agents SDK tutorial to use LiteLLM environment variables
* Enhance OpenAI Agents SDK tutorial with built-in LiteLLM extension details and updated configuration steps. Adjust section headings for clarity and improve the flow of information regarding model setup and usage.
* docs: add Google GenAI SDK tutorial for JS and Python
Add tutorial for using Google's official GenAI SDK (@google/genai for JS,
google-genai for Python) with LiteLLM proxy. Covers pass-through and
native router endpoints, streaming, multi-turn chat, and multi-provider
routing via model_group_alias. Also updates pass-through docs to use the
new SDK replacing the deprecated @google/generative-ai.
* fix(docs): correct Python SDK env var name in GenAI tutorial
GOOGLE_GENAI_API_KEY does not exist in the google-genai SDK.
The correct env var is GEMINI_API_KEY (or GOOGLE_API_KEY).
Also note that the Python SDK has no base URL env var.
* fix(docs): replace non-existent GOOGLE_GENAI_BASE_URL env var in interactions.md
The Python google-genai SDK does not read GOOGLE_GENAI_BASE_URL.
Use http_options={"base_url": "..."} in code instead.
* docs: add OpenClaw integration tutorial
* docs: simplify OpenClaw proxy start command
* docs: rewrite OpenClaw integration guide for clarity
- Use gpt-5 as default model
- Replace poetry run with standard litellm CLI
- Add prerequisites section and verification step
- Simplify onboarding instructions (table format)
- Move manual config and troubleshooting to bottom
- Add multi-model config (claude-sonnet, gemini-flash)
* docs: fix model name in OpenClaw manual config example
* docs: rewrite OpenClaw integration guide from scratch
Rewrote the guide based on hands-on testing of every command.
Key changes:
- Replace non-existent `openclaw chat` with verified commands
(dashboard, tui, agent --agent main)
- Add 3 onboarding options: QuickStart, Manual, and non-interactive
- Fix health check (requires Bearer token)
- Remove misleading "Starting from scratch" section
- Use gpt-4o instead of gpt-5 as the example model
- Clarify that API keys can come from export, .env, or any method
- Add config reference section showing openclaw.json structure
- Add real troubleshooting based on issues found during testing
* fix(budget): fix timezone config lookup and replace hardcoded timezone map with ZoneInfo
* fix(budget): update stale docstring on get_budget_reset_time
* docs: add Day 0 Sonnet 4.6 support blog post
Add concise blog post announcing Day 0 support for Claude Sonnet 4.6 with Docker image and usage examples across:
- Anthropic API
- Azure AI
- Vertex AI
- Bedrock
Includes both LiteLLM Proxy and SDK usage for all providers.
* docs: add Sonnet 4.6 blog post to sidebar navigation
Add link to Claude Sonnet 4.6 Day 0 support blog post in the Blog section of the documentation sidebar.
* Pyroscope: require PYROSCOPE_APP_NAME and PYROSCOPE_SERVER_ADDRESS, add UTF-8 locale hint
- No defaults for PYROSCOPE_APP_NAME or PYROSCOPE_SERVER_ADDRESS; fail at startup if unset when Pyroscope is enabled
- Set LANG/LC_ALL to C.UTF-8 when unset to reduce malformed_profile (invalid UTF-8) rejections
- Startup message suggests PYTHONUTF8=1 if server rejects profiles
- Simplify LITELLM_ENABLE_PYROSCOPE in config_settings; document Pyroscope env vars as required with no default
- Add pyroscope_profiling to sidebar (Alerting & Monitoring)
- pyproject.toml: pyroscope-io as required dep on non-Windows (marker), in proxy extra
* proxy: add PYROSCOPE_SAMPLE_RATE env, use verbose logging, fix int type
- Add optional PYROSCOPE_SAMPLE_RATE env (integer, no default)
- Pass sample_rate to pyroscope.configure() as int for pyroscope-io
- Replace print with verbose_proxy_logger (info/warning)
- Document PYROSCOPE_SAMPLE_RATE in config_settings.md
* Address Greptile PR feedback: Pyroscope optional, docs, tests, docstring
- pyproject.toml: mark pyroscope-io as optional=true (proxy extra only)
- Add docs/my-website/docs/proxy/pyroscope_profiling.md (fix broken sidebar link)
- Add tests/test_litellm/proxy/test_pyroscope.py for _init_pyroscope()
- proxy_server: fix _init_pyroscope docstring (required server/app name, sample rate as int)
* Update litellm/proxy/proxy_server.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* init schema with TAGS
* ui: add policy test
* resolvePoliciesCall
* add_policy_sources_to_metadata + headers
* types Policy
* preview Impact
* def _describe_match_reason(
* match based on TAGs
* TestTagBasedAttachments
* test fixes
* add policy_resolve_router
* add_guardrails_from_policy_engine
* TestMatchAttribution
* refactor
* fix
* fix: address Greptile review feedback on policy resolve endpoints
- Track unnamed keys/teams as separate counts instead of inflating
affected_keys_count with duplicate "(unnamed key)" placeholders.
Added unnamed_keys_count and unnamed_teams_count to response.
- Push alias pattern matching to DB via _build_alias_where() which
converts exact patterns to Prisma "in" and suffix wildcards to
"startsWith" filters.
- Gate sync_policies_from_db/sync_attachments_from_db behind
force_sync query param (default false) to avoid 2 DB round-trips
on every /policies/resolve request.
- Remove worktree-only conftest.py that cleared sys.modules at import
time — no longer needed since code moved to main repo.
- Rename MAX_ESTIMATE_IMPACT_ROWS → MAX_POLICY_ESTIMATE_IMPACT_ROWS.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: eliminate duplicate DB queries and fix header delimiter ambiguity
- Fetch teams table once in estimate_attachment_impact and reuse for
both tag-based and alias-based lookups (was querying teams twice when
both tag_patterns and team_patterns were provided).
- Convert tag/team filter functions from async DB queries to sync
filters that operate on pre-fetched data (_filter_keys_by_tags,
_filter_teams_by_tags).
- Fix comma ambiguity in x-litellm-policy-sources header: use '; '
as entry delimiter since matched_via values can contain commas.
- Use '+' as the within-value separator in matched_via reason strings
(e.g. "tag:healthcare+team:health-team") to avoid conflict with
header delimiters.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs v1 guide with UI imgs
* docs fix
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(callbacks): allow MAX_CALLBACKS override via env var (#20778)
* fix(callbacks): allow MAX_CALLBACKS override via env var
- Move MAX_CALLBACKS from logging_callback_manager.py to constants.py
- Add LITELLM_MAX_CALLBACKS env var override (default: 30)
- Add troubleshooting doc explaining the limit and override
Fixes issue where large deployments with 60+ teams using guardrails
would hit the hardcoded MAX_CALLBACKS=30 limit and fail to start.
* docs: add max_callbacks to sidebar navigation
---------
Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com>
* fix callbacks issue
---------
Co-authored-by: shin-bot-litellm <shin-bot-litellm@berri.ai>
Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com>
