* feat(proxy): add Prisma DB pool and engine health metrics to Prometheus
Add a PrismaMetricsCollector that periodically queries pg_stat_activity
and the Prisma engine process to expose connection pool and engine health
as Prometheus gauges/counters. Auto-enabled when prometheus_system is in
service_callback.
New metrics:
- litellm_db_pool_active_connections (Gauge)
- litellm_db_pool_idle_connections (Gauge)
- litellm_db_pool_total_connections (Gauge)
- litellm_db_pool_waiting_connections (Gauge)
- litellm_db_engine_up (Gauge)
- litellm_db_engine_restarts_total (Counter)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address Greptile review feedback
- Only increment engine_restarts counter on heavy reconnects (engine
actually dead), not lightweight network-blip reconnects
- Fix potential KeyError in _get_or_create_gauge/counter fallback path
when REGISTRY._names_to_collectors is absent
- Rename litellm_db_pool_waiting_connections to
litellm_db_pool_lock_waiting_connections to clarify it measures lock
contention, not pool slot queuing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: warn when prometheus_system enabled but watchdog disabled
Log a warning when users have prometheus_system in service_callback
but PRISMA_HEALTH_WATCHDOG_ENABLED=false, since DB pool and engine
metrics won't be collected in that configuration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: retrigger CI checks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: use labeled gauge for DB pool connection metrics
Replace 3 separate pool gauges (active, idle, total) with a single
`litellm_db_pool_connections` gauge using a `state` label. This is more
Prometheus-idiomatic and exposes all pg_stat_activity states (active,
idle, idle in transaction, etc.) without ambiguity about what "total"
includes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address Greptile review — stale labels and fallback re-registration
- Zero out known pg_stat_activity states that are absent from the current
query result, preventing stale gauge values from persisting.
- Simplify _get_or_create_gauge/counter by removing the fallback loop
that could re-register an already-registered metric (ValueError).
- Add test for stale label clearing across collection cycles.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: include "unknown" in _PG_STATES for stale label clearing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: collect immediately on start and consolidate into single query
- Move sleep to end of loop so metrics appear on /metrics immediately
after startup instead of after a 30s delay.
- Combine pool state and lock waiting queries into a single SQL query
using conditional aggregation, halving per-cycle DB overhead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: prevent tight spin loop on collection error
Move asyncio.sleep outside the try/except so it always executes even
when _collect_engine_health() or _collect_pool_metrics() raises.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: add multiprocess_mode to _get_or_create_gauge initialization
- Include `multiprocess_mode` parameter to properly support multiprocessing in Gauge creation.
- Ensure consistent behavior for labeled and unlabeled Gauges.
* fix: handle invalid env var and document watchdog prerequisite
- Add try/except ValueError for PRISMA_METRICS_COLLECTION_INTERVAL_SECONDS
to prevent proxy startup crash on non-numeric values (e.g. "30s")
- Document that DB metrics require both prometheus_system callback and
PRISMA_HEALTH_WATCHDOG_ENABLED=true
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: use defensive null coalescing for query_raw row values
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add invalid env var fallback test and fix mock signature
- Add test for non-numeric PRISMA_METRICS_COLLECTION_INTERVAL_SECONDS
- Add **kwargs to mock _patched_get_or_create_gauge for forward compat
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: enforce x-litellm-trace-id in header, if required
* feat: update spend for agent
* refactor: update agent table to follow similar format as other entities - also add a spend column - allows us to see spend of an agent
* fix: cleanup ui
* feat: return spend on agent endpoints
* feat: scope pr
* feat(agents/): support budgets + rate limiting on agents + agent sessions
* fix: address PR review feedback
- Add missing tpm_limit, rpm_limit, session_tpm_limit, session_rpm_limit
columns to root schema.prisma to match proxy and extras schemas
- Add backwards-compatible fallback to key metadata for max_iterations
so existing users don't silently lose enforcement
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: qa'ed RPM limiting on agents
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(docker): bump tar/minimatch/pypdf for CVE fixes + harden SBOM patching
- Bump tar 7.5.8→7.5.10, minimatch 10.2.1→10.2.4, pypdf 6.6.2→6.7.3
- Add sed-based SBOM metadata patching with properly indented find/sed
- Add npm package manager cleanup (apk del / apt-get purge) to remove
stale SBOM entries from image scanners
- Scope || true to only apk del via brace grouping { ... || true; }
- Guard npm root -g with non-empty assertion to prevent silent failures
- Scope minimatch sed regex to ^10.x to avoid matching other major versions
Addresses: CVE-2026-27903, CVE-2026-27904, GHSA-qffp-2rhf-9h96, CVE-2026-27888
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(docker): scope find to /usr/local/lib /usr/lib, drop autoremove
- Replace `find /` with `find /usr/local/lib /usr/lib` to avoid
traversing /proc, /sys, /dev during SBOM metadata patching
- Remove `apt-get autoremove -y` from Debian-based Dockerfiles to
prevent nodejs from being removed as an auto-installed dependency
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve ruff lint errors and mypy type error
- Remove unused import get_user_credential (F401)
- Add noqa: PLR0915 for 3 large functions exceeding 50 statements
- Cast result_data['q'] to str for _append_domain_filters (mypy arg-type)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add /vertex_ai/live to supported endpoints and azure gpt-5.1 reasoning flags
- Add /vertex_ai/live to JSON schema validation enum in test_utils.py
- Add supports_none_reasoning_effort=true to 10 azure/gpt-5.1 model entries
(matching the OpenAI gpt-5.1 behavior)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: handle non-string team_alias/key_alias in PolicyMatchContext
Prevent Pydantic validation errors when team_alias or key_alias are not
proper strings (e.g. MagicMock in tests). Only pass values that are
actually strings; default to None otherwise.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: initialize jwt_handler.litellm_jwtauth in JWT test
The test_jwt_non_admin_team_route_access test was failing because
user_api_key_auth now accesses jwt_handler.litellm_jwtauth.virtual_key_claim_field
before reaching the mocked JWTAuthManager.auth_builder. Initialize the
jwt_handler with a default LiteLLM_JWTAuth object.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add missing mock attributes to MCP server test
The test_add_update_server_fallback_to_server_id test was failing because
MagicMock auto-creates attributes when accessed. build_mcp_server_from_table
accesses many fields via getattr(), which on a MagicMock returns another
MagicMock instead of None, causing Pydantic validation errors in MCPServer.
Explicitly set all required mock attributes.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: update UI tests for leftnav, navbar, and KeyLifecycleSettings
- leftnav: Add mock for useTeams hook, add isUserTeamAdminForAnyTeam to
roles mock, update topLevelLabels to match current component menu items
- navbar: Add mocks for useDisableBouncingIcon, BlogDropdown, UserDropdown,
and serverRootPath. Update test to work with the new component structure.
- KeyLifecycleSettings: Fix placeholder and tooltip assertions to match
actual component behavior
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: update health check test assertion from 'connected' to 'healthy'
The /health/readiness endpoint now returns {"status": "healthy"} with the
DB status in a separate field, instead of the previous {"status": "connected"}.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: clear litellm.api_key in OpenRouter validate_environment test
The test_validate_environment_raises_without_key test was failing because
litellm.api_key may be set globally in the test environment. Clear it
along with OPENROUTER_API_KEY and OR_API_KEY env vars using monkeypatch.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: patch HTTPHandler class-level in VLLM embedding test
The test_encoding_format_not_sent_in_actual_request test was patching
client.post on an instance, but the handler uses the class method.
Patch HTTPHandler.post at class level, add caching=False to prevent
cache hits, and remove broad try/except that hid errors.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: make test_redaction_responses_api_stream resilient to async callback timing
Replace fixed 1s sleep with polling wait for async_log_success_event.
Streaming success handler runs via asyncio.create_task; 1s was insufficient
in CI. Add 0.5s initial sleep for event loop to schedule the task, then
poll up to 10s for the callback to fire.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: update dompurify and svgo to fix security CVEs
- CVE-2026-0540: dompurify XSS vulnerability - fix by upgrading to 3.3.2+
- CVE-2026-29074: svgo DoS via entity expansion - fix by upgrading to 3.3.3+
Added npm overrides in docs/my-website/package.json and regenerated
package-lock.json.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: remove unused json import in config_override_endpoints.py
Ruff F401: json is imported but unused (safe_json_loads/safe_dumps
are used instead)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add missing MCP mock attributes and provider documentation entries
- Add missing mock attributes to test_add_update_server_with_alias and
test_add_update_server_without_alias (same fix as fallback test)
- Add bedrock_mantle and searchapi to provider_endpoints_support.json
- Remove unused json import from config_override_endpoints.py
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: override _supports_reasoning_effort_level for Azure gpt5_series prefix
The Azure GPT-5 config uses 'gpt5_series/' as a routing prefix, but
_supports_factory(model='gpt5_series/gpt-5.1') fails to resolve because
'gpt5_series' is not a recognized provider. Override the method to strip
the prefix and prepend 'azure/' for correct model info lookup.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: accept both 'healthy' and 'connected' in health check test
The test_health_and_chat_completion test runs against both source builds
(which return 'healthy') and pip-installed versions (which may return
'connected'). Accept both values.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: mock extract_mcp_auth_context in streamable HTTP MCP handler test
The handle_streamable_http_mcp function now calls extract_mcp_auth_context
before session_manager.handle_request, but the test didn't mock it. The
auth extraction fails with the minimal mock scope, preventing
handle_request from being called. Also relax assertion to not check
exact args since the send wrapper may be modified by debug injection.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add test for _combine_fallback_usage to satisfy router code coverage
The router_code_coverage.py check requires all functions in router.py
to be called in test files. Add a basic test for _combine_fallback_usage.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add @log_guardrail_information decorator to CrowdStrike AIDR guardrail
The check_guardrail_apply_decorator.py CI check requires all guardrail
apply_guardrail methods to have the @log_guardrail_information decorator.
The CrowdStrike AIDR handler was missing it.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: document PRISMA_RECONNECT_ESCALATION_THRESHOLD and REDIS_CLUSTER_NODES env keys
Add missing environment variable documentation to config_settings.md
to satisfy the test_env_keys.py CI check.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: document enforced_file_expires_after and enforced_batch_output_expires_after in new_team docstring
The test_api_docs.py CI check validates that all Pydantic model fields
are documented in the function docstring. Add missing parameter docs
for enforced_file_expires_after and enforced_batch_output_expires_after.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: regenerate poetry.lock to match pyproject.toml
The poetry.lock file was out of sync with pyproject.toml, causing
proxy_e2e_azure_batches_tests to fail during dependency installation.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: set master_key=None in test_create_file_with_deep_nested_litellm_metadata
The test was missing the master_key monkeypatch that other tests in the
same file set. In CI with parallel execution (-n 4), another test may
set master_key to a non-None value, causing auth failures (500) when
the test sends 'Bearer test-key'.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: document enforced_*_expires_after in update_team docstring too
Same missing params as new_team - also needed in update_team docstring
for the test_api_docs.py CI check to pass.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: use get_async_httpx_client in a2a_protocol and add master_key monkeypatch to files tests
- Replace httpx.AsyncClient() with get_async_httpx_client() in a2a_protocol/main.py
to satisfy the ensure_async_clients_test CI check
- Add httpxSpecialProvider.A2AProvider enum value
- Add master_key=None monkeypatch to test_managed_files_with_loadbalancing
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: remove unused httpx import from a2a_protocol/main.py
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: use cache-key-only param for A2A extra_headers to avoid AsyncHTTPHandler init error
The 'extra_headers' key in params was being passed to AsyncHTTPHandler.__init__()
which doesn't accept it. Use 'disable_aiohttp_transport' as the cache-key-only
param since it's explicitly filtered out before reaching the constructor.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add additionalProperties:false and resolve $defs/$ref in Anthropic output_format schemas
Anthropic API now requires additionalProperties=false for all object-type
schemas in output_format. Also resolve $defs/$ref references by inlining
them using unpack_defs before sending to Anthropic, since Anthropic
doesn't support external schema references.
Fixes: llm_translation_testing Anthropic JSON schema failures
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: allowlist CVE-2026-2297 and GHSA-qffp-2rhf-9h96 in security scans
- CVE-2026-2297: Python 3.13 SourcelessFileLoader audit hook bypass,
no fix available in base image
- GHSA-qffp-2rhf-9h96: tar hardlink path traversal, from nodejs_wheel
bundled npm, not used in application runtime code
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: isolate files endpoint tests from shared proxy state in CI parallel execution
Override user_api_key_auth dependency to return a fixed UserAPIKeyAuth
with PROXY_ADMIN role, avoiding auth lookups via prisma_client,
user_api_key_cache, or master_key. Set prisma_client=None to prevent
DB state contamination. Use try/finally to clean up dependency overrides.
Fixes persistent test_create_file_with_deep_nested_litellm_metadata and
test_managed_files_with_loadbalancing 500 errors in CI with -n 4.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: apply same auth override to test_managed_files_with_loadbalancing
Same CI parallel execution fix as test_create_file_with_deep_nested -
override user_api_key_auth dependency and set prisma_client=None.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat: add LITELLM_WORKER_STARTUP_HOOKS for per-worker initialization (gflags support)
Add support for running user-defined startup hooks in each worker process
during proxy_startup_event. This enables re-initialization of in-process
state (like gflags.FLAGS) that doesn't survive uvicorn worker spawning.
Usage:
export LITELLM_WORKER_STARTUP_HOOKS=mymodule:init_fn,other:setup_fn
Hooks run early in proxy_startup_event (before config/DB loading).
Supports both sync and async callables. Errors propagate to prevent
broken workers from serving traffic. No-op when env var is unset.
Includes 5 tests covering sync/async hooks, multiple hooks, error
propagation, and no-hooks-set scenarios.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs: add Worker Startup Hooks page with gflags usage example
- New docs page: docs/proxy/worker_startup_hooks.md
- Explains the problem (per-process state lost in multi-worker deployments)
- Full gflags example with wrapper module and startup script
- Covers multiple hooks, async hooks, error behavior
- Architecture diagram showing master→worker flow
- Added LITELLM_WORKER_STARTUP_HOOKS to config_settings.md env var table
- Added to sidebar under Setup & Deployment
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Update litellm/proxy/proxy_server.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(vertex_ai): support explicit AWS credentials for WIF auth
The current Vertex AI AWS Workload Identity Federation implementation
exclusively uses google.auth.aws.Credentials.from_info(), which requires
EC2 instance metadata access to obtain AWS credentials. In environments
where the metadata service is blocked for security reasons, this makes
WIF unusable.
Add support for explicit AWS credentials by implementing a custom
AwsSecurityCredentialsSupplier (google-auth >= 2.29.0). When aws_* keys
(e.g. aws_role_name, aws_region_name) are present in the WIF credential
JSON, LiteLLM uses BaseAWSLLM.get_credentials() to obtain AWS creds via
STS AssumeRole (or any other supported AWS auth flow), wraps them in the
custom supplier, and passes them to aws.Credentials() — bypassing the
metadata service entirely.
When no aws_* keys are present, the existing from_info() flow is used
unchanged, preserving full backward compatibility.
* refactor(vertex_ai): extract AWS WIF auth to own class + add docs
Address PR review feedback:
- Move _AWS_CREDENTIAL_KEYS, _extract_aws_params(), and
_credentials_from_aws_with_explicit_auth() from VertexBase into
new VertexAIAwsWifAuth class in vertex_ai_aws_wif.py
- Add documentation for explicit AWS credentials WIF auth method
in vertex.md (supported params, JSON example, SDK/Proxy tabs)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(vertex_ai): use lazy credentials provider to prevent stale STS tokens
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Add forward_llm_provider_auth_headers support from litellm_settings
- When enabled, client x-api-key takes precedence over deployment keys
- Forward x-api-key when x-litellm-api-key or Authorization used for auth
- Fix duplicate patch lines in test_byok_oauth_endpoints.py
- Add Claude Code BYOK documentation with /login and ANTHROPIC_CUSTOM_HEADERS
- Add unit tests for clean_headers x-api-key forwarding logic
- Sync model_prices backup (pre-commit hook)
Made-with: Cursor
- Pass request_model to Azure AI cost calculator to detect router requests
- Add router flat cost ($0.14/M input tokens) even when Azure returns actual model in response
- Add test for router flat cost with response containing actual model
- Update docs with cost calculation flow and configuration requirements
Made-with: Cursor
Documents exactly how every request and response field gets translated
when LiteLLM routes an Anthropic /v1/messages call through the OpenAI
Responses API path (for OpenAI/Azure targets). Covers messages content
block mapping, tools, tool_choice, thinking→reasoning, context_management,
and the reverse response translation. Wired into the /v1/messages sidebar.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds provider documentation for bedrock_mantle including:
- API key and region configuration
- Supported models with pricing table
- SDK, streaming, and async usage examples
- LiteLLM Proxy config and usage
- Added to Bedrock category in sidebar
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* azure content enhancement...
* rafactored to increase confidence score
* improvements based on additional feedback
* removed unused import
* Force-split any word longer than max length allowed
* preserve whitespace in text splitting
* moving common initialization to base class
* consolidate enforcement into async_make_request as single point, remove redundant caller-side checks, extract shared init/HTTP logic into base, and fix stale log messages
* clean up
* clean up tests
shivam