* feat(proxy): add Prisma DB pool and engine health metrics to Prometheus
Add a PrismaMetricsCollector that periodically queries pg_stat_activity
and the Prisma engine process to expose connection pool and engine health
as Prometheus gauges/counters. Auto-enabled when prometheus_system is in
service_callback.
New metrics:
- litellm_db_pool_active_connections (Gauge)
- litellm_db_pool_idle_connections (Gauge)
- litellm_db_pool_total_connections (Gauge)
- litellm_db_pool_waiting_connections (Gauge)
- litellm_db_engine_up (Gauge)
- litellm_db_engine_restarts_total (Counter)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address Greptile review feedback
- Only increment engine_restarts counter on heavy reconnects (engine
actually dead), not lightweight network-blip reconnects
- Fix potential KeyError in _get_or_create_gauge/counter fallback path
when REGISTRY._names_to_collectors is absent
- Rename litellm_db_pool_waiting_connections to
litellm_db_pool_lock_waiting_connections to clarify it measures lock
contention, not pool slot queuing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: warn when prometheus_system enabled but watchdog disabled
Log a warning when users have prometheus_system in service_callback
but PRISMA_HEALTH_WATCHDOG_ENABLED=false, since DB pool and engine
metrics won't be collected in that configuration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: retrigger CI checks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: use labeled gauge for DB pool connection metrics
Replace 3 separate pool gauges (active, idle, total) with a single
`litellm_db_pool_connections` gauge using a `state` label. This is more
Prometheus-idiomatic and exposes all pg_stat_activity states (active,
idle, idle in transaction, etc.) without ambiguity about what "total"
includes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address Greptile review — stale labels and fallback re-registration
- Zero out known pg_stat_activity states that are absent from the current
query result, preventing stale gauge values from persisting.
- Simplify _get_or_create_gauge/counter by removing the fallback loop
that could re-register an already-registered metric (ValueError).
- Add test for stale label clearing across collection cycles.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: include "unknown" in _PG_STATES for stale label clearing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: collect immediately on start and consolidate into single query
- Move sleep to end of loop so metrics appear on /metrics immediately
after startup instead of after a 30s delay.
- Combine pool state and lock waiting queries into a single SQL query
using conditional aggregation, halving per-cycle DB overhead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: prevent tight spin loop on collection error
Move asyncio.sleep outside the try/except so it always executes even
when _collect_engine_health() or _collect_pool_metrics() raises.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: add multiprocess_mode to _get_or_create_gauge initialization
- Include `multiprocess_mode` parameter to properly support multiprocessing in Gauge creation.
- Ensure consistent behavior for labeled and unlabeled Gauges.
* fix: handle invalid env var and document watchdog prerequisite
- Add try/except ValueError for PRISMA_METRICS_COLLECTION_INTERVAL_SECONDS
to prevent proxy startup crash on non-numeric values (e.g. "30s")
- Document that DB metrics require both prometheus_system callback and
PRISMA_HEALTH_WATCHDOG_ENABLED=true
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: use defensive null coalescing for query_raw row values
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add invalid env var fallback test and fix mock signature
- Add test for non-numeric PRISMA_METRICS_COLLECTION_INTERVAL_SECONDS
- Add **kwargs to mock _patched_get_or_create_gauge for forward compat
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: enforce x-litellm-trace-id in header, if required
* feat: update spend for agent
* refactor: update agent table to follow similar format as other entities - also add a spend column - allows us to see spend of an agent
* fix: cleanup ui
* feat: return spend on agent endpoints
* feat: scope pr
* feat(agents/): support budgets + rate limiting on agents + agent sessions
* fix: address PR review feedback
- Add missing tpm_limit, rpm_limit, session_tpm_limit, session_rpm_limit
columns to root schema.prisma to match proxy and extras schemas
- Add backwards-compatible fallback to key metadata for max_iterations
so existing users don't silently lose enforcement
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: qa'ed RPM limiting on agents
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The validation method now reads use_redis_transaction_buffer directly
from the passed general_settings dict rather than delegating to
RedisUpdateBuffer._should_commit_spend_updates_to_redis() which
imports the global. Tests simplified to remove unnecessary patching.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When `use_redis_transaction_buffer: true` is set in general_settings but no
Redis cache is configured in litellm_settings, the proxy starts successfully
but silently drops all spend tracking data. This adds a startup validation
that raises a clear error, preventing the proxy from running in a broken state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(proxy): tool policies - auto-discover tools, manage policies, guardrail enforcement
- New LiteLLM_ToolTable in schema.prisma to store discovered tools
- Auto-discovery: tools seen in LLM responses get upserted via ToolDiscoveryQueue
(hooks into DBSpendUpdateWriter, same pipeline as spend tracking)
- Management endpoints: GET /v1/tool/list, GET /v1/tool/{name}, POST /v1/tool/policy
- ToolPolicyGuardrail: blocks tool_calls in responses based on policy setting
- UI: Tool Policies page under Guardrails section with policy selector,
filters by policy/team/key, live tail, sortable table
- Unit tests for queue, writer, endpoints, guardrail
* feat(tool-policies): track call_count + discover tools from request body and /messages API
- Add call_count column to LiteLLM_ToolTable; incremented on every flush
- Extract tools from request body too (not just response tool_calls):
- OpenAI /chat/completions: tools[].function.name
- Anthropic /messages pass-through: request_body.tools[].name
- Show call_count column in UI table (sortable)
- UI: drop dual_llm option, keep only trusted/blocked
* fix: address greptile review feedback
- Remove redundant @@index([tool_name]) from schema.prisma (tool_name has @unique which already creates an index)
- Replace gen_random_uuid()::text with str(uuid.uuid4()) for portability
- Rewrite test_tool_registry_writer.py to mock execute_raw/query_raw (actual implementation) instead of Prisma model methods
- Fix test patches in test_tool_management_endpoints.py to target source modules since imports are inside function bodies
- Add "Tool Policies" page title to ToolPolicies.tsx
* fix: address greptile review round 2
- Replace NOW() with Python datetime parameter in tool_registry_writer (SQLite portability)
- Fix cache key collision in tool_policy_guardrail: use null-byte separator instead of colon
- Remove type==function filter from request-side tool extraction to match response-side behavior
- Clear seen_tool_names on flush so call_count increments per batch cycle not per pod lifetime
* fix: address greptile review round 3
- Fix test_seen_names_persist_across_flushes to match actual per-flush-cycle behavior
- Update module docstring in tool_discovery_queue.py to accurately describe flush behavior
- Add created_at/updated_at to raw SQL INSERT in batch_upsert_tools and update_tool_policy
* fix: cache tool policies per tool name not per combination
Previously the cache key was built from the full set of tool names in a
request, so each unique combination of tools got its own cold cache entry
and triggered a separate DB query. With N distinct tools across requests
this was effectively a DB hit on every request.
Now each tool name is cached individually. Cache hits are checked per
tool, only missing tools are fetched from DB in a single batch query,
and each result is cached separately. Once a tool's policy is warm,
any subsequent request using that tool benefits from the cache regardless
of what other tools are in the request.
* Update ui/litellm-dashboard/src/components/ToolPolicies.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Replace 11 separate asyncio.create_task() calls per request with a
single batched task that runs all spend-update helpers sequentially.
This reduces task scheduling overhead at high RPS (11,000 -> 1,000
tasks/sec at 1K RPS) and cuts 5 copy.deepcopy(payload) calls to 1
shared copy.
Also fixes a mutation bug where the daily agent spend handler received
the raw payload without deepcopy, unlike all other daily helpers.
The test used a fixed side_effect list for time.time(), but the number
of calls varies by Python version, causing StopIteration on 3.12 and
AssertionError on 3.14. Replace with an infinite counter-based callable
and assert the timestamp was updated rather than checking for an exact
value.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Any PrismaError should be treated as a DB connection error for the
allow_requests_on_db_unavailable feature and 503 responses. The narrow
keyword-based check is now in is_database_transport_error, which is
what the reconnect logic in auth_checks.py should use.
Fixes test_delete_access_group_503_on_db_connection_error and
test_handle_authentication_error_db_unavailable failures caused by
PR #21706 narrowing is_database_connection_error.
Virtual keys only track created_at and updated_at, which don't indicate
when a key was last used. This adds a last_active field that gets updated
during the async batch spend update, giving admins visibility into which
keys are actively being used.
Changes:
- Add last_active DateTime? to VerificationToken and
DeletedVerificationToken in all 3 schema files and Python types
- Set last_active in the batch key spend update alongside spend increment
- Add Last Active column to virtual keys UI table with info popover
and hover tooltip showing full date/time with timezone
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add fallback in sort to prevent NoneType and str comparison
* Hide Default Team Settings from Proxy Admin Viewers
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* ensure original client is disconnected when re-creating
* test_recreate_prisma_client_successful_disconnect
* test_recreate_prisma_client_successful_disconnect
michelligabriele