3 people 0bb26c3f1b feat(proxy): add Prisma DB pool and engine health metrics to Prometheus (#22655) ...

* feat(proxy): add Prisma DB pool and engine health metrics to Prometheus

Add a PrismaMetricsCollector that periodically queries pg_stat_activity
and the Prisma engine process to expose connection pool and engine health
as Prometheus gauges/counters. Auto-enabled when prometheus_system is in
service_callback.

New metrics:
- litellm_db_pool_active_connections (Gauge)
- litellm_db_pool_idle_connections (Gauge)
- litellm_db_pool_total_connections (Gauge)
- litellm_db_pool_waiting_connections (Gauge)
- litellm_db_engine_up (Gauge)
- litellm_db_engine_restarts_total (Counter)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address Greptile review feedback

- Only increment engine_restarts counter on heavy reconnects (engine
  actually dead), not lightweight network-blip reconnects
- Fix potential KeyError in _get_or_create_gauge/counter fallback path
  when REGISTRY._names_to_collectors is absent
- Rename litellm_db_pool_waiting_connections to
  litellm_db_pool_lock_waiting_connections to clarify it measures lock
  contention, not pool slot queuing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: warn when prometheus_system enabled but watchdog disabled

Log a warning when users have prometheus_system in service_callback
but PRISMA_HEALTH_WATCHDOG_ENABLED=false, since DB pool and engine
metrics won't be collected in that configuration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* ci: retrigger CI checks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: use labeled gauge for DB pool connection metrics

Replace 3 separate pool gauges (active, idle, total) with a single
`litellm_db_pool_connections` gauge using a `state` label. This is more
Prometheus-idiomatic and exposes all pg_stat_activity states (active,
idle, idle in transaction, etc.) without ambiguity about what "total"
includes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address Greptile review — stale labels and fallback re-registration

- Zero out known pg_stat_activity states that are absent from the current
  query result, preventing stale gauge values from persisting.
- Simplify _get_or_create_gauge/counter by removing the fallback loop
  that could re-register an already-registered metric (ValueError).
- Add test for stale label clearing across collection cycles.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: include "unknown" in _PG_STATES for stale label clearing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: collect immediately on start and consolidate into single query

- Move sleep to end of loop so metrics appear on /metrics immediately
  after startup instead of after a 30s delay.
- Combine pool state and lock waiting queries into a single SQL query
  using conditional aggregation, halving per-cycle DB overhead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: prevent tight spin loop on collection error

Move asyncio.sleep outside the try/except so it always executes even
when _collect_engine_health() or _collect_pool_metrics() raises.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: add multiprocess_mode to _get_or_create_gauge initialization

- Include `multiprocess_mode` parameter to properly support multiprocessing in Gauge creation.
- Ensure consistent behavior for labeled and unlabeled Gauges.

* fix: handle invalid env var and document watchdog prerequisite

- Add try/except ValueError for PRISMA_METRICS_COLLECTION_INTERVAL_SECONDS
  to prevent proxy startup crash on non-numeric values (e.g. "30s")
- Document that DB metrics require both prometheus_system callback and
  PRISMA_HEALTH_WATCHDOG_ENABLED=true

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: use defensive null coalescing for query_raw row values

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: add invalid env var fallback test and fix mock signature

- Add test for non-numeric PRISMA_METRICS_COLLECTION_INTERVAL_SECONDS
- Add **kwargs to mock _patched_get_or_create_gauge for forward compat

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-09 08:49:46 -07:00

..

my-website

feat(proxy): add Prisma DB pool and engine health metrics to Prometheus (#22655)

2026-03-09 08:49:46 -07:00