Commit Graph

4 Commits

Author SHA1 Message Date
Ishaan Jaff 15fcd90b9c feat: add in_flight_requests metric to /health/backlog + prometheus (#22319)
* feat: add in_flight_requests metric to /health/backlog + prometheus

* refactor: clean class with static methods, add tests, fix sentinel pattern

* docs: add in_flight_requests to prometheus metrics and latency troubleshooting
2026-02-27 18:00:50 -08:00
Ryan Crabbe 498dad0af1 test: add backwards compatibility tests for PrometheusAuthMiddleware
Add tests verifying non-metrics requests pass through unaffected and
don't trigger auth even when auth is enabled.
2026-02-21 12:06:31 -08:00
Ishaan Jaff 6486db3646 fix: improve streaming proxy throughput by fixing middleware and logging bottlenecks (#21501)
* fix(middleware): replace BaseHTTPMiddleware with pure ASGI middleware

BaseHTTPMiddleware wraps streaming responses with receive_or_disconnect
per chunk, blocking the event loop and causing severe throughput
degradation under concurrent streaming load (53% of CPU in profiling).

Converts PrometheusAuthMiddleware to a pure ASGI middleware using the
__call__(scope, receive, send) protocol.

* fix(streaming): remove expensive debug logging and optimize usage stripping

- Remove print_verbose calls that format chunk/response Pydantic objects,
  triggering millions of __repr__ calls (8% of CPU in profiling)
- Guard remaining verbose_logger.debug with isEnabledFor(DEBUG) and use
  lazy %s formatting instead of f-strings
- Replace usage stripping round-trip (model_dump + delete + reconstruct)
  with a _usage_stripped flag, deferring exclusion to serialization time

* fix(proxy): remove per-chunk debug log and use _usage_stripped flag

- Remove verbose_proxy_logger.debug that formatted every streaming chunk
- Honor _usage_stripped flag from streaming handler to exclude usage
  during model_dump_json serialization instead of reconstructing objects

* fix(proxy): remove per-chunk debug log in async_data_generator

Remove verbose_proxy_logger.debug that formatted every streaming chunk,
which triggered expensive Pydantic serialization on the hot path.

* fix indentation and add clarifying comment for usage stripping

* fix: guard calculate_total_usage against None usage in chunks

* fix: store chunk copy to preserve usage for calculate_total_usage
2026-02-18 16:16:49 -08:00
Krish Dholakia ef42461c1e Litellm fix GitHub action testing (#11163)
* test: add __init__.py files

* refactor: rename test folder to avoid naming conflict

* test: update workflows

* test: update tests

* test: update imports

* test: update tests

* test: remove unused import

* ci(test-litellm.yml): add pytest retry to github workflow

* test: fix test
2025-05-26 14:41:42 -07:00