mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-23 02:24:18 +00:00
Cursor AgentandMateo Wang
1ef034bff6
When a client disconnects mid-stream from a Bedrock pass-through endpoint, Starlette calls aclose() on the async generator, raising GeneratorExit (a BaseException, not Exception) at the suspended yield. The previous `except Exception` blocks in _async_streaming/_sync_streaming (litellm/passthrough/main.py) and PassThroughStreamingHandler.chunk_processor did not catch GeneratorExit, so the post-loop flush that hands collected raw bytes to async_flush_passthrough_collected_chunks / _route_streaming_logging_to_handler never ran. All per-chunk usage data was silently dropped, undercounting spend for interrupted Bedrock invoke and converse streams. Move the flush into a finally block in all three sites and guard with a `flush_scheduled` flag so the success path still flushes exactly once. Also pull raise_for_status() out of the chunk-collection try block in _async_streaming so 4xx/5xx responses still raise and don't enter the flush path with zero bytes (preserving the behavior tested by test_async_streaming_error_propagation.py). Add regression coverage: - test_async_streaming_flushes_on_client_disconnect - test_async_streaming_flushes_on_upstream_exception_with_partial_data - test_sync_streaming_flushes_on_early_close - test_chunk_processor_logs_on_client_disconnect plus baseline tests for normal completion and the 4xx no-flush path. Fixes LIT-2642. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.