mirror of
https://github.com/tiennm99/litellm.git
synced 2026-06-30 21:06:03 +00:00
Alexsander Hamir
b02baf53a9
* fix: prevent memory blowout in LoggingWorker Tasks were being executed sequentially with each task awaited before processing the next one. When the queue had 10k+ tasks, only one could execute at a time. Since the request rate exceeded execution speed, objects accumulated in memory (50k+), holding references to heavy objects and causing memory blowout. The new implementation uses a semaphore to allow up to 1000 concurrent tasks while properly tracking and cleaning up each task, significantly improving throughput and preventing queue buildup. * fix: require semaphor before removing task from queue * fix: make worker concurrency configurable * fix: clean comments * fix: clarify new env purpose * fix: add missing lib * make constants configurable instead of hardcoded * add more aggressive cleaning when queue is full * add helpers function for the aggressive cleaning functionality * use envs instead of static constants * import and document constants * add unit test for new functionality * fix default value on config_settings * fix: remove unused variables and imports to resolve linter errors - Remove unused time_since_last_clear variable in logging_worker.py The variable was calculated but never used in _handle_queue_full() method, causing F841 linter error. - Remove unused TYPE_CHECKING import in mcp_server/server.py The import was not used anywhere in the file, causing F401 linter error. These changes improve code cleanliness and ensure the codebase passes all linter checks without affecting functionality. * add missing log expected by test_queue_full_handling * fix: clean config_setting.md file * fix: handle logging errors gracefully during shutdown in _flush_on_exit During process shutdown, logging handlers may be closed while _flush_on_exit tries to flush queued logging coroutines. This causes 'ValueError: I/O operation on closed file' errors when coroutines attempt to log. Changes: - Add _safe_log helper method that wraps logging calls and suppresses errors when logging handlers are closed (ValueError, OSError, AttributeError) - Replace all verbose_logger calls in _flush_on_exit with _safe_log - Remove logging from exception handler in coroutine execution loop to prevent cascading errors during shutdown This ensures graceful shutdown even when logging handlers are closed, which is common during process termination.
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.