mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-11 21:46:22 +00:00
Ryan Malloy
58c8c2b7b1
* fix: prevent HTTP client memory leaks in Presidio and OpenAI wrappers Fixes multiple memory leak issues reported in #14540 and related tickets: **Presidio Guardrail Fix (#14540)** - Problem: Every guardrail check created a new aiohttp.ClientSession - Impact: High-traffic proxies accumulated thousands of unclosed sessions - Solution: Share a single session across all guardrail checks - Added `self._http_session` instance variable - Lazy session creation via `_get_http_session()` - Proper cleanup via `_close_http_session()` and `__del__()` - Files: litellm/proxy/guardrails/guardrail_hooks/presidio.py **OpenAI HTTP Client Caching (#14540)** - Problem: `_get_async_http_client()` created new httpx.AsyncClient on each call - Impact: OpenAI/Azure completions bypassed client caching system - Solution: Route through `get_async_httpx_client()` for TTL-based caching - Caches clients by provider and SSL config - Fallback to direct creation if caching fails - Applied to both async and sync client methods - Files: litellm/llms/openai/common_utils.py **Test Script** - Added validation script to demonstrate fixes - Counts file descriptors and unclosed session objects - Files: test_oom_fixes.py Related issues: #14384, #13251, #12443 * fix(oom): prevent memory leaks in Presidio guardrails and OpenAI client creation Fixes two high-impact memory leaks: 1. Presidio Guardrail Session Leak (issue #14540) - Problem: Created new aiohttp.ClientSession on every guardrail check - Impact: Runs on EVERY proxy request when PII masking enabled - Fix: Shared session pattern with lifecycle management - Files: litellm/proxy/guardrails/guardrail_hooks/presidio.py 2. OpenAI HTTP Client Cache Bypass (issue #14540) - Problem: _get_async_http_client() created new httpx.AsyncClient, bypassing TTL cache - Impact: Every completion created new client with own connection pool - Fix: Route through get_async_httpx_client() for proper caching - Critical: Include SSL config in cache key for correctness - Files: litellm/llms/openai/common_utils.py Validation: - Presidio: 100 requests → 0 new sessions (was 100) - OpenAI: 100 calls → 1 unique client (was 100) - test_oom_fixes.py: Automated validation script * fix(oom): resolve Gemini aiohttp session leak (issue #12443) Fixes persistent "Unclosed client session" warnings when using Gemini models. Root Causes: 1. Broken atexit cleanup - get_event_loop() fails at exit time 2. On-demand session creation without reliable cleanup Changes: 1. Fixed atexit Cleanup (async_client_cleanup.py) - OLD: Used get_event_loop() which fails when loop is closed - NEW: Always create fresh event loop at exit time - Ensures cleanup runs successfully even when main loop is closed 2. Added __del__ Cleanup (aiohttp_handler.py) - Defense-in-depth: cleanup on garbage collection - Handles abnormal termination cases - Similar pattern to Presidio guardrail fix 3. Enhanced Cleanup Scope (async_client_cleanup.py) - Now closes global base_llm_aiohttp_handler instance - Previously only checked cache, missed module-level handler Validation: - Test 1: __del__ cleanup → 0 sessions leaked ✓ - Test 2: atexit cleanup → 0 sessions leaked ✓ - test_gemini_session_leak.py: Automated validation Related: #14540 (broader OOM issue tracking) * fix(types): use LlmProviders enum for get_async_httpx_client MyPy was failing because llm_provider parameter expects Union[LlmProviders, httpxSpecialProvider], not a string. Changed from string "openai" to LlmProviders.OPENAI enum value. * test: move validation tests to proper CI directories - Move test_oom_fixes.py to tests/test_litellm/llms/ - Move test_gemini_session_leak.py to tests/test_litellm/llms/custom_httpx/ - Fix pytest warning: use pytest.skip() instead of return True This ensures CI actually runs our OOM fix validation tests. * fix(oom): add asyncio.Lock to prevent race conditions in Presidio session creation - Make _get_http_session() async with asyncio.Lock protection - Prevents multiple concurrent requests from creating orphaned sessions - Add concurrent load test (50 parallel requests) to validate fix - Test confirms only 1 session created under concurrent load Critical fix: Previous implementation had race condition where concurrent guardrail checks could create multiple sessions, defeating the shared session pattern and causing memory leaks. * fix(presidio): eliminate race condition in session lock initialization Move asyncio.Lock creation from lazy initialization in _get_http_session() to __init__. The previous lazy init had a race condition where concurrent coroutines could both see _session_lock as None, both create locks, and end up with different lock instances - defeating the synchronization. asyncio.Lock() can be safely created without an event loop; it only requires one when awaited.