* fix: preserve tool output ordering for gemini in responses bridge
- Keep function_call_output adjacent to its function_call when building chat messages
- Normalize function_call_output.output lists (input_* parts) into tool message content
* fix test
* small improvements
Update Pillar Security integration to use the generic_guardrail_api
instead of the dedicated pillar guardrail type. This aligns with
the Generic Guardrail API specification introduced in previous PRs.
Changes:
- Rewrite pillar_security.md with new generic_guardrail_api config
- Add Pillar Security example to generic_guardrail_api.md
- Add Pillar Security to quick_start.md guardrails examples
Related PRs: #17175, #18647, #18932, #19023
Emit Responses API streaming events for tool calls when the underlying chat stream contains tool_call deltas, and recover tool calls into the stream when they only appear in the final response.
The documentation incorrectly used `vertex_region` as the parameter name,
but the actual parameter expected by LiteLLM is `vertex_location` as defined
in VertexPassThroughCredentials and other type definitions.
Co-authored-by: Claude <noreply@anthropic.com>
* fix: prevent HTTP client memory leaks in Presidio and OpenAI wrappers
Fixes multiple memory leak issues reported in #14540 and related tickets:
**Presidio Guardrail Fix (#14540)**
- Problem: Every guardrail check created a new aiohttp.ClientSession
- Impact: High-traffic proxies accumulated thousands of unclosed sessions
- Solution: Share a single session across all guardrail checks
- Added `self._http_session` instance variable
- Lazy session creation via `_get_http_session()`
- Proper cleanup via `_close_http_session()` and `__del__()`
- Files: litellm/proxy/guardrails/guardrail_hooks/presidio.py
**OpenAI HTTP Client Caching (#14540)**
- Problem: `_get_async_http_client()` created new httpx.AsyncClient on each call
- Impact: OpenAI/Azure completions bypassed client caching system
- Solution: Route through `get_async_httpx_client()` for TTL-based caching
- Caches clients by provider and SSL config
- Fallback to direct creation if caching fails
- Applied to both async and sync client methods
- Files: litellm/llms/openai/common_utils.py
**Test Script**
- Added validation script to demonstrate fixes
- Counts file descriptors and unclosed session objects
- Files: test_oom_fixes.py
Related issues: #14384, #13251, #12443
* fix(oom): prevent memory leaks in Presidio guardrails and OpenAI client creation
Fixes two high-impact memory leaks:
1. Presidio Guardrail Session Leak (issue #14540)
- Problem: Created new aiohttp.ClientSession on every guardrail check
- Impact: Runs on EVERY proxy request when PII masking enabled
- Fix: Shared session pattern with lifecycle management
- Files: litellm/proxy/guardrails/guardrail_hooks/presidio.py
2. OpenAI HTTP Client Cache Bypass (issue #14540)
- Problem: _get_async_http_client() created new httpx.AsyncClient, bypassing TTL cache
- Impact: Every completion created new client with own connection pool
- Fix: Route through get_async_httpx_client() for proper caching
- Critical: Include SSL config in cache key for correctness
- Files: litellm/llms/openai/common_utils.py
Validation:
- Presidio: 100 requests → 0 new sessions (was 100)
- OpenAI: 100 calls → 1 unique client (was 100)
- test_oom_fixes.py: Automated validation script
* fix(oom): resolve Gemini aiohttp session leak (issue #12443)
Fixes persistent "Unclosed client session" warnings when using Gemini models.
Root Causes:
1. Broken atexit cleanup - get_event_loop() fails at exit time
2. On-demand session creation without reliable cleanup
Changes:
1. Fixed atexit Cleanup (async_client_cleanup.py)
- OLD: Used get_event_loop() which fails when loop is closed
- NEW: Always create fresh event loop at exit time
- Ensures cleanup runs successfully even when main loop is closed
2. Added __del__ Cleanup (aiohttp_handler.py)
- Defense-in-depth: cleanup on garbage collection
- Handles abnormal termination cases
- Similar pattern to Presidio guardrail fix
3. Enhanced Cleanup Scope (async_client_cleanup.py)
- Now closes global base_llm_aiohttp_handler instance
- Previously only checked cache, missed module-level handler
Validation:
- Test 1: __del__ cleanup → 0 sessions leaked ✓
- Test 2: atexit cleanup → 0 sessions leaked ✓
- test_gemini_session_leak.py: Automated validation
Related: #14540 (broader OOM issue tracking)
* fix(types): use LlmProviders enum for get_async_httpx_client
MyPy was failing because llm_provider parameter expects Union[LlmProviders, httpxSpecialProvider], not a string.
Changed from string "openai" to LlmProviders.OPENAI enum value.
* test: move validation tests to proper CI directories
- Move test_oom_fixes.py to tests/test_litellm/llms/
- Move test_gemini_session_leak.py to tests/test_litellm/llms/custom_httpx/
- Fix pytest warning: use pytest.skip() instead of return True
This ensures CI actually runs our OOM fix validation tests.
* fix(oom): add asyncio.Lock to prevent race conditions in Presidio session creation
- Make _get_http_session() async with asyncio.Lock protection
- Prevents multiple concurrent requests from creating orphaned sessions
- Add concurrent load test (50 parallel requests) to validate fix
- Test confirms only 1 session created under concurrent load
Critical fix: Previous implementation had race condition where
concurrent guardrail checks could create multiple sessions,
defeating the shared session pattern and causing memory leaks.
* fix(presidio): eliminate race condition in session lock initialization
Move asyncio.Lock creation from lazy initialization in _get_http_session()
to __init__. The previous lazy init had a race condition where concurrent
coroutines could both see _session_lock as None, both create locks, and
end up with different lock instances - defeating the synchronization.
asyncio.Lock() can be safely created without an event loop; it only
requires one when awaited.
* Add Volcengine responses adapter
* fix llms/volcengine/responses/transformation.py:507:9: F841 Local variable `origin` is assigned to but never used
fix llms/volcengine/responses/transformation.py:95: error: Argument "headers" to "VolcEngineError" has incompatible type
add more supported optional params
removed redundant manual logging/utils fallbacks so litellm/__init__.py uses the registry only.
* feat: add support for keda in helm chart
Signed-off-by: R.Sicart <roger.sicart@gmail.com>
* chore: bump chart version
---------
Signed-off-by: R.Sicart <roger.sicart@gmail.com>
- Add health_check_client.py for monitoring model availability
- Add health_check_client_README.md with usage documentation
- Add health_check_requirements.txt for dependencies
- Add run_parallel_health_checks.ps1 (PowerShell version)
- Add run_parallel_health_checks.sh (Bash version)
- Organize all scripts under scripts/health_check/ directory
* docs: update UI contributing guide with correct commands
- Replace outdated proxy_cli.py command with poetry run litellm
- Add config.yaml example with required settings
- Clarify that UI comes pre-built in the repo
- Add two development options: Build Mode and Dev Mode (hot reload)
- Note about redirect issues in Dev Mode
* docs: add hot reload login flow and PR submission section
- Document the 3000 -> 4000 -> 3000 login flow for hot reload
- Reorder: Hot Reload as Option A, Build Mode as Option B
- Add section 4 on submitting PRs
- Add note that UI changes don't require tests
* Update login flow navigation URL in contributing.md
