* fix: prevent memory blowout in LoggingWorker
Tasks were being executed sequentially with each task awaited before
processing the next one. When the queue had 10k+ tasks, only one could
execute at a time. Since the request rate exceeded execution speed,
objects accumulated in memory (50k+), holding references to heavy
objects and causing memory blowout.
The new implementation uses a semaphore to allow up to 1000 concurrent
tasks while properly tracking and cleaning up each task, significantly
improving throughput and preventing queue buildup.
* fix: require semaphor before removing task from queue
* fix: make worker concurrency configurable
* fix: clean comments
* fix: clarify new env purpose
* fix: add missing lib
* make constants configurable instead of hardcoded
* add more aggressive cleaning when queue is full
* add helpers function for the aggressive cleaning functionality
* use envs instead of static constants
* import and document constants
* add unit test for new functionality
* fix default value on config_settings
* fix: remove unused variables and imports to resolve linter errors
- Remove unused time_since_last_clear variable in logging_worker.py
The variable was calculated but never used in _handle_queue_full()
method, causing F841 linter error.
- Remove unused TYPE_CHECKING import in mcp_server/server.py
The import was not used anywhere in the file, causing F401 linter error.
These changes improve code cleanliness and ensure the codebase passes
all linter checks without affecting functionality.
* add missing log expected by test_queue_full_handling
* fix: clean config_setting.md file
* fix: handle logging errors gracefully during shutdown in _flush_on_exit
During process shutdown, logging handlers may be closed while _flush_on_exit
tries to flush queued logging coroutines. This causes 'ValueError: I/O
operation on closed file' errors when coroutines attempt to log.
Changes:
- Add _safe_log helper method that wraps logging calls and suppresses
errors when logging handlers are closed (ValueError, OSError, AttributeError)
- Replace all verbose_logger calls in _flush_on_exit with _safe_log
- Remove logging from exception handler in coroutine execution loop
to prevent cascading errors during shutdown
This ensures graceful shutdown even when logging handlers are closed,
which is common during process termination.
* feat(anthropic/chat/transformations): for claude-4-5-sonnet and opus-4-1 support passing structured output to anthropic api
* docs: document new feature
* fix: fix output format
* fix: cleanup
* fix(transformation.py): conditionally pass in json tool call
* fix: support ARIZE_SPACE_ID instead of ARIZE_SPACE_KEY
* docs(arize_integration.md): cleanup arize docs
* feat(callback_info_helpers.tsx): allow setting arize space id via ui
* fix: fix linting error
* fix(opentelemetry.py): working arize phoenix root span tracing
* docs: Add mini-swe-agent to projects page
Add mini-swe-agent to the documentation projects page.
mini-swe-agent is a minimal AI coding agent that resolves >70% of
GitHub issues in SWE-bench, built on LiteLLM for model flexibility.
- Added projects/mini-swe-agent.md documentation
- Updated sidebars.js to include mini-swe-agent in projects list
* docs: Update Singularity to Apptainer in mini-swe-agent.md
- Add scope and url attributes to WebSocket mock in test_user_api_key_auth_websocket
- Add shared_realtime_ssl_context initialization in realtime handler test
This commit fixes two critical test failures and two test isolation issues
in the SSL configuration tests.
## Critical Test Failures Fixed
### 1. test_get_ssl_configuration
**Problem:** Test was failing with assertion error that ssl.create_default_context
was never called (expected 1 call, got 0).
**Root Cause:** The get_ssl_configuration() function uses a caching mechanism
(_ssl_context_cache) to avoid creating duplicate SSL contexts with the same
configuration. When tests run in sequence, a previous test may have created an
SSL context with the same configuration (same cafile, ssl_security_level,
ssl_ecdh_curve). When this test runs, it retrieves the cached context instead
of creating a new one, so ssl.create_default_context() is never called, causing
the mock assertion to fail.
**Fix:** Clear the SSL context cache at the start of the test to ensure a fresh
context is created, allowing the mock to be called and verified.
### 2. test_ssl_ecdh_curve
**Problem:** Test was failing with assertion error that set_ecdh_curve was
never called (expected 1 call, got 0).
**Root Cause:** Same caching issue as above. Additionally, the test needed to
use a real SSLContext instance instead of a MagicMock because _create_ssl_context
calls methods like set_ciphers() and minimum_version that require a real context.
**Fix:**
- Clear the SSL context cache at the start of the test
- Use a real SSLContext instance and patch set_ecdh_curve on it specifically
- Added explanatory comment about why a real context is needed
## Test Isolation Issues Fixed
### 3. test_ssl_security_level
**Problem:** Test was failing because it expected LiteLLMAiohttpTransport but
got httpx.AsyncHTTPTransport instead.
**Root Cause:** Test isolation issue. Other tests in the file (test_force_ipv4_transport,
test_aiohttp_disabled_transport) set litellm.disable_aiohttp_transport = True
but don't restore the original value. When this test runs after those tests,
aiohttp transport is disabled, causing it to use httpx transport instead.
**Fix:** Explicitly enable aiohttp transport at the start of the test and restore
the original value in a finally block, ensuring the test works regardless of
test execution order.
### 4. test_ssl_verification_with_aiohttp_transport
**Problem:** Same as above - expected LiteLLMAiohttpTransport but got
httpx.AsyncHTTPTransport.
**Root Cause:** Same test isolation issue - aiohttp transport disabled by
previous tests.
**Fix:** Same approach - explicitly enable aiohttp transport and restore
original value in finally block.
## Why These Fixes Work
1. **Cache clearing:** By clearing _ssl_context_cache before each test, we
ensure that get_ssl_configuration() creates a fresh SSL context, allowing
mocks to be properly called and verified.
2. **Test isolation:** By saving and restoring the disable_aiohttp_transport
setting, tests are independent of each other and work correctly regardless
of execution order.
These are minimal, targeted fixes that address the root causes without
modifying production code or affecting other functionality.
* Cache realtime websocket request body
Move the realtime request payload builder out of the websocket handler and wrap it with an LRU cache so repeated connections reuse the same bytes object. This keeps the JSON formatting cost down while bounding memory usage.
* Optimize realtime websocket caching
Refactored /v1/realtime to use cached helpers for both the JSON body and query params, introduced a reusable request-scope template, and optimized header handling to avoid redundant work.
* Refine realtime websocket header handling
* Reuse websocket scope headers in auth
* Refactor realtime request body helper
Move the realtime request body formatter into proxy common utils so it can be reused across modules. Reuse it in the websocket auth flow to share LRU caching and avoid ad hoc byte builders.
* fix: revert to old pattern
The old pattern was necessary, we can just return the optimized function instead.
* Reuse SSL context for realtime
Create a shared SSLContext for OpenAI realtime websocket dials and pass it into websockets.connect so we stop re-reading verify paths on every session.
* feat: reuse shared TLS context for realtime websockets
- add `SHARED_REALTIME_SSL_CONTEXT` helper so all realtime websocket clients share the same TLS settings
- wire the shared context into OpenAI, Azure, custom HTTPX handlers, and realtime health checks
- update realtime tests to assert that the expected SSL context is passed to `websockets.connect`
This keeps TLS configuration consistent and avoids recreating SSL contexts per connection.
* Reuse HTTP SSL context for realtime
Remove the standalone realtime SSL helper, expose a shared context directly from the HTTP handler, and point all realtime websocket clients and tests to it. Add the websocket header comparison tool.
* Lazy-load shared realtime SSL context
Fix circular imports introduced by eagerly instantiating the shared TLS context. Make the HTTP handler lazily create the context and have realtime clients/tests fetch it on demand, keeping configuration consistent without breaking startup.
* add: unit test for realtime LRU caches
* fix: merge conflict with imports
yuneng-jiang