* fix: fix getting mcp servers
* fix(litellm_logging.py): handle list objects for final response in standard logging payload
Fixes issue where mcp tool call response wouldn't show up
* fix(litellm_responses_transformation/): remove invalid item error for unmapped objects - breaks stream and there's no real value to this as outside of a few of them, not all can be mapped to chat completions
resolves error for web search calls via chat completions to responses api
* feat(teams.py): param for disabling guardrails by team
allows use-case where you don't run global guardrails for team - only run team-specific guardrails
* feat(custom_guardrail.py): add support for disabling global guardrails
only run guardrails requested for in the request/key/team
* feat: support adding disable_global_guardrails to metadata if present in key/team metadata
* feat(create_key_button.tsx): new disable global guardrails field
* feat(key_edit_view.tsx): support disabling global guardrails on key edit
* feat(teams.tsx): add disable global guardrails on create team on UI
* feat(team_info.tsx): allow disabling global guardrails on team update
* fix: prevent memory blowout in LoggingWorker
Tasks were being executed sequentially with each task awaited before
processing the next one. When the queue had 10k+ tasks, only one could
execute at a time. Since the request rate exceeded execution speed,
objects accumulated in memory (50k+), holding references to heavy
objects and causing memory blowout.
The new implementation uses a semaphore to allow up to 1000 concurrent
tasks while properly tracking and cleaning up each task, significantly
improving throughput and preventing queue buildup.
* fix: require semaphor before removing task from queue
* fix: make worker concurrency configurable
* fix: clean comments
* fix: clarify new env purpose
* fix: add missing lib
* make constants configurable instead of hardcoded
* add more aggressive cleaning when queue is full
* add helpers function for the aggressive cleaning functionality
* use envs instead of static constants
* import and document constants
* add unit test for new functionality
* fix default value on config_settings
* fix: remove unused variables and imports to resolve linter errors
- Remove unused time_since_last_clear variable in logging_worker.py
The variable was calculated but never used in _handle_queue_full()
method, causing F841 linter error.
- Remove unused TYPE_CHECKING import in mcp_server/server.py
The import was not used anywhere in the file, causing F401 linter error.
These changes improve code cleanliness and ensure the codebase passes
all linter checks without affecting functionality.
* add missing log expected by test_queue_full_handling
* fix: clean config_setting.md file
* fix: handle logging errors gracefully during shutdown in _flush_on_exit
During process shutdown, logging handlers may be closed while _flush_on_exit
tries to flush queued logging coroutines. This causes 'ValueError: I/O
operation on closed file' errors when coroutines attempt to log.
Changes:
- Add _safe_log helper method that wraps logging calls and suppresses
errors when logging handlers are closed (ValueError, OSError, AttributeError)
- Replace all verbose_logger calls in _flush_on_exit with _safe_log
- Remove logging from exception handler in coroutine execution loop
to prevent cascading errors during shutdown
This ensures graceful shutdown even when logging handlers are closed,
which is common during process termination.
* feat(anthropic/chat/transformations): for claude-4-5-sonnet and opus-4-1 support passing structured output to anthropic api
* docs: document new feature
* fix: fix output format
* fix: cleanup
* fix(transformation.py): conditionally pass in json tool call
* fix: support ARIZE_SPACE_ID instead of ARIZE_SPACE_KEY
* docs(arize_integration.md): cleanup arize docs
* feat(callback_info_helpers.tsx): allow setting arize space id via ui
* fix: fix linting error
* fix(opentelemetry.py): working arize phoenix root span tracing
- Add scope and url attributes to WebSocket mock in test_user_api_key_auth_websocket
- Add shared_realtime_ssl_context initialization in realtime handler test
This commit fixes two critical test failures and two test isolation issues
in the SSL configuration tests.
## Critical Test Failures Fixed
### 1. test_get_ssl_configuration
**Problem:** Test was failing with assertion error that ssl.create_default_context
was never called (expected 1 call, got 0).
**Root Cause:** The get_ssl_configuration() function uses a caching mechanism
(_ssl_context_cache) to avoid creating duplicate SSL contexts with the same
configuration. When tests run in sequence, a previous test may have created an
SSL context with the same configuration (same cafile, ssl_security_level,
ssl_ecdh_curve). When this test runs, it retrieves the cached context instead
of creating a new one, so ssl.create_default_context() is never called, causing
the mock assertion to fail.
**Fix:** Clear the SSL context cache at the start of the test to ensure a fresh
context is created, allowing the mock to be called and verified.
### 2. test_ssl_ecdh_curve
**Problem:** Test was failing with assertion error that set_ecdh_curve was
never called (expected 1 call, got 0).
**Root Cause:** Same caching issue as above. Additionally, the test needed to
use a real SSLContext instance instead of a MagicMock because _create_ssl_context
calls methods like set_ciphers() and minimum_version that require a real context.
**Fix:**
- Clear the SSL context cache at the start of the test
- Use a real SSLContext instance and patch set_ecdh_curve on it specifically
- Added explanatory comment about why a real context is needed
## Test Isolation Issues Fixed
### 3. test_ssl_security_level
**Problem:** Test was failing because it expected LiteLLMAiohttpTransport but
got httpx.AsyncHTTPTransport instead.
**Root Cause:** Test isolation issue. Other tests in the file (test_force_ipv4_transport,
test_aiohttp_disabled_transport) set litellm.disable_aiohttp_transport = True
but don't restore the original value. When this test runs after those tests,
aiohttp transport is disabled, causing it to use httpx transport instead.
**Fix:** Explicitly enable aiohttp transport at the start of the test and restore
the original value in a finally block, ensuring the test works regardless of
test execution order.
### 4. test_ssl_verification_with_aiohttp_transport
**Problem:** Same as above - expected LiteLLMAiohttpTransport but got
httpx.AsyncHTTPTransport.
**Root Cause:** Same test isolation issue - aiohttp transport disabled by
previous tests.
**Fix:** Same approach - explicitly enable aiohttp transport and restore
original value in finally block.
## Why These Fixes Work
1. **Cache clearing:** By clearing _ssl_context_cache before each test, we
ensure that get_ssl_configuration() creates a fresh SSL context, allowing
mocks to be properly called and verified.
2. **Test isolation:** By saving and restoring the disable_aiohttp_transport
setting, tests are independent of each other and work correctly regardless
of execution order.
These are minimal, targeted fixes that address the root causes without
modifying production code or affecting other functionality.
* Cache realtime websocket request body
Move the realtime request payload builder out of the websocket handler and wrap it with an LRU cache so repeated connections reuse the same bytes object. This keeps the JSON formatting cost down while bounding memory usage.
* Optimize realtime websocket caching
Refactored /v1/realtime to use cached helpers for both the JSON body and query params, introduced a reusable request-scope template, and optimized header handling to avoid redundant work.
* Refine realtime websocket header handling
* Reuse websocket scope headers in auth
* Refactor realtime request body helper
Move the realtime request body formatter into proxy common utils so it can be reused across modules. Reuse it in the websocket auth flow to share LRU caching and avoid ad hoc byte builders.
* fix: revert to old pattern
The old pattern was necessary, we can just return the optimized function instead.
* Reuse SSL context for realtime
Create a shared SSLContext for OpenAI realtime websocket dials and pass it into websockets.connect so we stop re-reading verify paths on every session.
* feat: reuse shared TLS context for realtime websockets
- add `SHARED_REALTIME_SSL_CONTEXT` helper so all realtime websocket clients share the same TLS settings
- wire the shared context into OpenAI, Azure, custom HTTPX handlers, and realtime health checks
- update realtime tests to assert that the expected SSL context is passed to `websockets.connect`
This keeps TLS configuration consistent and avoids recreating SSL contexts per connection.
* Reuse HTTP SSL context for realtime
Remove the standalone realtime SSL helper, expose a shared context directly from the HTTP handler, and point all realtime websocket clients and tests to it. Add the websocket header comparison tool.
* Lazy-load shared realtime SSL context
Fix circular imports introduced by eagerly instantiating the shared TLS context. Make the HTTP handler lazily create the context and have realtime clients/tests fetch it on demand, keeping configuration consistent without breaking startup.
* add: unit test for realtime LRU caches
* fix: merge conflict with imports
* Add openai metadata filed in the request
* Add docs related to openai metadata
* Add utils
* test_completion_openai_metadata[True]
* Added support for though signature for gemini 3 in responses api (#16872)
* Added support for though signature for gemini 3
* Update docs with all supported endpoints and cost tracking
* Added config based routing support for batches and files
* fix lint errors
* Litellm anthropic image url support (#16868)
* Add image as url support to anthropic
* fix mypy errors
* fix tests
* Fix: Populate spend_logs_metadata in batch and files endpoints (#16921)
* Add spend-logs-metadata to the metadata
* Add tests for spend logs metadata in batches
* use better names
* Remove support for penalty param for gemini 3 (#16907)
* Remove support for penalty param
* remove halucinated model names
* fix mypy/test errors
* fix tests
* fix too many lines error
* fix too many lines error
* Add config for cicd test case
* Fix final tests
* fix batch tests
* fix batch tests
* attempt to implement the passthrough feature
* Formatting and small change
* Fix formatting
* Format test file
---------
Co-authored-by: Xiaohan Fu <xiaohan@grayswan.ai>
This fix addresses the same issue that was resolved for OpenAI video in PR #16708.
The GeminiVideoConfig class was importing BaseVideoConfig only within TYPE_CHECKING,
causing it to be 'Any' at runtime. This prevented the async_transform_video_content_response
method from being available during video content downloads.
Changes:
- Moved BaseVideoConfig import from TYPE_CHECKING to top-level imports
- Added test_gemini_video_config_has_async_transform() to verify the fix
- Ensures GeminiVideoConfig properly inherits BaseVideoConfig at runtime
Fixes video generation errors for Gemini Veo models:
'GeminiVideoConfig' object has no attribute 'async_transform_video_content_response'
* though signature tool call id
* [stripe] refactor and tests
* [stripe] remove md and move to factory
* [stripe] remove redudant test
* [stripe] ran black formatting
* [stripe] add thought signature docs
* [stripe] remove unused import
* Add thought signature support to v1/messages api
* update the thinking level handling logic
* update the thinking level handling logic
* Add streaming support
* fix intalling litellm error
Gemini 3 models require 'includeThoughts: True' in the thinkingConfig to return the actual thought text. Previously, using reasoning_effort set the 'thinkingLevel' but missed the boolean flag, resulting in empty reasoning_content.
This fix:
1. Updates `_map_reasoning_effort_to_thinking_level` to include `includeThoughts: True` for low/medium/high.
2. Adds unit tests to verify the config mapping.
Fixes#16805
When using Gemini models (2.5/3.0) with streaming + tools enabled,
the reasoning_content field was missing from stream chunks, even though
thinking_blocks were present in non-streaming responses.
Changes:
- Convert thinking_blocks to reasoning_content for streaming responses
- Extract "thinking" field from each thinking_block
- Concatenate multiple thinking parts with newlines
- Assign to reasoning_content in chat_completion_message for streaming
Testing:
- Added test_streaming_chunk_with_tool_calls_includes_reasoning_content
- Test verifies reasoning_content appears with tool calls in streaming
- All 39 existing Gemini tests pass
- Fix blank function name in completions response when using native function calling
- Fix Enum name being used instead of Enum value for comparison in chunk conversion
- Added additional tests to cover changes
Thanks to @mcowger for the invaluable assitance with figuring this issue out!
Fixed#16863
