* Added ability to customize logfire base url through env var
* Added test to check if env var is used correctly for logfire
* Document the env var
* Documented env var in config_settings.md
When extract_and_raise_litellm_exception tries to raise a LiteLLM exception
from an error string, it was always passing the response parameter. However,
some exceptions like APIConnectionError don't accept this parameter, causing
a TypeError.
This fix tries to raise the exception with the response parameter first,
and falls back to raising without it if a TypeError occurs.
This fixes the error:
TypeError: APIConnectionError.__init__() got an unexpected keyword argument 'response'
Which was occurring when Gemini returned UNEXPECTED_TOOL_CALL finish reason
and LiteLLM tried to convert the error to an APIConnectionError.
Fixes: cascading error when Gemini uses thinking feature (__thought__ tool calls)
The `_get_request_tags` function was returning a reference to the
original tags list from metadata, then mutating it with `.extend()`.
This caused duplicate User-Agent tags when the function was called
multiple times during a single request lifecycle (e.g., by logging,
prometheus, and guardrails).
The fix uses `.copy()` to create a new list before extending, ensuring
the original metadata tags are not mutated.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Tianduo Zhai <tzhai@firsthandadmins-MacBook-Pro.local>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: prisma migrate deploy with lock
Author: Mini Jeong <mini.jeong@navercorp.com>
* fix: use redis cache from proxy server
Author: Mini Jeong <mini.jeong@navercorp.com>
* fix: add type checks and fix unit tests for migration lock
- Add DATABASE_URL validation in _create_baseline_migration() and _resolve_all_migrations()
- Fix MyPy type errors by adding None checks before using database_url in subprocess calls
- Add _resolve_all_migrations mock to failing unit tests to prevent filesystem errors
- Apply Black formatting to modified files
Fixes:
- MyPy type errors: database_url could be None when passed to subprocess
- Unit test failures: _resolve_all_migrations tried to create directories in read-only /test path
* fix: resolve MyPy type error in vertex_ai vertex_llm_base
Fix MyPy type checking error where vertex_api_version parameter type
was incompatible with function signature expectation.
* fix: Return 403 exception when calling GET responses api
* fix: added new step into rotate master key function for processing credentials table
* Add redisvl in requirements.txt
* fix: fixed the issue of handling root paths when processing Discovery protected resource metadata and authorization server metadata URLs.
* fix: added additional grant type into oauth_authorization_server response for fixing mcp auth register bad request issue
* fix: added RFC RECOMMENDED property(scopes_supported) to protected resource and authorization server metadata
* fix: removed initialize the tool name to MCP server name mapping(oauth2) on startup for avoiding 401 error
* fix: upgraded mcp sdk depency version for fixing ClosedResourceError
* Use already configured opentelemetry providers
Users that instrument using opentelemetry-instrument can now setup exporters as per their environment.
* Handle all protocols for all telemetry
* Add more tests
* feat(mcp): parallelize tool fetching from multiple MCP servers (#18627)
* feat(mcp): parallelize tool fetching from multiple MCP servers
Replace sequential tool fetching with asyncio.gather() to reduce
client timeouts when using multiple MCP servers.
Changes:
- mcp_server_manager.py: list_tools() now fetches tools in parallel
- server.py: _get_tools_from_mcp_servers() now fetches tools in parallel
Real-world impact (7 MCP servers example):
- Sequential: ~4.5+ seconds (exceeds typical 5-second client timeouts)
- Parallel: ~1.2 seconds (max of all servers)
Fixes#18626
* fix: copy oauth2_headers to avoid shared dict mutation in parallel tasks
* feat: add display_name, model_vendor, and model_version metadata
* added the option of adding langsmith tenant id in the env (#18623)
* fix(router): Validate routing_strategy at startup to fail fast with helpful error. (#18624)
Invalid routing_strategy values (e.g., "simple" instead of "simple-shuffle") previously failed silently, causing confusing "No deployments available" errors downstream. This change adds upfront validation in routing_strategy_init() to:
- Check if the provided strategy matches valid string values or RoutingStrategy enum
- Raise a clear ValueError listing valid options if invalid
- Fail fast at startup instead of at request time
Fixes behavior reported in #11330 where users had to debug cryptic errors.
Valid strategies: simple-shuffle, least-busy, usage-based-routing, latency-based-routing, cost-based-routing, usage-based-routing-v2
Co-authored-by: Flibbert E. Gibbitz <flibbertygibbitz@runelabs.ai>
* Add libsndfile to database Docker image for audio processing (#18612)
The litellm-database Docker image was missing the libsndfile system
library, which is required by the soundfile Python package for audio
file processing. This caused failures when using audio transcription
endpoints that attempt to calculate audio duration.
This adds libsndfile to the runtime dependencies in Dockerfile.database,
consistent with Dockerfile.alpine which already includes this library.
* Fix: Map Gemini cached_tokens to Langfuse cache_read_input_tokens (#18614)
* Fix: Map Gemini cached_tokens to Langfuse cache_read_input_tokens
Fixes#18520
## Problem
Langfuse integration was not capturing cached tokens from Gemini models.
Gemini returns cached tokens in `usage.prompt_tokens_details.cached_tokens`,
but Langfuse only read from top-level `usage.cache_read_input_tokens`
(which only Anthropic populates).
## Solution
Updated langfuse.py to check both locations:
1. First check top-level cache_read_input_tokens (for Anthropic)
2. Then check prompt_tokens_details.cached_tokens (for Gemini, OpenAI, others)
This ensures all providers' cached tokens are properly reported to Langfuse.
## Changes
- Modified litellm/integrations/langfuse/langfuse.py (lines 742-761)
- Added 3 unit tests in tests/test_litellm/integrations/langfuse/test_gemini_cached_tokens.py
- All existing Langfuse tests still pass (11/11)
## Testing
- test_cached_tokens_extraction: Verifies Gemini cached_tokens extraction
- test_cached_tokens_not_present: Backward compatibility (no cached_tokens)
- test_cached_tokens_is_zero: Edge case when cached_tokens = 0
* Refactor: Extract cache token logic into helper function
Address review feedback from @officer47p
- Created _extract_cache_read_input_tokens() helper function
- Reduces code bloat in _log_langfuse_v2 method
- Improves testability and reusability
- All tests still passing (11/11)
* Adding Role Mappings
* Fixing Edit SSO Settings Modal
* feat: add user_mcp_management_mode for view_all visibility
* Fixing tests
* fix: missing mcp_allow_all_ui.png
* docs: add user_mcp_management_mode
* Align responses API streaming hooks with chat pipeline
* Clarify responses API streaming context
* Address review comments
* feat: Add GigaChat provider support (#18564)
* feat: Add GigaChat provider support
Add native support for GigaChat API (Sber AI, Russia's leading LLM).
Supported features:
- Chat completions (sync/async)
- Streaming (sync/async)
- Function calling / Tools
- Structured output via JSON schema (emulated through function calls)
- Image input (base64 and URL)
- Embeddings
Closes#18515
* fix: resolve mypy type errors in GigaChat handler
- Fix _prepare_file_data return type (use 3-tuple for cleaner type flow)
- Add type annotations for lists in _process_content_parts methods
- Add type annotations in _collapse_user_messages
- Use ChatCompletionToolCallChunk for proper tool_use typing
- Add type: ignore[override] for astreaming async generator
* refactor(gigachat): migrate to BaseConfig pattern
* fix: remove unused imports
* fix: resolve mypy type errors
* fix: mypy type errors
* refactor: address review feedback for GigaChat provider
- Remove singleton pattern, reuse litellm HTTPHandler
- Move constants/errors to transformation files, delete common_utils.py
- Add models to model_prices_and_context_window.json
- Fix ssl_verify not passed to HTTP client for embeddings
* docs: update GigaChat documentation with ssl_verify requirement
* Revert "Add redisvl in requirements.txt"
* Put reasoning summary behind feat flag
* fix: model eol
* fix: anthropic claude-3-opus-20240229 EOL
* Revert "fix: model eol"
This reverts commit 5aa1665d79.
* Fix: ImportError: qualifire package is required for QualifireGuardrail. Install it with: pip install qualifire
* fix: test_secret_manager_failure_does_not_block_email
* fix: test_update_ui_settings_allowlisted_value
* fix: test_aaamodel_prices_and_context_window_json_is_valid
* fix: test_all_models_have_display_name
* fix: async def test_bedrock_apply_guardrail_blocked()
* fix: test_databricks_embeddings[True]
* fix:test_anthropic_beta_header
* fix:test_api_error_handling
* fix:mypy mcp management
* Revert "feat(model_cost): add display_name, model_vendor, and model_version metadata to model entries"
* [Feat] New API Endpoint - Responses API (v1/responses/compact) (#18697)
* init transform_compact_response_api_request
* init acompact_responses
* init async_compact_response_api_handler in llm http handler
* init transform_compact_response_api_request for openai
* init acompact_responses
* fix acompact_responses
* add OAI Compact API
* docs responses API Compact
* code qa checks
* test_openai_compact_responses_api
* fix mypy linting
* fix: remove display name
* Add the LITELLM_REASONING_AUTO_SUMMARY in doc
* fix model map
* [UI] - Feat add request provider form on UI (#18704)
* add request provider form
* fix link to github
* add button
* fix link
* fix(streaming): normalize status code extraction to prevent 4xx errors from triggering mid-stream fallback (#18698)
在流式处理错误时，添加状态码标准化逻辑，确保 4xx 客户端错误直接抛出而不是被包装成 MidStreamFallbackError。
- 新增 _normalize_status_code 函数用于从异常对象提取状态码
- 优先从异常的 status_code 属性获取，其次从 response.status_code 获取
- 当映射异常或原始异常的状态码在 400-499 范围内时，直接抛出映射异常
- 添加单元测试验证 Vertex AI 400 错误正确抛出为 BadRequestError
- 确保流式处理中的客户端错误能够正确传播，而不会触发回退机制
---------
Co-authored-by: Eric84626 <lixiannan@gmail.com>
Co-authored-by: Eric84626 <97266539+Eric84626@users.noreply.github.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: mangabits <1457532+mangabits@users.noreply.github.com>
Co-authored-by: Costa Tsaousis <costa@tsaousis.gr>
Co-authored-by: Nik <nikolas.garza5@gmail.com>
Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com>
Co-authored-by: FlibbertyGibbitz <seth@evenkeelconsultingllc.com>
Co-authored-by: Flibbert E. Gibbitz <flibbertygibbitz@runelabs.ai>
Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com>
Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com>
Co-authored-by: Yuta Saito <uc4w6c@bma.biglobe.ne.jp>
Co-authored-by: LingXuanYin <3546599908@qq.com>
Co-authored-by: YutaSaito <36355491+uc4w6c@users.noreply.github.com>
Co-authored-by: 0717376 <103773680+0717376@users.noreply.github.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Kris Xia <xiajiayi0506@gmail.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Fix Ollama_chatException "illegal base64 data at input byte 4" error
when using images with ollama_chat provider. Ollama expects pure base64
data, not the full data URL format (data:image/png;base64,...).
Fixes#18338
Signed-off-by: majiayu000 <1835304752@qq.com>
Fixes#18599
When OpenAI models (gpt-5-nano, o1-*, o3-*) and other providers return
reasoning_tokens in completion_tokens_details but don't provide text_tokens,
LiteLLM was incorrectly calculating costs using only reasoning_tokens,
ignoring the remaining completion tokens.
Changes:
- Modified generic_cost_per_token() in llm_cost_calc/utils.py to calculate
text_tokens as: completion_tokens - reasoning_tokens - audio_tokens - image_tokens
when text_tokens is not explicitly provided
- Added comprehensive test case test_reasoning_tokens_without_text_tokens_gpt5_nano()
to verify all completion_tokens are billed correctly
Example:
- completion_tokens: 977
- reasoning_tokens: 768
- Before: only 768 tokens billed (21% less)
- After: all 977 tokens billed correctly
Affected models:
- OpenAI: gpt-5-nano, o1-*, o3-*
- Perplexity: sonar-reasoning*
- Any model returning reasoning_tokens without text_tokens
* Allow get_nested_value dot notation to support escaping for Kubernetes JWT Support
* Add support for team and org alias fields, add docs, tests
* Fix lint issue with max statements in handle jwt logic
Fixes#18282
This PR fixes two issues with Gemini context window error handling:
1. **Pattern matching for Gemini 2.0 Flash**: The previous pattern
'input token count exceeds the maximum number of tokens allowed'
doesn't match Gemini 2.0 Flash errors which include dynamic token
counts like '(2800010)' in the message. Split into shorter patterns
that work with both formats.
2. **Add context window check to Gemini block**: The
is_error_str_context_window_exceeded() check was only called for
OpenAI-compatible providers, not for Gemini/Vertex AI. Added the
check to the Gemini-specific error handling block.
Test cases added for both Gemini 2.0 Flash and 2.5/3 error formats.
Add detection for Cerebras's context window exceeded error format:
"Current length is X while limit is Y"
This ensures LiteLLM raises ContextWindowExceededError instead of
generic BadRequestError when Cerebras API calls exceed the model's
context limit, enabling downstream libraries like DSPy to properly
catch and handle these errors for automatic context management.
- Filter async_log_success_event calls by expected input message
- Bridge models (openai/codex-mini-latest) may make internal calls that also log
- Test now asserts exactly one call with the expected input 'Hey' instead of asserting total call count
- Makes test robust to bridge-related double logging while still validating core behavior
When Gemini image generation models return `text_tokens=0` with `image_tokens > 0`,
the cost calculator was assuming no token breakdown existed and treating all
completion tokens as text tokens, resulting in ~10x underestimation of costs.
Changes:
- Fix cost calculation logic to respect token breakdown when image/audio/reasoning
tokens are present, even if text_tokens=0
- Add `output_cost_per_image_token` pricing for gemini-3-pro-image-preview models
- Add test case reproducing the issue
- Add documentation explaining image token pricing
Fixes#17410
Fixes#17425
- Add length check for tool_calls in model_response.choices[0].delta
- Prevents empty tool call objects from appearing in streaming responses
- Add regression tests for empty and valid tool_calls scenarios
* Fix Bedrock guardrail apply_guardrail method and test mocks
Fixed 4 failing tests in the guardrail test suite:
1. BedrockGuardrail.apply_guardrail now returns original texts when guardrail
allows content but doesn't provide output/outputs fields. Previously returned
empty list, causing test_bedrock_apply_guardrail_success to fail.
2. Updated test mocks to use correct Bedrock API response format:
- Changed from 'content' field to 'output' field
- Fixed nested structure from {'text': {'text': '...'}} to {'text': '...'}
- Added missing 'output' field in filter test
3. Fixed endpoint test mocks to return GenericGuardrailAPIInputs format:
- Changed from tuple (List[str], Optional[List[str]]) to dict {'texts': [...]}
- Updated method call assertions to use 'inputs' parameter correctly
All 12 guardrail tests now pass successfully.
* fix: remove python3-dev from Dockerfile.build_from_pip to avoid Python version conflict
The base image cgr.dev/chainguard/python:latest-dev already includes Python 3.14
and its development tools. Installing python3-dev pulls Python 3.13 packages
which conflict with the existing Python 3.14 installation, causing file
ownership errors during apk install.
* fix: disable callbacks in vertex fine-tuning tests to prevent Datadog logging interference
The test was failing because Datadog logging was making an HTTP POST request
that was being caught by the mock, causing assert_called_once() to fail.
By disabling callbacks during the test, we prevent Datadog from making any
HTTP calls, allowing the mock to only see the Vertex AI API call.
* fix: ensure test isolation in test_logging_non_streaming_request
Add proper cleanup to restore original litellm.callbacks after test execution.
This prevents test interference when running as part of a larger test suite,
where global state pollution was causing async_log_success_event to be
called multiple times instead of once.
Fixes test failure where the test expected async_log_success_event to be
called once but was being called twice due to callbacks from previous tests
not being cleaned up.
* fix: fix getting mcp servers
* fix(litellm_logging.py): handle list objects for final response in standard logging payload
Fixes issue where mcp tool call response wouldn't show up
* fix(litellm_responses_transformation/): remove invalid item error for unmapped objects - breaks stream and there's no real value to this as outside of a few of them, not all can be mapped to chat completions
resolves error for web search calls via chat completions to responses api
* fix: prevent memory blowout in LoggingWorker
Tasks were being executed sequentially with each task awaited before
processing the next one. When the queue had 10k+ tasks, only one could
execute at a time. Since the request rate exceeded execution speed,
objects accumulated in memory (50k+), holding references to heavy
objects and causing memory blowout.
The new implementation uses a semaphore to allow up to 1000 concurrent
tasks while properly tracking and cleaning up each task, significantly
improving throughput and preventing queue buildup.
* fix: require semaphor before removing task from queue
* fix: make worker concurrency configurable
* fix: clean comments
* fix: clarify new env purpose
* fix: add missing lib
* make constants configurable instead of hardcoded
* add more aggressive cleaning when queue is full
* add helpers function for the aggressive cleaning functionality
* use envs instead of static constants
* import and document constants
* add unit test for new functionality
* fix default value on config_settings
* fix: remove unused variables and imports to resolve linter errors
- Remove unused time_since_last_clear variable in logging_worker.py
The variable was calculated but never used in _handle_queue_full()
method, causing F841 linter error.
- Remove unused TYPE_CHECKING import in mcp_server/server.py
The import was not used anywhere in the file, causing F401 linter error.
These changes improve code cleanliness and ensure the codebase passes
all linter checks without affecting functionality.
* add missing log expected by test_queue_full_handling
* fix: clean config_setting.md file
* fix: handle logging errors gracefully during shutdown in _flush_on_exit
During process shutdown, logging handlers may be closed while _flush_on_exit
tries to flush queued logging coroutines. This causes 'ValueError: I/O
operation on closed file' errors when coroutines attempt to log.
Changes:
- Add _safe_log helper method that wraps logging calls and suppresses
errors when logging handlers are closed (ValueError, OSError, AttributeError)
- Replace all verbose_logger calls in _flush_on_exit with _safe_log
- Remove logging from exception handler in coroutine execution loop
to prevent cascading errors during shutdown
This ensures graceful shutdown even when logging handlers are closed,
which is common during process termination.
Sameer Kankute