* feat: Add Prometheus metrics for request queue time and guardrails
- Add litellm_request_queue_time_seconds metric to track time from request arrival to processing start
- Add guardrail metrics: latency, errors_total, and requests_total counters
- Track arrival time in litellm_pre_call_utils.py
- Calculate queue time in common_request_processing.py
- Record guardrail metrics in pre_call_hook and during_call_hook
- Add comprehensive unit tests for all new metrics
Fixes#17863
* perf: optimize timing calls for queue time and guardrail metrics
* fix: resolve conflicts in utils.py - integrate Prometheus metrics with guardrail load balancing
* feat: prisma migrate deploy with lock
Author: Mini Jeong <mini.jeong@navercorp.com>
* fix: use redis cache from proxy server
Author: Mini Jeong <mini.jeong@navercorp.com>
* fix: add type checks and fix unit tests for migration lock
- Add DATABASE_URL validation in _create_baseline_migration() and _resolve_all_migrations()
- Fix MyPy type errors by adding None checks before using database_url in subprocess calls
- Add _resolve_all_migrations mock to failing unit tests to prevent filesystem errors
- Apply Black formatting to modified files
Fixes:
- MyPy type errors: database_url could be None when passed to subprocess
- Unit test failures: _resolve_all_migrations tried to create directories in read-only /test path
* fix: resolve MyPy type error in vertex_ai vertex_llm_base
Fix MyPy type checking error where vertex_api_version parameter type
was incompatible with function signature expectation.
* fix: Return 403 exception when calling GET responses api
* fix: added new step into rotate master key function for processing credentials table
* Add redisvl in requirements.txt
* fix: fixed the issue of handling root paths when processing Discovery protected resource metadata and authorization server metadata URLs.
* fix: added additional grant type into oauth_authorization_server response for fixing mcp auth register bad request issue
* fix: added RFC RECOMMENDED property(scopes_supported) to protected resource and authorization server metadata
* fix: removed initialize the tool name to MCP server name mapping(oauth2) on startup for avoiding 401 error
* fix: upgraded mcp sdk depency version for fixing ClosedResourceError
* Use already configured opentelemetry providers
Users that instrument using opentelemetry-instrument can now setup exporters as per their environment.
* Handle all protocols for all telemetry
* Add more tests
* feat(mcp): parallelize tool fetching from multiple MCP servers (#18627)
* feat(mcp): parallelize tool fetching from multiple MCP servers
Replace sequential tool fetching with asyncio.gather() to reduce
client timeouts when using multiple MCP servers.
Changes:
- mcp_server_manager.py: list_tools() now fetches tools in parallel
- server.py: _get_tools_from_mcp_servers() now fetches tools in parallel
Real-world impact (7 MCP servers example):
- Sequential: ~4.5+ seconds (exceeds typical 5-second client timeouts)
- Parallel: ~1.2 seconds (max of all servers)
Fixes#18626
* fix: copy oauth2_headers to avoid shared dict mutation in parallel tasks
* feat: add display_name, model_vendor, and model_version metadata
* added the option of adding langsmith tenant id in the env (#18623)
* fix(router): Validate routing_strategy at startup to fail fast with helpful error. (#18624)
Invalid routing_strategy values (e.g., "simple" instead of "simple-shuffle") previously failed silently, causing confusing "No deployments available" errors downstream. This change adds upfront validation in routing_strategy_init() to:
- Check if the provided strategy matches valid string values or RoutingStrategy enum
- Raise a clear ValueError listing valid options if invalid
- Fail fast at startup instead of at request time
Fixes behavior reported in #11330 where users had to debug cryptic errors.
Valid strategies: simple-shuffle, least-busy, usage-based-routing, latency-based-routing, cost-based-routing, usage-based-routing-v2
Co-authored-by: Flibbert E. Gibbitz <flibbertygibbitz@runelabs.ai>
* Add libsndfile to database Docker image for audio processing (#18612)
The litellm-database Docker image was missing the libsndfile system
library, which is required by the soundfile Python package for audio
file processing. This caused failures when using audio transcription
endpoints that attempt to calculate audio duration.
This adds libsndfile to the runtime dependencies in Dockerfile.database,
consistent with Dockerfile.alpine which already includes this library.
* Fix: Map Gemini cached_tokens to Langfuse cache_read_input_tokens (#18614)
* Fix: Map Gemini cached_tokens to Langfuse cache_read_input_tokens
Fixes#18520
## Problem
Langfuse integration was not capturing cached tokens from Gemini models.
Gemini returns cached tokens in `usage.prompt_tokens_details.cached_tokens`,
but Langfuse only read from top-level `usage.cache_read_input_tokens`
(which only Anthropic populates).
## Solution
Updated langfuse.py to check both locations:
1. First check top-level cache_read_input_tokens (for Anthropic)
2. Then check prompt_tokens_details.cached_tokens (for Gemini, OpenAI, others)
This ensures all providers' cached tokens are properly reported to Langfuse.
## Changes
- Modified litellm/integrations/langfuse/langfuse.py (lines 742-761)
- Added 3 unit tests in tests/test_litellm/integrations/langfuse/test_gemini_cached_tokens.py
- All existing Langfuse tests still pass (11/11)
## Testing
- test_cached_tokens_extraction: Verifies Gemini cached_tokens extraction
- test_cached_tokens_not_present: Backward compatibility (no cached_tokens)
- test_cached_tokens_is_zero: Edge case when cached_tokens = 0
* Refactor: Extract cache token logic into helper function
Address review feedback from @officer47p
- Created _extract_cache_read_input_tokens() helper function
- Reduces code bloat in _log_langfuse_v2 method
- Improves testability and reusability
- All tests still passing (11/11)
* Adding Role Mappings
* Fixing Edit SSO Settings Modal
* feat: add user_mcp_management_mode for view_all visibility
* Fixing tests
* fix: missing mcp_allow_all_ui.png
* docs: add user_mcp_management_mode
* Align responses API streaming hooks with chat pipeline
* Clarify responses API streaming context
* Address review comments
* feat: Add GigaChat provider support (#18564)
* feat: Add GigaChat provider support
Add native support for GigaChat API (Sber AI, Russia's leading LLM).
Supported features:
- Chat completions (sync/async)
- Streaming (sync/async)
- Function calling / Tools
- Structured output via JSON schema (emulated through function calls)
- Image input (base64 and URL)
- Embeddings
Closes#18515
* fix: resolve mypy type errors in GigaChat handler
- Fix _prepare_file_data return type (use 3-tuple for cleaner type flow)
- Add type annotations for lists in _process_content_parts methods
- Add type annotations in _collapse_user_messages
- Use ChatCompletionToolCallChunk for proper tool_use typing
- Add type: ignore[override] for astreaming async generator
* refactor(gigachat): migrate to BaseConfig pattern
* fix: remove unused imports
* fix: resolve mypy type errors
* fix: mypy type errors
* refactor: address review feedback for GigaChat provider
- Remove singleton pattern, reuse litellm HTTPHandler
- Move constants/errors to transformation files, delete common_utils.py
- Add models to model_prices_and_context_window.json
- Fix ssl_verify not passed to HTTP client for embeddings
* docs: update GigaChat documentation with ssl_verify requirement
* Revert "Add redisvl in requirements.txt"
* Put reasoning summary behind feat flag
* fix: model eol
* fix: anthropic claude-3-opus-20240229 EOL
* Revert "fix: model eol"
This reverts commit 5aa1665d79.
* Fix: ImportError: qualifire package is required for QualifireGuardrail. Install it with: pip install qualifire
* fix: test_secret_manager_failure_does_not_block_email
* fix: test_update_ui_settings_allowlisted_value
* fix: test_aaamodel_prices_and_context_window_json_is_valid
* fix: test_all_models_have_display_name
* fix: async def test_bedrock_apply_guardrail_blocked()
* fix: test_databricks_embeddings[True]
* fix:test_anthropic_beta_header
* fix:test_api_error_handling
* fix:mypy mcp management
* Revert "feat(model_cost): add display_name, model_vendor, and model_version metadata to model entries"
* [Feat] New API Endpoint - Responses API (v1/responses/compact) (#18697)
* init transform_compact_response_api_request
* init acompact_responses
* init async_compact_response_api_handler in llm http handler
* init transform_compact_response_api_request for openai
* init acompact_responses
* fix acompact_responses
* add OAI Compact API
* docs responses API Compact
* code qa checks
* test_openai_compact_responses_api
* fix mypy linting
* fix: remove display name
* Add the LITELLM_REASONING_AUTO_SUMMARY in doc
* fix model map
* [UI] - Feat add request provider form on UI (#18704)
* add request provider form
* fix link to github
* add button
* fix link
* fix(streaming): normalize status code extraction to prevent 4xx errors from triggering mid-stream fallback (#18698)
在流式处理错误时，添加状态码标准化逻辑，确保 4xx 客户端错误直接抛出而不是被包装成 MidStreamFallbackError。
- 新增 _normalize_status_code 函数用于从异常对象提取状态码
- 优先从异常的 status_code 属性获取，其次从 response.status_code 获取
- 当映射异常或原始异常的状态码在 400-499 范围内时，直接抛出映射异常
- 添加单元测试验证 Vertex AI 400 错误正确抛出为 BadRequestError
- 确保流式处理中的客户端错误能够正确传播，而不会触发回退机制
---------
Co-authored-by: Eric84626 <lixiannan@gmail.com>
Co-authored-by: Eric84626 <97266539+Eric84626@users.noreply.github.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: mangabits <1457532+mangabits@users.noreply.github.com>
Co-authored-by: Costa Tsaousis <costa@tsaousis.gr>
Co-authored-by: Nik <nikolas.garza5@gmail.com>
Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com>
Co-authored-by: FlibbertyGibbitz <seth@evenkeelconsultingllc.com>
Co-authored-by: Flibbert E. Gibbitz <flibbertygibbitz@runelabs.ai>
Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com>
Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com>
Co-authored-by: Yuta Saito <uc4w6c@bma.biglobe.ne.jp>
Co-authored-by: LingXuanYin <3546599908@qq.com>
Co-authored-by: YutaSaito <36355491+uc4w6c@users.noreply.github.com>
Co-authored-by: 0717376 <103773680+0717376@users.noreply.github.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Kris Xia <xiajiayi0506@gmail.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Calculate `total_tokens` in usage data in Response manually if:
- `total_tokens` is missing
- `total_tokens` can be calculated from input and output tokens
Run the test for this feature with:
`poetry run pytest tests/test_litellm/responses/test_responses_utils.py -k "test_transform_response_api_usage_calculates_total_from_input_and_output_tokens_if_available" -v`
* init guardrails
* init guardrails
* some fixes
* some fixes
* ruff
* some fixes
* some fixes
* some fixes
* some fixes
* some fixes
* some fixes
* docs
* Fix: Map Gemini cached_tokens to Langfuse cache_read_input_tokens
Fixes#18520
## Problem
Langfuse integration was not capturing cached tokens from Gemini models.
Gemini returns cached tokens in `usage.prompt_tokens_details.cached_tokens`,
but Langfuse only read from top-level `usage.cache_read_input_tokens`
(which only Anthropic populates).
## Solution
Updated langfuse.py to check both locations:
1. First check top-level cache_read_input_tokens (for Anthropic)
2. Then check prompt_tokens_details.cached_tokens (for Gemini, OpenAI, others)
This ensures all providers' cached tokens are properly reported to Langfuse.
## Changes
- Modified litellm/integrations/langfuse/langfuse.py (lines 742-761)
- Added 3 unit tests in tests/test_litellm/integrations/langfuse/test_gemini_cached_tokens.py
- All existing Langfuse tests still pass (11/11)
## Testing
- test_cached_tokens_extraction: Verifies Gemini cached_tokens extraction
- test_cached_tokens_not_present: Backward compatibility (no cached_tokens)
- test_cached_tokens_is_zero: Edge case when cached_tokens = 0
* Refactor: Extract cache token logic into helper function
Address review feedback from @officer47p
- Created _extract_cache_read_input_tokens() helper function
- Reduces code bloat in _log_langfuse_v2 method
- Improves testability and reusability
- All tests still passing (11/11)
This extends the previous fix to handle capitalized fields across ALL guardrail types,
including Presidio, Azure, Lakera, Bedrock, etc.
Discovery:
- Database investigation revealed the issue affects multiple guardrail types
- Found 4 affected guardrails in staging: 3 Presidio + 1 Azure
- All had default_action: 'Deny' causing the same validation failures
- The initial fix only covered ToolPermissionGuardrailConfigModel
Root Cause (Deeper):
- LitellmParams inherits from 13+ different guardrail config models
- Models use ConfigDict(extra="allow") allowing any field to be set
- Users can set default_action/on_disallowed_action on ANY guardrail type
- Only ToolPermissionGuardrailConfigModel was validating these fields
Solution:
- Added field validators to LitellmParams class (parent of all guardrails)
- Validators run for ALL guardrail types: Presidio, Azure, Bedrock, Lakera, etc.
- Added comprehensive tests covering multiple guardrail types
Changes:
- litellm/types/guardrails.py:
* Added @field_validator for default_action in LitellmParams
* Added @field_validator for on_disallowed_action in LitellmParams
* Added normalization in LitellmParams.__init__ as backup
* Imported field_validator from pydantic
- tests/test_litellm/types/test_guardrails_case_normalization.py:
* New test file with 7 tests covering multiple guardrail types
* Tests verify Presidio, Azure, Tool Permission, Lakera, Bedrock
* All tests passing
Impact:
- Previous fix: Only tool_permission guardrails protected
- This fix: ALL guardrail types now protected (13+ types)
- Handles both new writes and existing database records
- Tested against actual database with Presidio/Azure guardrails
Testing:
- 7 new cross-guardrail tests (all passing)
- 27 existing tool_permission tests (all passing)
- Verified fix works for real database records
Invalid routing_strategy values (e.g., "simple" instead of "simple-shuffle") previously failed silently, causing confusing "No deployments available" errors downstream. This change adds upfront validation in routing_strategy_init() to:
- Check if the provided strategy matches valid string values or RoutingStrategy enum
- Raise a clear ValueError listing valid options if invalid
- Fail fast at startup instead of at request time
Fixes behavior reported in #11330 where users had to debug cryptic errors.
Valid strategies: simple-shuffle, least-busy, usage-based-routing, latency-based-routing, cost-based-routing, usage-based-routing-v2
Co-authored-by: Flibbert E. Gibbitz <flibbertygibbitz@runelabs.ai>
This fixes a critical issue where capitalized values in tool_permission guardrail
configurations (e.g., "Deny" instead of "deny") caused Pydantic validation errors
during proxy startup, leading to repeated initialization failures and latency issues.
Problem:
- Users could save guardrails with capitalized values through UI/API
- Data was written to database without validation (e.g., default_action: "Deny")
- On proxy startup, loading from DB triggered strict Pydantic validation
- ValidationError caused guardrail initialization to fail in a retry loop
- This resulted in startup delays and repeated error logging
Root Cause:
- Write path had no case normalization
- Read path enforced strict lowercase Literal validation
- Asymmetry between write and read caused latent data corruption
Solution:
Added field validators to normalize case before Pydantic validation:
1. ToolPermissionRule.decision ("allow"/"deny")
- Normalizes decision field in rules array
2. ToolPermissionGuardrailConfigModel.default_action ("allow"/"deny")
- Normalizes default fallback action
3. ToolPermissionGuardrailConfigModel.on_disallowed_action ("block"/"rewrite")
- Normalizes disallowed tool behavior
4. ToolPermissionGuardrail.__init__ normalization
- Defensive normalization for direct instantiation
- Ensures normalization regardless of code path
Impact:
- Prevents validation errors during guardrail initialization
- Eliminates startup retry loops and latency issues
- Handles existing database records with capitalized values
- Accepts case-insensitive input from all sources (UI, API, direct calls)
- Fully backward compatible with existing lowercase configurations
Testing:
- Added 3 comprehensive tests for case-insensitive handling
- All 27 existing tests still pass
- Tests verify normalization across all affected fields
Files Changed:
- litellm/types/proxy/guardrails/guardrail_hooks/tool_permission.py
Added @field_validator decorators for case normalization
- litellm/proxy/guardrails/guardrail_hooks/tool_permission.py
Added runtime normalization in __init__ method
- tests/test_litellm/proxy/guardrails/guardrail_hooks/test_tool_permission.py
Added case-insensitive validation tests
* refactor: migrate utils.py lazy imports to registry pattern
- Refactored utils.py __getattr__ to use cached registry pattern (similar to __init__.py)
- Added UTILS_MODULE_NAMES tuple and _UTILS_MODULE_IMPORT_MAP to _lazy_imports_registry.py
- Added _get_utils_globals() helper function to _lazy_imports.py
- Added _lazy_import_utils_module() handler function for utils module lazy imports
- Updated _get_lazy_import_registry() to include utils module lazy imports
- Removed redundant _get_utils_globals() from utils.py (now in _lazy_imports.py)
- Added comprehensive tests for utils module lazy imports in test_lazy_imports.py
This refactoring:
- Reduces code duplication (from 670+ lines to ~10 lines in __getattr__)
- Improves maintainability (new lazy imports just need registry entry)
- Maintains consistency with __init__.py lazy import pattern
- All existing functionality preserved and tested
* Fix NameError: get_coroutine_checker not defined in check_coroutine function
* Fix lazy loading for get_coroutine_checker in function_setup and check_coroutine
- Add lazy loading for get_coroutine_checker at start of function_setup to ensure all calls use lazy-loaded version
- Fix check_coroutine function to use lazy loading pattern via getattr
- All direct calls to get_coroutine_checker() now properly use lazy import mechanism
