* fix(add_model_modes.tsx): add 'batch' mode to ui
* fix(main.py): support health checks on batches + support litellm_credentials on batches
* fix(add_model_tab.tsx): clarify what 'team' on add model means
- Changed groq/moonshotai-kimi-k2-instruct to groq/moonshotai/kimi-k2-instruct in model_prices_and_context_window.json
- Added groq/moonshotai/kimi-k2-instruct and groq/qwen-qwq-32b to the supported models table in Groq documentation
* fix(prompt_templates/factory.py): handle anthropic cache control on individual tool results
Fixes issue where cache control on individual tool result was being ignored
* test(test_vertex_And_google_ai_studio_gemini.py): initial unit test covering translation for grounding metadata on streaming chunk
* fix(vertex_and_google_ai_studio.py): ensure grounding metadata is preserved on streaming
Closes https://github.com/BerriAI/litellm/issues/10237
* fix(core_helpers.py): include usage in expected openai keys
* feat(bulk_edit_user.tsx): initial working ui for editing users in bulk on the ui
easier to give access / assign to a default team
* feat(team_endpoints-+-bulk_edit_users.tsx): add bulk adding users to teams
make it easier to add existing users to a default team
* fix(bulk_edit_user.tsx): fix ui linting error
* fix: fix linting error
* feat: add v0 provider support to LiteLLM
- Add v0 as a new OpenAI-compatible provider
- Support all three v0 models: v0-1.0-md, v0-1.5-md, v0-1.5-lg
- Configure correct token limits and pricing for each model
- Enable vision support for all v0 models (multimodal)
- Add provider detection for v0/ prefix and api.v0.dev endpoint
- Include comprehensive unit tests for the provider
The v0 provider uses the standard OpenAI-compatible implementation
and supports all standard features including streaming, function
calling, and system messages.
* fix: add v0 provider to ProviderConfigManager
Add V0ChatConfig to the get_provider_chat_config method to fix
test_supports_tool_choice test failure. The v0 provider needs to
be included in the provider config manager to return the correct
configuration for tool choice support detection.
* docs: add documentation for v0 provider
- Add comprehensive v0 provider documentation
- Cover all supported models and their capabilities
- Include examples for SDK usage, proxy configuration, and all features
- Document supported OpenAI parameters based on v0 API docs
- Add v0 to the providers sidebar navigation
* fix: correct v0 supported OpenAI parameters
Based on review feedback and v0 API documentation:
- v0 only supports: messages, model, stream, tools, tool_choice
- Remove unsupported parameters like temperature, max_tokens, etc.
- Update tests to verify correct parameter set
- Update documentation to reflect actual API capabilities
- Remove JSON mode example as response_format is not supported
Reference: https://v0.dev/docs/v0-model-api#request-body
* fix: remove supports_response_schema from v0 models
Remove the supports_response_schema property from all v0 models in the model configuration files as v0 does not support this feature.
Models updated:
- v0/v0-1.0-md
- v0/v0-1.5-md
- v0/v0-1.5-lg
* fix(lowest_latency.py): Handle ZeroDivisionError with zero completion tokens (#12641)
Fixes ZeroDivisionError when LLM responses have zero completion tokens, which can
occur with Gemini models on very long contexts that only use tool calls.
Changes:
- Add check for completion_tokens > 0 before division in log_success_event
- Handle both timedelta and float types for response times (supporting both time.time() and datetime)
- Apply fix to both occurrences of the division operation in the file
- Add comprehensive tests for zero completion token scenarios
This ensures the lowest latency routing continues to work properly even when
models return responses with no completion tokens.
* test: Move lowest_latency zero tokens test to test_litellm for CI execution
* refactor: Use safe_divide helper to eliminate code duplication
- Added safe_divide utility function to litellm_core_utils.core_helpers
- Handles both timedelta and float types for numerator
- Prevents ZeroDivisionError and negative denominator issues
- Replaced duplicated division logic in lowest_latency.py
- Added comprehensive unit tests for safe_divide function
This improves code quality by reducing duplication and centralizing the
division safety logic in a reusable helper function.
* refactor: Rename to safe_divide_seconds for clarity
- Renamed safe_divide to safe_divide_seconds to better indicate it handles time durations
- Simplified implementation with single-line ternary for seconds conversion
- Updated all references and tests accordingly
The function name now clearly indicates it's specifically for dividing
time durations (in seconds) by a denominator.
* refactor: Simplify safe_divide_seconds to only accept float arguments
- Changed safe_divide_seconds to accept only float parameters for consistency
- Callers now handle timedelta to seconds conversion explicitly
- Removed timedelta-specific test cases
* fix: Remove unused timedelta import
* fix: Remove Union[timedelta, float] type annotations
Since safe_divide_seconds now only accepts floats, we handle the
timedelta conversion explicitly in the code. The type annotations
are no longer needed and can be simplified.
* Revert "fix: Remove Union[timedelta, float] type annotations"
This reverts commit 19c6c62154fceb9c431fea3e448753495dec35d2.
* fix: Clean up implementation
- Remove unnecessary type annotations
- Use safe_divide_seconds utility for zero-division protection
- Handle both timedelta and float types explicitly
- Maintain compatibility with both datetime and time.time() usage
* feat: integrate Google Cloud Model Armor guardrails (LIT-298)
- Add ModelArmorGuardrail class that extends CustomGuardrail and VertexBase
- Support for both pre-call (sanitizeUserPrompt) and post-call (sanitizeModelResponse) sanitization
- Integrate with existing Vertex AI authentication using VertexBase
- Add configuration model for Model Armor in guardrail types
- Register Model Armor in guardrail initializers and registry
- Include comprehensive test suite for Model Armor functionality
- Support content masking for both requests and responses
- Handle streaming responses with content sanitization
This integration allows LiteLLM to use Google Cloud Model Armor API for
content moderation and sanitization, providing similar functionality to
Bedrock Guards but using Google Cloud's security infrastructure.
* fix: remove unused imports flagged by ruff linter
- Remove unused asyncio import at top level (moved to local import where needed)
- Remove unused TextCompletionResponse import
* fix: remove additional unused imports
- Remove unused TextChoices import from line 34
- Remove duplicate asyncio import from line 384
- Replace asyncio.iscoroutine() with hasattr check for __await__
* fix: remove final unused imports
- Remove unused 'import sys' from line 9
- Remove unused 'StreamingChoices' from imports
* fix: remove unused import from model_armor.py
- Remove unused 'import os' from the top of the file
* fix: remove commented-out header from model_armor.py
- Eliminate unnecessary comments at the top of the file to improve code clarity.
* feat(guardrails): Add Model Armor UI support
- Convert model_armor.py to a directory structure with __init__.py for dynamic discovery
- Add get_config_model() method to ModelArmorGuardrail class for UI integration
- Add ui_friendly_name() to ModelArmorConfigModel returning "Google Cloud Model Armor"
- Remove manual registration from guardrail_registry.py to use dynamic discovery
- Model Armor now appears in the UI guardrails dropdown with proper configuration fields
This enables users to configure Model Armor guardrails through the LiteLLM UI interface.
* fix(guardrails): Fix undefined name 'GuardrailConfigModel' in Model Armor
- Import TYPE_CHECKING and GuardrailConfigModel from base module
- Fixes F821 linting error for undefined name in type annotation
- Follows same pattern as other guardrail implementations
* fix(guardrails): Fix Model Armor type errors and config model inheritance
- Create ModelArmorGuardrailConfigModel that properly inherits from GuardrailConfigModel base class
- Move config model to litellm/types/proxy/guardrails/guardrail_hooks/model_armor.py following convention
- Update get_config_model() to return the properly typed config model
- Remove ModelArmorConfigModel from LitellmParams inheritance chain
- Add template_id field to BaseLitellmParams instead
This fixes the mypy type errors and follows the same pattern as other guardrail implementations.
* fix(guardrails): Add missing Model Armor fields to BaseLitellmParams
- Add location, credentials, api_endpoint, and fail_on_error fields
- Fixes mypy errors about missing attributes in LitellmParams
- All Model Armor configuration parameters are now properly defined
* fix: Apply PR review feedback for Model Armor guardrail
- Move test file to tests/test_litellm/ for GitHub Actions
- Extract only last consecutive user messages to avoid context limits
- Use get_content_from_model_response helper for response extraction
- Handle non-ModelResponse types (e.g., TTS) gracefully
- Maintain newline separation for multi-part content
* refactor: Simplify message content extraction in ModelArmorGuardrail
- Removed the custom _extract_content_from_messages method.
- Integrated get_last_user_message helper for improved content extraction.
- Updated test to reflect changes in content formatting.
* refactor: Remove unused import in model_armor.py
- Deleted the unused import of AllMessageValues to clean up the codebase.
* Add unit tests for Model Armor guardrail functionality
- Implement tests for pre-call and post-call hooks, including content sanitization and blocking behavior.
- Validate error handling for API responses and credential management.
- Test streaming responses and handling of list content in user messages.
- Ensure proper assertions for API interactions and response sanitization.
* Add comprehensive test coverage for Model Armor guardrail
- Add test for requests with no messages field
- Add test for empty message content handling
- Add test for system/assistant-only messages
- Add test for fail_on_error=False behavior
- Add test for custom API endpoint configuration
- Add test for dictionary credentials (non-file path)
- Add test for action=NONE response handling
- Add test for missing sanitized_text field fallback
- Add test for non-text response types (TTS/image)
- Add test for auth token refresh behavior
All tests ensure robust edge case handling and proper error management.
* feat: improve Model Armor handling of non-ModelResponse types
- Add debug logging when skipping non-text responses (TTS, images, etc.)
- Improve docstring to clarify behavior for non-text responses
- Add test coverage for non-ModelResponse handling
- Ensure guardrail gracefully skips processing for response types it cannot handle
* build: move build_and_test to use prisma migrate
* fix(guardrails_ai.py): default to guardrail accepting 'llmOutput' as the input param
enables same guardrail to work for pre call and post call
* fix(__init__.py): set default value
* fix(guardrails_ai.py): updates
* fix: fix linting error
* fix(prompt_templates/factory.py): handle anthropic cache control on individual tool results
Fixes issue where cache control on individual tool result was being ignored
* test(test_vertex_And_google_ai_studio_gemini.py): initial unit test covering translation for grounding metadata on streaming chunk
* fix(google_genai/adapters/transformation.py): enable calling non-googlegenai models via streaming
Fixes https://github.com/BerriAI/litellm/issues/12562
* test(test_openai.py): add unit test asserting streaming works as expected
* Vllm rerank (#12737)
* Add Hosted VLLM rerank provider integration
This commit implements the Hosted VLLM rerank provider integration for LiteLLM. The integration includes:
Adding Hosted VLLM as a supported rerank provider in the main rerank function
Implementing the HostedVLLMRerank handler class for making API requests
Creating a transformation class to convert Hosted VLLM responses to LiteLLM's standardized format
The integration supports both synchronous and asynchronous rerank operations. API credentials can be provided directly or through environment variables (HOSTED_VLLM_API_KEY and HOSTED_VLLM_API_BASE).
Notable features:
Proper error handling for missing credentials
Standard response transformation
Support for common rerank parameters (top_n, return_documents, etc.)
Proper token usage tracking
This expands LiteLLM's rerank provider ecosystem to include Hosted VLLM alongside existing providers like Cohere, Together AI, Azure AI, and Bedrock.
* refactor(rerank): use base_llm_http_handler for hosted_vllm rerank
- Replace custom HostedVLLMRerank handler with base_llm_http_handler
- Implement proper HostedVLLMRerankConfig inheriting from BaseRerankConfig
- Follow Cohere-compatible implementation pattern
- Clean up unnecessary comments
* Fix lint errors in hosted_vllm rerank transformer: remove unused imports
* Fix linting errors in rerank transformation modules
* fix: resolve type errors in Hosted VLLM rerank module
---------
Co-authored-by: Philip D'Souza <philip.dsouza@macro4.com>
Co-authored-by: Philip D'Souza <philip.a.dsouza@gmail.com>
* added a few tests
---------
Co-authored-by: Philip D'Souza <philip.dsouza@macro4.com>
Co-authored-by: Philip D'Souza <philip.a.dsouza@gmail.com>
Ishaan Jaff