* test(test_router.py): initial unit test confirming router.afile_content uses dynamic api key / api base
* fix(managed_files.py): filter deployments for only those within file id mapping
ensure call works - only route to models where the file was written
* fix(proxy_server.py): fix loading in model ids from config, if config id is int
* fix(router.py): return all model file id mappings on create_file
if multiple deployments - this ensures all the file id mappings are bubbled up
Fixes issue when trying to use loadbalanced deployments - only 1 file id mapping was being stored
* feat(router_utils/common_utils.py): filter models by team id when selecting for routing
Prevents team only models from being used by other teams
* fix(common_utils.py): additional fixes around filtering team-based models
* fix(batches_endpoints/endpoints): support list batches with target model names specified
* fix(common_utils.py): more testing for team deployment filters
* test(test_router.py): initial unit test confirming router.afile_content uses dynamic api key / api base
* fix(managed_files.py): filter deployments for only those within file id mapping
ensure call works - only route to models where the file was written
* fix(proxy_server.py): fix loading in model ids from config, if config id is int
* fix(router.py): return all model file id mappings on create_file
if multiple deployments - this ensures all the file id mappings are bubbled up
Fixes issue when trying to use loadbalanced deployments - only 1 file id mapping was being stored
* init litellm google gen ai methods
* feat init structure of functions for generate content
* add init
* add BaseGoogleGenAIGenerateContentConfig
* add generate_content_handler
* add get_provider_google_genai_generate_content_config
* fixes for generate content
* add get_vertex_ai_project etc to base
* use VertexBase
* fixes for BaseGoogleGenAIGenerateContentConfig
* working validate env for google gemini
* feat - add transform google response
* fixes for transform_generate_content_request
* fix get_supported_generate_content_optional_params
* add BaseGoogleGenAITest
* working e2e test
* fixes init config
* use correct types
* fix test for google gen ai
* fix types
* add sync_get_auth_token_and_url
* fixes for transform
* add llm http handler for google
* working non-streaming google endpoints
* add BaseGoogleGenAIGenerateContentStreamingIterator
* add GoogleGenAIGenerateContentStreamingIterator
* fix working sync stream
* fixes for litellm logging obj
* working async streaming
* add google gen ai types
* fix - required imports
* fix readme
* fix deps
* fix deps
* fix ruff code QA checks
* fix linting
* fixes TYPE_CHECKING
* fixes for typing
* add google gemini methods to litellm router
* [Feat] Add initial endpoints for using Gemini SDK (gemini-cli) with LiteLLM (#12040)
* init with google endpoints
* add Depends
* feat - add gemini endpoints
* google_generate_content
* fix init
* fixes import
* fixes for streaming
* fixes for sync/async
* working streaming with google gemini cli
* add google endpoints to llm api routes
* add VertexAIGoogleGenAIConfig
* use aiter_bytes
* use common request for streaming data
* re-use logic for anthropic streaming
* add GoogleAIStudioDataGenerator
* init litellm google gen ai methods
* feat init structure of functions for generate content
* add init
* add BaseGoogleGenAIGenerateContentConfig
* add generate_content_handler
* add get_provider_google_genai_generate_content_config
* fixes for generate content
* add get_vertex_ai_project etc to base
* use VertexBase
* fixes for BaseGoogleGenAIGenerateContentConfig
* working validate env for google gemini
* feat - add transform google response
* fixes for transform_generate_content_request
* fix get_supported_generate_content_optional_params
* add BaseGoogleGenAITest
* working e2e test
* fixes init config
* use correct types
* fix test for google gen ai
* fix types
* add sync_get_auth_token_and_url
* fixes for transform
* add llm http handler for google
* working non-streaming google endpoints
* add BaseGoogleGenAIGenerateContentStreamingIterator
* add GoogleGenAIGenerateContentStreamingIterator
* fix working sync stream
* fixes for litellm logging obj
* working async streaming
* add google gen ai types
* fix - required imports
* fix readme
* fix deps
* fix deps
* fix ruff code QA checks
* fix linting
* fixes TYPE_CHECKING
* fixes for typing
* docs: add Elasticsearch logging tutorial and update sidebar
* docs: update Elasticsearch logging tutorial to include OpenTelemetry setup and configuration
* docs: remove sections from Elasticsearch logging tutorial
* docs: remove analytics examples from Elasticsearch logging tutorial
* Update Elasticsearch version and logging exporter configuration in the Elasticsearch logging tutorial
* Add visualization instructions for LLM telemetry data in Kibana to Elasticsearch logging tutorial
* Add Elasticsearch demo image to documentation
* Move demo image for Elasticsearch logging tutorial
* fix(handler.py): support routing custom llm's to chat completion handler
Adds custom llm support for anthropic
* test(test_anthropic_experimental_pass_through_messages_handler.py): add unit test confirming custom llm respected
* docs(custom_llm_server.md): document anthropic custom llm translation
* test(volcengine.py): map thinking in extra body
Fixes https://github.com/BerriAI/litellm/issues/11879
* feat(main.py): support `azure/responses/<deployment-name>` model string
this allows us to route the model correctly
Closes https://github.com/BerriAI/litellm/issues/11879
* docs(azure_responses.md): document calling azure responses api models via chat completions bridge
Closes https://github.com/BerriAI/litellm/issues/11917
* fix: fix custom provider check
* test: update tests
* fix(team_endpoints.py): support setting default key expiry
allows admin to set key expiry on all team member keys
makes it easier to setup default team for experimentation
* feat(key_management_endpoints.py): allows admin to set duration for keys created by team members
* feat(team_endpoints.py): support team_member_key_duration on `/team/update`
allows setting max time team member keys are valid for
* fix(team_info.tsx): ui component to update team member key duration
* fix(team_info.tsx): support updating team member key duration, if set
* feat(teams.tsx): add team member key duration param ui component
allow admin to set this on UI
* feat(ui_sso.py): support restricting ui access by sso group
allows controlling who can/can't access the UI
* feat(ssomodals.tsx): add initial commit adding sso group access to admin ui
* feat(proxy_server.py): support reading + writing ui_access_mode from db
allows admin to configure allowed sso groups from UI
* feat(ui_sso.py): support enforcing all teams on sso jwt handler
if ui access mode set via ui, support reading the value and enforcing it
* feat(ui/): ui component for controlling sso access group
allow admin to only allow users within specific sso group to log into UI
* fix(uiaccesscontrolform.tsx): fix field names
* feat(ui_sso.py): return received sso response in the clientside error message - enables easier debugging
* test: add unit tests
* fix: minor fixes
* Refactor unpack_defs to use iterative approach instead of recursion
- Replace recursive depth-first traversal with iterative queue-based approach
- Add collections.deque import for efficient queue operations
- Avoid potential stack overflow issues with deeply nested schemas
- Maintain same functionality while improving performance and safety
* Remove unused import of Set in common_utils.py
* Enhance type hinting for queue in unpack_defs function in common_utils.py
* Enhance unpack_defs function to handle key validation for parent structures in common_utils.py
- Added checks to ensure that the parent is a dictionary or list and that the key is of the appropriate type (string for dicts, integer for lists) before assigning the resolved schema.
- This improves the robustness of the unpack_defs function when dealing with various schema structures.
* SCIM fix new user roles
* test_create_user_defaults_to_viewer
* test_create_user_uses_default_internal_user_params_role
* fix default user for SCIM
* fix linting error
* refactor(unpack_defs): enhance handling of schema properties and anyOf structures
- Improved the unpack_defs function to handle top-level properties and nested structures more effectively.
- Added recursion for items in schemas and refined the handling of anyOf branches to ensure proper unpacking of references.
- Streamlined the logic for resolving $ref keys and managing nested schemas.
* test(unpack_defs): add test for resolving nested $ref in anyOf items
- Introduced a new test to verify that unpack_defs correctly resolves references within items of anyOf structures, addressing a specific bug scenario (Issue #11372).
- The test includes a minimal schema to ensure proper unpacking and validation of the resolved items schema.
* refactor(unpack_defs): implement a generic resolver for $ref entries
- Redesigned the unpack_defs function to provide a more robust and dependency-free implementation for resolving all $ref entries in JSON schemas.
- Introduced a depth-first traversal method that efficiently handles nested structures, including anyOf, allOf, and items, while avoiding infinite recursion.
- Enhanced memory management by resolving nodes in-place without creating a full dereferenced copy, improving performance and reducing overhead.
* Remove test for unpack_defs resolving nested references in anyOf items from test_utils.py
* Add test for unpack_defs resolving nested references in anyOf items
This commit introduces a new test to ensure that the unpack_defs function correctly resolves $ref references within items of anyOf schemas, addressing issue #11372. The test verifies that the unpacked schema contains the expected properties and structure.
* fix(litellm_logging.py): fix using router model id for logging calls
Fixes https://github.com/BerriAI/litellm/issues/11975#issuecomment-2995882238
* test(test_litellm_logging.py): add unit test for custom price tracking
* fix(vertex_ai/): don't send invalid format parameter to vertex
causes calls to fail
* fix(vertex_ai_context_caching.py): if cached content present and tools in message, cache tools as well
gemini throws errors if tools passed in alongside cached content
* test: add unit tests
* fix: fix linting errors
* test: test_vertex_ai_common_utils.py
update test
* fix(streaming_handler.py): unset response cost when creating model response
* feat(model_prices_and_context_window.json): add mistral-small-3.2-24b-instruct model with token costs and chat mode support
* fix(model_prices_and_context_window.json): update model paths to include 'openrouter' prefix for mistral-small-3.1 and 3.2
* feat: add citation_cost_per_token and search_queries_cost_per_1000 fields to ModelInfoBase
- Add citation_cost_per_token field to ModelInfoBase for Perplexity citation token costs
- Add search_queries_cost_per_1000 field to ModelInfoBase for Perplexity search query costs
- Update _get_model_info_helper to include these fields in model info responses
- Enables proper cost calculation for Perplexity-specific usage metrics
* feat: update Perplexity sonar-deep-research model pricing configuration
- Update input/output token costs to / per million tokens respectively
- Add reasoning token cost at per million tokens
- Add citation_cost_per_token at per million tokens (same as input)
- Add search_queries_cost_per_1000 at /bin/zsh.005 per 1000 search queries
- Remove deprecated search_context_cost_per_query structure
- Aligns with Perplexity's updated pricing model for deep research capabilities
* feat: implement Perplexity-specific cost calculator
- Create cost_per_token function for Perplexity provider
- Calculate standard input/output token costs
- Add citation token cost calculation using citation_cost_per_token rate
- Add reasoning token cost calculation with fallback to completion_tokens_details
- Add search query cost calculation using search_queries_cost_per_1000 rate
- Return separate prompt_cost and completion_cost for accurate billing
- Handles all Perplexity-specific usage metrics: citation_tokens, num_search_queries, reasoning_tokens
* feat: integrate Perplexity cost calculator with main cost calculation system
- Import perplexity_cost_per_token function in main cost calculator
- Add perplexity provider case to cost_per_token function
- Enables automatic routing of Perplexity cost calculations to provider-specific logic
- Maintains compatibility with existing cost calculation patterns
- Supports all Perplexity-specific cost metrics through unified interface
* feat: enhance Perplexity response transformation to extract cost-related fields
- Override transform_response method to extract Perplexity-specific usage fields
- Add _enhance_usage_with_perplexity_fields method to process API responses
- Extract citation_tokens from citations array using character-based estimation (~4 chars/token)
- Extract num_search_queries from both usage field and root level with priority handling
- Create usage object when none exists to ensure cost fields are always captured
- Handle empty citations and missing fields gracefully
- Enables automatic extraction of cost metrics from Perplexity API responses
* test: add comprehensive test suite for Perplexity cost calculation features
Add 82 comprehensive tests across 3 test files:
- test_perplexity_cost_calculator.py (59 tests):
* Cost calculation with citation tokens, search queries, reasoning tokens
* Various combinations and edge cases
* Integration with main cost calculator
* Model info access and validation
* Zero values and missing fields handling
- test_perplexity_chat_transformation.py (12 tests):
* Citation token extraction from API responses
* Search query extraction from usage and root fields
* Priority handling and field aggregation
* Empty citations and missing fields handling
* Token estimation accuracy validation
- test_perplexity_integration.py (11 tests):
* End-to-end cost calculation workflows
* High-volume and edge case scenarios
* Model info integration validation
* Case-insensitive provider matching
* Transformation preservation of existing fields
Ensures reliability and correctness of all Perplexity cost features with comprehensive coverage of happy path, edge cases, and error conditions.
* fix: remove unused Union import from Perplexity transformation
- Remove unused typing.Union import from litellm/llms/perplexity/chat/transformation.py
- Fixes F401 linting error: 'typing.Union imported but unused'
- Maintains only necessary imports: Any, List, Optional, Tuple
* Fix JSON schema validation and use web_search_requests field
- Add citation_cost_per_token and search_queries_cost_per_1000 to JSON schema
- Update Perplexity transformation to use web_search_requests in PromptTokensDetailsWrapper
- Update Perplexity cost calculator to read from web_search_requests field
- Maintain backward compatibility while using standard LiteLLM fields
* Fix type errors in Perplexity cost calculator
- Add null checks for token counts and cost values to prevent None multiplication errors
- Use .get() with fallback values instead of direct dictionary access
- Ensure all arithmetic operations handle None values safely
This fixes the failing job 44517525148 type errors.
* Refactor Perplexity cost calculation tests to improve accuracy and consistency
- Replace absolute difference assertions with math.isclose for better precision in cost comparisons
- Update tests to utilize PromptTokensDetailsWrapper for handling web search requests
- Ensure all test cases correctly reflect the new structure of usage fields, enhancing clarity and maintainability
* fix: address type hinting issues in PerplexityChatConfig usage handling
- Add type ignore comments to model_response.usage assignments to resolve type checking errors
- Ensures compatibility with type definitions while maintaining existing functionality
* Update model pricing configuration in JSON backup
- Add citation_cost_per_token and search_queries_cost_per_1000 fields to enhance cost tracking
- Remove deprecated search_context_cost_per_query structure to streamline pricing model
- Aligns with recent updates in Perplexity's pricing strategy
* Update search queries cost structure in model_prices_and_context_window.json to use search_context_cost_per_query
* Refactor search queries cost structure in model_prices_and_context_window_backup.json and update related code to use search_queries_cost_per_query. Remove deprecated search_queries_cost_per_1000 references across model info and tests.
* Enhance cost calculation in cost_calculator.py by introducing a safe float casting function to handle potential None and invalid values. Update cost calculations for input, citation, output, reasoning, and search query tokens to use this new function, ensuring more robust handling of model pricing data.
* Refactor cost calculation in cost_calculator.py to support both legacy and current search cost keys. Enhance handling of search cost values by accommodating both dictionary and float formats, ensuring robust cost computation for search queries.
* Update test cases to reflect changes in cost structure, renaming search_queries_cost_per_query to search_context_cost_per_query for consistency with recent refactor. Ensure assertions in tests align with updated cost keys.
* Update test_perplexity_integration.py to rename search_queries_cost_per_query to search_context_cost_per_query, ensuring consistency with recent cost structure changes. Adjust assertions to align with updated cost keys.
Ishaan Jaff