* Add Azure OpenAI assistant features cost tracking
Implements cost tracking for Azure's new assistant features:
- File Search: $0.1 USD per 1 GB/Day (storage-based pricing)
- Code Interpreter: $0.03 USD per session
- Computer Use: $0.003 input + $0.012 output per 1K tokens
Features:
- Provider-specific pricing (Azure vs OpenAI)
- Model-specific pricing overrides via JSON config
- Environment variable configuration
- Backwards compatible with existing OpenAI pricing
* Add comprehensive tests for Azure assistant features cost tracking
- Unit tests for file search, code interpreter, computer use, vector store
- Integration tests for combined cost calculation
- Provider-specific pricing tests (Azure vs OpenAI)
- Model-specific pricing override tests
- Edge case handling (None inputs, zero values)
- All 17 tests passing
* Fix test and ensure all Azure assistant cost tracking tests pass
- Fixed integration test approach
- All 17 tests now passing
- Comprehensive coverage of Azure assistant features cost tracking
* Enhance cost tracking for Azure assistant features
- Safely convert and extract parameters for file search, computer use, and code interpreter sessions.
- Ensure model_info is consistently converted to a dictionary format.
- Improve error handling for input values to prevent type-related issues.
- Maintain compatibility with existing cost calculation methods.
* Refactor cost tracking for Azure assistant features
- Introduced separate methods for handling costs related to web search, file search, vector store, computer use, and code interpreter.
- Enhanced parameter extraction and conversion for file search and computer use.
- Improved error handling and type safety throughout the cost calculation process.
- Maintained compatibility with existing cost calculation methods while streamlining the overall structure.
* Fix user-team association issues in LiteLLM proxy
- Update list_team function to properly filter teams using user's teams array instead of only checking members_with_roles field
- Add Field descriptions and docstring to TeamMemberAddRequest and related models for better Swagger/OpenAPI documentation
- Maintain backward compatibility with fallback to members_with_roles if user lookup fails
This ensures users created with teams parameter appear correctly in team views and improves API documentation.
* Fix duplicate member checking in team_member_add endpoint
- Enhanced team_member_add_duplication_check to check both user_id and user_email
- Added additional duplicate prevention logic after user creation/lookup
- Fixed issue where users added by email could be duplicated in teams
- Added logging for debugging duplicate detection
This addresses the bug where adding the same user by email multiple times would create duplicate entries in the team's members_with_roles array.
* Improve duplicate member prevention in team_member_add
- Enhanced early duplicate check to handle both user_id and user_email
- Added late-stage duplicate prevention after user lookup/creation
- Fixed issue where users could be added multiple times by email
- Cleaned up debug logging
Note: There's still an edge case where the duplicate prevention may not work
correctly in all scenarios. This needs further investigation and testing.
* Refactor team_member_add endpoint for improved member management
- Split team_member_add functionality into smaller, dedicated functions for permission validation, member processing, and team member list updates.
- Enhanced permission checks to ensure only authorized users can add members.
- Streamlined member addition logic to reduce redundancy and improve readability.
- Maintained existing functionality while improving code structure and maintainability.
* Add tests for team_member_add helper functions
- Add test for _validate_team_member_add_permissions with admin user
- Add test for _validate_team_member_add_permissions with non-admin user
- Add test for _process_team_members with single member
- Add test for _process_team_members with multiple members
- Add test for _update_team_members_list with new member
- Add test for _update_team_members_list duplicate prevention
These tests ensure the refactored helper functions work correctly
after fixing the PLR0915 linting error.
* refactor(passthrough_endpoints-success-handler): refactor llm passthrough logging logic
isolate the llm translation work to enable cost tracking on sdk
* feat: initial implementation of passthrough SDK cost calculation
enables bedrock passthrough cost tracking to work
* feat(cost_calculator.py): working cost calculation for bedrock passthrough
* feat(litellm_logging.py): consider allm_passthrough in cost tracking
allows async calls (e.g. via proxy) to work
* feat(bedrock/passthrough): working event stream decoding for bedrock passthrough calls + logging instrumentation for passthrough sdk calls (log on stream completion)
Enables bedrock streaming cost calculation
* feat(litellm_logging.py): support streaming passthrough cost tracking
* feat(passthrough/main.py): working async streaming cost calculation
Closes https://github.com/BerriAI/litellm/issues/11359
* feat(proxy_server.py): fix passthrough routing when llm router enabled
* feat: further fixes
* feat(bedrock/): working bedrock passthrough cost tracking (non-streaming)
* feat(litellm_logging.py): working usage tracking for bedrock passthrough calls
ensures tokens are logged
* feat(bedrock/passthrough): add converse passthrough cost tracking support
* feat(base_llm/passthrough): remove redundant function
* refactor(litellm_logging.py): refactor function to be below 50 LOC
* test: update test
* test: remove redundant test
- Change verbose_logger.info() to verbose_logger.debug() for cost calculation messages
- Ensures cost calculation details are only shown when DEBUG logging is enabled
- Reduces log noise in production environments
Part 2/2 of fix for #9815
- Initialize verbose_logger, verbose_proxy_logger, and verbose_router_logger with the correct log level
- Ensures loggers respect the LITELLM_LOG environment variable setting
- Fixes part of issue #9815 where logs were shown regardless of log level setting
Part 1/2 of fix for #9815
* [Bug Fix] Anthropic - Token Usage Null Handling in calculate_usage (BerriAI/litellm#11920)
* [Fix] Missed a null check and used a cast instead by error
* feat: initial commit adding bedrock support via the new sdk passthrough logic
ensures correct sequencing of tasks (pre call checks etc. can run before signing request)
* fix(route_llm_requests.py): passthrough to allm_passthrough_route if no model found
* feat(bedrock/passthrough): working bedrock passthrough via sdk support
* fix(passthrough/main.py): re-add data and json
* feat(passthrough/main): support async passthrough calls to bedrock
* feat(passthrough/main.py): async streaming + completion support
* feat(llm_passthrough_endpoints.py): migrate bedrock passthrough calls to to new bedrock passthrough sdk
Enables calls to work correctly
* fix: fix linting errors
* test: update test
* fix(llm_http_handler.py): raise clearer error on anthropic unified route
Fixes https://github.com/BerriAI/litellm/issues/12063
* fix(key_management_endpoints.py): add new param `new_key` for setting the regenerated key value
user request
* test: add unit tests
* fix(pass_through_endpoints.py): use data instead of json for passthrough requests
fixes bedrock latency issue
* Revert "fix(pass_through_endpoints.py): use data instead of json for passthrough requests"
This reverts commit 021dfd9165f837d37e6aad247ccbd0d2e8ca6043.
Add reasoning content support when Responses API falls
back to completions API, enabling reasoning content for
all LLM providers (Anthropic, Vertex AI, etc.) since
OpenAI is currently the only native Responses API
provider.
* Add ReasoningSummaryTextDeltaEvent for streaming
reasoning deltas
* Update streaming iterator to detect and transform
reasoning content
* Extract reasoning content as separate output items in
transformations
* Support reasoning content alongside regular message
content
Closes https://github.com/BerriAI/litellm/issues/11302
The test was failing because the managed_files hook was not configured.
This fix adds a DummyManagedFiles class and mocks the create_file
function as an AsyncMock to ensure proper test execution.
* test(test_router.py): initial unit test confirming router.afile_content uses dynamic api key / api base
* fix(managed_files.py): filter deployments for only those within file id mapping
ensure call works - only route to models where the file was written
* fix(proxy_server.py): fix loading in model ids from config, if config id is int
* fix(router.py): return all model file id mappings on create_file
if multiple deployments - this ensures all the file id mappings are bubbled up
Fixes issue when trying to use loadbalanced deployments - only 1 file id mapping was being stored
* feat(router_utils/common_utils.py): filter models by team id when selecting for routing
Prevents team only models from being used by other teams
* fix(common_utils.py): additional fixes around filtering team-based models
* fix(batches_endpoints/endpoints): support list batches with target model names specified
* fix(common_utils.py): more testing for team deployment filters
* test(test_router.py): initial unit test confirming router.afile_content uses dynamic api key / api base
* fix(managed_files.py): filter deployments for only those within file id mapping
ensure call works - only route to models where the file was written
* fix(proxy_server.py): fix loading in model ids from config, if config id is int
* fix(router.py): return all model file id mappings on create_file
if multiple deployments - this ensures all the file id mappings are bubbled up
Fixes issue when trying to use loadbalanced deployments - only 1 file id mapping was being stored
* init litellm google gen ai methods
* feat init structure of functions for generate content
* add init
* add BaseGoogleGenAIGenerateContentConfig
* add generate_content_handler
* add get_provider_google_genai_generate_content_config
* fixes for generate content
* add get_vertex_ai_project etc to base
* use VertexBase
* fixes for BaseGoogleGenAIGenerateContentConfig
* working validate env for google gemini
* feat - add transform google response
* fixes for transform_generate_content_request
* fix get_supported_generate_content_optional_params
* add BaseGoogleGenAITest
* working e2e test
* fixes init config
* use correct types
* fix test for google gen ai
* fix types
* add sync_get_auth_token_and_url
* fixes for transform
* add llm http handler for google
* working non-streaming google endpoints
* add BaseGoogleGenAIGenerateContentStreamingIterator
* add GoogleGenAIGenerateContentStreamingIterator
* fix working sync stream
* fixes for litellm logging obj
* working async streaming
* add google gen ai types
* fix - required imports
* fix readme
* fix deps
* fix deps
* fix ruff code QA checks
* fix linting
* fixes TYPE_CHECKING
* fixes for typing
* add google gemini methods to litellm router
* [Feat] Add initial endpoints for using Gemini SDK (gemini-cli) with LiteLLM (#12040)
* init with google endpoints
* add Depends
* feat - add gemini endpoints
* google_generate_content
* fix init
* fixes import
* fixes for streaming
* fixes for sync/async
* working streaming with google gemini cli
* add google endpoints to llm api routes
* add VertexAIGoogleGenAIConfig
* use aiter_bytes
* use common request for streaming data
* re-use logic for anthropic streaming
* add GoogleAIStudioDataGenerator
* init litellm google gen ai methods
* feat init structure of functions for generate content
* add init
* add BaseGoogleGenAIGenerateContentConfig
* add generate_content_handler
* add get_provider_google_genai_generate_content_config
* fixes for generate content
* add get_vertex_ai_project etc to base
* use VertexBase
* fixes for BaseGoogleGenAIGenerateContentConfig
* working validate env for google gemini
* feat - add transform google response
* fixes for transform_generate_content_request
* fix get_supported_generate_content_optional_params
* add BaseGoogleGenAITest
* working e2e test
* fixes init config
* use correct types
* fix test for google gen ai
* fix types
* add sync_get_auth_token_and_url
* fixes for transform
* add llm http handler for google
* working non-streaming google endpoints
* add BaseGoogleGenAIGenerateContentStreamingIterator
* add GoogleGenAIGenerateContentStreamingIterator
* fix working sync stream
* fixes for litellm logging obj
* working async streaming
* add google gen ai types
* fix - required imports
* fix readme
* fix deps
* fix deps
* fix ruff code QA checks
* fix linting
* fixes TYPE_CHECKING
* fixes for typing
* fix(handler.py): support routing custom llm's to chat completion handler
Adds custom llm support for anthropic
* test(test_anthropic_experimental_pass_through_messages_handler.py): add unit test confirming custom llm respected
* docs(custom_llm_server.md): document anthropic custom llm translation
* test(volcengine.py): map thinking in extra body
Fixes https://github.com/BerriAI/litellm/issues/11879
* feat(main.py): support `azure/responses/<deployment-name>` model string
this allows us to route the model correctly
Closes https://github.com/BerriAI/litellm/issues/11879
* docs(azure_responses.md): document calling azure responses api models via chat completions bridge
Closes https://github.com/BerriAI/litellm/issues/11917
* fix: fix custom provider check
* test: update tests
* fix(team_endpoints.py): support setting default key expiry
allows admin to set key expiry on all team member keys
makes it easier to setup default team for experimentation
* feat(key_management_endpoints.py): allows admin to set duration for keys created by team members
* feat(team_endpoints.py): support team_member_key_duration on `/team/update`
allows setting max time team member keys are valid for
* fix(team_info.tsx): ui component to update team member key duration
* fix(team_info.tsx): support updating team member key duration, if set
* feat(teams.tsx): add team member key duration param ui component
allow admin to set this on UI
* feat(ui_sso.py): support restricting ui access by sso group
allows controlling who can/can't access the UI
* feat(ssomodals.tsx): add initial commit adding sso group access to admin ui
* feat(proxy_server.py): support reading + writing ui_access_mode from db
allows admin to configure allowed sso groups from UI
* feat(ui_sso.py): support enforcing all teams on sso jwt handler
if ui access mode set via ui, support reading the value and enforcing it
* feat(ui/): ui component for controlling sso access group
allow admin to only allow users within specific sso group to log into UI
* fix(uiaccesscontrolform.tsx): fix field names
* feat(ui_sso.py): return received sso response in the clientside error message - enables easier debugging
* test: add unit tests
* fix: minor fixes
* SCIM fix new user roles
* test_create_user_defaults_to_viewer
* test_create_user_uses_default_internal_user_params_role
* fix default user for SCIM
* fix linting error
* refactor(unpack_defs): enhance handling of schema properties and anyOf structures
- Improved the unpack_defs function to handle top-level properties and nested structures more effectively.
- Added recursion for items in schemas and refined the handling of anyOf branches to ensure proper unpacking of references.
- Streamlined the logic for resolving $ref keys and managing nested schemas.
* test(unpack_defs): add test for resolving nested $ref in anyOf items
- Introduced a new test to verify that unpack_defs correctly resolves references within items of anyOf structures, addressing a specific bug scenario (Issue #11372).
- The test includes a minimal schema to ensure proper unpacking and validation of the resolved items schema.
* refactor(unpack_defs): implement a generic resolver for $ref entries
- Redesigned the unpack_defs function to provide a more robust and dependency-free implementation for resolving all $ref entries in JSON schemas.
- Introduced a depth-first traversal method that efficiently handles nested structures, including anyOf, allOf, and items, while avoiding infinite recursion.
- Enhanced memory management by resolving nodes in-place without creating a full dereferenced copy, improving performance and reducing overhead.
* Remove test for unpack_defs resolving nested references in anyOf items from test_utils.py
* Add test for unpack_defs resolving nested references in anyOf items
This commit introduces a new test to ensure that the unpack_defs function correctly resolves $ref references within items of anyOf schemas, addressing issue #11372. The test verifies that the unpacked schema contains the expected properties and structure.
* fix(litellm_logging.py): fix using router model id for logging calls
Fixes https://github.com/BerriAI/litellm/issues/11975#issuecomment-2995882238
* test(test_litellm_logging.py): add unit test for custom price tracking
* fix(vertex_ai/): don't send invalid format parameter to vertex
causes calls to fail
* fix(vertex_ai_context_caching.py): if cached content present and tools in message, cache tools as well
gemini throws errors if tools passed in alongside cached content
* test: add unit tests
* fix: fix linting errors
* test: test_vertex_ai_common_utils.py
update test
* fix(streaming_handler.py): unset response cost when creating model response
