Commit Graph
2702 Commits
Author SHA1 Message Date
Krrish Dholakia 6322aef0e3 fix(streaming_handler.py): fix streaming chunk calculation 2025-08-16 14:25:29 -07:00
Krrish Dholakia eb66daeef7 test: update test
we now return correct token usage on clientside
2025-08-16 14:14:35 -07:00
Ishaan Jaff d96df5e9be Revert "test_stream_token_counting_anthropic_with_include_usage"
This reverts commit c3aee1194b.
2025-08-16 13:07:00 -07:00
Ishaan Jaff c3aee1194b test_stream_token_counting_anthropic_with_include_usage 2025-08-16 13:06:31 -07:00
Krrish Dholakia 8bb5ee2ba5 test: update unit test 2025-08-16 12:39:03 -07:00
Krrish Dholakia 1ec0808713 test: update test - anthropic returns input tokens in final usage chunk now 2025-08-16 12:31:52 -07:00
Ishaan Jaff f522f40228 test_passing_tool_result_as_list 2025-08-16 08:08:25 -07:00
Krish DholakiaandGitHub 1b2ec16eee Merge pull request #13640 from yytdfc/fix_bedrock_epc
[Bug Fix] Add cachePoint support for assistant and tool messages in Bedrock
2025-08-16 01:20:28 -07:00
Ishaan JaffandGitHub 40395598da [Feat] UI - Allow editing team member rpm/tpm limits (#13669)
* show team member tpm/rpm limits

* ui - allow setting team settings

* fix better debugging

* fix types: TeamMemberUpdateRequest

* add _upsert_budget_and_membership

* allow updating team member RPM/TPM in teamMemberUpdateCall

* editing team member rpm/tpm

* UI - fixes for team member component

* fix info

* test_upsert_rpm_only_creates_new_budget
2025-08-15 17:29:44 -07:00
0x-fangandGitHub 9f17bedb48 bump up boto3 and aioboto3 versions (#13665) 2025-08-15 17:06:08 -07:00
Ishaan Jaff f5a1c14929 Revert "update boto3 version to 1.37.38 (#13656)"
This reverts commit 6f2264a13d.
2025-08-15 11:24:59 -07:00
Daniel BarkerandGitHub 47edecd5bc Fixed incorrect key info endpoint (#13633) 2025-08-15 11:10:06 -07:00
0x-fangandGitHub 6f2264a13d update boto3 version to 1.37.38 (#13656) 2025-08-15 11:06:54 -07:00
Ishaan JaffandGitHub c3608adb0f [Bug Fix] Using stream=True + background=True with Responses API (#13654)
* test_update_responses_api_response_id_with_model_id_handles_dict

* fix linting
2025-08-15 08:58:28 -07:00
FuChen 511363d3a7 test: Add comprehensive test cases for cachePoint support
- test_assistant_message_cache_control: Tests assistant messages with string content and cache_control
- test_assistant_message_list_content_cache_control: Tests assistant messages with list content and cache_control
- test_tool_message_cache_control: Tests tool messages with list content and cache_control
- test_tool_message_string_content_cache_control: Tests tool messages with string content and cache_control
- test_assistant_tool_calls_cache_control: Tests assistant tool_calls with cache_control
- test_multiple_tool_calls_with_mixed_cache_control: Tests multiple tool calls with mixed cache_control
- test_no_cache_control_no_cache_point: Tests that messages without cache_control don't generate cachePoint blocks

These tests ensure that cachePoint blocks are correctly generated for all message types when cache_control is present.
2025-08-15 11:56:34 +08:00
Ishaan JaffandGitHub 48c89812c4 [Bug Fix] /messages endpoint - ensure tool use arguments are returned for non-anthropic models (#13638)
* bug fix _translate_streaming_openai_chunk_to_anthropic

* test  test_translate_streaming_openai_chunk_to_anthropic_with_partial_json
2025-08-14 17:29:42 -07:00
Ishaan JaffandGitHub b78495d398 [Fix] Ensure /messages works when using `bedrock/converse/<model> with LiteLLM (#13627)
* get_bedrock_provider_config_for_messages_api

* fixes for get_bedrock_provider_config_for_messages_api

* test_anthropic_messages_litellm_router_bedrock

* fix merge conflicts

* fix - refactor based on jugal's comment
2025-08-14 16:50:05 -07:00
Jugal D. BhattandGitHub b8fe5f7b17 [MCP Gateway] LiteLLM Fix MCP gateway key auth (#13630)
* Fix - add safe divide by 0 for most places to prevent crash

* Enhance MCPRequestHandler to support permission inheritance and intersection logic for access groups. Added integration tests to verify behavior when keys have no permissions and when both keys and teams have overlapping permissions.

* Remove redundant assertions for permission checks in test_user_api_key_auth_mcp.py to streamline test logic.

* Refactor integration tests for MCPRequestHandler to simplify mocking. Replace complex database mocks with direct function mocks for permission inheritance and intersection scenarios, improving test clarity and maintainability.

* Revert "Fix - add safe divide by 0 for most places to prevent crash"

This reverts commit 265d40e39051e148996b9fb7f354730c57ff23ac.
2025-08-14 16:32:18 -07:00
aea0605eed [LLM Translation] Fix Realtime API endpoint for no intent (#13476)
* fix intent params

* Add responses

* fix unrelated test

* test fix - fireworks API endpoint is down

* test fix fireworks ai is having an active outage

* test_completion_cost_databricks

* dbrx fix test API currently not responding

* Update OpenAI Realtime handler to use the correct endpoint and include all query parameters. Adjusted error messages for missing API base and key. Updated health check URL construction to pass model as a query parameter.

* Enhance OpenAI Realtime handler tests to ensure model parameter inclusion in WebSocket URL. Added new tests to verify correct URL construction with model and additional parameters, preventing 'missing_model' errors. Updated existing tests for consistency.

* Remove debug print statements for API base and key in OpenAIRealtime handler to clean up the code.

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
2025-08-14 16:24:14 -07:00
Krish DholakiaandGitHub ab8bd70c5e Merge pull request #13635 from BerriAI/litellm_mock_test_fix
[Proxy] LiteLLM mock test fix
2025-08-14 15:47:41 -07:00
TomuHirataandGitHub f6e53deacd Update mlflow logger usage span attributes (#13561)
* test: sync mlflow request tags

* fix test
2025-08-14 15:20:50 -07:00
Jugal Bhatt f53599825d Merge branch 'main' into litellm_mock_test_fix 2025-08-14 15:17:31 -07:00
Jugal Bhatt 45f188b041 Add mock user API key authentication in tag management tests
This update integrates mock user API key authentication into the tag management endpoint tests, ensuring accurate simulation of user roles for creating, updating, and deleting tags. The changes enhance the reliability of the tests by properly setting up user authentication before executing test cases.
2025-08-14 15:03:21 -07:00
Krrish Dholakia aaf9c38a10 test: skip test - ran out of credits 2025-08-14 15:01:26 -07:00
Jugal Bhatt d21f467264 Revert "Add mock user API key authentication in tag management tests"
This reverts commit 5fc0803b94.
2025-08-14 14:58:49 -07:00
Jugal Bhatt 5fc0803b94 Add mock user API key authentication in tag management tests
This update introduces a helper function to create a mock user API key authentication object, which is utilized in the tag management endpoint tests. The mock authentication is integrated into the test cases for creating, updating, and deleting tags, enhancing the reliability of the tests by simulating user roles accurately.
2025-08-14 14:40:49 -07:00
Jugal Bhatt a6e55c0447 Revert "Revert "Fix - add safe divide by 0 for most places to prevent crash""
This reverts commit 5ad698f2cc.
2025-08-14 14:23:06 -07:00
Jugal Bhatt bfb0a3854e Enhance logging in cost calculation tests to ensure DEBUG level captures are accurate. Updated tests to set logger level before assertions and restored original logger level after execution. This improves reliability of log level checks in both cost and batch cost calculation tests. 2025-08-14 14:21:22 -07:00
Jugal Bhatt 5ad698f2cc Revert "Fix - add safe divide by 0 for most places to prevent crash"
This reverts commit 265d40e39051e148996b9fb7f354730c57ff23ac.
2025-08-14 14:21:07 -07:00
Jugal D. BhattandGitHub 3a38912bce [Proxy] Litellm fix mapped tests (#13634)
* Fix - add safe divide by 0 for most places to prevent crash

* mock prisma client

* Revert "Fix - add safe divide by 0 for most places to prevent crash"

This reverts commit 265d40e39051e148996b9fb7f354730c57ff23ac.
2025-08-14 14:19:36 -07:00
Krish DholakiaandGitHub f360e0ead2 Merge pull request #13590 from BerriAI/litellm_bedrock_api_header
[LLM translation] Refactor Anthropic Configurations and Add Support for `anthropic_beta` Headers
2025-08-14 11:32:18 -07:00
Jugal D. BhattandGitHub 1beba93cc8 Fix - add safe divide by 0 for most places to prevent crash (#13624) 2025-08-14 11:17:49 -07:00
Ishaan JaffandGitHub 5bb96af818 [Feat] Add reasoning_effort param for hosted_vllm provider (#13620)
* add reasoning_effort to hosted_vllm

* test_hosted_vllm_supports_reasoning_effort

* Reasoning Effort
2025-08-14 10:10:30 -07:00
Cole McIntoshandGitHub dea98a315b fix(volcengine): handle thinking disabled parameter properly (#13598)
* fix(volcengine): handle thinking disabled parameter properly

When thinking is set to {"type": "disabled"} in volcengine provider config,
skip adding the parameter to extra_body entirely instead of passing it through.
This prevents "thinking: undefined" from appearing in request logs.

Fixes #13039

* test(volcengine): fix and enhance thinking parameter tests

- Fixed existing test that expected broken behavior
- Added comprehensive test coverage for all thinking parameter scenarios:
  * thinking disabled → omitted from extra_body
  * thinking enabled → included in extra_body
  * thinking None → included in extra_body as None
  * custom thinking values → included in extra_body
  * no thinking parameter → empty result

All tests passing, verifying the fix for issue #13039
2025-08-14 09:10:06 -07:00
Krrish Dholakia 0288ed35da test: update tests 2025-08-13 23:33:32 -07:00
Krrish Dholakia b53962dee2 test: update test 2025-08-13 23:09:18 -07:00
Ishaan Jaff cd893134b7 test team endpoints 2025-08-13 18:43:50 -07:00
Ishaan Jaff 086621e3d3 test_handle_jwt.py 2025-08-13 17:27:50 -07:00
Ishaan JaffandGitHub 8e76f8e7d0 [Feat] Team Member Rate Limits + Support for using with JWT Auth (#13601)
* fix - assign tpm/rpm limit onJWT

* add team member rpm/tpm limits

* update - rate limiter v3 with team member rate limits

* update utils

* fixes for LiteLLM_BudgetTable

* undo change

* add TeamMemberBudgetHandler

* add _process_team_member_budget_data

* add get_team_membership

* add safe_get_team_member_rpm_limit and safe_get_team_member_tpm_limit

* LiteLLM_TeamMembership

* add LiteLLM_TeamMembership rate limit for JWTs

* fix

* tests
2025-08-13 17:21:36 -07:00
Ishaan JaffandGitHub 76d25926d4 [Feat] New provider - Azure AI Flux Image Generation (#13592)
* init files

* add AzureFoundryModelInfo

* fix api_version property

* add azure_ai img gen

* use AzureFoundryModelInfo

* get_base_image_generation_call_args

* add azure_ai/FLUX-1.1-pro

* add util for route_image_generation_cost_calculator

* docs azure ai flux

* fixes for flux

* fixes for AzureFoundryFluxImageGenerationConfig

* ruff fix
2025-08-13 17:20:30 -07:00
Ishaan JaffandGitHub 75bcfbb76a [Feat] New model vertex_ai/deepseek-ai/deepseek-r1-0528-maas (#13594)
* add ertex_ai/deepseek-ai/deepseek-r1-0528-maas

* fix init

* test_model_info_for_vertex_ai_deepseek_model
2025-08-13 13:44:45 -07:00
Jugal Bhatt 26e62c9bd8 Update Test for Anthropic Messages Configuration
- Renamed `AmazonAnthropicClaude3MessagesConfig` to `AmazonAnthropicClaudeMessagesConfig` in the test file to align with recent refactoring.
- Adjusted the instantiation of the configuration class in the test to reflect the new naming convention.
2025-08-13 11:53:11 -07:00
Jugal Bhatt c2310bcccc Refactor Anthropic Configurations in Tests
- Updated test cases to use the renamed `AmazonAnthropicClaudeConfig` instead of `AmazonAnthropicClaude3Config` for consistency with recent changes.
- Adjusted imports and assertions in test files to reflect the new configuration class name.
2025-08-13 11:52:41 -07:00
Jugal Bhatt 3990f61bed Refactor Anthropic Configurations and Add Support for anthropic_beta Headers
- Renamed `AmazonAnthropicClaude3Config` and `AmazonAnthropicClaude3MessagesConfig` to `AmazonAnthropicClaudeConfig` and `AmazonAnthropicClaudeMessagesConfig` respectively for consistency.
- Implemented `get_anthropic_beta_from_headers` function to extract and handle `anthropic-beta` headers across various transformations.
- Updated request transformations in `AmazonConverseConfig` and `AmazonInvokeConfig` to include `anthropic_beta` parameters based on user headers.
- Added tests to ensure proper handling of `anthropic_beta` headers in different scenarios.
2025-08-13 11:47:59 -07:00
Krish DholakiaandGitHub ffd165c9b7 Merge pull request #13549 from BerriAI/litellm_mcp_config_fix
[fix] Enhance MCPServerManager with access groups and description support
2025-08-13 10:51:19 -07:00
Krish DholakiaandGitHub 5e45fcdff0 Merge pull request #13375 from colesmcintosh/fix/ollama-gpt-oss-thinking-field 2025-08-13 07:58:57 -07:00
Krzysztof GąsiorowskiandGitHub 0b28930ad4 [Fix] Hide sensitive data in /model/info - azure entra client_secret (#13577)
* Remove litellm_params.client_secret from /model/info

Added pop of client_secret (Azure provider secret) from litellm_params in remove_sensitive_info_from_deployment used by /model/info endpoints

* Added test for litellm.proxy.common_utils.openai_endpoint_utils.remove_sensitive_info_from_deployment
2025-08-13 07:35:53 -07:00
Krrish Dholakia 5ae44e3275 fix(router.py): fix cooldown increment logic 2025-08-12 23:33:56 -07:00
Krish DholakiaandGitHub 6afaf5721a [Fix] Streaming - consistent 'finish_reason' chunk index (#13560)
* feat(model_response_utils.py): new function to check if modelresponsestream is empty

used for checking https://github.com/BerriAI/litellm/issues/13348

* fix(streaming_handler.py): skip chunk if empty

Fixes https://github.com/BerriAI/litellm/issues/13348

* fix(streaming_handler.py): add is_empty logic to async flow
2025-08-12 23:21:57 -07:00
Krish DholakiaandGitHub f00e891004 LiteLLM SDK <-> Proxy: support user param + Prisma - remove use_prisma_migrate flag - redundant as this is now default (#13555)
* fix(litellm_proxy/chat/transformation.py): support 'user' and all other openai chat completion params

Fixes issue where 'user' was not being sent in request to litellm proxy via sdk

* fix(prisma_migration.py): remove 'use_prisma_migrate' flag, is now default

* docs: cleanup docs

* fix(proxy_cli.py): remove --use_prisma_migrate flag

* refactor: remove references to use_prisma_migrate env var

This is now the default flow for db migrations
2025-08-12 22:03:39 -07:00