Ishaan Jaff and GitHub
061b7b2713
[Fix] Bedrock Embeddings - Ensure correct aws_region is used when provided dynamically ( #16547 )
...
* test_bedrock_embedding_uses_correct_region_when_specified
* fix aws_region_name in bedrock embeddings
2025-11-12 12:38:58 -08:00
Ishaan Jaff and GitHub
abde56391b
[Fix] - Bedrock Knowledge Bases - add support for filtering kb queries ( #16543 )
...
* test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call_with_tools_and_filters
* fix vs registry
* fix merging params
* test_bedrock_kb_request_body_has_transformed_filters
* fix typing / linting
2025-11-12 12:38:50 -08:00
Mubashir Osmani and GitHub
e6bbf3ce81
fix: app_roles missing from jwt payload ( #16448 )
...
* fix: jwt app_roles missing
* add test
2025-11-12 12:07:04 -08:00
Jean-Adrien Ducastaing and GitHub
086e557d78
fix: use vllm passthrough config for hosted vllm provider instead of raising error ( #16537 )
...
* fix(passthrough): use VLLM passthrough config for hosted VLLM provider instead of raising an error
* test(passthrough): add tests for hosted VLLM passthrough configuration and routing
* fix account linking for CLA
2025-11-12 09:12:14 -08:00
Sameer Kankute and GitHub
517eb0ee10
Use safe loading of creds ( #16479 )
2025-11-11 19:16:29 -08:00
Andrew Maguire and GitHub
bd15250960
fix: Add atexit handlers to flush callbacks for async completions ( #16487 )
...
Fixes #16486
## Problem
Callbacks configured via litellm.success_callback (e.g., PostHog, LangSmith)
were not being invoked for litellm.acompletion() in short-lived scripts.
The callbacks worked correctly for synchronous completions but async
completions would queue callbacks that were lost when the script exited
before GLOBAL_LOGGING_WORKER could process them.
Root cause: asyncio.run() closes the event loop immediately after the
async function completes, preventing the background worker from processing
queued callbacks.
## Solution
Implemented a two-level atexit handler approach:
1. GLOBAL_LOGGING_WORKER atexit handler (logging_worker.py):
- Processes remaining callbacks from queue before exit
- Creates new event loop to run pending coroutines synchronously
- Applies time and iteration limits to prevent blocking shutdown
2. Integration-specific atexit handlers (posthog.py as example):
- Flushes internal queue to external service
- Uses synchronous HTTP client for reliable delivery
- Each integration needs its own handler due to varying sync APIs
## Changes
- litellm/litellm_core_utils/logging_worker.py:
- Added _flush_on_exit() method
- Registered atexit handler in __init__
- Processes up to MAX_ITERATIONS_TO_CLEAR_QUEUE events
- Time-limited to MAX_TIME_TO_CLEAR_QUEUE seconds
- litellm/integrations/posthog.py:
- Added _flush_on_exit() method
- Registered atexit handler in __init__
- Groups events by credentials for batch sending
- Uses sync_client for reliable HTTP delivery
- tests/logging_callback_tests/test_posthog.py:
- Added test_async_callback_atexit_handler_exists()
- Added test_posthog_atexit_flushes_internal_queue()
- Added test_sync_callback_not_affected_by_atexit()
## Testing
- All existing tests pass
- Manual end-to-end testing confirms fix:
- Async events now arrive in PostHog
- Sync events continue working (no regression)
- Unit tests verify atexit handlers registered and functional
## Impact
- Fixes async callback delivery for ALL integrations using GLOBAL_LOGGING_WORKER
- No breaking changes - only adds missing functionality
- Sync path unchanged - no performance impact
2025-11-11 19:12:53 -08:00
Alan Ponnachan and GitHub
8cb610b8b2
Sanitize null token values in provider responses ( #16493 )
2025-11-11 19:08:00 -08:00
yuneng-jiang and GitHub
ba674f241c
[Fix] Use user budget instead of key budget when creating new team ( #16074 )
...
* Use end user budget instead of key budget when creating new team
* Fixed implementation to use user's max budget from the UserTable instead of EndUserTable
2025-11-11 18:55:05 -08:00
yuneng-jiang and GitHub
67478a9074
[Fix] Litellm tags usage add request_id ( #16111 )
...
* Add request_id into tag spend
* Linting
2025-11-11 18:53:48 -08:00
Ishaan Jaff and GitHub
50b5cf5215
[Feat] New Provider - Add RunwayML Provider for video generations ( #16505 )
...
* add RUNWAYML
* init folders
* add RunwayMLVideoConfig
* add RUNWAYML_DEFAULT_API_VERSION
* add RunwayMLVideoConfig
* fix getting status
* add async_transform_video_content_response
* add runwayml transform_video_content_response
* fix config.yaml
* add runwayml docs
* add runwayml to videos
* docs runwayml video gen
* add new models to model cost map
* TestRunwayMLVideoTransformation
* fix linting errors
2025-11-11 18:48:23 -08:00
443bada425
Add Zscaler AI Guard hook ( #15691 )
...
* Add Zscaler AI Guard hook
Co-authored-by: Angela Tao <atao@zscaler.com >
* Fix lint error, update document
* Fix lint error, update document
* update document
* fix mypy type error
* fix mypy issue
* fix test
* fix test
* improve document
* remove unuseful code
* use litellm httphandler
* update test cases
* revover guardrail_initializers.py and guardrail_registry.py
* remove unuse import
* app apply_guardrail
* remove functions repleased by apply_guardrail, update test and doc
* remove functions repleased by apply_guardrail, update test and doc
---------
Co-authored-by: Angela Tao <atao@zscaler.com >
2025-11-11 15:34:27 -08:00
Ishaan Jaff and GitHub
5c9f50d584
[AI Gateway] - End User Budgets - Allow pointing max_end_user budget to an id, so the default ID applies to all end users ( #16456 )
...
* add _apply_budget_limits_to_end_user_params
* add _apply_budget_limits_to_end_user_params
* add _apply_budget_limits_to_end_user_params
* test_default_budget_applied_to_end_user_without_budget
* docs fix
* fix config
2025-11-11 08:20:13 -08:00
Ishaan Jaff and GitHub
e94186629d
[Fix] Bedrock Knowledge bases - ensure users can access search_results for both stream + non stream response to /chat/completions ( #16459 )
...
* fix message with provider_specific_fields
* test_provider_specific_fields_in_proxy_http_response
* test_provider_specific_fields_in_proxy_http_response
2025-11-11 08:19:57 -08:00
Daniel Sabanov and GitHub
0ecc38519e
[Bug] Updated spend would not be sent to CloudZero ( #16201 )
...
* Address a bug where cloudzero spend is not sent to cloudzero if a spend
update happens
* revert change unrelated to PR
* use polars for mocking instead of sqlite
2025-11-10 19:43:15 -08:00
yellowsubmarine372 and GitHub
b6dbd4fa28
fix: apply provided timeout value to ClientTimeout.total ( #16395 )
2025-11-10 19:37:42 -08:00
Alan Ponnachan and GitHub
5f12e4be1e
fix(langfuse): Handle null usage values to prevent validation errors ( #16396 )
...
* langfuse null validation fix
* formatting
2025-11-10 19:34:56 -08:00
Alan Ponnachan and GitHub
7b292ccdf5
router fallback for unknown models ( #16419 )
2025-11-10 19:33:01 -08:00
Matt Cowger and GitHub
e0d6774ec1
fix: update model_cost_map_url to use environment variable ( #16429 )
...
* fix: update model_cost_map_url to use environment variable, to match behavior to documentation in docs/my-website/docs/proxy/sync_models_github.md
* Add an appropriate test.
2025-11-10 19:30:44 -08:00
Alan Ponnachan and GitHub
df33f36c06
Correctly handle date filters in /spend/logs endpoint ( #16443 )
2025-11-10 19:26:55 -08:00
Val Miscenko and GitHub
19e6e60d30
fix: remove strict master_key check in add_deployment ( #16453 )
...
Allows proxy to save spend logs without requiring master_key.
Decryption now gracefully handles both encrypted and unencrypted values.
2025-11-10 19:23:13 -08:00
Sameer Kankute and GitHub
6cab77f53f
Added thinking streaming support for mistral ( #16434 )
2025-11-10 18:41:45 -08:00
Sameer Kankute and GitHub
be3c09e6d5
Add GET list of providers endpoint ( #16432 )
2025-11-10 18:38:09 -08:00
Ishaan Jaff and GitHub
8140d85d28
[Bug Fix] - LiteLLM Usage shows key_hash- ( #16471 )
...
* test_get_logging_payload_api_key_preserved_when_standard_logging_payload_is_none
* fix get_logging_payload
* test_api_key_preserved_through_failure_hook_to_database
2025-11-10 18:15:52 -08:00
912be308b2
fix: allow internal users to access video generation routes ( #16472 )
...
Fixes #16470
Video generation endpoints (/v1/videos, /videos/{video_id}, etc.) were
incorrectly restricted to proxy_admin role only. These routes are now
added to openai_routes list, making them accessible to internal_user
role as they should be - video generation is a legitimate user feature,
not a management/admin operation.
Changes:
- Added 8 video route patterns to LiteLLMRoutes.openai_routes in _types.py
- Added comprehensive tests verifying internal_user and virtual key access
- All existing route permission tests continue to pass
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-authored-by: Claude <noreply@anthropic.com >
2025-11-10 17:44:16 -08:00
Ishaan Jaff and GitHub
dc76b6c76e
[Fix] Management Endpoints - Fixes inconsistent error responses in customer management endpoints. Non-existent user errors now return proper 404 status codes with consistent error schema format across all endpoints. ( #16450 )
...
* fix: ensure end user endpoints use "handle_exception_on_proxy" correctly
* test 404 on info and update for non-existent user
* test 404 for no customer found
* fix 404 handling for customer endpoints
* test_error_schema_consistency
* test_customer_endpoints_error_schema_consistency
2025-11-10 17:16:53 -08:00
Ishaan Jaffer
8c1dd78c13
fix imports
2025-11-08 18:01:32 -08:00
Krish Dholakia and GitHub
06906534b3
feat(audio_transcriptions/): calculate duration of audio file for cost calculation + feat (image_generations): cost tracking accuracy improved with output_format, quality, size values fixed per openai model
...
* feat(audio_transcriptions/): calculate duration of audio file for cost calculation
Fixes https://github.com/BerriAI/litellm/issues/11846
Closes https://github.com/BerriAI/litellm/issues/14605
* fix(cost_calculator.py): correctly use base model, when set
Fixes issue where azure base model was being ignored
* feat(cost_calculator.py): fix default cost tracking quality param for image generation
* feat(image_generations/): return output_format, quality, size
aligns response to openai spec and improves cost tracking accuracy
* fix(cost_calculator.py): refactor cost calculation for image generation to use image response instead of hidden params
* build: update build
* fix: fix cost calculation
* build: update poetry lock
* fix: fix ruff checks
* fix: fix aembedding
* fix: fix ruff errors
* fix: modify to catch errors
* fix: test
* fix: loosen test to handle openai lib out of sync
* fix: fix base models
* fix: fix usage object
2025-11-08 16:24:31 -08:00
86d73c918c
Adds support for returning Azure Content Policy error information when exceptions from Azure OpenAI occur ( #16231 )
...
* add provider_specific_fields to ContentPolicyViolationError
* use provider_specific_fields in ProxyException
* update openai_exception_handler
* fix use exception checker for content policy violation azure
* add AzureOpenAIExceptionMapping
* test_azure_with_content_safety_error
* Accessing Provider-Specific Error Details
* TestExceptionCheckers
* unit test got provider_specific_fields=
* add clear types for error dict
* fix test_azure_with_content_safety_error
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com >
2025-11-08 16:04:36 -08:00
e037d9315d
Add Vertex and Gemini Videos API with Cost Tracking + UI support ( #16323 )
...
* Use video id for videos api
* remove mock code
* Potential fix for code scanning alert no. 3630: Clear-text logging of sensitive information
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* remove print statements
* Update video prefix for 'video_'
* Add veo with openai videos unified specs
* Add videos testing to UI
* remove mock code
* Remove not need ui changes:
* Fix mypy errors related to gemini
* fix test_transform_video_create_request
* Add vertex ai veo config
* Add vertex ai veo config
* Add cost tracking for gemini and add optional param passing
* fix bugs related to vertex ai veo
* Add Gemini Veo Video Generation in Openai Videos Unified Spec (#16229 )
* Add veo with openai videos unified specs
* Add videos testing to UI
* remove mock code
* Remove not need ui changes:
* Fix mypy errors related to gemini
* fix test_transform_video_create_request
* Add contant video duration for gemini and vertex
* Fix litellm_mapped_tests tests
* fix azure videos issue
* Added doc for videos vertex ai
* fix seconds param error
* fix lint errors
* test_transform_video_create_response_cost_tracking_no_duration
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com >
2025-11-08 16:03:51 -08:00
Krish Dholakia and GitHub
202eaeb1a2
Revert "(feat) Audio transcription - cost tracking + (feat) image generation …" ( #16409 )
...
This reverts commit c96da44265
2025-11-08 15:38:16 -08:00
Krish Dholakia and GitHub
c96da44265
(feat) Audio transcription - cost tracking + (feat) image generation - accurate cost tracking based on output_format/quality/size
...
* feat(audio_transcriptions/): calculate duration of audio file for cost calculation
Fixes https://github.com/BerriAI/litellm/issues/11846
Closes https://github.com/BerriAI/litellm/issues/14605
* fix(cost_calculator.py): correctly use base model, when set
Fixes issue where azure base model was being ignored
* feat(cost_calculator.py): fix default cost tracking quality param for image generation
* feat(image_generations/): return output_format, quality, size
aligns response to openai spec and improves cost tracking accuracy
* fix(cost_calculator.py): refactor cost calculation for image generation to use image response instead of hidden params
* build: update build
* fix: fix cost calculation
* build: update poetry lock
* fix: fix ruff checks
* fix: fix aembedding
* fix: fix ruff errors
* fix: modify to catch errors
* fix: test
* fix: loosen test to handle openai lib out of sync
2025-11-08 15:30:46 -08:00
Krish Dholakia and GitHub
bce8a5d6b7
Revert "(feat) audio transcriptions cost tracking (for azure/other non-openai…" ( #16402 )
...
This reverts commit 9a88fe0861
2025-11-08 14:24:51 -08:00
Ishaan Jaffer
e4c31add58
test_dd_llms_obs_redaction
2025-11-08 14:23:42 -08:00
Ishaan Jaffer
1bf284abca
test_delete_callbacks_in_db
2025-11-08 14:05:44 -08:00
Ishaan Jaffer
a4c008c433
test_datadog_message_redaction
2025-11-08 14:02:24 -08:00
Krish Dholakia and GitHub
9a88fe0861
(feat) audio transcriptions cost tracking (for azure/other non-openai models) + (fix) image generations - accurate cost tracking for dalle3/gpt-image-1 - uses the correct max image quality ( #16076 )
...
* feat(audio_transcriptions/): calculate duration of audio file for cost calculation
Fixes https://github.com/BerriAI/litellm/issues/11846
Closes https://github.com/BerriAI/litellm/issues/14605
* fix(cost_calculator.py): correctly use base model, when set
Fixes issue where azure base model was being ignored
* feat(cost_calculator.py): fix default cost tracking quality param for image generation
* feat(image_generations/): return output_format, quality, size
aligns response to openai spec and improves cost tracking accuracy
* fix(cost_calculator.py): refactor cost calculation for image generation to use image response instead of hidden params
* build: update build
* fix: fix cost calculation
* build: update poetry lock
* fix: fix ruff checks
2025-11-08 13:54:37 -08:00
Ishaan Jaffer
993c75d10b
test_redaction_responses_api
2025-11-08 13:43:22 -08:00
Ishaan Jaffer
c43fcf2ded
test_redaction_responses_api
2025-11-08 13:39:40 -08:00
6fb0a8fc58
Added xai responses support ( #16391 )
...
* Added xai responses support
* add the xai provider config above
* remove init file
* remove init file
* Fix f string lint error
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com >
2025-11-08 12:30:09 -08:00
Ishaan Jaffer
eef8c01d4e
test_redaction_responses_api
2025-11-08 12:29:38 -08:00
Ishaan Jaffer
1d2bdaebb6
test_openai_streaming_logging
2025-11-08 11:49:36 -08:00
Ishaan Jaffer
9b4e32c880
Revert "Added support for desabling thoughts by setting budget to 0 ( #16347 )"
...
This reverts commit b6f792f301
2025-11-08 11:34:05 -08:00
Ishaan Jaffer
4fb521c251
test_basic_openai_responses_api_non_streaming_with_logging
2025-11-08 11:12:35 -08:00
Ishaan Jaffer
c60a13c919
fixes
2025-11-08 11:11:22 -08:00
Ishaan Jaffer
da7ecb3ea9
test_gemini_reasoning_effort_zero_budget_disables_thoughts
2025-11-08 11:10:54 -08:00
Ishaan Jaffer
bbcdf6f996
test_basic_openai_responses_api_non_streaming_with_logging
2025-11-08 10:47:48 -08:00
Ishaan Jaffer
0148b8d2f7
test_basic_openai_responses_api_non_streaming_with_logging
2025-11-08 10:36:47 -08:00
Ishaan Jaffer
89157f4b5c
test_basic_openai_responses_api_streaming_with_logging
2025-11-08 10:30:41 -08:00
Ishaan Jaffer
6bb963dca2
test_basic_openai_responses_api_non_streaming_with_logging
2025-11-08 10:13:27 -08:00
Ishaan Jaffer
68d24b40c6
test_gemini_25_implicit_caching_cost
2025-11-08 10:13:04 -08:00
