Commit Graph
23118 Commits
Author SHA1 Message Date
Krish DholakiaandGitHub ee6e76e1f9 Bedrock Passthrough cost tracking (/invoke + /converse routes - streaming + non-streaming) (#12123)
* refactor(passthrough_endpoints-success-handler): refactor llm passthrough logging logic

isolate the llm translation work to enable cost tracking on sdk

* feat: initial implementation of passthrough SDK cost calculation

enables bedrock passthrough cost tracking to work

* feat(cost_calculator.py): working cost calculation for bedrock passthrough

* feat(litellm_logging.py): consider allm_passthrough in cost tracking

allows async calls (e.g. via proxy) to work

* feat(bedrock/passthrough): working event stream decoding for bedrock passthrough calls + logging instrumentation for passthrough sdk calls (log on stream completion)

Enables bedrock streaming cost calculation

* feat(litellm_logging.py): support streaming passthrough cost tracking

* feat(passthrough/main.py): working async streaming cost calculation

Closes https://github.com/BerriAI/litellm/issues/11359

* feat(proxy_server.py): fix passthrough routing when llm router enabled

* feat: further fixes

* feat(bedrock/): working bedrock passthrough cost tracking (non-streaming)

* feat(litellm_logging.py): working usage tracking for bedrock passthrough calls

ensures tokens are logged

* feat(bedrock/passthrough): add converse passthrough cost tracking support

* feat(base_llm/passthrough): remove redundant function

* refactor(litellm_logging.py): refactor function to be below 50 LOC

* test: update test

* test: remove redundant test
2025-06-27 20:01:12 -07:00
tanjiroandGitHub db2a55a3d2 Add "Get Code" Feature (#11629)
* added get code

* modify template

* fixed code template

* put the codesnippets in a different file

* minor
2025-06-27 18:18:56 -07:00
Alessandro MarrellaandGitHub 0e10ae1cf1 change Message init type annotation to support other roles (#11942) 2025-06-27 18:12:13 -07:00
ohmeowandGitHub 9ab7d97b85 adds model to metadata to estimated cost is calculated; corrects how to pass metadata in braintrust in the docs (#12022) 2025-06-27 18:11:07 -07:00
Ishaan Jaff 723f24ef17 ci/cd new release 2025-06-27 18:07:44 -07:00
Ishaan Jaff 080372399f test_mock_router_testing_params_str_to_bool_conversion 2025-06-27 18:07:33 -07:00
Ishaan Jaff 0db60009b5 ci/cd new release 2025-06-27 18:06:06 -07:00
Ishaan Jaff d8dc84ee96 test fix - router utils 2025-06-27 18:05:37 -07:00
Ishaan Jaff c76f9be729 ui - new build 2025-06-27 18:00:59 -07:00
Ishaan JaffandGitHub 01b9e6efb1 Revert "fix: set logger levels based on LITELLM_LOG environment variable" (#12122) 2025-06-27 17:55:44 -07:00
Ishaan JaffandGitHub ebf6395bc1 [Feat] Add Eleven Labs - Speech To Text Support on LiteLLM (#12119)
* add ELEVENLABS as a provider

* add deepgram to main.py

* add ElevenLabsException

* add ElevenLabsAudioTranscriptionConfig

* add transform_audio_transcription_response

* TestElevenLabsAudioTranscription

* add elevenlabs/scribe_v1 to model cost map

* add ElevenLabsAudioTranscriptionConfig

* add AudioTranscriptionRequestData

* add ElevenLabs transform

* use AudioTranscriptionRequestData

* refactoring fixes

* add ProcessedAudioFile util for reading audio files

* test_elevenlabs_diarize_parameter_passthrough

* docs eleven labs

* docs fixes

* fix code qa checks

* fixes - audio transcription

* ui - add ElevenLabs logo

* add elevenlabs logo

* docs - ElevenLabs

* test fix elevenlabs
2025-06-27 17:50:49 -07:00
041db0268c [Bug fix] Router - handle cooldown_time = 0 for deployments (#12108)
* fix get cooldown time

* fixes for _should_run_cooldown_logic

* test_cooldown_time_zero_uses_zero_not_default

* Update litellm/router_utils/cooldown_cache.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update litellm/router_utils/cooldown_handlers.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-06-27 17:50:35 -07:00
Cole McIntoshandGitHub cdb5c6fb6b Merge pull request #12035 from colesmcintosh/add-gemini-md
docs(GEMINI.md): add development guidelines and architecture overview
2025-06-27 17:30:52 -06:00
Ishaan JaffandGitHub f04808e293 [Bug Fix] Exception mapping for context window exceeded - should catch anthropic exceptions (#12113)
* fix is is_error_str_context_window_exceeded

* test_is_error_str_context_window_exceeded

* fix is_error_str_context_window_exceeded
2025-06-27 15:49:07 -07:00
Jason RobertsandGitHub b57cfb8bff Adding Feature: Palo Alto Networks Prisma AIRS Guardrail (#12116)
* feat: Add PANW Prisma AIRS guardrail integration

- Add PANW_PRISMA_AIRS to SupportedGuardrailIntegrations enum
- Update guardrail registry and initializers
- Add complete documentation with curl examples and response formats
- Support pre_call, post_call, and during_call modes
- Include fail-safe error handling and comprehensive logging
- Integration with official PANW Prisma AIRS API

* feat: Add PANW Prisma AIRS guardrail integration

- Update to test file
- Fail-closed security on API errors

* update fail closed behavior

* fix: Update PANW Prisma AIRS guardrail integration pattern

* fix: Remove unused import

* fix: addressed MyPy error
2025-06-27 15:17:36 -07:00
Ishaan JaffandGitHub a42a058b0c [Bug Fix] Fix handling str, bool types formock_testing_fallbacks on router using /audio endpoints (#12117)
* use a dataclass for managing mock params

* fixes for mock testing
2025-06-27 15:11:09 -07:00
Ishaan JaffandGitHub f85cd9c54a [Docs] - Show how to use fallbacks with audio transcriptions endpoints (#12115)
* add overview

* docs audio transcriptions

* docs on fallbacks with audio endpoints
2025-06-27 12:43:55 -07:00
Ishaan Jaff 7a5e7f454d bump: version 1.73.3 → 1.73.4 2025-06-27 11:10:14 -07:00
Ishaan JaffandGitHub f1c7024e70 [Feat] Add Bridge from generateContent <> /chat/completions (#12081)
* add GenerateContentToCompletionHandler

* working - non streaming bridge

* add GoogleGenAIAdapter

* add google gen ai adapter

* working streaming bridge

* working streaming usage for adapter

* tool calling transform for generate content to openai

* fixes for accumulating tool calls

* fix code qa checks

* Best Practices for Production

* fix code qa checks

* test_streaming_partial_tool_calls_accumulation

* linting fixes

* add supported_openai_chat_completion_params

* fix translate_generate_content_to_completion

* test_google_genai_adapter.py
2025-06-27 11:08:55 -07:00
Cole McIntoshandGitHub 575d25ab16 Merge pull request #12111 from colesmcintosh/fix/logger-level-initialization
fix: set logger levels based on LITELLM_LOG environment variable
2025-06-27 11:58:05 -06:00
Cole McIntoshandGitHub cce2656376 Merge pull request #12112 from colesmcintosh/fix/cost-calculation-log-level
fix: change cost calculation logs from INFO to DEBUG level
2025-06-27 11:57:55 -06:00
Cole McIntosh bc41cee0a1 fix: change cost calculation logs from INFO to DEBUG level
- Change verbose_logger.info() to verbose_logger.debug() for cost calculation messages
- Ensures cost calculation details are only shown when DEBUG logging is enabled
- Reduces log noise in production environments

Part 2/2 of fix for #9815
2025-06-27 11:36:58 -06:00
Cole McIntosh 9d58d872ac fix: set logger levels based on LITELLM_LOG environment variable
- Initialize verbose_logger, verbose_proxy_logger, and verbose_router_logger with the correct log level
- Ensures loggers respect the LITELLM_LOG environment variable setting
- Fixes part of issue #9815 where logs were shown regardless of log level setting

Part 1/2 of fix for #9815
2025-06-27 11:34:38 -06:00
KishanandGitHub fc17da0aef [Bug Fix] Anthropic - Token Usage Null Handling in calculate_usage (#12068)
* [Bug Fix] Anthropic - Token Usage Null Handling in calculate_usage (BerriAI/litellm#11920)

* [Fix] Missed a null check and used a cast instead by error
2025-06-27 10:00:23 -07:00
Krish DholakiaandGitHub 8bd1f8f6ab Add o3 and o4-mini deep research models (#12109)
* build(model_prices_and_context_window.json): add o3-deep-research models

* build(model_prices_and_context_window.json): add o4-deep-research model

* build(model_prices_and_context_window.json): add o4-mini-deep-research versioned model
2025-06-27 09:58:53 -07:00
Cole McIntoshandGitHub a56c28ab2f fix: handle provider_config type error in passthrough error handler (#12101)
- Fix type error where _handle_error was called with provider_config=None
- Add proper provider config retrieval logic similar to llm_passthrough_route
- Handle cases where provider config may not be available
- Ensure type safety while maintaining backward compatibility
2025-06-27 09:11:28 -07:00
Ishaan JaffandGitHub e3c9318bd4 [Fix] MCP - Ensure internal users can access /mcp and /mcp/ routes (#12106)
* fixes for MCP route checks

* test is is_llm_api_route for MCP
2025-06-27 09:05:38 -07:00
Cole McIntoshandGitHub 60bf3c36bf fix(docs): Remove unused dotenv dependency from docusaurus config (#12102)
* fix(docs): Remove unused dotenv dependency from docusaurus config

The dotenv package was being required in docusaurus.config.js but was listed as
a devDependency, causing build failures. Since no environment variables are
actually used in the config, removed the unnecessary import.

* fix(docs): Remove reference to non-existent spending_monitoring doc

The sidebars.js file was referencing proxy/spending_monitoring which was deleted
in commit ba7463b9c. This was causing the documentation build to fail with missing
document errors.
2025-06-27 08:20:44 -07:00
Ishaan JaffandGitHub c994c22c13 [Bug Fix] Bedrock Guardrails - Ensure PII Masking is applied on response streaming or non streaming content when using post call (#12086)
* test_bedrock_guardrail_response_pii_masking_streaming

* fix - apply masking for response on bedrock
2025-06-27 07:41:32 -07:00
Davis FeatherstoneandGitHub 51074ffcae Fix Azure-OpenAI Vision API Compliance (#12075)
* Fix Azure-OpenAI Vision API Compliance

* Linting Fix
2025-06-26 22:54:15 -07:00
Krish DholakiaandGitHub dd92b1a0ac Refactor: bedrock passthrough fixes - migrate to Passthrough SDK (#12089)
* feat: initial commit adding bedrock support via the new sdk passthrough logic

ensures correct sequencing of tasks (pre call checks etc. can run before signing request)

* fix(route_llm_requests.py): passthrough to allm_passthrough_route if no model found

* feat(bedrock/passthrough): working bedrock passthrough via sdk support

* fix(passthrough/main.py): re-add data and json

* feat(passthrough/main): support async passthrough calls to bedrock

* feat(passthrough/main.py): async streaming + completion support

* feat(llm_passthrough_endpoints.py): migrate bedrock passthrough calls to to new bedrock passthrough sdk

Enables calls to work correctly

* fix: fix linting errors

* test: update test
2025-06-26 22:51:35 -07:00
Krish DholakiaandGitHub e8d7537b57 Raise clearer error on Anthropic Unified route + allow setting new_key on /key/regenerate
* fix(llm_http_handler.py): raise clearer error on anthropic unified route

Fixes https://github.com/BerriAI/litellm/issues/12063

* fix(key_management_endpoints.py): add new param `new_key` for setting the regenerated key value

user request

* test: add unit tests

* fix(pass_through_endpoints.py): use data instead of json for passthrough requests

fixes bedrock latency issue

* Revert "fix(pass_through_endpoints.py): use data instead of json for passthrough requests"

This reverts commit 021dfd9165f837d37e6aad247ccbd0d2e8ca6043.
2025-06-26 21:56:03 -07:00
Ishaan Jaff ba7463b9c6 docs - spend monitoring 2025-06-26 17:39:13 -07:00
Ishaan Jaff edf416c24f docs - Use LiteLLM with Gemini CLI 2025-06-26 17:31:19 -07:00
Ishaan Jaff 72ecac8962 docs - user agent 2025-06-26 16:23:59 -07:00
Ishaan Jaff 16126f0d6b docs Gemini Text-to-Speech 2025-06-26 15:59:48 -07:00
Ishaan Jaff 80acf6bc97 docs */speech to /chat/completions Bridge** 2025-06-26 15:53:25 -07:00
Ishaan Jaff fc39343599 docs update 2025-06-26 15:32:33 -07:00
Ryan CastnerandGitHub ac11bfabcb Responses API - Add reasoning content support for non-OpenAI providers (#12055)
Add reasoning content support when Responses API falls
back to completions API, enabling reasoning content for
all LLM providers (Anthropic, Vertex AI, etc.) since
OpenAI is currently the only native Responses API
provider.

* Add ReasoningSummaryTextDeltaEvent for streaming
  reasoning deltas
* Update streaming iterator to detect and transform
  reasoning content
* Extract reasoning content as separate output items in
  transformations
* Support reasoning content alongside regular message
  content

Closes https://github.com/BerriAI/litellm/issues/11302
2025-06-26 14:21:49 -07:00
Cole McIntoshandGitHub 98ba5c8ffe Merge pull request #12042 from colesmcintosh/fix-today-selector-date-mutation-bug
Fix today selector date mutation bug in dashboard components
2025-06-26 14:43:02 -06:00
Ishaan Jaff e556071544 docs - Use LiteLLM with Gemini CLI 2025-06-26 13:35:33 -07:00
Cole McIntosh 8f376a60fa fix(ui): fix Today filter not showing usage data in dashboard
- Fix date formatting in API calls from ISO format to YYYY-MM-DD
- Update userDailyActivityCall, teamDailyActivityCall, and tagDailyActivityCall
- Prevent date mutation by creating new Date objects before API calls
- Set proper time boundaries (00:00:00 to 23:59:59) for same-day selections

The API expects dates in YYYY-MM-DD format but the UI was sending
full ISO timestamps, causing the Today filter to return empty results.
2025-06-26 13:59:40 -06:00
Cole McIntoshandGitHub 92948b33b1 Merge branch 'BerriAI:main' into fix-today-selector-date-mutation-bug 2025-06-26 12:29:44 -06:00
Krrish Dholakia 1de2e4ce28 fix(anthropic_endpoints.py): publicly expose anthropic v1/messages endpoint 2025-06-26 10:47:37 -07:00
Krrish Dholakia 0e96f412a1 build(model_prices_and_context_window.json): mark azure o3-pro as responses api model
Fixes https://github.com/BerriAI/litellm/issues/12059
2025-06-26 10:41:03 -07:00
Ishaan JaffandGitHub 3c43b0f8ad add _pretty_print_redis_config (#12073) 2025-06-26 10:26:57 -07:00
tanjiroandGitHub bd0a5f7387 Inkeep searchbar and chat added to the Docs (#12030)
* inkeep chat added

* add prod key
2025-06-26 10:13:42 -07:00
Cole McIntoshandGitHub 7efb2c0f3c Merge pull request #12041 from colesmcintosh/ci-artifact
Enhance CircleCI integration in LLM translation testing workflow
2025-06-26 11:12:04 -06:00
Cole McIntoshandGitHub bd77321c6d Merge branch 'BerriAI:main' into ci-artifact 2025-06-26 10:59:17 -06:00
Cole McIntoshandGitHub 7b7472d1f1 Merge pull request #12072 from colesmcintosh/fix/test-mock-create-audio-file
fix(proxy): Fix test_mock_create_audio_file by adding managed_files hook
2025-06-26 10:58:58 -06:00