tiennm99/litellm
1
0
Fork 0
mirror of https://github.com/tiennm99/litellm.git synced 2026-08-17 04:26:20 +00:00
Code Issues Packages Projects Releases Wiki Activity
25,925 Commits 2,072 Branches 1,425 Tags
efa782d6d281e28d126dce8e0610c4e449d7a0cd
Commit Graph
25925 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ishaan JaffandGitHub efa782d6d2 [Feat] Add Nvidia NIM Rerank Support (#15152) 
* feat: add NvidiaNimRerankConfig

* fix: NvidiaNimRerankConfig

* fix: NvidiaNimRerankConfig

* fix routing to nvidia nim

* docs nvidia nim rerank

* TestNvidiaNim

* nvidia nim rerank fixes

* fix rerank

* transform_rerank_response

* Usage with LiteLLM Proxy

* fixes linting

* NvidiaNimRerankConfig.DEFAULT_NIM_RERANK_API_BASE

* fix Custom API Base URL

* fix rerank base

* fix main.py

* fix transform

* fix linting

* map_cohere_rerank_params

* ruff fix

* linting fixes

* ruff fix
 2025-10-02 18:58:52 -07:00
Alexsander HamirandGitHub 79c24be48b [Fix] - Router: optimize unhealthy deployment filtering in retry path (O(n*m) → O(n+m)) (#15110) 
* perf(router): optimize unhealthy deployment filtering in retry path

Convert unhealthy_deployments list to set for O(1) lookups in
_async_get_healthy_deployments, reducing complexity from O(n*m) to O(n+m).

This method is called on every retry attempt (inside the retry loop), so
the optimization compounds during failures:

Before: 100 deployments × 50 unhealthy = 5,000 operations per call
After:  100 deployments + 50 unhealthy = 150 operations per call

Impact during cascading failures:
- With 1000 req/sec, 40% error rate, 3 retries
- Prevents 32M+ operations/sec during incidents
- Critical for preventing router CPU spikes when you need performance most

Matches the pattern already used in _filter_cooldown_deployments (line 7272).

* fix: Undefined name HTTPException
 2025-10-02 18:55:18 -07:00
Sameer KankuteandGitHub 544db8d140 (feat)Litellm x twelvelabs bedrock[Async Invoke Support] (#14871) 
* Add async invoke support

* Add docs and correct embedding response

* fix cicd erros

* fix cicd erros

* fix mypy error

* Add litellm param input_type

* Update the docs
 2025-10-02 18:52:33 -07:00
Ishaan Jaffer 9c29f35c4b test_end_user_jwt_auth 2025-10-02 18:48:11 -07:00
Krish DholakiaandGitHub 54095a256a Merge pull request #15015 from anthony-liner/fix/langfuse-total-tokens 
fix: set usage_details.total in langfuse integration
 2025-10-02 17:33:43 -07:00
Ishaan JaffandGitHub 09106222c7 [Fix]: Handle non-serializable objects in Langfuse logging (#15148) 
* fix: safe_deep_copy

* fix: import copy
 2025-10-02 17:31:01 -07:00
Krish DholakiaandGitHub 7b8426af53 Merge pull request #15130 from deepanshululla/feature/logging_payload_spec_update 
Add provider name to payload specification
 2025-10-02 17:29:24 -07:00
Krish DholakiaandGitHub 6b3db2fd49 Merge pull request #15140 from niharm/fix-sonnet-4-5-200k-pricing 
Price Fix: Add 200K prices for Sonnet 4.5
 2025-10-02 17:28:47 -07:00
Georg WölfleinandGitHub dbfa8ec921 Fix end user cost tracking in the responses API (#15124) 
#13860
 2025-10-02 15:13:57 -07:00
Ishaan JaffandGitHub f8f4207994 [Security Fix] fix: don't log JWT SSO token on .info() log (#15145) 
* fix: get_redirect_response_from_openid

* fix info log check

* fix: forward_upstream_to_client
 2025-10-02 15:07:37 -07:00
Amir RefaeeandGitHub 8991657d67 added railtracks to projects using litellm (#15144) 2025-10-02 14:21:38 -07:00
Mubashir OsmaniandGitHub f2107a189d add azure_ai grok-4 model family (#15137) 
* added oauth mcp to docs

* added azure ai/grok-4 model family

* Revert "added oauth mcp to docs"

This reverts commit 950b7cef44f14b2db1429f6fbd32548a7c95d325.
 2025-10-02 14:21:12 -07:00
nihar cd7acd2eb2 Add 200K prices for Sonnet 4.5 2025-10-02 12:39:44 -07:00
Krrish Dholakia ebba9e0b2a docs: cleanup docs 2025-10-02 09:51:16 -07:00
deepanshu 2512d89872 Add provider name to payload specification 2025-10-02 10:23:26 -04:00
Krish DholakiaandGitHub f6b67fd9bd Merge pull request #14799 from tyler-liner/chore/generation-name-opentelemetry 
fix (opentelemetry): use generation_name for span naming in logging method
 2025-10-01 21:39:38 -07:00
Krish DholakiaandGitHub cb39a1bb60 Merge pull request #15024 from kowyo/main 
docs: use docker compose instead of docker-compose
 2025-10-01 21:34:42 -07:00
Krish DholakiaandGitHub 93e347b3d8 Merge pull request #15106 from plafleur/ISSUE-15105 
Guardrails - Don't run post_call guardrail if no text returned from Bedrock
 2025-10-01 21:29:14 -07:00
Ishaan JaffandGitHub d538cf489a [Feat] Fixes to dynamic rate limiter v3 - add saturatation detection (#15119) 
* test cases dynamic rate limits

* fix _handle_generous_mode

* docs add readme

* use configs for vars

* fix debug

* add comment

* test_dynamic_rate_limiter_v3.py

* test_concurrent_pre_call_hooks_stress
 2025-10-01 18:35:34 -07:00
Ishaan Jaffer 2bc5d93f23 use_callback_in_llm_call 2025-10-01 18:32:37 -07:00
Deepanshu LullaGitHubdeepanshu
 		68adca04c8 Gitlab based Prompt manager (#14988) 
* add prompt

* add prompt

* add prompt

* add prompt

* add prompt management via gitlab

* gitlab client

* gitlab client

* gitlab client

* fix lint issues

* fix lint issues

* remove router changes

---------

Co-authored-by: deepanshu <deepanshu.lulla@hq.bill.com>
 2025-10-01 18:13:11 -07:00
Ishaan JaffandGitHub 388761f52d [Fix] LiteLLM UI - Ensure OTEL settings are saved in DB after set on UI (#15118) 
* fix: fix _add_callback_from_db_to_in_memory_litellm_callbacks

* test_add_callback_from_db_to_in_memory_litellm_callbacks

* fix otel

* fix: fix _add_callback_from_db_to_in_memory_litellm_callbacks
 2025-10-01 15:33:22 -07:00
Ishaan JaffandGitHub d9664a3ee4 fix gpt-5-chat-latest on model cost map (#15116) 2025-10-01 14:35:57 -07:00
Ishaan JaffandGitHub e73d053de3 [Fix] Proxy Auth - Ensure LLM_API_KEYs can access pass through routes (#15115) 
* test_virtual_key_llm_api_routes_allows_registered_pass_through_endpoints

* fix: is_registered_pass_through_route

* docs fix
 2025-10-01 14:09:01 -07:00
Patrick Lafleur 8e5efd29df Fix comment 2025-10-01 16:39:26 -04:00
Patrick Lafleur d6be26dba6 Merge branch 'main' into ISSUE-15105 2025-10-01 14:44:40 -04:00
Luiz Rennó CostaGitHubLuiz Rennó Costa
 		7e56600896 fix: model_group not always present in litellm_params, and metadata reference location (#15108) 
Co-authored-by: Luiz Rennó Costa <luiz.renno@ifood.com.br>
 2025-10-01 11:39:49 -07:00
Patrick LafleurandGitHub e7fd1fb96b Fix missing HTTPException import (#15111) 2025-10-01 11:39:30 -07:00
Patrick Lafleur 7ef71d4885 Fix text 2025-10-01 12:08:03 -04:00
Sameer KankuteandGitHub 7ec7e5332c Add generateContent cost tracking (#15014) 2025-10-01 09:03:45 -07:00
Patrick Lafleur 3e5d585f7d Don't run post_call guardrail if no text returned from bedrock 2025-10-01 11:56:50 -04:00
Ishaan Jaffer ab00ca2de9 bump: version 1.77.6 → 1.77.7 2025-10-01 08:54:03 -07:00
Krrish Dholakia 7fd24a2632 bump: version 1.77.6 → 1.77.7 2025-09-30 21:23:42 -07:00
Krish DholakiaandGitHub 1503435d91 Merge pull request #15029 from henryhwang/gemini-adapter-fixes 
feat(gemini): Add full support for native Gemini API translation
 2025-09-30 21:19:18 -07:00
Ishaan JaffandGitHub 73bfef1a1f Revert "[Feature]: Replace HTTPException with ParallelRequestLimitError in pa…" (#15095) 
This reverts commit 71b9b58fa9.
 2025-09-30 21:17:04 -07:00
Ishaan Jaffer 395c32c38d test_azure_openai_assistants_e2e_operations_stream 2025-09-30 21:16:29 -07:00
Krish DholakiaandGitHub 64083111d3 (Feat) Add Vertex AI Live API WebSocket Passthrough with Cost Tracking 
(Feat) Add Vertex AI Live API WebSocket Passthrough with Cost Tracking
 2025-09-30 21:14:16 -07:00
Henry Wang acc23b9757 fix issue from pr review 2025-10-01 11:57:10 +08:00
Henry H WangandGitHub 7c4439ba0a Merge branch 'BerriAI:main' into gemini-adapter-fixes 2025-10-01 09:53:51 +08:00
Ishaan Jaffer 8dd31a5fe8 test_azure_openai_assistants_e2e_operations_stream 2025-09-30 18:47:09 -07:00
Ishaan Jaffer 04b3ac89b8 test: QueryParams 2025-09-30 18:45:38 -07:00
Uzair AliandGitHub fcfe856e10 Add support for GPT 5 codex models (#14841) 
* Add support for GPT 5 codex models

* lint

* fixes
 2025-09-30 18:44:35 -07:00
Alexsander HamirandGitHub 26145da3e7 perf(router): optimize _filter_cooldown_deployments to O(n) (#15091) 
Refactored to use set-based lookup and list comprehension instead
of two-pass approach with list.remove().

Old complexity: O(n×m + k×n)
- First loop: n deployments × m list lookups = O(n×m)
- Second loop: k removals × n list.remove() scans = O(k×n)

New complexity: O(m + n)
- Convert to set: O(m)
- Filter with O(1) set lookups: O(n)

Example with 100 deployments, 5 in cooldown:
- Old: ~1000 operations
- New: ~105 operations

Called on every request - high impact for production.
 2025-09-30 18:39:12 -07:00
Ishaan JaffandGitHub 0ca11eefde [Feat] Guardrails - add logging for important status fields (#15090) 
* add StandardLoggingPayloadStatusFields

* add status_fields

* add StandardLoggingPayloadStatusFields

* noma guard: add_standard_logging_guardrail_information_to_request_data

* fix: StandardLoggingPayloadStatusFields

* fix tests

* fix StandardLoggingPayloadStatus

* get_standard_logging_object_payload

* test_bedrock_guardrail_status_failure

* fix: _get_status_fields

* fixes new guardrail tracing

* fix ruff
 2025-09-30 18:38:07 -07:00
Henry Wang 4eee54b157 fix the test issue from the pr review 2025-10-01 09:08:25 +08:00
Ishaan Jaffer f205b2c0a5 test fixes 2025-09-30 17:08:49 -07:00
Ishaan Jaffer aac1129761 fix is_sensitive_key 2025-09-30 17:05:57 -07:00
malagsandGitHub 68189d1c04 [Performance] Reduce complexity of InMemoryCache.evict_cache from O(n*log(n)) to O(log(n)) (#15000) 
* Improved performance by reducing complexity

* Improved logic to prevent memory from increasing too much, added test

* Restore indent

* Restore indent

* Added type annotation

* Updated test to correctly initialize the expiration_heap
 2025-09-30 16:49:35 -07:00
Ishaan Jaffer f46f9d3fd9 docs azure passthrough api fixes 2025-09-30 15:54:09 -07:00
Krish DholakiaandGitHub c475723c67 Merge pull request #15044 from BerriAI/litellm_dev_09_29_2025_p1 
MCP - enforce server permissions on call tools + Teams - add model specific tpm/rpm limits to teams on LiteLLM
 2025-09-30 15:32:53 -07:00