mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-13 01:04:54 +00:00
f5b11b72a6dcc8f4e7a16f00a61bdd124cc171d2
22 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ryan-crabbe-berri
|
770fff7058 |
test(proxy): stop running real-DB tests in GitHub Actions unit jobs (#29700)
* test(proxy): stop running real-DB tests in GitHub Actions unit jobs GitHub Actions unit jobs were spinning up a Postgres service container, but the only active tests that touched it either used the DB incidentally (a cargo-culted prisma_client.connect()) or were genuine integration tests mislabeled as unit. Mock the incidental ones so the proxy-db job needs no container, and move the tests that genuinely need a database (proxy management behavior, master-key-not-persisted, schema-migration sync) to CircleCI, which is already the real-infrastructure lane. * test(proxy): restore no-unexpected-startup-writes canary in master-key test Greptile noted the hash-match assertion no longer catches other unexpected startup writes (a default key, a rotation artifact). The CircleCI job gives each run a fresh DB, so a clean startup must leave the table empty; add that canary back alongside the precise master-key assertion. |
||
|
Sameer Kankute
|
b7e978a5c3 |
Litellm oss staging 04 21 2026 2 (#26569)
* fix(bedrock): use model info lookup for output_config support instead of hardcoded check Replace hardcoded _is_claude_4_6_model() string matching with supports_output_config flag in model_prices_and_context_window.json, accessed via _supports_factory(). This follows the project's established pattern for model capability checks (per AGENTS.md rule #8). Bedrock Invoke now conditionally preserves output_config for models that declare supports_output_config=true (currently Claude 4.6 models), while stripping it for older models to avoid request rejection. Ref: https://github.com/BerriAI/litellm/issues/22797 * fix(vertex_ai): single-flight credential refresh to prevent thundering herd (#26024) * fix(vertex_ai): single-flight credential refresh to prevent thundering herd When GCP credentials expire under high concurrency, all requests simultaneously call credentials.refresh() via asyncify, saturating the 40-thread anyio pool and blocking the proxy for 20+ seconds. This adds: - Per-credential asyncio.Lock in get_access_token_async for single-flight refresh (1 coroutine refreshes, others wait on the lock) - Background refresh when token_state is STALE (usable but near expiry), returning the current token immediately with zero added latency - threading.Lock on the sync get_access_token path - Uses google-auth's TokenState enum (FRESH/STALE/INVALID) instead of reimplementing expiry logic Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review comments - Use asyncio.create_task() instead of deprecated get_event_loop().create_task() - Track in-flight background refresh tasks to prevent duplicate refreshes when multiple STALE-path callers pass through the lock before the first background task completes - Add token validation in the STALE branch (consistent with FRESH/INVALID) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: lazy-import TokenState to avoid breaking when google-auth is not installed Also extract helper methods to bring get_access_token_async under the PLR0915 statement limit (50). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: apply Black formatting to test file and update uv.lock Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove user-provided project_id from log messages (CodeQL log injection) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: avoid leaking token value in error message, log type instead Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: restore uv.lock to match litellm_oss_branch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove project_id from remaining log message (CodeQL log injection) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove remaining project_id from log and error messages Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: reuse cached credentials in VertexAIPartnerModels (#26065) * fix: reuse cached credentials in VertexAIPartnerModels instead of creating new VertexLLM per request VertexAIPartnerModels.completion() was creating a throwaway VertexLLM() instance on every call to get an access token, bypassing the credential cache inherited from VertexBase. This caused a fresh token fetch for every single request, adding significant latency overhead. Fix: call super().__init__() to initialize VertexBase's credential cache, and use self._ensure_access_token() instead of a new VertexLLM instance. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: apply same credential caching fix to VertexAIGemmaModels and VertexAIModelGardenModels Same bug as VertexAIPartnerModels: both classes had `pass` in __init__ instead of `super().__init__()`, and created throwaway VertexLLM() instances per request instead of using self._ensure_access_token(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(fireworks): add glm-5p1 metadata and parallel_tool_calls (#26069) * fix(chatgpt): preserve responses routing and recover empty output (#25403) (#26219) - preserve existing shared backend `mode` when router deployment registration reuses a provider/model key already in `litellm.model_cost` (prevents alias with `mode: chat` from downgrading shared `chatgpt/gpt-5.4` from `responses` to `chat` and triggering 403s on /v1/chat/completions) - teach the ChatGPT Responses parser to recover `response.output_item.done` entries when `response.completed.output` is empty - add defensive /responses -> /chat/completions bridge fallback that reconstructs output items from raw SSE when `raw_response.output` is empty - regression coverage for shared alias routing, empty completed.output parsing, and SSE bridge recovery Closes #25403 Co-authored-by: afoninsky <andrey.afoninsky@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(deps): relax core runtime dependency pins from exact == to ranges When litellm migrated from Poetry to uv (PR #24905, v1.83.1), the core dependency specifications in pyproject.toml changed from Poetry bare-version strings (e.g. openai = "2.30.0") to PEP 621 exact pins (openai==2.24.0). Poetry bare-version strings are actually caret ranges (^X.Y.Z == >=X.Y.Z,<X+1), but PEP 621 == is exact. This means every downstream package that installs litellm as a library dependency is now forced to downgrade aiohttp, pydantic, openai, click, and 8 other common packages to exact old versions. Fix: restore range specifiers for the 12 core runtime dependencies. The optional extras (proxy, proxy-runtime, etc.) are consumed primarily by Docker images where exact pins are appropriate and are left unchanged. The uv.lock file continues to provide exact reproducibility for Docker builds and CI. Fixes: #26154 * Add Rubrik as officially-supported guardrail plugin (#25305) * Add Rubrik as officially-supported guardrail plugin Adds tool blocking and batch logging integration with an external Rubrik webhook service. The plugin validates LLM tool calls against a policy service (fail-open on errors) and batch-logs all requests/responses. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Update Rubrik docs: config.yaml as primary, env vars as fallback Restructures the Quick Start to present config.yaml as the recommended approach with tabbed UI, and environment variables as an alternative fallback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add Rubrik env vars to config_settings reference Fixes documentation validation by adding RUBRIK_API_KEY, RUBRIK_BATCH_SIZE, RUBRIK_SAMPLING_RATE, and RUBRIK_WEBHOOK_URL to the environment settings reference table. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add fallback message when blocking service returns empty explanation Prevents whitespace-only violation message when the tool blocking service blocks tools but returns an empty content field. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(ocr): add Reducto parse OCR support (#26068) * feat(ocr): add Reducto parse OCR support * fix(reducto): address OCR review feedback * chore: refresh uv lockfile * Revert "chore: refresh uv lockfile" This reverts commit 47200c0e603275108335aee852d0a96586165337. * Fix failing tests * Fix code qa * Replaced the async client violation * Replaced black formatting * Fix failing tests * Fix failing tests * Fix failing tests * Fix failing tests * Fix tests * Fix vertex ai cred test * Fix test * fix(xai): normalize usage total_tokens for prompt caching xAI can return total_tokens inconsistent with prompt_tokens + completion_tokens when caching is enabled. Align with OpenAI-style usage so shared LLM tests and downstream consumers see coherent totals. Apply to non-streaming responses and streaming usage chunks. Made-with: Cursor * Fix stale Vertex token refresh fallback * Fix OCR zero credit and Bedrock support checks * Fix OCR and Fireworks capability handling * fix: evict completed background refresh tasks from _background_refresh_tasks Completed asyncio.Task objects were never removed from _background_refresh_tasks. In long-running proxies with many distinct credential keys the dict grows indefinitely, retaining references to finished tasks and their results. Fix: - Pop the existing (done) entry before creating a replacement task. - Attach a done_callback to each new task that removes its entry from the dict once the task finishes (success or failure). Tests: - test_background_refresh_task_removed_after_completion: verifies the done-callback cleans up a single entry after the task completes. - test_background_refresh_tasks_no_accumulation_across_many_keys: drives 20 distinct credential keys and confirms the dict is empty after all background refreshes finish. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: guard asyncio.create_task in RubrikLogger.__init__ against missing event loop asyncio.create_task() raises RuntimeError when called outside a running event loop. Wrap the call in a try/except RuntimeError so that RubrikLogger can be instantiated in synchronous contexts (e.g. during startup, testing) without crashing. The periodic_flush background task simply won't start in those cases; it starts normally when the constructor is called inside an event loop. Add a test that verifies instantiation outside an event loop does not raise (does not patch asyncio.create_task). Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: preserve async batch and reauth coordination * Fix mypy * Fix xAI usage and Fireworks parallel tool params * Fix Rubrik batch drain and SSE recovery mutation * Fix router mode preservation and Rubrik batch flushing * fix(responses): merge text-only items with output items in SSE recovery When recovering output from raw SSE, OUTPUT_ITEM_DONE and OUTPUT_TEXT_DONE events were treated as mutually exclusive fallbacks. If a stream emitted OUTPUT_ITEM_DONE for some output indices and only OUTPUT_TEXT_DONE for others, the text-only items at the missing indices were silently dropped. Merge both dicts before returning, with OUTPUT_ITEM_DONE entries taking precedence at any shared index (preserving the existing behavior covered by test_transform_response_preserves_output_item_when_text_done_arrives_later). Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(rubrik): preserve events on batch send failure Previously, _log_batch_to_rubrik swallowed all HTTP errors and exceptions, and the parent flush_queue unconditionally drained the queue afterwards. On Rubrik 5xx responses, network errors, or timeouts the in-flight events were silently dropped without ever being delivered. - Re-raise from _log_batch_to_rubrik so failures surface to the caller. - In CustomBatchLogger.flush_queue, catch exceptions from async_send_batch and leave the queue intact for retry on the next flush. Existing loggers that override flush_queue (e.g. Datadog) or that swallow their own errors inside async_send_batch (e.g. Langsmith, GCS, Argilla) are unaffected. - Tests now assert events are preserved on HTTP errors, network errors, and that mid-flush appended events are also preserved on failure. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(chatgpt/responses): strip whitespace before parsing SSE chunks _parse_sse_json_chunk in ChatGPTResponsesAPIConfig passed the raw chunk directly to _strip_sse_data_from_chunk, which only matches the 'data:' prefix at position 0. Chunks with leading whitespace (e.g. ' data: {...}') were returned unchanged and silently failed JSON parsing, dropping the contained event. Mirror the existing fix in LiteLLMResponsesTransformationHandler._parse_raw_sse_chunk by calling chunk.strip() before stripping the SSE prefix. Adds a regression test using whitespace-padded data: lines and verifies that the response.output_item.done payload is recovered into the final ResponsesAPIResponse output. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(rubrik): override flush_queue so a single snapshot drives send and drain Previously RubrikLogger relied on CustomBatchLogger.flush_queue, which captured len(self.log_queue) separately from the snapshot taken inside async_send_batch. Although both happen without an intervening await today (so they agree in practice), they are semantically disconnected: a future refactor that adds an await between the two captures, or that changes the async_send_batch contract, could cause the parent to delete a different number of items than were actually sent and trigger duplicate deliveries to Rubrik. Override flush_queue on RubrikLogger so a single snapshot drives both the HTTP POST and the queue truncation. async_send_batch is preserved for direct callers/tests but no longer participates in the canonical flush path. Existing tests (including the one that explicitly invokes the base CustomBatchLogger.flush_queue path) still pass. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: register reducto/parse-v3 and reducto/parse-legacy in active model pricing file Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bedrock): restore output_config forwarding and black formatting Use model-map lookup with _model_supports_effort_param fallback so Bedrock Invoke keeps output_config for Claude 4.6/4.7 when pricing flags are missing. Revert custom_llm_provider=bedrock for supports_output_config checks, fix allowlist test model, and apply black to xai/vertex files failing lint CI. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(greptile): address remaining review concerns - fireworks: resolve supports_reasoning lookup for short model names by also trying the full accounts/fireworks/models/ path in model_cost - ocr_cost: drop reducto-specific guard in shared utility; treat missing pages_processed as zero cost when no per-page pricing is configured - docs: remove reducto/rubrik markdown stubs from this repo (canonical docs live in litellm-docs) * fix(model_prices): register mistral/ministral-8b-2512 Mistral's API now returns model='ministral-8b-2512' when 'mistral-tiny' is requested. Adding the entry so completion_cost can resolve the cost for that response. * fix(greptile): prune async refresh locks and lazy-start rubrik flush - vertex: back `_async_refresh_locks` with a WeakValueDictionary so a per-key Lock is auto-evicted once no coroutine holds it, preventing unbounded growth in deployments with many credential combinations while keeping single-flight semantics intact. - rubrik: defer the periodic flush task to the first log event when the logger is constructed without a running event loop, so low-traffic batches still get drained instead of being silently stranded by a swallowed RuntimeError. * Remove duplicate supports_max_reasoning_effort key in claude-opus-4-7 entries Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(vertex_ai): stabilize background refresh task tracking - Guard background refresh done_callback with an identity check so a stale callback cannot remove a newer task that already replaced it in the tracking dict (done_callbacks are scheduled via call_soon, so a fresh task can be stored for the same credential key before the old callback fires). - Replace WeakValueDictionary with a regular dict for _async_refresh_locks so the per-key asyncio.Lock identity is stable across concurrent callers; otherwise a lock can be GC'd between two coroutines arriving for the same key, breaking single-flight. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: surface OCR pricing gaps and recover OUTPUT_TEXT_DONE in ChatGPT SSE - cost_calculator.ocr_cost: log a warning when pages_processed is reported but no ocr_cost_per_page is configured, instead of silently billing zero via an implicit '(... or 0.0) * pages_processed' fallback. Behavior is preserved (zero cost) so free-tier / unpriced models still work, but configuration gaps are now visible in logs. - ChatGPTResponsesAPIConfig._extract_completed_response_from_sse: also collect response.output_text.done events into a text-only items map and merge them into the recovered output (OUTPUT_ITEM_DONE wins on duplicate output_index), mirroring the LiteLLMResponses handler. This recovers text content when a provider only emits OUTPUT_TEXT_DONE and the final response.completed event has an empty output list. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(cicd): drop obsolete async refresh locks auto-prune test Commit |
||
|
Sameer Kankute
|
e59e34bed3 |
Gemini managed agents support (#28270)
* Add support for environment variable in interactions api * Add sdk support for gemini create agent * Add agents endpoint support via proxy * Add outputs of each api * Add routing for model and agents param * Remove redundant condition in get_provider_agents_api_config LlmProviders.GEMINI.value is literally the string "gemini", so the second clause of the or was checking the exact same thing as the first. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: forward query-param credentials to list/get/delete/versions Gemini agent endpoints The list_gemini_agents, get_gemini_agent, delete_gemini_agent, and list_gemini_agent_versions endpoints previously constructed a hardcoded data dict with no mechanism to pass provider credentials. Unlike create_gemini_agent (POST, reads litellm_params_template from body), these GET/DELETE endpoints gave no way for multi-tenant callers to supply a per-request api_key or other LiteLLM params. Fix: - Add _merge_query_params_into_data() helper that reads query parameters from the request and merges them into the data dict without overwriting already-set keys (e.g. path params like 'name'). - Support a JSON-encoded litellm_params_template query parameter (matching the POST body pattern) as well as flat key=value pairs (e.g. api_key=AIza...). - Apply the helper in all four affected endpoints. - Add 13 unit tests covering the helper and each endpoint. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: pass model=None for managed agent proxy endpoints to prevent agent name polluting data["model"] Endpoints acreate_agent, aget_agent, adelete_agent, and alist_agent_versions were passing model=<agent_name> to base_process_llm_request. This caused common_processing_pre_call_logic to write the agent name into self.data["model"], which then triggered spurious model-alias mapping, rate-limiting lookups, and logging tied to a non-existent model deployment. The agent name is already carried in data["name"] and is passed correctly to the SDK functions (litellm.interactions.agents.*). There is no reason to also set model=<agent_name>; the correct value is model=None for all five managed-agent management routes. Adds tests/test_litellm/proxy/google_endpoints/test_managed_agents_model_param.py to verify all five managed-agent endpoints pass model=None. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: address greptile P1/P2 review comments P1 (router.py): Restore fallback/retry support for acreate_interaction and create_interaction. Both were silently moved to _init_interactions_api_endpoints (direct call, no fallbacks). Moved them back to _ageneric_api_call_with_fallbacks so users with configured fallback models keep retry behaviour. P1 security (agents_endpoints.py): Remove flat query-param credential path (e.g. ?api_key=AIza...) from _merge_query_params_into_data. Credentials in URL query strings appear verbatim in server access logs, CDN edge logs, and browser history. Only the JSON-encoded litellm_params_template query param (matching the POST body pattern) is retained. P2 (interactions/http_handler.py): Extract _BaseHTTPHandler with shared _handle_error, _sync_client, and _async_client helpers. InteractionsHTTPHandler now extends _BaseHTTPHandler. The _async_client reads the provider from litellm_params instead of hardcoding GEMINI. P2 (interactions/agents/http_handler.py): AgentsHTTPHandler now extends InteractionsHTTPHandler (which inherits _BaseHTTPHandler) so all shared HTTP infrastructure is reused rather than duplicated. Removes the hardcoded LlmProviders.GEMINI from the async client path. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: address CI failures from greptile review fixes - black: format interactions/agents/main.py and utils.py - tests: update test_gemini_agents_endpoints.py to match new _merge_query_params_into_data behaviour (flat credential params are rejected; only JSON-encoded litellm_params_template is accepted) - ci: add test_gemini_agents_endpoints.py to endpoints-and-responses shard in test-unit-proxy-db.yml so assert-shard-coverage passes - tests: add _initialize_managed_agents_endpoints and _init_managed_agents_api_endpoints test coverage so router_code_coverage passes; also fix TestRouterCreateInteractionRouting to reflect that acreate_interaction now correctly routes through _ageneric_api_call_with_fallbacks (restoring fallback support) Co-authored-by: Cursor <cursoragent@cursor.com> * fix: remove InteractionsHTTPHandler._handle_error override to fix type errors AgentsHTTPHandler extends InteractionsHTTPHandler and calls self._handle_error(provider_config=agents_api_config) where agents_api_config is BaseAgentsAPIConfig. Python MRO resolved _handle_error to InteractionsHTTPHandler._handle_error which expected BaseInteractionsAPIConfig, causing 10 mypy arg-type errors in interactions/agents/http_handler.py. Removing the redundant override lets both classes inherit _BaseHTTPHandler._handle_error (provider_config: Any) which is structurally correct for both config types. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: agent-only interactions and managed agents provider routing Resolve None custom_llm_provider in agents HTTP client lookup and set custom_llm_provider on GenericLiteLLMParams for all agent CRUD paths. Stop mapping agent names to proxy model routing; route interactions through _init_interactions_api_endpoints with fallbacks only when model is set. Consolidate duplicate router elif branches for interaction APIs. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix greptile review * test(agents): add unit tests for managed agents SDK and HTTP handler Adds coverage for the new `litellm.interactions.agents` surface area: - main.py: sync/async entry points (create/list/get/delete/list_versions), provider config lookup, logging-obj helper, async error wrapping - http_handler.py: every CRUD method (sync + async paths), `_is_async` dispatch branches, and provider error mapping through GeminiAgentsConfig - utils.py: get_provider_agents_api_config for supported / unsupported providers Brings patch coverage on these files from <25% to ~100% so codecov/patch is satisfied. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * docs(gemini-agents): fix misleading credential-passing examples in GET/DELETE docstrings (#28293) The four GET/DELETE endpoint docstrings (list_gemini_agents, get_gemini_agent, delete_gemini_agent, list_gemini_agent_versions) documented passing per-request credentials as flat query parameters (e.g. ?api_key=AIza...). However, _merge_query_params_into_data only reads the JSON-encoded litellm_params_template query parameter and intentionally ignores flat params (URL query strings appear verbatim in access logs, browser history, and Referer headers). Callers following the documented curl examples would have their credentials silently dropped and hit auth failures against Gemini. Update the examples to use the supported JSON-encoded litellm_params_template query parameter, matching _merge_query_params_into_data's own docstring. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(agents): rename provider-agnostic agent response types Move GeminiAgent{ListResponse,DeleteResult,VersionsResponse} to provider-neutral names (AgentListResponse, AgentDeleteResult, AgentVersionsResponse) so the BaseAgentsAPIConfig interface no longer references Gemini-specific type names. * fix(gemini-agents): close veria-flagged credential-escalation gaps Two high-severity findings from the veria-ai PR review are addressed: 1. **api_base override could leak the shared Gemini key** GeminiAgentsConfig.validate_environment falls back to GOOGLE_API_KEY / GEMINI_API_KEY when no api_key is supplied. Combined with caller-controlled api_base on the proxy CRUD endpoints, an authenticated user could redirect the outbound request to an attacker-controlled host and capture the operator's shared Gemini key from the x-goog-api-key header. The config now refuses env-fallback whenever api_base is explicitly overridden. 2. **Managed-agent CRUD exposed to ordinary LLM keys** The new /v1beta/agents routes live in google_routes (i.e. llm_api_routes), so any non-admin LLM key can reach them. Unlike /v1beta/models/...: generateContent these endpoints are NOT model-routed and have no model_list-supplied credentials, so env-fallback would let any LLM key list / create / delete agents inside the operator's Gemini project. Each endpoint now calls _enforce_caller_supplied_provider_key, which requires non-admin callers to supply their own Gemini api_key via litellm_params_template. Proxy admins keep the env-fallback convenience. Tests cover non-admin rejection, admin allow-through, the api_base override guard, and SDK env-fallback when api_base is not overridden. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(router): restore strict assert_called_once_with on interactions default-provider test --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
Krrish Dholakia
|
8bbc61e03c |
fix: harden /key/update authorization checks (#27878)
* fix: patch Host-header auth bypass in get_request_route Starlette reconstructs request.url from the Host header. A malformed Host like `localhost/?x=1` causes Starlette to build the full URL as `http://localhost/?x=1/health`, which url-parses to path="/". Since "/" is in LiteLLMRoutes.public_routes, all protected routes became reachable without authentication. Fix: read scope["path"] (set by uvicorn from the HTTP request line, not derivable from headers) instead of request.url.path. Sub-path deployments are handled via scope["app_root_path"] / scope["root_path"], mirroring Starlette's own base_url construction logic. Affected variants confirmed fixed: Host: localhost/?x=1 Host: localhost:4000/?x=1 Host: localhost/#test Host: localhost:4000/#test Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * style: reduce comments in route fix Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block credential fields in RAG ingest vector_store options Credential fields (vertex_credentials, aws_access_key_id, api_key, etc.) in ingest_options.vector_store are now rejected at the API boundary with a 400 error. Credentials must be configured server-side. Previously any authenticated user could supply a vertex_credentials dict with type=external_account pointing credential_source.file at an arbitrary path (e.g. /proc/1/environ) and token_url at an attacker-controlled server. google-auth's identity_pool.Credentials refresh() would read the file and POST its contents to the attacker. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block /key/update self-escalation by assigned users Non-admin users who were assigned a key (created_by != caller) could update any non-budget field — models, rpm_limit, guardrails, etc. — without admin authorization, allowing privilege self-escalation. Gate: only the key creator (created_by == caller) may edit their own key without admin check; budget changes always require admin regardless of creator status. All other callers must pass _check_key_admin_access. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block user-controlled api_base in RAG ingest vector_store options A user-supplied api_base in ingest_options.vector_store caused the server to forward its configured provider credentials (Gemini, OpenAI) to an attacker-controlled endpoint via SSRF. Add api_base to the blocked credential params set alongside api_key and the existing credential fields. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: restrict /utils/transform_request to PROXY_ADMIN and apply body safety check Any authenticated internal_user could POST arbitrary provider config (aws_sts_endpoint, api_base, etc.) to /utils/transform_request and have the server forward its credentials to an attacker-controlled endpoint. - Gate the endpoint on PROXY_ADMIN role (403 for all other roles) - Call is_request_body_safe() to reject banned params even for admins - Convert ValueError from safety check to HTTP 400 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: apply banned-param check to /utils/transform_request Without is_request_body_safe(), any authenticated user could pass aws_sts_endpoint, api_base, or aws_web_identity_token to /utils/transform_request and have the server forward its configured provider credentials to an attacker-controlled endpoint during SDK credential resolution. Applies the same banned-param blocklist already used by LLM endpoints. Endpoint remains accessible to all authenticated users. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block SSRF via api_base in /prompts/test dotprompt YAML frontmatter Any frontmatter key not in ["model","input","output"] flowed into optional_params and was merged into the LLM call data dict, bypassing is_request_body_safe. An attacker with any bearer key could set api_base in YAML to redirect the outbound LLM request — including the provider API key — to an attacker-controlled host. Fix: call is_request_body_safe on the constructed data dict after optional_params are merged, before invoking ProxyBaseLLMRequestProcessing. ValueError from the banned-param check is surfaced as HTTP 400. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Update litellm/proxy/rag_endpoints/endpoints.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * fix: coerce nested config strings before banned-param check _NESTED_CONFIG_KEYS descent used isinstance(nested, dict) which silently skipped litellm_embedding_config when delivered as a JSON string via multipart/form-data. Banned params (api_base, aws_sts_endpoint, etc.) nested inside the stringified value were invisible to is_request_body_safe. _NESTED_METADATA_KEYS already used _coerce_metadata_to_dict which parses JSON strings before checking. Apply the same coercion to _NESTED_CONFIG_KEYS. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: replace substring match with prefix match in is_llm_api_route mapped_pass_through_routes used `_llm_passthrough_route in route` (substring) so any admin-only path whose URL contained a provider name (openai, anthropic, azure, bedrock, etc.) was misclassified as an LLM API route and bypassed the admin gate in non_proxy_admin_allowed_routes_check. Confirmed live: non-admin key could GET /credentials/by_name/openai (read masked provider API key) and DELETE /credentials/openai (delete credential). Fix: use exact match or startswith(prefix + "/") — the same pattern used everywhere else in RouteChecks — so only routes that actually start with a passthrough prefix are allowed through. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: stabilize PR #27878 test failures - key_management_endpoints: extend can_skip_admin_check to team keys so team members with /key/update permission can update non-budget fields. can_team_member_execute_key_management_endpoint already validates team membership + permission and raises if unauthorized; reaching the admin check on a team key means the caller was authorized. - test: set created_by on mock key in test_update_key_non_budget_fields_allowed_for_internal_user so caller_is_creator resolves correctly (MagicMock default ≠ user_id). - auth_utils.get_request_route: guard against non-dict request.scope (e.g. MagicMock in unit tests) to prevent a MagicMock leaking into UserAPIKeyAuth.request_route and failing Pydantic validation. - ci: assign test_multipart_bypass_repro.py to the proxy-runtime shard in test-unit-proxy-db.yml to satisfy the shard-coverage check. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(lint): add explicit str() cast in get_request_route for MyPy scope.get() returns Any|None which MyPy cannot coerce to str implicitly. Wrap both scope.get() calls in str() to satisfy the type checker. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: guard bare-/ root_path strip + make total_spend migration idempotent auth_utils.get_request_route: when Starlette sets scope["app_root_path"] to "/" (e.g. behind some middleware), the old stripping logic would remove the leading slash from every path ("/team/new" → "team/new"), breaking route matching and causing auth to misclassify protected routes. Skip stripping when root_path is bare "/". migration: add IF NOT EXISTS to total_spend ALTER TABLE so the migration is safe to replay when a prior partial run already created the column. Without this guard, prisma migrate deploy fails on CI DBs that were partially migrated, causing all subsequent DB operations (including /team/new) to 500. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: require creator still owns key for personal-key bypass in /key/update caller_is_creator now requires both created_by == caller AND user_id == caller. Previously checking only created_by let a demoted admin who originally created a key for another user continue editing non-budget fields on it after reassignment, bypassing _check_key_admin_access. Adds regression test: creator whose key was reassigned is blocked (403). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: extract auth checks to fix PLR0915 + broaden max_budget assertion internal_user_endpoints._update_single_user_helper exceeded 50 statements (PLR0915). Extract authorization checks into _check_user_update_authz helper to bring statement count under the limit. test_validate_max_budget: assert "negative" (substring of both the local "cannot be negative" and the CI "non-negative finite number" messages) so the test is stable regardless of which exact wording the function uses. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> |
||
|
harish-berri
|
8f25942ecf |
Litellm key rotation bug (#27756)
* fix(proxy): resolve cache handling issues in _lookup_deprecated_key - Updated the in-memory cache for deprecated key lookups to store a 3-tuple (active_token_id, cache_expires_at_ts, revoke_at_ts) instead of a 2-tuple, ensuring proper unpacking and backward compatibility. - Removed duplicate cache reads and added logic to handle legacy cache entries gracefully. - Enhanced unit tests to cover scenarios for cache hits, DB misses, and respect for revoke_at timestamps, ensuring robust handling of the grace-period key-rotation feature. * refactor(proxy): streamline cache handling in _lookup_deprecated_key - Simplified the cache retrieval logic by directly unpacking the 3-tuple cache entries, removing the need for backward compatibility checks for 2-tuple entries. - Updated unit tests to ensure that pre-warmed 3-tuple cache entries are served correctly without unnecessary database lookups. * chore(ci): add new unit test for deprecated key grace period - Included `test_deprecated_key_grace_period.py` in the CI workflow to enhance coverage for deprecated key handling scenarios. * fix(proxy): remove unnecessary check for revoke_at in _lookup_deprecated_key - Eliminated the redundant check for None on revoke_at, streamlining the logic for handling deprecated keys in the cache. This change enhances the efficiency of the key lookup process. * test(proxy): add end-to-end tests for deprecated key lookup behavior - Introduced a new test class `TestDeprecatedKeyLookupDbE2E` to validate the behavior of deprecated key lookups against a real Prisma-backed database. - The test ensures that old key hashes resolve correctly and that repeated lookups utilize the in-memory cache without errors. - Cleaned up the `_lookup_deprecated_key` function by removing an unnecessary check for `revoke_at`, enhancing the efficiency of the key lookup process. |
||
|
yuneng-jiang
|
a3a42c6c47 |
[Chore] CI: Assign test_request_size_limit_middleware To Proxy-Runtime Shard (#27341)
The assert-shard-coverage guard in test-unit-proxy-db.yml failed because test_request_size_limit_middleware.py was added under tests/proxy_unit_tests/ but not referenced by any matrix entry. Assigning it to the proxy-runtime shard, which already covers other server-runtime tests (proxy_routes, proxy_gunicorn, server_root_path). |
||
|
Yuneng Jiang
|
d747e4c248 |
Remove stale test_project_endpoints_prisma.py path from proxy-db workflow
The file was moved to tests/enterprise/litellm_enterprise/proxy/management_endpoints/ and is covered by the CircleCI litellm_mapped_enterprise_tests job. The stale path was causing pytest to error with 'file or directory not found'. |
||
|
Yuneng Jiang
|
66bf890226 |
[Infra] Stop attaching push-only postgres workflows to a GHA environment
The `_test-unit-services-base.yml` reusable workflow attached every job to the `integration-postgres` GHA environment to read three "secrets": DATABASE_URL, POSTGRES_USER, POSTGRES_PASSWORD. These are not secrets — the postgres service container is spawned per-job on localhost and destroyed with the job, so the user/password are bootstrap values for a throwaway container and the URL is always `postgresql://…@localhost:…`. Each environment attachment produces a "temporarily deployed to integration-postgres" deployment record, which the PR timeline renders as a message per matrix shard per push. With 14 proxy-db shards that's ~14 notifications per push, drowning the PR conversation. Changes: * Hardcode POSTGRES_USER/POSTGRES_PASSWORD/POSTGRES_DB and the derived DATABASE_URL in `_test-unit-services-base.yml`. * Delete the `environment: integration-postgres` attachment. * Delete the `secrets:` declarations on the reusable workflow and on the two callers (test-unit-proxy-db.yml, test-unit-security.yml). * The `services:` container still starts a fresh postgres per job; the connection string now matches what the container boots up with. Security review: no regression. The environment wasn't gating anything real — no protection rules configured, no approval gates, and the branch restriction is already enforced by `on: push: branches: [...]` on both caller workflows. Zizmor pedantic-mode findings are identical before and after (same 6 pre-existing findings, zero new ones). The `integration-postgres` environment and its three "secrets" in repo settings are now unreferenced and can be deleted from repo admin. |
||
|
Yuneng Jiang
|
21e08b0bb5 |
[Infra] Run schema-migration shard serially (workers: 0)
test_db_schema_migration.py has exactly one test, and that test is mostly
waiting on prisma subprocesses (~170s: prisma migrate deploy + prisma
migrate diff). No CPU-bound Python work inside the test body, and only
one test in the file means xdist's parallelism is unused regardless.
Previous run on commit
|
||
|
Yuneng Jiang
|
5df9f397e6 |
[Infra] Match xdist workers to runner cores; revert test_proxy_utils -k split
Two changes: 1. workers: 8 -> 4 on every non-serial proxy-db shard. ubuntu-latest is a 4-core runner; -n 8 oversubscribes 2x and workers block each other during their cold-start imports (pytest-cov instruments every litellm module per worker). Measured ~441% CPU locally with -n 8 on 8 cores (i.e. ~55% effective). Matching -n to physical cores should give ~2x faster worker startup, which is where most of the ~9m wall-clock per shard goes (7+ minutes is plugin load + xdist imports before any test runs). 2. Revert the -k split on test_proxy_utils.py. It was split into proxy-utils-a-h / proxy-utils-i-z as a semantic-adjacent hack; merge back to a single proxy-utils shard. Still uses --dist=worksteal so xdist can balance the 188 parametrized cases across workers. Also drops the now-unused `keyword` input from _test-unit-services-base.yml and its matching matrix field across all proxy-db entries. Shard count: 14 -> 13 (+ the assert-shard-coverage guard). |
||
|
Yuneng Jiang
|
584a7cd40f |
[Infra] Clean up proxy-db matrix job display names
Default GHA matrix job names join every matrix field, producing unreadable
check labels like:
'proxy-db (logging-misc, tests/proxy_unit_tests/test_proxy_reject_logging.py
tests/proxy_unit_tests/test_audit_logs_proxy.py ..., 8, loadscope, "", 15)'
Set the job's display name to '${{ matrix.test-group }}' so each check
shows just 'logging-misc', 'proxy-utils-a-h', etc.
|
||
|
Yuneng Jiang
|
e0201ece1e |
[Infra] Split slow proxy-db shards to hit 7m wall-clock target
Previous run (13.8m total) was bottlenecked by shards with 9-12m wall-clock. Setup + xdist spawn + coverage teardown is ~3m per shard, so each shard's pytest runtime must stay under ~4m to fit inside 7m total. Observed per-shard pytest times (before split): db-and-spend 9:08 (170s outlier: test_aaaasschema_migration_check) proxy-server 7:15 logging-and-callbacks 6:45 guardrails-budget-hooks 6:37 proxy-utils 6:23 auth-and-jwt 6:54 Split 6 shards into 12, keeping key-generation and endpoints-and-responses (already <7m). Adds a `keyword` input to _test-unit-services-base.yml so test_proxy_utils.py can be split by -k expression (same file, two runners). New matrix entries: auth-and-jwt -> auth-checks + jwt-and-keys proxy-server -> proxy-server-core + proxy-runtime logging-and-callbacks -> custom-logging + logging-misc db-and-spend -> schema-migration (isolated 170s test) + db-and-spend guardrails-budget-hooks-> guardrails-hooks + budgets proxy-utils -> proxy-utils-a-h + proxy-utils-i-z (-k split) The -k expression split is verified to cover every one of the 64 test functions in test_proxy_utils.py exactly once. The assert-shard-coverage guard still catches any file not in any shard. |
||
|
Yuneng Jiang
|
32c390a0f6 |
fix(tests): restore proxy_server.master_key in realtime fixture; add shard-coverage guard
Two fixes to proxy-db CI: 1. test_realtime_webrtc_endpoints.py's `proxy_app` fixture mutated the module-global `proxy_server.master_key` without restoring it, leaking state into any test that shared the same xdist worker. Under --dist=loadscope with 2 workers (GHA proxy-endpoints), this caused the google_endpoints tests to fail with "No api key passed in." because user_api_key_auth saw a set master_key and a missing API key on the test request. The fixture now saves and restores the original value. 2. Address the Greptile note that the semantic shard design has no catch-all, so a new test file added to tests/proxy_unit_tests/ without a matrix entry would silently skip CI. Adds an assert-shard-coverage job that enumerates test_*.py files and fails the workflow if any are not referenced by a matrix entry, with a clear message telling the author which semantic shard to place it in. All proxy-db shards now depend on this guard. |
||
|
Yuneng Jiang
|
c2f40e89d5 |
[Infra] Remove CCI/GHA test duplication and semantically shard proxy DB tests
Split into two related cleanups:
1. Delete CCI jobs that duplicate GHA coverage:
- mcp_testing (tests/mcp_tests) — already run by test-mcp.yml
- litellm_mapped_tests_proxy_part1/part2 (tests/test_litellm/proxy) —
already run across test-unit-proxy-auth.yml, test-unit-proxy-endpoints.yml,
and test-unit-proxy-infra.yml
Add rag_endpoints and realtime_endpoints to test-unit-proxy-endpoints.yml
(they were only covered by the deleted CCI part2 job).
Remove the corresponding workflow wiring, coverage combine entries, and
upload-coverage dependencies in .circleci/config.yml.
2. Re-shard test-unit-proxy-db.yml from 4 alphabetic buckets to 8 semantic
ones (auth-and-jwt, proxy-server, logging-and-callbacks, db-and-spend,
guardrails-budget-hooks, endpoints-and-responses, plus the existing
serial key-generation and test_proxy_utils.py shards). New test files are
placed in whichever group they belong to instead of reshuffling slices.
Add a dist input to _test-unit-services-base.yml so the test_proxy_utils.py
shard can use --dist=worksteal to spread its ~64 (many parametrized)
functions across workers; the default --dist=loadscope pins a single file
to a single worker, which was the root cause of that shard running 10m+.
|
||
|
Yuneng Jiang
|
7b43f5981f |
[Fix] CI: split test_proxy_utils.py into its own proxy-db matrix entry
The "remaining" proxy-db job was consistently timing out at ~98% because --dist=loadscope pins every test in test_proxy_utils.py (168+ parametrized tests) to a single xdist worker. 7 workers finished their files in ~15 minutes, then one worker ran alone for another 8+ minutes and hit the 30-minute job cap. Give test_proxy_utils.py its own matrix entry so its tests spread across all 8 workers, and add it to the "remaining" ignore list. |
||
|
Yuneng Jiang
|
aff4717494 |
[Infra] Expand CI branch filters for non-main PR targets
Required test-unit-* and related workflows only triggered on PRs targeting main, so feature PRs routed through litellm_internal_staging or litellm_oss_branch never dispatched the full suite. Branch protection reported BLOCKED even when CircleCI was green. Expand pull_request and push branch filters to also match litellm_internal_staging, litellm_oss_branch, and "litellm_**" (using ** so branch names containing "/" also match). |
||
|
Yuneng Jiang
|
83c459225c |
[Fix] CI: fix GHA timeouts and uv lock --check failures
1. exclude-newer: change from absolute "2026-04-10" to relative "3 days". All pinned deps were published before the 3-day cutoff. Re-locked so uv lock --check passes in test-mcp.yml and test-linting.yml. 2. test_eager_tiktoken_load: run all 10 env var values in a single subprocess instead of spawning 10 separate processes. Each cold import litellm takes ~78s on CI, so the old loop took ~13 min on a single xdist worker. Now takes ~78s total. 3. proxy-db remaining timeout: increase from 20 to 30 minutes. The remaining group has 51 test files and was consistently timing out at 71% across all branches (pre-existing issue, not migration-related). |
||
|
Yuneng Jiang
|
3a02c0ac6b |
[Infra] Migrate Redis caching tests from GHA to CircleCI
Redis caching unit tests (test_dual_cache, test_redis_batch_optimizations, test_router_utils) required Redis secrets that should live in CircleCI. - Add redis_caching_unit_tests job to CircleCI config - Delete test-unit-caching-redis.yml GHA workflow - Remove all Redis plumbing (inputs, secrets, env vars) from _test-unit-services-base.yml and its callers |
||
|
yuneng-jiang
|
d132b1bf51 |
[Infra] Remove Redundant Matrix Unit Test Workflow (#25251)
* Remove redundant matrix unit test workflow All test paths in test-litellm-matrix.yml are fully covered by the newer semantic unit test workflows (test-unit-*.yml), making the matrix workflow redundant CI spend. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add Codecov coverage reporting to semantic unit test workflows Add coverage collection (--cov) and Codecov OIDC upload to both reusable base workflows and all 12 caller workflows, replacing the coverage reporting that was previously only in the matrix workflow. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Move id-token/pull-requests permissions to job level for multi-job workflows For workflows with multiple jobs (llm-providers, proxy-db), move id-token: write and pull-requests: write from workflow level to job level so permissions are scoped to only the jobs that need them. Removes zizmor inline suppressions that were masking the issue. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Yuneng Jiang
|
3ae80407dd |
[Fix] Move Postgres username and password to environment secrets
Move POSTGRES_USER and POSTGRES_PASSWORD from hardcoded values to environment secrets so no credentials appear in workflow files at all. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Yuneng Jiang
|
d42e2f6429 |
[Fix] Move Postgres DATABASE_URL to environment secret to avoid credential leak warnings
The hardcoded postgresql://postgres:postgres@localhost connection string was being flagged by secret scanners. Move DATABASE_URL to a GHA environment secret (integration-postgres) so the password is never in the workflow file. Changes: - _test-unit-services-base.yml: DATABASE_URL now comes from secrets, environment is derived from enable-* flags (integration-postgres, integration-redis, or integration-redis-postgres) - test-unit-proxy-db.yml: switched to push-only trigger (uses secrets now) - test-unit-security.yml: switched to push-only trigger (uses secrets now) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Yuneng Jiang
|
6549f3eb1a |
[Infra] Add unit test workflows for Postgres, Redis, and security test suites
Add three new GHA workflows for tests requiring service containers, plus a
reusable base workflow that provides Postgres and cloud Redis support.
New workflows:
- test-unit-proxy-db.yml: proxy DB tests (key generation, auth checks,
remaining) using a local Postgres container with a 3-way descriptive matrix
- test-unit-caching-redis.yml: caching tests that need Redis but no provider
API keys, using cloud Redis via the integration-redis environment
- test-unit-security.yml: proxy security tests using a local Postgres container
Reusable base (_test-unit-services-base.yml):
- Local Postgres pinned by digest (postgres@sha256:705a5d5b...)
- Cloud Redis credentials scoped to the integration-redis GHA environment
- Environment binding is derived from enable-redis flag inside the base
(not caller-controllable) to prevent secret scope bypass
- Supports workers=0 for tests that cannot run in parallel
Security hardening:
- All actions pinned to commit SHAs
- persist-credentials: false on all checkouts
- permissions: contents: read only
- Postgres-only workflows (proxy-db, security) use zero secrets and trigger on
both pull_request and push to main/litellm_*
- Redis workflow triggers on push only (not pull_request) to prevent external
PRs from accessing Redis Cloud credentials
- Added ${TEST_PATH:?} guard to both _test-unit-base.yml and
_test-unit-services-base.yml to fail fast on empty test paths
- All files pass zizmor --pedantic with zero findings
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|