test_virtual_key_max_budget_alert_check_per_key_overrides_global asserted
override semantics but the implementation does additive merge. Renamed test
and updated assertion to match: per-key and global thresholds are unioned,
not replaced.
- Guard empty recipients in _handle_multi_threshold_max_budget_alert:
log warning and skip instead of falling through to old path error loop
- Widen max_budget_alert_emails type to Dict[str, Union[str, List[str]]]
to match _parse_email_list runtime behavior (accepts comma-separated strings)
- Pre-filter asyncio.create_task with min threshold check to avoid
unnecessary task allocation on every request when spend is below
all configured thresholds
- Add `default_key_max_budget_alert_emails` litellm_settings config as
global fallback for all virtual keys (per-key metadata takes priority)
- Fix crash when key has no user_id/user_email by passing recipient email
to _get_email_params (same pattern as team soft budget path)
- Use owner email for greeting, falling back to key_alias or token
- Rename setting from default_max_budget_alert_emails to
default_key_max_budget_alert_emails for clarity
Users can set metadata.max_budget_alert_emails as a JSON map of threshold
percentages to email recipients on virtual keys. When configured, the email
handler loops over each threshold, checks per-threshold dedup cache, and
sends to the configured recipients (auto-including the key owner's email).
When no map is set, the existing single 80% threshold behavior is preserved
unchanged. Teams support is out of scope for this v0.
Close three variant bypasses adjacent to VERIA-28 found during post-fix
variant audit:
1. _guardrail_modification_check had the same isinstance(dict) bypass
Veria-AI just flagged on the pre-call strip. A caller sending
`{"metadata": "{…}"}` as a JSON-encoded string (multipart/form-data
or extra_body) skipped the guard, got parsed to dict downstream, and
reached guardrail logic with bypass flags intact. Coerce strings via
safe_json_loads before evaluating.
2. The allow_client_tags strip only covered body metadata.tags and
litellm_metadata.tags — caller-supplied tags arriving via the
x-litellm-tags header or root-level data["tags"] bypassed it. Gate
add_request_tag_to_metadata's result on the same flag.
3. requester_metadata was deepcopied BEFORE the strip, so attacker
injections (user_api_key_metadata shadows, disallowed tags,
_pipeline_managed_guardrails) persisted in the snapshot. The PANW
guardrail (and any future consumer) trusting requester_metadata
would see forged values. Move the deepcopy to after the strip.
Regression tests added for each.
Per VERIA-28's secondary recommendation. The existing check only gated
metadata.guardrails. User-supplied values for disable_global_guardrails
(plural and the original singular typo variant) and opted_out_global_guardrails
are already silently ignored by _get_admin_metadata at read time, but the
silent-ignore makes diagnosis confusing and relies on one specific read
site catching them.
Reject at auth time with a 403 when any of:
- guardrails list (existing)
- disable_global_guardrails (new)
- disable_global_guardrail (new — historical singular-key variant)
- opted_out_global_guardrails (new)
are present in metadata, litellm_metadata, or at the request root, and the
caller's team lacks can_modify_guardrails. Defense in depth: the strip at
the pre-call layer still runs; this check fails loudly one layer earlier
so operators see an explicit 403 rather than a silent-ignore.
* fix: remove leading space from license public_key.pem
PEM must begin with -----BEGIN; a leading ASCII space breaks
cryptography.load_pem_public_key on older cryptography (e.g. 41.x),
causing OpenSSL no start line / deserialize errors.
Made-with: Cursor
* test: assert license public_key.pem loads as valid PEM
Regression guard for leading whitespace before -----BEGIN, which breaks
load_pem_public_key on older cryptography (e.g. 41.x).
Made-with: Cursor
Allow JWT tokens matching routing_overrides to use OAuth2 introspection without enabling global OAuth2 while keeping OAuth2 routing limited to LLM/info routes. Add regression coverage for management-route boundary and tighten opaque-token assertions; update docs to reflect selective-mode route scope.
Made-with: Cursor
* feat: multiple concurrent budget windows per API key and team (#24883)
* feat(proxy): add BudgetLimitEntry type and wire budget_limits into key/team models
* feat(schema): add budget_limits Json column to VerificationToken and TeamTable
* feat(migrations): add migration for budget_limits column on keys and teams
* feat(keys): initialize budget_limits windows with reset_at on key create/update
* feat(teams): initialize budget_limits windows with reset_at on team create/update
* feat(auth): add _virtual_key_multi_budget_check and _team_multi_budget_check
* feat(auth): call multi-budget checks from common_checks for keys and teams
* feat(proxy): increment per-window Redis spend counters after each request
* feat(budget): reset individual budget windows on schedule via reset_budget_job
* feat(ui): add hourly option to BudgetDurationDropdown
* feat(ui): add budget_limits field to KeyResponse type
* feat(ui): add Budget Windows editor to key edit view
* feat(ui): add Budget Windows editor to create key form
* fix(proxy): strip budget_limits=None before Prisma upsert to fix login 500
Prisma rejects nullable JSON fields (Json? without @default) when passed as
Python None — it needs the field omitted entirely so the DB stores NULL via
the column's nullable constraint. This was breaking /v2/login because the UI
session key creation path hit the upsert with budget_limits=None.
* ui(key-edit): use antd InputNumber+Button for budget windows, add reset hints
* ui(create-key): use antd InputNumber+Button for budget windows, add reset hints
* docs(users): add multiple budget windows section with API + dashboard walkthrough
* fix: BudgetExceededError returns HTTP 429 instead of 400
- Add status_code=429 to BudgetExceededError class
- auth_exception_handler hardcoded code=400 → code=429
* fix: no-op else branch in multi-budget auth checks causes KeyError
- BudgetLimitEntry objects must be coerced via model_dump() not left as-is
- Move _virtual_key_multi_budget_check into common_checks (was asymmetric
with _team_multi_budget_check which already lived there)
* fix: len() on JSON string returns char count not window count
Guard with isinstance check + json.loads() before iterating per-window
Redis counters in increment_spend_counters
* fix: silent except:pass hides Redis reset failures in reset_budget_windows
Log Redis counter reset failures as warnings so they are observable
* test: add unit tests for multi-budget window enforcement
5 tests covering: no budget_limits passes, under budget passes,
over hourly window raises 429, over monthly window raises 429,
BudgetLimitEntry objects coerced without KeyError
* fix: key per-window counters stable across reorders (duration key, not index)
* fix: team+key per-window spend increments use duration key, not index
* fix: budget window reset uses duration key; log failures instead of swallowing
* refactor: extract BudgetWindowsEditor to shared component
* refactor: key_edit_view imports BudgetWindowsEditor from shared component
* refactor: create_key_button imports BudgetWindowsEditor from shared component
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* fix(reset_budget_job): extract _reset_expired_window helper to fix PLR0915 too many statements
* feat(skills): Skills Registry & Hub — register skills, browse in AI Hub, public skill hub (#25118)
* feat(skills): add domain and namespace fields to plugin types
* feat(skills): store and return domain/namespace inside manifest_json
* feat(skills): add /public/skill_hub endpoint for unauthenticated access
* feat(skills): whitelist /public/skill_hub from auth requirements
* feat(skills): add domain, namespace to Plugin and RegisterPluginRequest types
* feat(skills): smart URL parser — paste github URL, auto-detect source type and name
* feat(skills): replace enable toggle with Public badge, make rows clickable
* feat(skills): add skill detail view with Overview and How to Use tabs
* feat(skills): add MakeSkillPublicForm modal for publishing skills to the hub
* feat(skills): rename panel to Skills, wire in skill detail view on row click
* feat(skills): add skill hub table columns — name, description, domain, source, status
* feat(skills): add SkillHubDashboard with stats row, domain dropdown filter, and table
* feat(skills): add Skill Hub tab to AI Hub with Select Skills to Make Public button
* feat(skills): move Skills to top-level nav item directly under MCP Servers
* feat(skills): add skillHubPublicCall and NEXT_PUBLIC_BASE_URL support
* feat(skills): add Skill Hub tab to public AI Hub page
* feat(skills): add skills page routing in main app router
* feat(skills): add /skills page route
* chore: update package-lock after npm install
* docs(skills): add Skills Gateway doc page with mermaid architecture diagram
* docs(skills): add Skills Gateway to sidebar under Agent & MCP Gateway
* docs(skills): add loom walkthrough video to Skills Gateway doc
* chore: fixes
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* feat: add brave/search to model_prices_and_context_window.json (#25042)
Brave Search is supported by litellm as a search provider (documented at
docs.litellm.ai/docs/search/brave and listed in provider_endpoints_support.json)
but was missing from model_prices_and_context_window.json, making it invisible
to any code that discovers search providers from litellm.model_cost.
Cost: $0.005/query ($5 per 1,000 requests) per https://brave.com/search/api/
* feat(models): add NVIDIA Nemotron 3 Super 120B on Bedrock (#24588)
* feat(models): add NVIDIA Nemotron 3 Super 120B on Bedrock
Add model definition for nvidia.nemotron-3-super-120b-a12b-v1 via
Bedrock Converse API with pricing, context window (256k/32k), and
capability flags (function calling, tool choice, system messages).
* fix model ID to nvidia.nemotron-super-3-120b + add tests
Correct the Bedrock model ID from nvidia.nemotron-3-super-120b-a12b-v1
(NVIDIA's internal name) to nvidia.nemotron-super-3-120b (the actual
AWS Bedrock programmatic model ID). Add unit tests verifying model
resolution, pricing, and context window.
* fix(proxy): allow JWT auth for /v1/mcp/server sub-paths (#24698)
mcp_routes only contained "/v1/mcp/server" (exact match). Starlette's
compile_path produces an end-anchored regex, so sub-paths like
/register, /health, /submissions, /oauth/* all failed the JWT
allowed_routes_check. Add a {path:path} wildcard entry so all
sub-paths are covered.
---------
Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
* fix(proxy): enforce key-level model allowlist for custom auth
custom_auth_run_common_checks only runs common_checks (team/user/project model checks).
Custom auth now also enforces key-level model restrictions via can_key_call_model.
Move the custom-auth key-access regression tests to test_user_api_key_auth.py and keep test_custom_auth_end_user_budget.py focused on end-user budget behavior.
Made-with: Cursor
* fix(proxy): gate custom-auth key model checks behind opt-in
Keep key-level model allowlist enforcement in custom auth behind `custom_auth_run_common_checks` to preserve backwards compatibility, and update tests to verify default non-enforcement and opt-in enforcement behavior.
Made-with: Cursor
* test(proxy): isolate custom auth default check from shared settings state
Patch `proxy_server.general_settings` to an empty dict in the default custom-auth key-access test so it remains deterministic under shared module state.
Made-with: Cursor
* test(proxy): strengthen custom auth post-check assertions
Tighten custom auth regression tests by asserting exact can_key_call_model args and remove an unused common_checks mock from the default behavior path.
Made-with: Cursor
* fix(agentcore): parse A2A JSON-RPC responses in AgentCore provider
* fix(prompt-templates): ensure_alternating_roles handles tool-call chains
* feat(auth): add JWT claim routing overrides for OAuth2 validation
Made-with: Cursor
* docs(auth): document JWT-to-OAuth2 routing overrides
Add generic docs for running JWT and OAuth2 together, including routing_overrides YAML examples and list-based selector behavior for iss/client_id/aud.
Made-with: Cursor
---------
Co-authored-by: Milan <milan@berri.ai>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
PR #24755 renamed `azure_api_key_header` to `AZURE_AI_API_KEY_header` in
the test file but did not update the actual function signatures of
`get_api_key()` and `_user_api_key_auth_builder()`, causing TypeError
on all affected test cases.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL to both async_set_cache
calls in sync_user_role_and_teams for consistency with all other user cache
writes. Add 3 tests covering cache invalidation on role change, team change,
and no-op when nothing changes.
Add None-token test cases to both proxy_unit_tests and test_litellm
to cover the guard added in the previous commit. Also add -> bool
return type annotation to is_jwt().
Budget checks on API keys, teams, and team members were not enforced in
multi-pod deployments because user_api_key_cache is intentionally
in-memory-only. Each pod tracked spend independently, so with N pods
the effective budget was N × max_budget.
Introduces a separate spend_counter_cache (DualCache wired to
redis_usage_cache) with atomic increment/read helpers:
- increment_spend_counters(): awaited in cost callback (not create_task)
to update both in-memory and Redis before the next auth check
- get_current_spend(): reads Redis first (cross-pod authoritative),
falls back to in-memory, then to cached object .spend from DB
Budget check functions (_virtual_key_max_budget_check,
_team_max_budget_check, _check_team_member_budget) now read spend via
get_current_spend() instead of cached object .spend fields.
When Redis is not configured, falls back to in-memory-only counters
(same as current single-instance behavior).
Fixes#23714
* feat(redis): add circuit breaker to RedisCache to fast-fail when Redis is down (#24181)
* feat(redis): add circuit breaker env var constants
* feat(redis): add RedisCircuitBreaker and apply guard decorator to all async ops
* fix(dual_cache): fall back to L1 instead of re-raising on Redis increment failures
* test(caching): add circuit breaker unit tests
* fix(redis): fast-fail concurrent HALF_OPEN probes — only one probe at a time
* fix(dual_cache): return None fallback when in_memory_cache is absent and Redis fails
* test(caching): add regression tests for HALF_OPEN concurrency and None fallback
* Fix blocking sync next in __anext__ (#24177)
* Fix blocking sync next
* Update tests/test_litellm/litellm_core_utils/test_streaming_handler.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix PEP 479 regression in __anext__ sync iterator exhaustion
asyncio.to_thread re-raises thread exceptions inside a coroutine, where
PEP 479 converts StopIteration to RuntimeError before any except clause
can catch it. Add _next_sync_or_exhausted() module-level helper that
catches StopIteration in the thread and returns a sentinel instead, then
raise StopAsyncIteration in the coroutine.
Also rewrites the non-blocking test to use asyncio.gather() instead of
asyncio.create_task() (which returned None on Python 3.9 / pytest-asyncio
in CI), and adds an exhaustion regression test that drains the wrapper
fully and asserts no RuntimeError leaks out.
---------
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat: add git-subdir source type to claude-code/plugins API (#24223)
Support a third plugin source type `git-subdir` alongside the existing
`github` and `url` types, as documented in the official Claude Code
plugin marketplaces spec.
New format: {"source": "git-subdir", "url": "...", "path": "subdir/path"}
- Validates url and path fields are present and non-empty
- Rejects absolute paths, '..' segments, backslashes, and percent-encoded
traversal sequences (including double-encoded variants via regex check)
- Extracts path validation into _validate_git_subdir_path() helper
- Updates Pydantic field description to document all three source types
- Adds isValidUrl() check for url/git-subdir source types in the UI form
- Adds "Git Subdir" option to the UI form with a required Path field
- Adds unit tests covering success, update, missing/empty fields,
path traversal variants, and unknown source type
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* [FEAT] add extract_header and extract_footer to Mistral OCR supported params (#24213)
* docs: add git-subdir source type to claude-code plugin marketplace docs (#24289)
* fix(ui): swap J/K keyboard navigation in log details drawer (#24279) (#24286)
J should navigate down (next) and K should navigate up (previous),
matching vim/standard conventions.
* fix: use async_set_cache in user_api_key_auth hot path (#24302)
* fix: use async_set_cache in auth hot path to avoid blocking event loop
* test: assert no blocking set_cache call in _user_api_key_auth_builder
* test: broaden blocking call check to all sync DualCache methods
* test: fix regression test to actually catch blocking cache calls
* fix: ruff lint unused variable + UI build MessageManager error
- litellm/caching/redis_cache.py: remove unused variable 'e' in circuit
breaker exception handler (F841)
- add_plugin_form.tsx: use MessageManager.error() instead of undefined
message.error() for git URL validation
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs: add REDIS_CIRCUIT_BREAKER env vars to config_settings reference
Add REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD and
REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT to the environment variables
reference table so test_env_keys.py passes.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Vincenzo Barrea <manamana88@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Robert Kirscht <rkirscht242@gmail.com>
Co-authored-by: Imgyu Kim <kimimgo@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
- Change `if team_limit:` to `if team_limit is not None:` in both
get_key_model_rpm_limit and get_key_model_tpm_limit so that an
explicitly-empty team rate-limit map ({}) is returned as-is instead
of silently falling through to deployment defaults (P1 fix).
- Replace the bare `int()` list comprehension in _get_deployment_default_limit
with a loop that catches ValueError/TypeError so malformed config strings
do not raise an unhandled exception during request handling (P2 fix).
- Add corresponding unit tests for both edge cases.
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
Replace bare _get_deployment_default_tpm/rpm_limit calls in the
async_log_success_event condition with get_key_model_tpm/rpm_limit
(model_name=model_group). The higher-level getters short-circuit on
key/team metadata hits before ever reaching the router, so requests
that don't use deployment defaults incur no extra router lookup. Remove
the now-unused bare helper imports.
Also fix invalid `int = None` type hints in test helper signatures
to `Optional[int] = None`.
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
- Use min() across all matching deployments instead of first-wins when
resolving default_api_key_tpm/rpm_limit for a model group, so
load-balanced setups with different per-deployment limits always apply
the most conservative value
- Replace the global SensitiveDataMasker non_sensitive_overrides change
with a targeted excluded_keys set at the remove_sensitive_info_from_deployment
call site, avoiding unintended suppression of other fields
- Update the v1 parallel request limiter to pass model_name to
get_key_model_tpm/rpm_limit so deployment defaults apply there too
- Add 4 tests covering multi-deployment min semantics
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
Adds `default_api_key_tpm_limit` and `default_api_key_rpm_limit` to
`GenericLiteLLMParams` so operators can set per-deployment rate limit
defaults in config.yaml. When a key has no model-specific tpm/rpm limit
configured, the proxy falls back to these deployment defaults (Case 2 in
spec). Key-level limits always take priority (Case 1).
- Extends `get_key_model_tpm_limit` / `get_key_model_rpm_limit` with a
`model_name` param and a priority-4 deployment-default fallback
- Passes `model_name=requested_model` in the parallel request limiter so
the fallback is triggered at enforcement time
- Adds `"limit"` to `SensitiveDataMasker` non-sensitive overrides so
`*_limit` fields are not masked in `/model/info` responses
- Adds 17 unit tests covering both spec cases and the `/model/info` path
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
- Only remove wildcard path from openai_routes when the route entry has
type="subpath", avoiding accidental removal when two endpoints share
the same base path but differ in include_subpath
- Clean up _registered_pass_through_routes in the test finally block to
prevent stale entries from polluting subsequent tests on failure
- Add dedup guard for base path registration (prevents unbounded list
growth on config reload)
- Clean up base path and wildcard path from openai_routes when an
endpoint is removed via remove_endpoint_routes
- Rewrite test to exercise initialize_pass_through_endpoints directly,
covering registration, dedup on reload, and cleanup on removal
When a pass-through endpoint has both auth=true and include_subpath=true,
non-admin users got 401 errors on subpath requests because only the base
path was registered in openai_routes. Now the wildcard path is also
registered so the auth check recognizes subpath requests as LLM API routes.
Also fixes pre-existing pyright error where logging_obj was possibly
unbound in the except block.
Keep both sets of tests: upstream's OAuth2 token injection test and
our case-insensitive tool matching tests. Use upstream's version of
the bedrock output_config test (more comprehensive).
Ryan Crabbe