mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-12 21:04:10 +00:00
f5b11b72a6dcc8f4e7a16f00a61bdd124cc171d2
3625 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Sameer Kankute
|
f5b11b72a6 |
feat(proxy): publish /v2/model/info in Swagger OpenAPI spec (#29900)
* feat(proxy): publish /v2/model/info in Swagger OpenAPI spec Expose the v2 model info endpoint in /docs by removing include_in_schema=False and documenting query parameters used by the admin UI and proxy CLI consumers. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(ui): regenerate schema.d.ts for /v2/model/info OpenAPI docs Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
ryan-crabbe-berri
|
aaf1e2444b |
feat(ui): include internal routes in the dashboard's generated OpenAPI types (#29885)
The dashboard calls UI-internal proxy routes that the public /openapi.json hides with include_in_schema=False, so they never reached schema.d.ts and could not be typed. The type generator now force-includes those routes when it dumps the spec for openapi-typescript; this mutates a throwaway interpreter only, so the spec the proxy actually serves is unchanged. Regenerates schema.d.ts so 86 internal route families (for example /v2/model/info, /global/spend/*, /config/*, /v2/login, /sso/*) are now typed, with no public route removed. This unblocks migrating the dashboard's data fetching onto the typed $api client. Branch CI note: schema.d.ts is generated; CI regenerates and diffs it via the same gen:api script. |
||
|
Yassin Kortam
|
5e2db7eee4 | feat(litellm): add models and repository layers (#29686) | ||
|
yuneng-jiang
|
3448bf79f8 |
fix(ui): default guardrails page to first tab for admins, not submitted (#29872)
The Guardrails page hardcoded defaultActiveKey="submitted", so admins landed on the "Submitted Guardrails" tab (the last of their four tabs) instead of the primary view. The original intent was for non-admins, whose only tab is Submitted Guardrails, to default there; admins should open on their first tab. Make the default role-aware: admins default to the first tab (Guardrail Garden), non-admins keep Submitted Guardrails. |
||
|
Mateo Wang
|
13924fa1d6 |
feat: standardize rate limit errors with category, rate_limit_type, model, and llm_provider fields (#27687)
* feat(exceptions): add RateLimitErrorCategory + headers/detail fields on RateLimitError
LiteLLM previously surfaced rate-limit conditions through several unrelated
error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException).
This commit adds the data model needed to consolidate them under a single
class:
* RateLimitErrorCategory enum exposing four categorical values
(vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit,
litellm_batch_rate_limit) so callers can switch on the rate-limit source.
* New optional fields on RateLimitError:
- category (defaults to vendor_rate_limit, preserving today's behavior for
every existing call site in exception_mapping_utils);
- headers (preserves retry-after / rate_limit_type / reset_at across the
proxy boundary instead of dropping them on the floor);
- detail (mirrors FastAPI HTTPException.detail so the same instance can be
serialized through both paths).
litellm.RateLimitErrorCategory is re-exported at the package root to match
the existing exception-export pattern.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(proxy): add ProxyRateLimitError unifying RateLimitError + HTTPException
Adds a single proxy-side error class that subclasses BOTH
litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative
multiple inheritance.
Why both bases:
* Subclassing RateLimitError lets user code catch every rate-limit source
with one 'except RateLimitError' and switch on the new .category field.
* Subclassing HTTPException keeps every existing FastAPI plumbing path (the
isinstance(e, HTTPException) branches in proxy_server.py route handlers,
FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException))
working without modification, and preserves retry-after / rate_limit_type /
reset_at headers on the wire.
The class declaration order is (HTTPException, RateLimitError) so the MRO
puts HTTPException's no-super-call __init__ ahead of openai's cooperative
__init__ chain — preventing openai.APIError.super().__init__(message) from
landing in HTTPException.__init__(status_code=message).
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(proxy/hooks): raise ProxyRateLimitError from budget + iteration limiters
Replaces three bare HTTPException(status_code=429, ...) call sites with
ProxyRateLimitError, which is both a RateLimitError (catchable by category)
and an HTTPException (preserves existing FastAPI serialization). Drops the
now-unused HTTPException import in the iteration / per-session limiters.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(proxy/hooks): raise ProxyRateLimitError from parallel-request limiters
Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3
parallel-request limiters (key/team/user/model/customer rate limits) with
ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type
annotation accordingly.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(proxy/hooks): raise ProxyRateLimitError from dynamic rate limiters
Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3
dynamic rate limiters (project-level TPM/RPM allocation, model-saturation
checks, priority-based limits, fail-closed guards) with ProxyRateLimitError.
The v3 limiter still imports HTTPException for an unrelated bare 'except
HTTPException:' branch.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(proxy/hooks): raise ProxyRateLimitError from batch rate limiter
Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error
with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT
so users can distinguish batch-level throttling (which counts requests/tokens
across an uploaded batch input file before submission) from the generic
key/team/user RPM/TPM limiter.
The HTTPException import is retained because the same module raises
HTTPException for unrelated 403/IO error paths.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): pin down unified rate-limit error contract
Adds a dedicated test module covering the new RateLimitErrorCategory enum,
RateLimitError.category default + override behavior, ProxyRateLimitError's
dual nature (RateLimitError + HTTPException), and a parametrized regression
guard that asserts every proxy hook module imports the unified class.
The regression guard catches the failure mode the refactor is designed to
prevent: someone re-introducing a bare HTTPException(status_code=429, ...)
in one of the hook modules instead of going through ProxyRateLimitError.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(logging): expose rate-limit category via StandardLoggingPayload
Adds an optional 'error_rate_limit_category' field to
StandardLoggingPayloadErrorInformation, populated from the unified
RateLimitError.category attribute (introduced in the previous commits on
this branch).
Why: the .category attribute is reachable off the raw exception today via
getattr(e, 'category', None), but the structured contract that downstream
custom callbacks / loggers / spend log writers consume is the
StandardLoggingPayload. Without this field, a user building custom
rate-limit metrics on top of callback data has to special-case the raw
exception object — which defeats the purpose of the StandardLoggingPayload
abstraction.
The field is None for non-rate-limit exceptions (so consumers can read it
unconditionally without isinstance checks) and is one of the
RateLimitErrorCategory string values otherwise.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): assert StandardLoggingPayload carries the category
Five tests covering: vendor default, explicit litellm_rate_limit and
litellm_batch_rate_limit values, None for non-rate-limit exceptions, and
None when no exception is provided. Pins down the contract that custom
callbacks can read 'error_information.error_rate_limit_category' off the
StandardLoggingPayload to drive custom rate-limit metrics without ever
reaching for the raw exception.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(types): silence mypy [misc] on intentional dual-base attr overlap
mypy emits two [misc] errors on the ProxyRateLimitError class line because
its two bases declare overlapping attributes with related-but-not-identical
annotations:
* status_code: int on starlette HTTPException vs. Literal[429] on openai's
RateLimitError (every openai status-error subclass narrows it the same
way and silences pyright with the same convention).
* headers: Mapping[str, str] | None on HTTPException vs. our Optional[
Dict[str, str]] (the proxy hooks always carry a stringified dict).
Both narrowings are intentional and enforced at construction time. Add a
type: ignore[misc] with an inline explanation rather than relax the
annotations on the parent or change the wire-format guarantees.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): add direct hook-invocation tests to lift patch coverage
Adds six end-to-end tests that drive each refactored hook past its
limit and assert the unified ProxyRateLimitError is raised with the
correct category and dual-base shape. Complements the
import-shape-only parametrized guard above by actually executing the
new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage
sees them as hit.
Hooks covered (one test each):
* parallel_request_limiter v1 — direct call to raise_rate_limit_error()
* parallel_request_limiter v3 — direct call to _handle_rate_limit_error
with a fabricated OVER_LIMIT response
* max_iterations_limiter — full async_pre_call_hook with mocked agent
registry, second call exceeds budget=1
* max_budget_limiter — async_pre_call_hook with mocked get_current_spend
* dynamic_rate_limiter v1 — async_pre_call_hook with mocked
check_available_usage forcing available_tpm == 0
* batch_rate_limiter — direct _raise_rate_limit_error call, asserts
category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the
generic LITELLM_RATE_LIMIT)
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: guard rate_limit_category extraction with isinstance check
* test(rate-limit): cover remaining hook raise sites for codecov
Adds five more direct hook-invocation tests so every PR-touched line
in the proxy hooks is exercised by tests in tests/test_litellm/, which
codecov measures:
* parallel_request_limiter v1 — check_key_in_limits inline raise
(the second raise site, separate from the raise_rate_limit_error
helper covered earlier)
* dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already
covered)
* dynamic_rate_limiter v3 — parametrized over all three raise sites:
model_saturation_check, priority_model, and the fail-closed
fallback for an unrecognized descriptor_key
* max_budget_per_session_limiter — full async_pre_call_hook with a
mocked agent registry and over-budget cached spend
All 42 tests in test_rate_limit_error_unification.py now pass and
together exercise every changed import + raise line across the eight
refactored proxy hooks.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: use computed error_message in ProxyRateLimitError detail
* fix(parallel-request-limiter): drop None from detail; annotate raise_rate_limit_error as NoReturn
The v1 ' raise_rate_limit_error' helper built an unused 'error_message'
variable and then assembled the actual ' detail' via an f-string that
interpolated 'additional_details' verbatim — producing
'Max parallel request limit reached None' when invoked without
arguments (flagged by code review).
Fix the helper to:
- use the constructed 'error_message' as the detail
- annotate the helper as NoReturn since it always raises
- drop the redundant 'raise'/'return' at the two call sites
Add two regression tests covering both the with- and without-
additional_details paths.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): drop literal 'None' from raise_rate_limit_error detail
The v1 parallel_request_limiter's raise_rate_limit_error helper has a
long-standing bug: it computes a None-guarded 'error_message' string but
then ignores it and emits an f-string that interpolates the raw
'additional_details' arg. Callers that pass no argument get
'Max parallel request limit reached None' as the user-facing detail.
This commit:
* wires error_message into the detail kwarg so the None-guard actually
applies and operators see a clean message;
* changes the return-type annotation from ProxyRateLimitError to NoReturn
(the function always raises) so type-checkers know callers after this
invocation are unreachable.
Greptile P1 + P2 review feedback on PR #27687.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(types): demote TypedDict floating string to a # comment
A string literal placed after a field declaration in a TypedDict body is
not a per-field docstring — it's an orphaned string expression Python
discards. Tools like mypy / pyright that inspect TypedDict fields won't
surface that text either.
Move the documentation for error_rate_limit_category to a real comment
so the intent is visible to readers and type-checker tooling without
the misleading docstring framing.
Greptile P2 review feedback on PR #27687.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* security(exceptions): do not auto-copy vendor response headers to e.headers
A vendor 429 response can set arbitrary headers (Set-Cookie, CORS
overrides, …). Previously, when RateLimitError was constructed with only
a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to
a copy of response.headers. If a downstream proxy serializer ever
forwarded e.headers to the client, a malicious upstream could inject
browser-interpreted headers for the proxy origin.
Drop the fallback. Only headers passed explicitly via the headers= kwarg
make it onto self.headers (proxy hooks pass retry-after etc. — they
control what's surfaced). Vendor response headers stay reachable on
e.response.headers for callers that explicitly want them.
Today's proxy_server.py route handlers don't actually forward e.headers
on the wire (they construct ProxyException without passing headers), so
no current behavior changes — this is a defensive narrowing so the
fallback can never be turned into a vector when someone wires
e.headers through later.
Veria-AI security review feedback on PR #27687.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): regression guards for review-pass fixes
Pins down the three review-pass fixes:
* test_parallel_request_limiter_v1_helper_no_additional_details — calls
raise_rate_limit_error() with no args and asserts the detail does NOT
contain the literal string 'None'. Pre-fix, callers got 'Max parallel
request limit reached None'.
* test_rate_limit_error_does_not_auto_copy_response_headers — passes a
vendor httpx.Response with a Set-Cookie header to RateLimitError
WITHOUT an explicit headers= kwarg, asserts self.headers stays None
(no leak), then re-checks that an explicit headers= kwarg DOES
populate self.headers. Vendor headers remain reachable on
e.response.headers for callers that explicitly want them.
* The existing v1-helper test now also asserts the additional_details
string makes it through to the detail.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(rate-limit): add orthogonal RateLimitType (requests/tokens/concurrent_requests/budget/max_iterations)
trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by
the dimension that was exceeded, not just by who rate-limited (vendor vs.
litellm). Adds:
- RateLimitType str-enum exposed at `litellm.RateLimitType` with values
requests / tokens / concurrent_requests / budget / max_iterations.
- `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError;
None default so existing callers (vendor-429 path in exception_mapping_utils)
remain a no-op.
- StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom
callbacks can split rate-limit failures by cause without parsing free-text
error messages. Mirror to error_rate_limit_category extraction in
get_error_information(); single isinstance(RateLimitError) check covers both.
- map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels
("requests", "tokens", "max_parallel_requests") onto the public enum so
the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive
None on unknown values rather than silently picking a wrong dimension.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(proxy/hooks): wire rate_limit_type onto every limiter raise site
Each refactored proxy hook now populates rate_limit_type with the dimension
that actually tripped the limit, so downstream consumers (custom callbacks,
prometheus exporters via the StandardLoggingPayload) can split key/team/user
rate-limit failures by cause:
- parallel_request_limiter (v1): detect dimension from current vs. limit in
the post-cache branch (concurrent_requests > tokens > requests, matches the
boolean condition order). Base case (current is None, one limit set to 0)
picks the most-specific zero. raise_rate_limit_error() helper accepts an
explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches
every existing internal call site, including the global-limit branch).
- parallel_request_limiter (v3): forward status["rate_limit_type"] through
map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS
for the public field while the raw v3 jargon stays on the HTTP header for
wire-format backward compat.
- dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass
data["model"] through so callbacks see the model that hit the limit
(addresses the secondary "provider missing" complaint in the original
Slack thread, partially — the model is what dashboards typically split on).
- dynamic_rate_limiter (v3): forward status["rate_limit_type"] via
map_v3_rate_limit_type() at every raise site (model_saturation_check,
priority_model, fail-closed unknown-descriptor guard). Also pass model.
- batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map
directly without going through the helper's None branch.
- max_budget_limiter, max_budget_per_session_limiter: BUDGET.
- max_iterations_limiter: MAX_ITERATIONS.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): cover RateLimitType enum, hook wiring, and StandardLoggingPayload propagation
27 new tests across five new test classes:
- TestRateLimitType: enum exposed at litellm.RateLimitType, all five values
defined, RateLimitError default is None (vendor 429 path makes no claim
about which dimension), accepts both string and enum forms with
str-coercion guarantee for downstream JSON serializers.
- TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts
string or enum, doesn't break existing callers that pass nothing.
- TestMapV3RateLimitType: pins each v3-internal → public-enum mapping
(tokens, requests, max_parallel_requests → concurrent_requests, unknown
→ None) so a future v3 refactor can't silently swap dimensions.
- TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type
field reaches the structured payload for both ProxyRateLimitError and
plain RateLimitError, is None when unspecified, and is None for
non-rate-limit exceptions (symmetric with error_rate_limit_category).
- TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the
v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error
(both "tokens" and "max_parallel_requests" paths), and the batch
limiter (both tokens and requests paths) — coverage tools see the new
rate_limit_type= kwargs as exercised, not just the import shape.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): cover _coerce_message branches and v1 dimension detection
Drives the patch coverage on the new orthogonal RateLimitType wiring up
to (or close to) 100% on the touched files.
ProxyRateLimitError._coerce_message — was 22% covered, now 100%:
* nested {error: {message}} dict
* nested {message: {message}} dict (alt key)
* dict without 'error'/'message' keys → JSON dump fallback
* non-JSON-serializable dict value → str() fallback
* non-string non-mapping detail (int) → str() coercion
v1 parallel_request_limiter dimension detection — was 0% covered, now
exercised across 6 parametrized cases:
* check_key_in_limits else-branch: current at concurrent / TPM / RPM cap
→ asserts rate_limit_type is concurrent_requests / tokens / requests.
* check_key_in_limits base case (current is None): max_parallel_requests
/ tpm_limit / rpm_limit set to 0 → asserts the most-specific zero
attribution wins per the helper's order.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver
Introduces a small helper layer used by every proxy-side rate-limit
hook so that the 429 they raise carries a populated llm_provider /
model — instead of an empty exception.llm_provider that downstream
loggers (Prometheus failure metric, observability callbacks) read as
'no provider attribution'.
ProxyHTTPRateLimitError inherits from both fastapi.HTTPException
(so the proxy server still renders it as a 429) and
litellm.exceptions.RateLimitError (so isinstance checks and
PrometheusLogger._get_exception_class_name pick up llm_provider).
We deliberately don't call RateLimitError.__init__ — it constructs
an httpx.Response we don't need and would just add failure surface;
attribute parity is what downstream consumers care about.
resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider
defensively. Internal limiter hooks fire from async_pre_call_hook —
well before get_llm_provider runs anywhere else in the request
lifecycle — so we have to call it ourselves at raise time. If the
model is missing or unparseable (alias, router-only model) we fall
back to llm_provider='litellm_proxy' rather than letting a second
exception leak out and break the request path.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on parallel-request 429s
Both v1 and v3 parallel-request limiters fired bare HTTPException(429)
from inside async_pre_call_hook. The downstream Prometheus failure
metric reads exception.llm_provider via _get_exception_class_name —
the empty value showed up as exception_class='HTTPException' and
left model_id='None' on the time series.
Threads requested_model through every raise site in:
* parallel_request_limiter.py:
- check_key_in_limits (the per-key/per-model/per-user/per-team/
per-customer over-limit path)
- raise_rate_limit_error (zero-limit + global_max_parallel_requests
paths) — now takes an optional requested_model kwarg
* parallel_request_limiter_v3.py:
- _handle_rate_limit_error (the OVER_LIMIT translator), called
from both the should_rate_limit pre-check and the TPM
reservation path
Resolved via resolve_llm_provider_for_rate_limit so unknown / missing
models silently fall back to llm_provider='litellm_proxy' instead of
breaking the request path with a second exception.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s
Same plumbing change as the parallel limiters, applied to both
dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3:
* v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve
data['model'] -> (model, llm_provider) once and pass it into both
raises.
* v3: All three raise sites in _check_rate_limits — the
model_saturation_check enforced raise, the priority_model
enforced raise, and the fail-closed unknown-descriptor branch —
now attribute the 429 to the actual provider.
Falls back to llm_provider='litellm_proxy' when the model can't be
resolved.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s
batch_rate_limiter._raise_rate_limit_error now takes a
requested_model kwarg threaded from data['model'] in
_check_and_increment_batch_counters. The batch-creation 429 is what
gets raised when the input file's tokens/requests count would push
the per-key TPM/RPM window over its limit.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on budget/iterations 429s
Final batch of internal raise sites — the user/session-budget and
max-iterations hooks. Same pattern: resolve data['model'] once at
raise time, attach to ProxyHTTPRateLimitError so Prometheus and
observability callbacks can attribute the 429.
Hooks updated:
* max_budget_limiter (per-user max_budget exceeded)
* max_iterations_limiter (per-session agent iteration cap)
* max_budget_per_session_limiter (per-session dollar cap)
All three fall back to llm_provider='litellm_proxy' when data['model']
is missing or unparseable. Drops the now-unused HTTPException import
from each module.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(proxy/hooks): pin provider field on internal rate-limit 429s
Regression coverage for the 'provider field missing' bug across every
proxy-side rate-limit hook + the helper layer:
* ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError,
dict-detail stringification, None-provider normalization).
* resolve_llm_provider_for_rate_limit happy paths
(gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback
branches (None, '', unknown name) plus a 'get_llm_provider raises'
case that asserts we swallow the secondary exception.
* For each limiter (parallel v1/v3, dynamic v1/v3, batch,
max_budget, max_iterations, max_budget_per_session): assert the
raised exception is a RateLimitError carrying the resolved
model + llm_provider, and a sibling test that asserts the
fallback path returns 'litellm_proxy' without leaking a second
exception.
* Two PrometheusLogger._get_exception_class_name pins so the
Prometheus failure metric label flips from 'HTTPException' to
'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on
fallback) — that's what dashboards consume.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* perf(proxy/hooks): defer provider resolution to over-limit branches
* fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail
* Consolidate rate_limiter_utils imports in dynamic_rate_limiter
* fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError
ProxyHTTPRateLimitError inherits from RateLimitError but did not call
RateLimitError.__init__, so num_retries/max_retries were never set.
When Starlette's HTTPException lacks __str__, MRO falls through to
RateLimitError.__str__, which unconditionally reads these attributes
and raises AttributeError during logging/traceback formatting.
Initialize them to None defensively.
* fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError
HTTPException declares 'status_code: int' while openai.RateLimitError
(via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy
flags the multi-base override as [misc] in CI lint. The runtime semantics
are fine (we set self.status_code in __init__), so silence the
class-level annotation conflict with a targeted ignore.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: annotate batch limiter _raise_rate_limit_error as NoReturn
* feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to #27687) (#27706)
* feat(prometheus): add rate_limit_category and rate_limit_type labels
Adds two new labels to litellm_proxy_failed_requests_metric so dashboards
can split 429s by rate-limit source (vendor vs. litellm-internal) and by
the dimension that was exceeded (requests/tokens/concurrent_requests/
budget/max_iterations) without parsing free-text error messages.
Closes the Prometheus side of LIT-2718. The unified RateLimitError.category
and .rate_limit_type fields landed in PR #27687 but were only surfaced on
StandardLoggingPayload (custom-callback channel); this exposes them on
the metric label set as well.
Both labels are populated only when the underlying exception is a
litellm.RateLimitError; non-rate-limit failures keep them empty.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(prometheus): populate rate-limit labels + preserve exception_class back-compat
Two coupled changes in the Prometheus integration:
1. async_post_call_failure_hook now extracts the new RateLimitError
.category / .rate_limit_type fields (added in PR #27687) via a
_extract_rate_limit_labels helper and forwards them through
UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric.
Empty for non-rate-limit failures.
2. _get_exception_class_name special-cases ProxyRateLimitError and
keeps emitting 'HTTPException' for the exception_class label.
Without this shim, ProxyRateLimitError (which multi-inherits from
HTTPException + RateLimitError) would silently flip the label
from 'HTTPException' (the historical value for proxy-side 429s)
to 'ProxyRateLimitError', breaking existing dashboards / alerts
that key off exception_class='HTTPException'. Distinguishing
vendor vs. litellm 429s is now the job of the new
rate_limit_category label.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(prometheus): cover rate-limit labels and exception_class back-compat
Adds 19 tests across:
- enum / label-list registration
- _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError,
non-rate-limit and None inputs (incl. parametrized over every
RateLimitErrorCategory x RateLimitType combo)
- _get_exception_class_name back-compat: ProxyRateLimitError keeps the
legacy 'HTTPException' string while vendor RateLimitError keeps the
historical 'Provider.ClassName' format
- end-to-end through async_post_call_failure_hook with both
ProxyRateLimitError and vendor RateLimitError, asserting both new
labels populate and exception_class stays back-compat
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import
Address greptile feedback:
- async_post_call_failure_hook docstring: drop the stale labelnames listing
and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric
as the source of truth so the doc cannot drift from the actual labelset.
- _get_exception_class_name: guard the lazy ProxyRateLimitError import with
ImportError so router-side fallback callsites don't blow up in non-proxy
installs that don't have fastapi (a transitive dep of
proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when
fastapi is available.
Also fix the existing enterprise callback test that asserted the old
labelset on litellm_proxy_failed_requests_metric — it now expects the new
rate_limit_category / rate_limit_type labels populated for vendor 429s.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(bugbot): simplify rate-limit label coercion + guard None detail
- prometheus.py _extract_rate_limit_labels: RateLimitError.__init__ already
normalizes category/rate_limit_type to plain str, so the getattr(.value)
+ isinstance dance was dead code. Reduce to str(value) if not None.
- proxy_rate_limit_error.py _coerce_message: short-circuit None to ''
instead of falling through to str(None) = 'None', which produced the
literal message 'litellm.RateLimitError: None'.
* fix(rate-limit): surface unified category/type fields on BudgetExceededError
The most common budget cap (virtual-key max_budget enforcement in
auth_checks.py) raises litellm.BudgetExceededError, a bare Exception
subclass that bypassed the unified rate-limit error class introduced
by PR #27687. Custom callbacks reading
StandardLoggingPayload.error_information saw category=None and
rate_limit_type=None for these 429s, missing the most common budget
case (team / org / end-user budgets all hit the same code path).
Surface the fields off BudgetExceededError as plain attributes:
- category = RateLimitErrorCategory.LITELLM_RATE_LIMIT
- rate_limit_type = RateLimitType.BUDGET
- llm_provider = "" (or caller-supplied)
Switch get_error_information and _extract_rate_limit_labels from
isinstance(RateLimitError) gating to duck-typed attribute reads,
guarded by membership in the rate-limit enums so unrelated third-party
exceptions exposing a .category attribute can't leak garbage values
into the payload.
This is strictly additive: BudgetExceededError keeps its bare-Exception
base class, so `except BudgetExceededError:` handlers keep firing and
`except RateLimitError:` does not start catching budget errors.
* fix(rate-limit): validate enum membership at duck-typed read sites + enrich BudgetExceededError llm_provider
Two follow-ups uncovered during the second QA pass on PR #27687:
1. Guard third-party `.category` / `.rate_limit_type` attribute leakage.
The duck-typed read in `get_error_information` and
`_extract_rate_limit_labels` would forward any string attribute named
`category` / `rate_limit_type` on an unrelated third-party exception
into the StandardLoggingPayload and Prometheus labels — silently
mislabeling custom-callback payloads and blowing out Prometheus label
cardinality. Add `validate_rate_limit_category` /
`validate_rate_limit_type` helpers that gate on the documented enum
value sets; non-matching values are dropped to None.
2. Enrich BudgetExceededError.llm_provider from request_data.
Budget checks live in tenant-scoped helpers (key / team / org / tag /
end-user / project) that don't see the request model, so the
BudgetExceededError they raise carried llm_provider="" — leaving
custom-metrics consumers without provider attribution for the most
common 429 case. Resolve it once at the central
UserAPIKeyAuthExceptionHandler seam, before post_call_failure_hook
fires, so the StandardLoggingPayload the callback sees has the same
provider attribution as RPM/TPM 429s.
Regression tests pin both: 4 leakage tests + 4 enrichment tests. The
leakage tests would fail under the pre-validation version of either read
site; the enrichment tests would fail if the handler skipped the
resolver call.
* fix(rate-limit): resolve router model_name aliases to real provider (#27914)
* fix(rate-limit): resolve router model_name aliases to real provider
For nearly every real LiteLLM proxy deployment the request model is a
router model_name alias (e.g. 'tpm-locked' -> litellm_params.model:
openai/gpt-4o-mini), and 'litellm.get_llm_provider' doesn't know about
router aliases — it raises 'LLMProviderNotProvidedError'. The resolver
then fell through to the defensive 'litellm_proxy' fallback, so the
'llm_provider' field this PR adds was effectively always
'litellm_proxy' in the field, defeating its purpose for the most common
proxy configuration.
Add a router-alias fallback step: when 'get_llm_provider' raises, scan
the active 'llm_router.model_list' for a deployment whose 'model_name'
matches the request model and resolve from its 'litellm_params.model'
instead. If multiple deployments share the same alias (load-balancing
case) the first one wins — every deployment under one alias should
agree on provider in any sensible config, and 'first' is deterministic
so the Prometheus label stays stable.
Defensive throughout: an uninitialized router, a malformed deployment,
a 'litellm_params.model' that itself fails 'get_llm_provider' — every
branch falls through to the existing 'litellm_proxy' fallback rather
than letting a secondary exception escape and mask the rate-limit
error we're trying to surface.
Tests:
- test_router_alias_resolves_to_underlying_provider: alias
'tpm-locked' -> 'openai/gpt-4o-mini' produces provider='openai',
model='gpt-4o-mini'.
- test_router_alias_with_multiple_deployments_uses_first.
- test_router_alias_unknown_falls_back.
- test_router_alias_with_malformed_deployment_falls_back.
- Existing fallback test updated to also stub
'litellm.proxy.proxy_server.llm_router' so it exercises the
full 'no resolution anywhere' path.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(rate-limit): harden router alias resolver + test isolation
- Wrap _resolve_provider_from_router_alias loop in top-level try/except so
a non-iterable model_list / unexpected deployment shape can't escape and
mask the 429 with a 500.
- Type-check litellm_params before .get() to handle non-dict truthy values.
- Patch llm_router=None in the parametrized fallback test so a router left
by another test in the session can't redirect the unknown-model path.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(bugbot): preserve "BudgetExceededError" Prometheus label
Adding llm_provider to BudgetExceededError (so callbacks get provider
attribution from StandardLoggingPayload) made the provider-prefix step in
_get_exception_class_name silently flip the label from "BudgetExceededError"
to e.g. "Openai.BudgetExceededError", breaking dashboards keyed on the
historical value.
Short-circuit BudgetExceededError in _get_exception_class_name the same way
ProxyRateLimitError already is. Provider/category attribution still lands on
the new rate_limit_category / rate_limit_type labels.
* test: fix invalid 'rpm' rate_limit_type in v3 limiter test mocks
The v3 rate limiter only emits 'requests', 'tokens', or
'max_parallel_requests'. Using 'rpm' caused map_v3_rate_limit_type to
return None, leaving the expected RateLimitType.REQUESTS untested.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(bugbot): hoist provider resolver + opt-in prom rate-limit labels
- dynamic_rate_limiter.py: hoist resolve_llm_provider_for_rate_limit
above the TPM/RPM if/elif so the lookup runs once per request, matching
the pattern in dynamic_rate_limiter_v3.py.
- prometheus.py: gate the new rate_limit_category / rate_limit_type
labels on litellm_proxy_failed_requests_metric behind
litellm.prometheus_emit_rate_limit_labels (default False). Mirrors the
existing prometheus_emit_stream_label opt-in. Preserves the metric's
pre-unification label set so existing dashboards / recording rules
keep matching after upgrade; operators can enable the new labels once
downstream consumers include them.
- Tests updated: default-off back-compat case, opt-in path enables the
flag before asserting label presence.
* fix: stabilize prometheus label sets and drop redundant model normalization
- Cache PrometheusLogger.get_labels_for_metric per metric_name so that
the label set used to construct counters at __init__ time stays in
sync with the label set used at increment time, even if module-level
toggles like prometheus_emit_rate_limit_labels or
prometheus_emit_stream_label are flipped at runtime. Without this,
toggling these flags after the logger was created would cause
ValueError from prometheus_client because the runtime labels would
not match the counter's declared labelnames.
- Drop redundant 'model or ""' guard in ProxyRateLimitError.__init__
where model is already normalized one step earlier.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* perf(dynamic_rate_limiter): only resolve provider when rate limit hit
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(prometheus): clear cached metric labels after toggling rate-limit flag
The PrometheusLogger caches each metric's label set at construction
time so that labels used at counter.labels(...) time stay consistent
with the labels the metric was registered with. The enterprise
async_post_call_failure_hook test toggles
litellm.prometheus_emit_rate_limit_labels = True AFTER the fixture
has already built the logger, so without invalidating the cache the
rate_limit_category / rate_limit_type labels never reach the mocked
counter and the assert_called_once_with check fails.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test: fix CI failures from prom label cache + flaky time-window assertion
PrometheusLogger.get_labels_for_metric now caches the per-metric label
set at first read so the labels passed to counter.labels(...) stay in
lock step with the labels the counter was registered with. This broke
two existing test patterns:
- test_prometheus_labels.py: tests bind the real method onto a
MagicMock, but MagicMock auto-creates a Mock for _cached_metric_labels
whose .get(...) returns a truthy Mock — treated as a populated cache
and returned as the label set, producing empty filtered labels and
KeyError on labels["requested_model"] / ["route"]. Seed real {}
containers for _cached_metric_labels and label_filters before binding.
- test_prometheus_logging_callbacks.py::test_set_team_budget_metrics_with_custom_labels:
the fixture builds the logger before the test monkeypatches
litellm.custom_prometheus_metadata_labels, so the cached label set
never picks up the new metadata labels. Clear the cache after the
monkeypatch (same pattern already used for the rate-limit toggle in
test_async_post_call_failure_hook).
UI: view_logs/index.test.tsx "Last Minute" window assertion is off by
one at the minute boundary. start_date is floored to the minute, so the
dropped sub-minute fraction can push the truncated-seconds diff up to
(minMinutes+1)*60 exactly when the click lands near a minute rollover.
Switch the upper bound to toBeLessThanOrEqual.
* feat(otel-v2): surface rate_limit_category + rate_limit_type on failed LLM-call spans
PR #28909 introduced the typed v2 OTel engine that builds spans from
StandardLoggingPayload, with SpanError carrying error_type + message and
the genai mapper stamping error.type onto every failed LLM-call span.
This PR's earlier commits added error_rate_limit_category and
error_rate_limit_type to the same StandardLoggingPayload.error_information
the v2 engine reads — but neither field reached a span attribute, so v2
OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm,
RPM vs TPM vs concurrent vs budget vs max_iterations) even after the
custom-callback and prometheus surfaces gained that decomposition.
Three coupled changes:
1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY /
LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace
(no GenAI semconv equivalent exists for who-rate-limited /
which-dimension).
2. payloads.py: extend SpanError with rate_limit_category +
rate_limit_type, populated by _parse_error() from the same
error_information.error_rate_limit_* fields the custom-callback
channel and prometheus rate_limit_category / rate_limit_type labels
read. Single source of truth across all three observability surfaces.
3. mappers/genai.py: stamp the two attributes on the LLM-call span when
present. drop_none guarantees they stay absent (not 'None') for
non-rate-limit failures so trace consumers can read them
unconditionally.
Three regression tests in test_otel_v2_emitter.py pin: a vendor /
litellm-internal RateLimitError lands category=litellm_rate_limit +
rate_limit_type=requests on the span; a BudgetExceededError lands
rate_limit_type=budget; a non-rate-limit failure (BadRequestError)
keeps the rate_limit_* attributes absent. Mutation-tested against
reverting either the SpanError extension or the _parse_error read site
— both new tests fail under either mutation.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test: align prometheus user-budget + logs quick-select tests with merged code
The merge into this branch left two test patterns out of step with the code
they exercise.
test_set_user_budget_metrics_includes_user_email_and_alias_labels_when_opted_in
flipped litellm.prometheus_user_budget_label_include_email_alias after the
fixture had already built the PrometheusLogger. get_labels_for_metric now
snapshots each metric's label set at construction time, so the runtime flip
no longer reached the cached labels. Enable the flag before constructing the
logger, matching how the proxy applies config at startup.
view_logs/index.test.tsx referenced uiSpendLogsCall and moment without
importing them, and the merged index.tsx now fetches through
useLogFilterLogic (the hook the file stubs out) rather than calling
uiSpendLogsCall directly. Add the imports and restore the real hook for the
Quick Select window assertions so the call is actually observed.
* refactor(otel/v2): drop rate-limit decomposition from the LLM-call span
Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are
rejected at the gate before any upstream call, so async_post_call_failure_hook
tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the
v2 OTel logger never opens an LLM-call span for them; the
litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes
were dead for exactly the cases they were meant to surface. The only failure
that does open an LLM-call span carrying a RateLimitError is a vendor 429, where
rate_limit_type is always None and the category just restates
error.type=RateLimitError.
The decomposition still reaches downstream consumers through
StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus
rate_limit_category / rate_limit_type labels, both unchanged.
Removes the SpanError fields, the _parse_error reads, the genai mapper
attributes, the semconv keys, and the three span tests that asserted a scenario
that never reaches the mapper in production.
* fix(batch_rate_limiter): map max_parallel_requests to concurrent_requests
* refactor(prometheus): drop transitive fastapi import from _get_exception_class_name
Read the legacy exception_class label from a prometheus_exception_class_name
marker on ProxyRateLimitError instead of importing the proxy module, keeping
the integrations layer free of a transitive fastapi dependency.
* chore(ui): sync schema.d.ts with unified rate-limit error spec
The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429
response description, so the generated dashboard types were out of sync.
Regenerated via npm run gen:api (Check UI API Types Sync).
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
|
||
|
yuneng-jiang
|
7bfce053a9 |
fix(ui): make workflow runs page fill full width (#29868)
The Workflow Runs page rendered its table at roughly a quarter of the available width. Its root container is a flex child of the dashboard content row but set only padding, min-height and background, so with no width it shrank to the table's natural content size. Sibling pages (logs, memory) fill the area with a full-width root; mirror that by setting width 100% on the container. Fixes LIT-3636 |
||
|
ryan-crabbe-berri
|
f31d059aa3 |
feat(ui): add budget duration to edit team member form (#29717)
* feat(ui): add budget duration to edit team member form Editing a team member created a member budget with no duration, so the budget never reset. This threads a budget reset period through the edit flow end to end and reuses the shared duration dropdown so the options stay in sync with the rest of the UI. Resolves LIT-2651 * fix(proxy): validate member budget_duration and persist clears Reject budget_duration values that can't be parsed, are non-positive, or overflow date math before any write, so a bad value can't be persisted and later crash the budget reset job. Clearing the budget duration in the edit-member form now sends null and clears the column end to end, so the dropdown's clear control reflects a real change instead of being a no-op * chore(ui): regenerate schema.d.ts for member budget_duration Adds budget_duration to TeamMemberUpdateRequest/Response in the generated dashboard types so the Check UI API Types Sync gate passes |
||
|
tin-berri
|
21d2c3aa83 | fix(ui): stop MCP playground tool calls from sending twice (#29821) | ||
|
ryan-crabbe-berri
|
001bda37d9 | refactor(ui): route query-building networking calls through apiClient (#29815) | ||
|
milan-berri
|
1f171ee018 | fix(ui): require new expiration when regenerating an expired key (#29838) | ||
|
tin-berri
|
22186f457a | fix(ui): persist Tools-tab MCP OAuth token to DB (#29809) | ||
|
ryan-crabbe-berri
|
6955e6f2c2 |
refactor(ui): route behavior-preserving networking calls through apiClient (#29806)
* refactor(ui): route callbacks/nudges calls through apiClient * refactor(ui): route alerting + key/user/team delete calls through apiClient * fix(ui): late-bind fetch in apiClient so global.fetch swaps take effect createApiClient captured fetch at construction time, so reassigning global.fetch (as tests do) had no effect and a real network call leaked. Resolve fetch per request instead; harmless in production where fetch is never swapped, and required for apiClient-based calls to be testable. * refactor(ui): route behavior-preserving networking calls through apiClient Collapse ~61 hand-rolled fetch() calls whose semantics already match the shared apiClient (auth header, JSON body, json-error + deriveErrorMessage + handleError) into apiClient.get/post/etc. Query-string builders and the divergent error-handling functions (no-check, custom messages, text-error) are intentionally left for a follow-up normalization pass, since converting them changes wire encoding or error behavior. Prunes the now-stale no-restricted-syntax suppressions for the removed fetch calls. * refactor(ui): convert remaining admin/guardrail GETs, guard late-bind fetch Routes adminspendByProvider, adminGlobalActivity, and the three guardrail submission calls (list/approve/reject) through apiClient so they match their already-converted siblings instead of staying on raw fetch. Adds a client.test.ts case that swaps globalThis.fetch after createApiClient() and asserts the swap takes effect, which fails on the pre-fix captured-fetch line and locks in the per-call resolution |
||
|
Mateo Wang
|
4ec4ab99d0 | feat(mcp): per-server env vars with global + per-user scopes (#28917) | ||
|
ryan-crabbe-berri
|
e53bd7cbd1 |
feat(ui): generate dashboard API types from the proxy OpenAPI spec (#29816)
* feat(ui): generate dashboard API types from the proxy OpenAPI spec Introduces the shared type foundation for the dashboard without touching any runtime code. The proxy's FastAPI app is the source of truth; app.openapi() emits the spec and openapi-typescript turns it into src/lib/http/schema.d.ts. Adds an npm run gen:api script (a Python spec dump piped into openapi-typescript) and a Check UI API Types Sync CI job that regenerates the file from the live spec and fails if it drifts, so the committed types can never silently fall out of step with the backend. The generated file is pinned to openapi-typescript 7.13.0 and excluded from prettier, eslint, and knip, and marked linguist-generated so it collapses in diffs. No openapi-fetch and no call-site changes yet; this only makes the types exist. * chore(ui): tidy gen-api-types script per review Write the spec dump inside a with-block and clean up the temp dir in a finally, so repeated local runs don't leave stray ~MB JSON files behind. |
||
|
Sameer Kankute
|
d671a09c20 |
Litellm oss staging 050626 (#29774)
* Mark xAI models retiring on 2026-05-15 (#28788) Per https://docs.x.ai/developers/migration/may-15-retirement, xAI is retiring the following slugs on 2026-05-15 (auto-redirect to grok-4.3 with various reasoning efforts; callers continuing to use the old slugs will be billed at grok-4.3 pricing): grok-4-1-fast-reasoning{,-latest} -> grok-4.3 (low effort) grok-4-1-fast-non-reasoning{,-latest} -> grok-4.3 (none) grok-4-fast-reasoning -> grok-4.3 (low effort) grok-4-fast-non-reasoning -> grok-4.3 (none) grok-4-0709 -> grok-4.3 (low effort) grok-code-fast-1{,-0825} -> grok-build-0.1 grok-3 -> grok-4.3 (none) Only the direct xai/ slugs are tagged; third-party hosts (azure_ai, oci, vercel_ai_gateway, perplexity/xai) run their own schedules. The grok-3 retirement list explicitly names only the base grok-3 slug — the -mini / -fast / -beta / -latest variants are not listed, so they remain untouched. * feat(moonshot): advertise json_schema response support on live models (#29683) litellm.responses() already routes Moonshot through the responses->chat-completions bridge, and Moonshot honors response_format json_schema on chat completions. The cost-map entries left supports_response_schema unset, so discovery layers that gate on that flag dropped Moonshot from structured-output / responses listings even though the capability works end to end. Set supports_response_schema on the nine models currently live on api.moonshot.ai: kimi-k2.5, kimi-k2.6, the moonshot-v1 8k/32k/128k text and vision-preview variants, and moonshot-v1-auto. Verified against the live API that each honors json_schema and that litellm.responses() returns schema-valid structured output through the bridge. * chore(moonshot): mark models retired from api.moonshot.ai as deprecated (#29685) Thirteen Moonshot/Kimi models in the cost map no longer resolve on api.moonshot.ai (all return 404). Stamp each with its deprecation_date from platform.kimi.ai/docs/models rather than deleting the entries, so historical cost calculation keeps resolving the names while tooling can surface the retirement. Dates: kimi-thinking-preview 2025-11-11; kimi-latest and its 8k/32k/128k context variants 2026-01-28; the kimi-k2 preview/turbo/thinking series 2026-05-25; the moonshot-v1 -0430 snapshots use their own 2024-04-30 snapshot date (Moonshot publishes no discontinuation date for them). * fix(moonshot): drop temperature for reasoning models (kimi-k2.5/k2.6) (#29687) Kimi reasoning models reject every temperature except 1; a request with temperature=0.2 returns "invalid temperature: only 1 is allowed for this model". litellm only clamped temperature into [0.3, 1], so any value below 1 still 400'd. Drop the temperature param entirely for reasoning models (gated on supports_reasoning, the same signal transform_request already uses) so the model default is used; the non-reasoning moonshot-v1 models keep the existing clamp. Co-authored-by: Sameer Kankute <sameer@berri.ai> * feat(mcp): add per-server timeout configuration (#29672) * feat(mcp): add per-server timeout configuration * fix(mcp): address timeout field review comments - use is not None guard instead of or for 0.0 edge case - copy timeout in both LiteLLM_MCPServerTable constructions (health check path + _build_mcp_server_table) - add timeout Float? column to all three schema.prisma files - extend round-trip test to cover _build_mcp_server_table direction - add test for zero timeout not treated as falsy * fix(mcp): forward timeout in _build_temporary_mcp_server_record * fix(mcp): return 504 instead of 500 when per-server timeout fires * test(mcp): add 504 timeout regression test; fix black formatting * Add jp. Bedrock cross-region inference profile for claude-opus-4-7 (#28567) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add jp. Bedrock cross-region inference profile for claude-opus-4-7 AWS Bedrock documents jp.anthropic.claude-opus-4-7 alongside the existing us./eu./au./global. profiles for Claude Opus 4.7 (ap-northeast-1 Tokyo / ap-northeast-3 Osaka), but the entry is missing from model_prices_and_context_window.json. Tokyo-region users currently get an "unknown model" error when routing through the JP geo profile. Adds the entry to both the canonical file and the bundled backup, mirroring the recent pattern for sonnet-4-6 (#27831). Pricing matches the other regional profiles (10% premium over base/global). Regression test pins all six documented profiles (base, global, us, eu, au, jp) and asserts pricing parity between jp. and au. variants. Source: https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-anthropic-claude-opus-4-7.html --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * feat(soniox): add soniox audio transcription integration (#29508) * feat(openmeter): add OPENMETER_TRUST_REQUEST_USER to prevent forged attribution (#29650) The OpenMeter callback resolves the CloudEvent subject from kwargs["user"] first, then falls back to the key-bound user_api_key_user_id. For multi-tenant proxy deployments, a client can set `"user": "..."` in the request body and cause their usage to be attributed to that arbitrary string — a billing-attribution forgery risk. Adds OPENMETER_TRUST_REQUEST_USER env var (default "true" for backward compatibility). When set to "false", the request-supplied `user` field is ignored and the subject is resolved solely from user_api_key_user_id. Matches the existing env-var-driven config pattern in this file (OPENMETER_API_KEY, OPENMETER_API_ENDPOINT, OPENMETER_EVENT_TYPE). * feat(search): add you_com as a search provider (#28370) * feat(search): add you_com as a search provider Registers You.com Search API as a first-class `search_provider` in the `search_tools` registry, alongside Tavily, Exa, Perplexity, etc. - New adapter: litellm/llms/you_com/search/transformation.py - POSTs to https://ydc-index.io/v1/search - Auth: X-API-Key from YOUCOM_API_KEY (or explicit api_key) - Maps Perplexity unified spec: max_results -> count, search_domain_filter -> include_domains, country -> country - Flattens results.web + results.news into a single SearchResult list; snippet prefers snippets[0], falls back to description; page_age -> date - Registry: SearchProviders.YOU_COM in litellm/types/utils.py and wired into ProviderConfigManager.get_provider_search_config() - Pricing entry: model_prices_and_context_window.json (placeholder $0.0; happy to adjust to maintainers' preferred public number) - Docs: example router config snippet and example proxy yaml updated - Tests: tests/search_tests/test_you_com_search.py - 5 mocked tests (payload shape, domain filter mapping, snippet fallback, news flattening, missing-api-key error) Refs upstream expansion signal: #15942 * review fixups: normalize api_base, lowercase country, scope env-var to test Addresses Greptile inline review comments on #28370: - get_complete_url: strip trailing slashes from api_base *before* the endswith("/v1/search") check, so a custom base like ".../v1/search/" doesn't become ".../v1/search/v1/search". - transform_search_request: .lower() country before sending, matching Tavily's convention so callers using the unified spec form ("US") get consistent behavior across providers. - Tests: replace direct os.environ writes with an autouse monkeypatch fixture so YOUCOM_API_KEY is set per-test and removed afterwards. The missing-key test now uses monkeypatch.delenv. New test asserts the trailing-slash normalization above. Reverts the ARCHITECTURE.md / example yaml edits per the reviewer note that documentation changes belong in the litellm-docs repo. * support keyless free tier (api.you.com/v1/agents/search) as default You.com offers an IP-throttled keyless endpoint that returns the same response shape as the keyed one (~100 queries/day, no signup). This is a significant onboarding lever - mirrors the keyless DuckDuckGo/SearXNG providers already in the search_tools registry. Behavior: - YOUCOM_API_KEY set -> keyed: POST https://ydc-index.io/v1/search (X-API-Key header) - no key -> free: POST https://api.you.com/v1/agents/search (no auth) - YOUCOM_API_BASE override -> honored as-is Tests: - New: test_you_com_search_keyless_free_tier - asserts URL + absence of X-API-Key when no key is configured. - New: test_you_com_search_validate_environment_keyless - asserts the config no longer raises when the key is absent. - Removed: test_you_com_search_raises_without_api_key (the precondition no longer holds). - Existing payload/domain-filter/etc tests still cover keyed mode via the autouse YOUCOM_API_KEY fixture. Verified both endpoints accept POST + return identical JSON shape: results.web[] / results.news[] with title, url, snippets, description, page_age. * register you_com in provider_endpoints_support.json Adding `litellm/llms/you_com/` requires a corresponding entry in provider_endpoints_support.json or the code-quality/check_provider_folders_documented CI check fails. Follows the compact tavily/serper pattern - endpoints: { search: true }. Local run of the check now reports "All 114 provider folders are documented". * move tests under tests/test_litellm/llms/ so CI exercises them The litellm CI workflows scope unit tests to `tests/test_litellm/...` (see test-unit-llm-providers.yml: `tests/test_litellm/llms` path), so tests living under `tests/search_tests/` are never run in CI - which is why codecov reports 0% patch coverage for the new adapter even though the unit tests exist and pass locally. Move test_you_com_search.py into `tests/test_litellm/llms/you_com/` so the test-unit-llm-providers job picks it up. 7/7 tests still pass at the new location. (Sibling search-only providers - tavily, exa_ai, brave, etc. - still live only in `tests/search_tests/` and would benefit from the same move, but that is out of scope for this PR.) * fix(you_com): pin Accept-Encoding: identity to dodge keyless gzip bug The keyless free-tier endpoint (api.you.com/v1/agents/search) advertises Content-Encoding: gzip but returns a body that httpx's decoder rejects with `zlib.error: Error -3 while decompressing data: incorrect header check`, surfacing as litellm.APIConnectionError in user code. curl works because it doesn't request compression by default. Pin Accept-Encoding: identity in validate_environment so the upstream server skips compression entirely. Harmless on the keyed endpoint (ydc-index.io/v1/search) which negotiates content-encoding correctly. The header uses setdefault so a caller-supplied Accept-Encoding still takes precedence. (Server-side bug has been flagged to the You.com team separately - once fixed there, this workaround can be removed.) New unit test: test_you_com_search_pins_identity_accept_encoding. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * docs: fix README typo (#29419) Correct clear spelling mistakes in documentation without changing behavior. Confidence: high Scope-risk: narrow Tested: git diff --check; uvx codespell on changed files Not-tested: Full docs build not run; text-only changes * Fix(langfuse): pass httpx_client to Langfuse in langfuse_prompt_management to respect SSL_VERIFY (#29480) * fix(langfuse): pass ssl_verify to Langfuse httpx client * fix_langfuse_ * add unit tests * addressed comments --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * feat(models): add minimax/MiniMax-M3 to model cost map (#29412) Add MiniMax's new flagship MiniMax-M3 to the native minimax provider: 512K context, 128K max output, native multimodal (supports_vision), reasoning, prompt caching. Pricing (USD/M tokens): input 0.6 / output 2.4 / cache read 0.12. M3 has no active prompt-cache-write tier, so cache_creation_input_token_cost is omitted. Updated both the root model_prices_and_context_window.json (remote source) and the bundled litellm/model_prices_and_context_window_backup.json (local fallback), keeping them in sync. * fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log (#29394) * fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log * fix(logging): extend terminal event handling to ResponseIncompleteEvent and ResponseFailedEvent; fix return type annotation * feat(provider): Add Neosantara provider as OpenAI Compatible (#29646) * Add Neosantara provider * Register Neosantara provider enum * Address Neosantara provider review feedback * Add Neosantara packaged endpoint support --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix: address greptile and veria review feedback - langfuse: guard httpx_client injection behind version check (>= 2.7.3) - soniox: propagate audio_transcription_duration in _hidden_params for spend tracking - soniox: give SONIOX_API_BASE env var priority over caller-supplied api_base - mcp: replace CancelledError catch with asyncio.wait_for + TimeoutError * chore(mcp): add migration for per-server timeout column * fix(test): add tool_use_system_prompt_tokens to model prices schema validator * fix: mcp timeout test uses real asyncio.wait_for timeout; you_com get_complete_url respects resolved api_key * fix: forward resolved api_key into you_com endpoint selection and apply timeout to soniox polling GETs The search flow resolves api_key in validate_environment but never passed it into get_complete_url, so a programmatic api_key (with no YOUCOM_API_KEY in the env) set the X-API-Key header yet still selected the keyless free-tier endpoint. Forward api_key through both the search entrypoint and the http handler so the keyed endpoint is chosen. HTTPHandler.get/AsyncHTTPHandler.get had no timeout parameter, so the Soniox poll and transcript-fetch GETs silently used the client global default instead of the caller timeout. Add a per-request timeout to get() and forward the configured timeout from the Soniox handler. * fix(soniox): price stt-async-v4 per second so transcriptions are billed The handler stores audio_transcription_duration in _hidden_params, but the model carried only token cost fields and the response has no token usage, so the transcription cost path fell through to cost_per_second and returned $0. An authenticated caller could transcribe Soniox audio without decrementing their budget. Switch the entry to output_cost_per_second at Soniox's published $0.10/hour async rate so the stored duration produces a real charge. * fix(langfuse): use a dedicated httpx client for the SDK injection The httpx_client handed to the Langfuse SDK came from _get_httpx_client(), which returns LiteLLM's globally cached HTTPHandler. If Langfuse closed that client on teardown it would invalidate the shared client used by every other LiteLLM HTTP call. Build a dedicated httpx.Client instead, still resolving SSL verification and client certificate from LiteLLM's configuration. * fix(soniox): prefer caller-supplied api_base over SONIOX_API_BASE env var * fix(cohere): support max_completion_tokens on cohere v2 chat (default route) (#29779) * fix(cohere): support max_completion_tokens on cohere v2 chat The default cohere_chat route resolves to CohereV2ChatConfig, which did not list or map max_completion_tokens, so get_optional_params raised UnsupportedParamsError for the standard OpenAI parameter (the modern replacement for the deprecated max_tokens). The v1 config already maps it to cohere's max_tokens; mirror that in v2 and add v2 regression tests. * fix(cohere): make max_completion_tokens take precedence over max_tokens on v2 When both max_tokens and max_completion_tokens are supplied, prefer max_completion_tokens explicitly rather than relying on dict iteration order, and cover both orderings with a regression test. --------- Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com> Co-authored-by: hectorc98 <hector.chamorroalvarez@adyen.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Dan Lemon <dan@danlemon.com> Co-authored-by: Saswat <saswatds@users.noreply.github.com> Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com> Co-authored-by: Zhao73 <156770117+Zhao73@users.noreply.github.com> Co-authored-by: Urain Ahmad Shah <60431964+urainshah@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: kape <168134658+kapelame@users.noreply.github.com> Co-authored-by: danisalvaa <159898202+danisalvaa@users.noreply.github.com> Co-authored-by: Just R <remixingmagelang@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: abhay23-AI <abhaytrivedi22@gmail.com> |
||
|
ryan-crabbe-berri
|
4a5644d51e |
refactor(ui): centralize proxy base URL resolution into tested resolver (#29793)
* refactor(ui): centralize proxy base URL resolution into tested resolver The API base URL join logic was hand-rolled inside networking.tsx and re-derived inline at hundreds of call sites, with no test coverage and a latent double-slash bug when the base carried a trailing slash. This pulls the join into a single pure resolveApiBase() with full unit coverage and routes the existing resolution through it, also de-duplicating the env precedence ladder that was copied in two places. * test(ui): assert root-path redirect joins prefix exactly once The existing toContain check accepts a doubled separator; tighten it to a strict prefix match plus a no-double-slash assertion so a regression in the resolveApiBase origin+SERVER_ROOT_PATH join is caught end-to-end. |
||
|
tin-berri
|
a4f57032e0 |
fix(ui): route MCP playground auth by oauth2 mode instead of token_url (#29714)
Interactive PKCE and OBO servers were mislabeled as M2M, so passthrough never showed the Authorize gate; classify by oauth2_flow + delegate_auth_to_upstream instead. |
||
|
Sameer Kankute
|
89f177b7b6 |
fix(galileo): use ingest traces API and standard logging payload (#29651)
* fix(galileo): use ingest traces API and standard logging payload Switch hosted Galileo logging to /ingest/traces with nested trace/span payloads, read metrics from standard_logging_object, and include cost and total tokens on trace metrics. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(galileo): route username/password auth to v2 traces ingest Hosted Galileo no longer serves /observe/ingest; JWT login should post the same trace payload to /v2/projects/{project_id}/traces. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(galileo): address Greptile review on logging and timestamps Use debug-level logs for per-request Galileo callback messages and fall back to start_time/end_time when standard_logging_object omits startTime/endTime. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(galileo): add Galileo to proxy UI callback configuration Expose Galileo in the admin callback selector and config APIs so credentials can be configured through the dashboard instead of YAML only. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(galileo): align response type logging with Langfuse Mirror Langfuse input/output handling for rerank, speech, transcription, realtime, pass-through, and other response types so Galileo ingest no longer skips supported call types. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(galileo): redact trace payload in debug logs and format with black Avoid logging prompts and model responses in flush debug output while keeping structural metadata for troubleshooting. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(galileo): stop logging full trace payload in debug output Log only flush URL and trace count so prompts and model responses are not written to application logs when debug logging is enabled. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix Galileo token totals and prompt messages --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
yuneng-jiang
|
56aa55b991 |
fix(proxy): stop team BYOK model name corruption on model edit (#29731)
* fix(proxy): stop team model name corruption on edit (#28382) (#29001) Team-scoped ("Team-BYOK") models store an internal routing key model_name_{team_id}_{uuid} in the model_name column and the user-facing name in model_info.team_public_model_name. The internal name leaked into /v1, /v2, and /model/info responses; the dashboard bound its edit form to it, so any non-rename save (e.g. a TPM tweak) PATCHed the internal name back. The update path then treated it as a rename, overwriting team_public_model_name and rewriting the team's models[] ACL with the mangled string -- breaking team key calls with team_model_access_denied. Two-layer fix: - Read path (root cause): add _translate_model_name_for_response and apply it in model_info_v2 and _get_proxy_model_info so /v1, /v2, and /model/info surface the public name for team-scoped rows. The DB column and router index keep the internal name as the routing key; this is a presentation-layer swap on a shallow copy (never mutates input). - Write path (defense in depth): harden _get_public_model_name so a value matching the internal shape, or a no-op against the current DB column, is never treated as a rename -- for both the top-level model_name and an explicit model_info.team_public_model_name. Tests: regression for the reported scenario, full branch coverage of _get_public_model_name, two internal-shape guard cases, an end-to-end PATCH through _update_team_model_in_db (asserts the team ACL is untouched), and four response-translation cases. 60 passed (model management), 181 passed (proxy server). * fix(ui): key Agent Builder agent selection on model_info.id (#29729) * fix(ui): key Agent Builder agent selection on model_info.id Once team-scoped BYOK models can share a public name (the backend now returns the public name on /model/info instead of the internal routing key), selecting agents by model_name collides. Key selection, create, update and delete on the stable model_info.id instead, falling back to model_name only for config-defined agents that have no id. * fix(ui): add name-match fallback to post-create agent selection If the just-created agent's id is not yet present in the re-fetched list, try matching by name before falling back to the first agent. Addresses greptile review on #29729. --------- Co-authored-by: tushar8408 <32977767+tushar8408@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
f3811ce63b |
refactor(ui): shared HTTP client + location-pinned fetch() lint rule (#29723)
* refactor(ui): add shared HTTP client and pin raw fetch() to one file Introduce src/lib/http/client.ts, a single typed wrapper that owns the only fetch() in the dashboard. It centralizes the base URL, the auth header, error parsing (deriveErrorMessage), non-2xx -> thrown ApiError, and JSON parsing, and is framework-agnostic (no React) so it can run from client and, later, server components. The base URL, auth header name and the logout side effect are injected through createApiClient. networking.tsx builds one configured apiClient and the 29 functions whose boilerplate maps exactly to the client's default behavior (canonical deriveErrorMessage + handleError + res.json() template) now call it instead of hand-rolling fetch. Names, signatures, return types and error behavior are unchanged; this is a pure refactor that drops ~440 lines. The no-restricted-syntax fetch rule now points at the client and a files: ["src/lib/http/**"] override makes that the only place fetch() is allowed. Re-baselined eslint-suppressions.json: networking.tsx fetch suppressions drop 270 -> 241; no other rule's counts change. The remaining networking.tsx fetches and the ~61 scattered component/hook fetches diverge from the default client behavior (text() error bodies, no res.ok check, no handleError side effect) and stay grandfathered for a follow-up burndown. * fix(ui): make the HTTP client tolerate non-JSON error bodies The non-2xx branch parsed the error body with response.json(), so a gateway returning HTML (502/503 from a reverse proxy) threw a SyntaxError before onError fired or ApiError was built, dropping the user-facing notification. This matched the old per-function behavior, but the client is now the single error path so it is the right place to harden. Read the body as text once, try JSON.parse for the existing deriveErrorMessage path, and fall back to the raw text (or the HTTP status) otherwise. The success path stays strict json() so return types are unchanged. * fix(ui): await the returned apiClient promise in 6 migrated functions The codemod rendered the `return response.json()` tail as `return apiClient.x()` without `await`. Inside the surrounding try/catch that returns an unawaited promise, so the catch never runs and its console.error log is dropped on failure; 4 of the 6 were `return await response.json()` originally, so this restores their exact behavior. Use `return await apiClient.x()` in all six. * refactor(ui): widen onError type and handle empty success bodies Address review notes on the shared client. Type onError as (message: string) => void | Promise<void> so the fire-and-forget async contract (networking passes the async handleError) is explicit rather than silently discarded by void. On the success path, read the body as text and return undefined for an empty body (e.g. a 204 No Content) instead of throwing a SyntaxError, while still parsing non-empty bodies strictly so a malformed JSON response surfaces rather than being masked. Add tests for the 204 case. |
||
|
ryan-crabbe-berri
|
41e90a6ada |
chore(ui): remove the bare-fetch lint rule (#29712)
* fix(ui): only flag bare fetch() outside React Query queryFn/mutationFn The frontend lint rule banned every fetch() call by static AST name match, so a fetch wrapped in a React Query queryFn/mutationFn tripped it just like a loose fetch in a component. esquery (no-restricted-syntax) can't express "has ancestor", so this replaces that selector with a small custom rule (local/no-bare-fetch) that exempts a fetch lexically inside a queryFn or mutationFn and reports everything else. Re-baselined eslint-suppressions.json under the new rule id (same 44 files / 331 violations) so existing code keeps its grandfathered suppressions. Adds a RuleTester suite covering wrapped (valid) vs unwrapped, the standalone *Api.ts function pattern, queryKey, and computed-key cases. * chore(ui): remove the bare-fetch lint rule Drop the fetch lint gate (and its 331 grandfathered suppressions) ahead of the networking refactor. The plan is to centralize all fetching in a single shared http client and enforce that with a location-based rule, so keeping a fetch rule in place now would only block CI while functions are routed through the new client. Removing it unblocks that work; the location-based rule lands with the client in a follow-up. |
||
|
ryan-crabbe-berri
|
7edf3a9cb5 |
style(ui): run prettier --write across the dashboard (#29622)
Formatting-only pass; no logic changes. Brings the UI into compliance with .prettierrc so the new format-check CI job passes |
||
|
ryan-crabbe-berri
|
443f0ca4cd |
ci(ui): frontend-lint job enforcing prettier + eslint on changed files (#29633)
* ci(ui): add frontend-lint job enforcing prettier and eslint on changed files
Lints only the files a PR adds or modifies under ui/litellm-dashboard,
so new and touched code must be prettier-clean and eslint-clean while the
existing tree is grandfathered. Skips cleanly when a PR touches no
lintable UI files. This lets us adopt the formatters incrementally
without a repo-wide reformat
* ci(ui): write frontend-lint file lists to $RUNNER_TEMP
Keep the prettier/eslint changed-file lists out of the checkout dir so
they cannot collide with a future source file of the same name
* lint(ui): baseline existing eslint findings so only new ones block
Capture the current error-level eslint findings (318 across 183 files)
in a committed suppressions baseline via eslint --suppress-all. Every
rule stays at its error severity, so any newly introduced violation
fails the frontend-lint gate, while the existing tree is grandfathered;
touching a legacy file never forces fixing its pre-existing issues. CI
runs eslint with --pass-on-unpruned-suppressions so that fixing a
baselined issue does not fail on a now-stale suppression, and the
generated baseline is prettier-ignored since eslint owns its format.
Burn the baseline down over time with eslint --prune-suppressions
* lint(ui): enforce a count budget for explicit any
Make @typescript-eslint/no-explicit-any a warning and cap the total
instead of hard-blocking each new one. A frontend-lint step counts the
repo-wide explicit any and fails only when it exceeds the committed
budget in eslint-any-budget.json. max starts at 2031, ten above the
current 2021, so the next ten land as warnings and the build fails once
that headroom is gone. Lower max over time toward target to ratchet the
count down. New anys still surface as warnings on changed files via the
normal eslint step
* lint(ui): enable zero-cost rules no-var, no-self-assign, react/no-danger
These have no existing violations, so they need no baseline; turning them
on purely blocks new instances. react/no-danger guards against new
dangerouslySetInnerHTML (XSS), no-var enforces let/const, and
no-self-assign catches self-assignment typos. no-debugger is already
enforced by the recommended preset
* lint(ui): add baselined complexity rules
Enable complexity:20, max-depth:4, max-params:4, max-nested-callbacks:4,
with thresholds set near the codebase p99 so only genuine outliers are
flagged. The 272 existing over-threshold functions are grandfathered in
the suppressions baseline; new over-threshold functions block. Lower the
thresholds over time to ratchet complexity down. max-lines-per-function
is intentionally left off since React components are legitimately long
* lint(ui): ban new raw fetch, standardize on React Query
Add a no-restricted-syntax rule flagging bare fetch() calls, pointing
contributors at React Query (@tanstack/react-query). The rule is not
exempted anywhere, including the already-bloated networking.tsx, so all
331 existing fetch calls are grandfathered but no new ones can be added
there or elsewhere. New data access goes through React Query, and the
networking layer can be migrated out and pruned from the baseline over
time
* lint(ui): ban new @tremor/react imports
Add a no-restricted-imports rule flagging imports from @tremor/react so
tremor is phased out rather than spread further. The 232 existing tremor
imports are grandfathered in the baseline; new ones block and point at
antd. Migrate components off tremor and prune the baseline over time
* lint(ui): widen explicit-any budget headroom to 2040
Raise max from 2031 to 2040, giving ~19 of slack over the current 2021
instead of 10
* style(ui): prettier-format eslint.config.mjs
The frontend-lint gate flagged its own config file. Format it so the
prettier check on this PR's changed files passes
* lint(ui): soften complexity and max-depth to warnings
These two are smell metrics with arbitrary thresholds where a legit new
function can trip them, so make them advisory rather than hard-blocking.
They drop out of the baseline (now 963). max-params, max-nested-callbacks,
and the react-hooks rules stay strict since those are clear-cut
* lint(ui): move complexity and max-depth to the count-budget pattern
Generalize the explicit-any budget into a shared lint-budget mechanism:
eslint-budgets.json maps a rule to {max, target} and check-lint-budgets.mjs
counts each across the repo and fails when a count exceeds its max.
complexity (129, max 140) and max-depth (61, max 70) now use the same
slack-plus-counter model as explicit-any (2021, max 2040): they warn
per-file and the build only fails if the repo-wide total crosses the
ceiling. Lower each max toward its target over time
* docs(ui): note pruning the eslint suppressions baseline when fixing lint debt
|
||
|
ryan-crabbe-berri
|
c7f1bcfd0d |
build(ui): migrate eslint to flat config and bump eslint-config-next to 16 (#29626)
ESLint 9 defaults to flat config and eslint-config-next was pinned at 15 while Next is on 16, so eslint only ran with ESLINT_USE_FLAT_CONFIG=false and next lint is gone on Next 16. Replace .eslintrc.json with a native flat eslint.config.mjs (config-next 16 ships flat configs, so no FlatCompat shim is needed), bump eslint-config-next to 16.2.6, add @eslint/js and typescript-eslint as explicit devDeps for the recommended rule sets, and point the lint script at eslint directly. This only makes eslint runnable on modern tooling; it does not wire it into CI. The same rules carry over (next/core-web-vitals, eslint and typescript-eslint recommended, prettier, unused-imports) |
||
|
Sameer Kankute
|
c7ab9adde5 |
Litellm oss staging 030626 (#29578)
* Fix incorrect agent API request example payload structure (#29556) * fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs (#29427) * fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs On /v1/messages and other LITELLM_METADATA_ROUTES, the parent OTel span is stored in litellm_params['litellm_metadata'] instead of litellm_params['metadata']. When the request body contains a native 'metadata' field (e.g. Anthropic's {"user_id": "..."}), litellm_params['metadata'] gets overwritten and the parent span is lost, producing orphan root spans with a different trace_id. Add fallback checks to litellm_metadata in: - _get_span_context(): so child spans find the correct parent - _end_proxy_span_from_kwargs(): so the proxy span gets closed Fixes: https://github.com/BerriAI/litellm/issues/27934 * test(otel): tighten assertions per Greptile review - test_span_context_metadata_takes_priority: assert litellm_metadata span is never accessed, proving metadata takes priority - test_span_context_no_parent_when_neither_has_span: assert both ctx and detected_span are None --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: remove premature end-user budget check from get_end_user_object (#29420) * fix(proxy): remove premature end-user budget check from get_end_user_object Problem: - `_check_end_user_budget()` was called inside `get_end_user_object()` - This caused budget checks to run BEFORE `skip_budget_checks` could be evaluated - Zero-cost models (e.g., local vLLM) were incorrectly blocked when end-users exceeded their budget, even though they should bypass budget checks Solution: - Remove `_check_end_user_budget()` calls from `get_end_user_object()` - Budget enforcement now happens exclusively in `common_checks()` where `skip_budget_checks` context is available - `get_end_user_object()` keeps `route` as optional in function parameter for backwards compatibility and future implementation. * refactor(tests): update budget enforcement tests to reflect changes in get_end_user_object - test_get_end_user_object() verifies data fetching - test_check_end_user_budget() verifies enforcement - test_budget_enforcement_blocks_over_budget_users() integrates _check_end_user_budget() - test_resolve_end_user_reraises_budget_exceeded() is now test_resolve_end_user since no budget exceeded is thrown in get_end_user_object() * Gemini /images/generate and /images/edits billing fixes + add support for size and aspect ratio params (#29534) * Fix Gemini image config mapping * Address Gemini image config review * Format Gemini image generation transform * Fix Gemini image token usage logging * Share Gemini image request helpers * Fix Gemini Imagen model routing * Fixes as per self code review * Fixes per internal code review * Stop gating Imagen imageSize forwarding * Document Gemini image size mapping source * chore: retrigger lint * Clarify Gemini candidate count precedence * Add Inception provider (#29522) * add inception as provider (chat, fim) * linting * seperate test suite for chat and fim * fix test coverage * fix: model hub custom pricing model info (#29293) * Opik user auth key metadata extractors (#28397) * fix: enhance Opik metadata extraction to include user API key auth context fixed after refactoring to extractor logic * test: add unit tests for OPik metadata extraction logic * fix: enhance extract_opik_metadata function to prioritize metadata sources for improved accuracy * fix(ci): clarified comments and edited unit tests * test: add unit tests for OPik metadata extraction with auth and requester overrides * fix(ui): replace fixed favicon.ico with current api get /get_favicon (#29532) Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar> * fix(vertex/gemini): keep tool_call reference when a text-only assistant message follows (#29561) `_gemini_convert_messages_with_history` tracks `last_message_with_tool_calls` so a following tool result can be matched back to its tool call. The assignment was inside a branch guarded by `assistant_msg.get("tool_calls", []) is not None`, which is also True for a text-only assistant message (an empty list is not None). As a result, an assistant message with no tool calls that appears between a tool call and its tool result overwrote the reference, and conversion failed with: Exception: Missing corresponding tool call for tool response message. This shape is common: a model emits a short narration/assistant message after a tool call before the tool result is appended. Only update `last_message_with_tool_calls` when the assistant message actually carries tool_calls (or a function_call). Adds a regression test. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models (#28572) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models The 1-hour prompt-cache write tier (`cache_creation_input_token_cost_above_1hr`) was added to the us./global. variants of the Claude 4.5/4.6/4.7 family on Bedrock, but the eu./au./jp. cross-region inference profiles were left without it. AWS Bedrock pricing applies the same +10% regional premium across all geo profiles, so eu./au./jp. should carry the same 1-hour rates as us. (1.6x the 5-minute regional rate). Without these fields, cost tracking on EU/AU/JP Bedrock 1-hour-TTL prompt caching falls back to the 5-minute write rate and undercounts spend by ~60% for European, Australian, and Japanese tenants. Adds the 1-hour tier (and Sonnet 4.5's long-context >200K tier where AWS publishes one) to 14 regional Bedrock entries in both `model_prices_and_context_window.json` and the bundled `model_prices_and_context_window_backup.json`: - eu./au. Opus 4.6 ($11.00 / MTok) - eu./au. Opus 4.7 ($11.00 / MTok) - eu./au./jp. Sonnet 4.6 ($6.60 / MTok) - eu./au./jp. Sonnet 4.5 ($6.60 / MTok regular, $13.20 / MTok LC) - eu./au./jp. Haiku 4.5 ($2.20 / MTok) Also extends `tests/test_litellm/test_bedrock_anthropic_1hr_cache_pricing.py` with a `REGIONAL_EXPECTED` parametrized block covering all 13 new entries plus the existing 1.6x ratio invariant. Note: `eu.anthropic.claude-opus-4-5-20251101-v1:0` carries the wrong 5m rate today (base 6.25e-06 instead of regional 6.875e-06), which would break the 1.6x ratio check. It is intentionally left out of this PR so the scope stays "1-hour cache tier addition" — a separate follow-up should correct the EU 5m rates for Opus 4.5. --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * Add 1-hour cache write pricing tier for Vertex AI Anthropic models (#28569) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add 1-hour cache write pricing tier for Vertex AI Anthropic models GCP Vertex AI publishes a separate 1-hour cache write column for the Claude family (1.6x the 5-minute write rate, matching the documented Bedrock ratio). LiteLLM's Vertex AI Anthropic entries only carry the 5-minute tier, so any request that uses `cache_control: {"ttl": "1h"}` on Vertex AI Claude is undercounted in cost tracking by ~60%. The runtime side already supports the 1-hour tier — `VertexAIAnthropicConfig` extends `AnthropicConfig`, populating `ephemeral_1h_input_tokens`, and `_calculate_cache_creation_cost` reads `cache_creation_input_token_cost_above_1hr`. Only the price registry was missing data. Adds the field to 19 vertex_ai/claude-* entries across both `model_prices_and_context_window.json` and the bundled `model_prices_and_context_window_backup.json`: - Haiku 4.5 ($1.25 -> $2.00 / MTok) - Sonnet 3.7 / 4 / 4.5 / 4.6 ($3.75 -> $6.00 / MTok) - Opus 4.5 / 4.6 / 4.7 ($6.25 -> $10.00 / MTok) - Opus 4 / 4.1 ($18.75 -> $30.00 / MTok) Adds `tests/test_litellm/test_vertex_anthropic_1hr_cache_pricing.py` mirroring the Bedrock equivalent — pins each (5m, 1h) pair per model and asserts the 1.6x ratio across the family. Fixes #27781. --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * Fix Gemini multimodal function responses (#29325) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * address greptile review: add _transform_image_usage method and model-map supports_image_size flag - Add _transform_image_usage instance method to GoogleImageGenConfig that delegates to transform_gemini_image_usage, fixing the regression test - Replace hardcoded "2.5-flash" string check in supports_gemini_image_size with a get_model_info lookup on supports_image_size (default true) - Add supports_image_size: false to all gemini-2.5-flash model entries in model_prices_and_context_window.json so capability is controlled via the model map rather than embedded in code * fix test failures: schema validation, mypy type, model info plumbing, pricing test - Add supports_image_size to ModelInfoBase TypedDict so get_model_info surfaces it - Pass supports_image_size through _get_model_info_helper constructor call - Fix supports_gemini_image_size to use value is not False (None means unset, defaults to True) - Add supports_image_size to JSON schema in test_aaamodel_prices_and_context_window_json_is_valid - Correct gemini-3.1-flash-lite pricing assertions in test to match JSON values * Add Azure AI Kimi K2.6 metadata (#27052) * Add Azure AI Kimi K2.6 metadata * Scope Kimi metadata test cost map setup * fall back to substring check for models not in model_prices_and_context_window.json Models like gemini-2.5-flash-image-preview are not in the pricing JSON, so get_model_info raises. Fall back to "2.5-flash" not in model when the JSON has no explicit supports_image_size entry for the model. * fix(inception): don't forward global litellm.api_key to Inception FIM Match the Inception chat config: resolve only an Inception-specific key (param, litellm.inception_key, or INCEPTION_API_KEY) for the text-completion FIM path. The global litellm.api_key (often an OpenAI key) was both leaking to api.inceptionlabs.ai and taking precedence over the configured Inception key when set. * fix(auth): enforce end-user budget on custom-auth path that skips common_checks get_end_user_object() no longer raises BudgetExceededError, so custom-auth deployments with custom_auth_run_common_checks unset (which skip the centralized common_checks gate) stopped enforcing the end-user budget, letting an over-budget end user keep making requests. Re-enforce the budget in _run_post_custom_auth_checks on that path. --------- Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar> Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com> Co-authored-by: aneeshsangvikar <aneeshsangvikar@fiddler.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com> Co-authored-by: Suleiman Elkhoury <108065141+suleimanelkhoury@users.noreply.github.com> Co-authored-by: Dmitriy Alergant <93501479+DmitriyAlergant@users.noreply.github.com> Co-authored-by: Yanis Miraoui <yanis.miraoui19@imperial.ac.uk> Co-authored-by: Lovro Seder <vrovro@gmail.com> Co-authored-by: Thomas Mildner <12685945+Thomas-Mildner@users.noreply.github.com> Co-authored-by: José Luis Di Biase <josx@interorganic.com.ar> Co-authored-by: Lai Quang Huy <64073540+1qh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: ZHONG Ziwen <67355585+zzw-math@users.noreply.github.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
d991c47018 |
fix(ui/agents): make A2A skill tags enterable and validated (#29512)
* fix(ui/agents): make A2A skill tags enterable and validated
Skill tags were marked required but rendered as a comma-split text input
that couldn't surface validation and let empty values save. Switch tags
and examples to Select tag inputs, drop the misleading "Required" skills
label (the API allows zero skills), and validate the full configure step
so an added skill must be complete before advancing.
Resolves LIT-3153
* fix(ui/agents): allow Enter to create skill tags/examples
Drop open={false} from the tags and examples Select inputs. With the
dropdown forced closed, AntD suppresses the "create from input" option,
so pressing Enter (as the placeholder instructs) did nothing. Matches the
existing extra_headers Select.
|
||
|
Sameer Kankute
|
ae7ac72331 |
feat(agents): add LangFlow agent provider with A2A session bridging (#28963)
* feat(agents): add LangFlow agent provider with A2A session bridging Register LangFlow as a completion provider and agent type (UI + /api/v1/run), and map A2A contextId to LangFlow session_id for multi-turn conversations. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(providers): document langflow in provider_endpoints_support.json Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agents): address Greptile review for LangFlow integration Move A2A contextId→session_id mapping into LangFlow A2A provider config, add langflow.svg logo, remove live integration test, use model for token count. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(langflow): prevent flow_id override via request optional_params Derive flow_id only from the authorized model name and reject flow_id kwargs so callers cannot invoke a different LangFlow run endpoint. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(langflow): remove redundant flow_id branch in _get_flow_id * fix(langflow): surface an error when the run response has no extractable message Previously the response parser returned the raw JSON blob as the assistant message when it could not find message text, silently presenting an unparseable payload as a valid answer. It now returns None and the caller raises a LangFlowError so the failure is visible to the client. * fix(langflow): URL-encode flow_id path segment to prevent path injection flow_id is taken from the model suffix and interpolated into /api/v1/run/{flow_id}. Without path-segment encoding a model such as langflow/../../x (or one containing ?) could move the request off the run endpoint to another path on the configured LangFlow server using the operator x-api-key. Encode the segment with quote(safe="") so it always stays a single path segment. * fix(langflow): reject empty flow_id from model name * fix(langflow): return stripped flow_id so validation matches URL path * fix(langflow): reject caller-supplied tweaks to prevent flow component override * fix(langflow): reject caller-supplied tweaks injected via extra_body The transform_request guard only inspected optional_params, but extra_body is popped before transform_request runs and merged into the request body afterward, letting a caller reintroduce tweaks and override the operator-configured LangFlow flow components. Validate the final request body in sign_request so tweaks cannot reach LangFlow through extra_body. * test(langflow): move provider tests into mirrored coverage path The langflow tests lived under tests/llm_translation/, whose CircleCI job runs without --cov and uploads nothing to Codecov, so none of the new langflow code counted toward patch coverage (codecov/patch reported 9.78% of the diff hit against a 70.83% target). Relocate them to tests/test_litellm/llms/langflow/, which the GitHub Actions provider job runs with --cov=./litellm and uploads, and add regression tests for the previously untested happy paths (transform_response building the ModelResponse with usage, non-JSON body handling, last-user message extraction, outputs-dict response shape, sign_request pass-through, error class and stream flags). Patch coverage on the diff is now ~88%. * fix(langflow): require litellm_params in A2A config instead of silent empty fallback * fix(langflow): scope A2A session_id to the authenticated key The LangFlow A2A bridge used the LangFlow session_id verbatim from the client-controlled A2A contextId, so two distinct virtual keys authorized for the same agent could read or append to each other's LangFlow conversation memory by reusing a contextId. Hand the authenticated key hash to the completion bridge through litellm_params and namespace the forwarded session_id with it. The same key keeps a stable session across turns, while different keys can no longer collide on a shared contextId. The principal is hashed before it is embedded in the session_id, so the stored token is never sent to the LangFlow backend; the original contextId is preserved as a suffix for operator-side correlation. * fix(langflow): wire authenticated key hash through A2A bridge and tests Define A2A_USER_API_KEY_HASH_PARAM in the completion bridge handler, strip it before litellm.acompletion, inject the authenticated key hash at the proxy A2A endpoint, and add regression tests for per-key LangFlow session scoping. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
6d6eda8101 |
[internal copy of #28008] Support MCP OAuth passthrough and issuer-scoped JWT auth (#28356)
* fix(proxy): point /metrics 401 at the opt-out flag Operators upgrading past |
||
|
Sameer Kankute
|
b7bbddbd4d |
fix(mcp): clear allowed_tools and tool overrides on MCP server edit (#29411)
* fix(mcp): clear allowed_tools and tool overrides on MCP server edit Send empty arrays/objects from the dashboard instead of null, coerce legacy null payloads before Prisma, and stop auto-selecting all tools when the stored allowlist is empty. Co-authored-by: Cursor <cursoragent@cursor.com> * style(mcp): simplify CRUD panel value ternary per review Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): enforce empty tool allowlist when cleared in dashboard Set mcp_info.tool_allowlist_enforced on UI save so [] blocks all tools while legacy servers with default [] remain unrestricted. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix legacy MCP tool allowlist edit state * test(mcp): pin allowlist fields on mock server in tools test MagicMock auto-attributes are truthy and trigger server_applies_tool_allowlist after the empty-allowlist enforcement change. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): avoid locking legacy servers on quick edit save Only set tool_allowlist_enforced when already enforced or the user selected tools; skip allowlist fields on save for unrestricted servers; do not auto-select all tools when editing legacy servers before load. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): type mcp_info base for allowlist flag read Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): use MCPInfo type for tool_allowlist_enforced in edit save Co-authored-by: Cursor <cursoragent@cursor.com> * Update ui/litellm-dashboard/src/components/mcp_tools/mcp_server_edit.tsx Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * Remove unused MCP allowlist variable * Fix MCP legacy tool state display * Fix legacy MCP tool allowlist saves * fix(mcp): enforce allowlist when create flow deselects all tools Track explicit allowlist interaction in the create form so deselecting every tool persists tool_allowlist_enforced=true. Previously an empty selection sent the flag as false with allowed_tools=[], which the proxy treats as allow-all, contradicting the UI's 0 tools enabled state. This mirrors the existing edit-flow handling. * fix(mcp): enforce disallowed_tools on REST listing and keep restored tool selection on legacy edit --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Sameer Kankute
|
e8fcb01215 |
Litellm OSS Staging (#29161)
* Cato Networks guardrail, based on Aim (#26597) * Aim was acquired by Cato Networks, creating Cato Networks guardrail based on Aim * Add more tests * Move test so they are reached by codecov coverage * base URL trailing slashes * Support Lemonade runtime context metadata (#28135) * Support Lemonade runtime context metadata * Add provider hook for runtime model metadata * Address provider model info review feedback Keep the runtime model info hook duck-typed instead of extending the base model-info class, and avoid importing ModelInfoBase from Ollama common utilities to reduce CodeQL cyclic-import noise. Co-authored-by: openhands <openhands@all-hands.dev> * Fix CI after staging rebase Relax the Ollama runtime metadata return annotation to match the provider-hook dict response and update the Google Interactions OpenAPI status expectation for the current live spec. Co-authored-by: openhands <openhands@all-hands.dev> * Normalize Lemonade runtime model metadata * Avoid leaking Ollama metadata auth * Avoid leaking Lemonade metadata auth --------- Co-authored-by: Graham Neubig <398875+neubig@users.noreply.github.com> Co-authored-by: openhands <openhands@all-hands.dev> * fix(cato): address guardrail review feedback Use proxy-authenticated user identity, forward moderation hook return values, and ensure streaming sender tasks are cancelled and awaited on exit. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(vertex_ai): route google/gemma-*-maas through partner-models OpenAI path - clone of #28010 (#28846) * fix(vertex_ai): route google/gemma-*-maas through partner-models OpenAI path Fixes #26083 vertex_ai/google/gemma-4-26b-a4b-it-maas previously fell through to the NON_GEMINI route. Per owtaylor's plan on #26083: add the google/gemma- prefix to PartnerModelPrefixes so is_vertex_partner_model picks it up and should_use_openai_handler routes it to the OpenAI-compatible /endpoints/openapi/chat/completions URL. No gemma-detection exclusion needed (the "gemma/" check uses a slash, which google/gemma-... doesn't match). No OpenAIGPTConfig subclass needed — works with the base handler. * fix(vertex_ai): mark gemma-4-26b-a4b-it-maas as vision-capable (empirically verified) * fix(vertex_ai): address greptile feedback — provider category, canonical URL, sync backup * test(vertex_ai): add function-calling and vision pass-through tests for Gemma MaaS Addresses oss-pr-review-agent-shin feedback on PR #28010: supports_function_calling, supports_tool_choice, and supports_vision were marked true but had no tests proving the payloads actually reached the OpenAI-compatible endpoint. Added: - test_gemma_maas_supports_function_calling — verifies the utility returns True when the model_cost entry carries supports_function_calling=true - test_gemma_maas_supports_vision — same for supports_vision - test_vertex_ai_gemma_function_calling_passthrough — verifies tools + tool_choice appear in the JSON body POSTed to /endpoints/openapi/chat/completions - test_vertex_ai_gemma_vision_passthrough — verifies image_url content parts survive transformation and reach the global endpoint URL * fix: Delete uv.lock * test(vertex_ai): add function-calling and vision pass-through tests for Gemma MaaS Addresses oss-pr-review-agent-shin feedback on PR #28010: P1 (patch target): Added a comment explaining why patching litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler is correct — get_async_httpx_client() (defined in http_handler.py) instantiates AsyncHTTPHandler within that module's scope, so the definition-site patch intercepts it. Without the mock the test raises AuthenticationError, confirming it never silently passes. P2 (partner-provider regression guard): Added test_gemma_routes_through_openai_handler() which calls VertexAIPartnerModels.should_use_openai_handler() directly, so if Gemma's routing to VertexPartnerProvider.llama ever changes the URL-shape tests below it become a real regression guard rather than an unanchored unit test. Also added: - test_gemma_maas_supports_function_calling / supports_vision — capability flag checks via patch.dict(litellm.model_cost) - test_vertex_ai_gemma_function_calling_passthrough — tools + tool_choice forwarded in the request body - test_vertex_ai_gemma_vision_passthrough — image_url part survives transformation to the global endpoint Added: - test_gemma_maas_supports_function_calling — verifies the utility returns True when the model_cost entry carries supports_function_calling=true - test_gemma_maas_supports_vision — same for supports_vision - test_vertex_ai_gemma_function_calling_passthrough — verifies tools + tool_choice appear in the JSON body POSTed to /endpoints/openapi/chat/completions - test_vertex_ai_gemma_vision_passthrough — verifies image_url content parts survive transformation and reach the global endpoint URL * fix: proper patch for unit tests --------- Co-authored-by: Iana <iana@Shivakumars-MacBook-Pro.local> * fix(cato): guardrail all completion choices on output When n > 1, only choices[0] was analyzed and redacted. Iterate every Choices entry so block and anonymize actions apply to all completions. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix review * fix(cato_networks): harden output anonymize handling and restructure nested UI routes Guard against empty redacted_output and empty all_redacted_messages from Cato. Restructure nested admin UI HTML exports to index.html so extensionless routes work. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix mypy * fix(cato): guard missing policy_drill_down and all_redacted_messages keys * fix(cato): avoid KeyError bypassing block action on missing analysis_result * fix(cato): preserve non-text message fields during anonymize Rebuild redacted messages from the original messages, overwriting only content, so tool_calls, tool_call_id, name and multimodal fields survive the anonymize action. * fix(cato): preserve trailing messages when fewer redacted messages returned Avoid silently truncating the conversation in _anonymize_request when Cato returns fewer redacted messages than were sent, and isolate the no-api-key config test from a pre-existing CATO_API_KEY environment variable. * fix(cato,model-info): preserve stream block signal on sender teardown; forward api_key in dynamic model-info lookup Suppress ConnectionClosed (alongside CancelledError) when tearing down the Cato streaming sender task so a backend ConnectionClosed cannot mask the original StreamingCallbackError (e.g. a guardrail block) raised by the receive loop. Thread api_key through get_model_info -> _get_model_info_helper so an explicit key reaches a provider's dynamic get_model_info for a caller-supplied api_base. Previously only api_base was forwarded, so authenticated Ollama and Lemonade servers at a custom base could only be queried unauthenticated. * fix(cato): surface mid-stream forwarding errors instead of blocking on recv If the upstream LLM stream errors mid-flight, the sender task dies before sending the terminal done frame, so the consumer would block on websocket.recv() until Cato closes the connection. Race recv against the sender task and raise the stored sender exception promptly as a StreamingCallbackError. * fix(cato): drop spoofable end_user_id from guardrail user identity Only the key/JWT-bound user_email is a trusted identity. end_user_id is resolved from caller-supplied request fields (OpenAI user param, headers, metadata), so an authenticated caller with no bound user_email could set it to another user's email and have LiteLLM forward x-cato-user-email for that victim, poisoning Cato audit and policy attribution. Forward only user_email and omit the header otherwise. * fix(cato): harden output anonymize path against missing content key * fix(cato): fall back to original message when redacted content key is missing * refactor(model-info): drop unused api_key from cached model-info helper _cached_get_model_info_helper is only called by the cost-tracking hot path, which never authenticates, so the api_key parameter was never populated. Keeping it in the lru_cache key offered no benefit and risked fragmenting the high-RPS cache and retaining credential strings per entry. * fix(cato): preserve None content on tool-call-only choices in output hook * fix(ollama): respect static-model guard in OllamaConfig.get_model_info Delegate to OllamaModelInfo.get_model_info so statically-priced Ollama models short-circuit before the /api/show network call instead of hitting the server unconditionally. * fix(lemonade,ollama): treat empty api_key as unset to avoid leaking server creds An empty-string api_key was treated as an explicit key, so it passed the guard meant to keep server-side credentials off caller-supplied bases and then fell back through the env/global key chain. A caller could point api_base at a server they control and send api_key="" to receive the configured provider key in the Authorization header. Gate the credential fallback on the api_key being truthy instead of merely not-None. * fix(cato): inspect and redact Responses-API input, not just messages The guardrail only read data["messages"], so /v1/responses requests, which carry their text in data["input"], reached Cato as an empty message list and bypassed inspection entirely. Send build_inspection_messages(data) so both shapes are analyzed, and write anonymized results back with apply_redacted_messages_back when the request used input. * perf(utils): keep api_key out of get_model_info lru_cache key * fix(cato): propagate ssl_verify to streaming WebSocket connection The streaming hook applied ssl_verify only to the HTTP handler; the websockets.connect() call used default verification, so a custom Cato instance behind TLS with a self-signed cert worked for non-streaming calls but failed every streaming request. Resolve the ssl_verify setting into the connect() ssl argument, mirroring the HTTP handler. * refactor(utils): rename shadowing local in _get_model_info_helper * fix(cato): flatten multimodal chat content before inspection Chat Completions requests whose message content is a multimodal parts array were posted to Cato as the raw OpenAI parts, so text inside content: [{"type":"text", ...}] reached the model without Cato ever inspecting the string. Flatten each message's list content to plain text while keeping the list 1:1 with the request so the index-based redaction write-back stays valid; Responses-API input requests still go through build_inspection_messages. * test(lemonade): clear get_model_info cache around api_base test * fix(cato): inspect and redact Responses-API input even when messages present _inspection_messages returned early once messages was non-empty, so a /v1/responses caller could place benign text in messages and disallowed text in input and have only messages reach Cato while the model used input. Inspect both fields and write anonymize redactions back to input as well as the index-aligned messages. * test(log_db_metrics): assert table_name event_metadata contract log_db_metrics now emits minimal event_metadata via _safe_db_event_metadata (table_name only, function_name/function_kwargs/function_args dropped as redundant with call_type and unsafe to stamp on a span). The success-path test still asserted function_name membership and crashed with TypeError on the None metadata returned when no table_name is passed. Pass a table_name and assert the surfaced contract instead. * fix(cato): inspect and redact completion prompt and Responses-API instructions The Cato guardrail only inspected chat messages and the Responses-API input field, so blocked text placed in the legacy /v1/completions prompt or the /v1/responses instructions field reached the model without ever being sent to Cato. Both fields are now appended as synthetic inspection messages, and the anonymize path slices Cato's redactions back to the field they came from. * fix(cato): serialize non-str/bytes websocket chunks before forwarding * fix(cato): inspect tool descriptions and tool-call arguments * fix(cato): map redacted output by assistant index; restore get_model_info.cache_info * fix(cato): block output even when detection_message is null/empty A block_action returned by Cato on the output hook whose detection_message was null or empty was let through to the caller: the truthiness guard on detection_message skipped the HTTPException and the unblocked response was returned. Raise the HTTPException directly in _handle_block_action_on_output so the output path blocks unconditionally, mirroring the input path. * fix(cato): inspect and redact nested tool param and legacy function descriptions Tool/function parameter descriptions and the legacy functions[] array are forwarded to the model but were not seen by Cato, so blocked text hidden there bypassed inspection and anonymization. Recursively walk every description string in tools[].function and functions[] schemas for both the analyze payload and the anonymize write-back. * fix(cato): traverse schema descriptions iteratively to satisfy recursive detector The nested walk() generator recursed over tool/function JSON schemas with no depth bound, which the recursive_detector code-quality gate rejects. Replace it with an explicit-stack DFS that yields the same (container, key) refs in the same pre-order, so schema description redaction is unchanged. * fix(cato): inspect and redact response_format JSON schema descriptions response_format json_schema descriptions are forwarded to the model, so blocked text hidden in nested schema descriptions could bypass Cato inspection and redaction. Extend the schema-description walk to cover response_format alongside tools and legacy functions. * fix(cato): skip output rewrite when Cato returns no redaction Return None from call_cato_guardrail_on_output on monitor/no-action so the post-call hook only mutates the message when there is an actual redaction, instead of redundantly re-writing the original content. * refactor(utils): resolve explicit api_key model info without the cache Move the model-info build into a non-cached _build_model_info helper and drop api_key from the lru-cached _cached_get_model_info signature. Both cached helpers now take the same (model, provider, api_base) key and never forward api_key, while explicit per-caller keys are resolved through the builder directly instead of reaching into the cache wrapper's __wrapped__. * fix(cato): inspect and redact non-description schema string values Tool, function and response_format JSON schemas forward more than just description text to the model. enum, const, default, examples and title values are sent verbatim, so blocked content hidden in any of them bypassed Cato inspection and redaction. Walk those schema string values alongside descriptions on both the inspection and anonymize paths. * fix(model-info): surface swallowed dynamic model-info errors The provider-specific get_model_info dispatch falls back to the static cost map when a provider's dynamic lookup raises, which is intentional graceful degradation. Previously the exception was discarded with a bare debug line, so a real failure (e.g. a provider whose get_model_info signature does not accept api_key) was invisible. Log the exception at warning level with the model and provider context so the fallback is diagnosable. * fix(cato): inspect and redact Responses API output in post-call hook The post-call success hook only handled ModelResponse, so /v1/responses (which returns a ResponsesAPIResponse) bypassed the Cato output guardrail. Extract and inspect/redact every output_text content block and function-call arguments string, blocking on a block action, so generated text cannot escape inspection by using the Responses API. * chore: reset _experimental/out folder * chore(ui): remove orphaned prebuilt dashboard chunk files The _experimental/out manifests are byte-identical to the base branch, so the served dashboard already matches base. 436 unreferenced Next.js chunk files had accumulated in the directory and are not loaded by any manifest; removing them restores the committed UI artifacts to the base build and drops the artifact churn from this PR's diff. * fix(guardrails,ollama): forward ssl_verify to Cato init and raise_for_status on /api/show --------- Co-authored-by: Alex Yaroslavsky <trexinc@gmail.com> Co-authored-by: Graham Neubig <neubig@gmail.com> Co-authored-by: Graham Neubig <398875+neubig@users.noreply.github.com> Co-authored-by: openhands <openhands@all-hands.dev> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Piotr Placzko <piotr@icep-design.com> Co-authored-by: Iana <iana@Shivakumars-MacBook-Pro.local> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
609e1e9763 |
fix(ui): render caller-supplied filter options in caller order (LIT-3151) (#29462)
FilterComponent iterated a hardcoded orderedFilters whitelist instead of the options prop, so any consumer whose filter names were not on that list rendered nothing. The Tool Policies page passes "Input Policy", "Output Policy", "Team Name" and "Key Name", none of which were whitelisted, so its Filters panel opened to an empty area. Drop the whitelist and render the options the caller passes, in the order they pass them, so each page owns its own filter set and ordering. The Logs page array is reordered to match its prior on-screen order; VirtualKeys and TeamVirtualKeys already matched the old whitelist order and are unaffected. |
||
|
ryan-crabbe-berri
|
1cce49b9d0 |
fix(vector-stores): support engines URL for Vertex AI Search (#27885)
Adds optional vertex_engine_id field to vertex_ai/search_api so users can route through a Discovery Engine search app instead of the data store directly. Required for website, healthcare, and connector-based data stores that return FAILED_PRECONDITION on the existing dataStores URL. Existing data-store-direct callers are unaffected. Resolves LIT-3036 |
||
|
yuneng-jiang
|
54ed5a4eb5 |
fix(e2e): tolerate trailing slash in SERVER_ROOT_PATH login redirect (#29369)
The Next.js admin UI is exported with trailingSlash: true, so the proxy
serves /ui/login at /ui/login/index.html and 308s /ui/login → /ui/login/.
The waitForURL predicate used endsWith("/ui/login"), which never matched
the canonicalized URL and timed out after 15s.
This was masked until the build artifacts were regenerated against the
AuthContext fix: the prior bundles still hit the racy redirect path that
fired before proxyBaseUrl was populated, producing /ui/login (no prefix,
no proxy round-trip, no trailing slash) which fortuitously satisfied the
predicate. The first PR to ship the corrected bundle exposed the
assertion bug.
Switch the predicate to includes("/ui/login"); the prefix assertion below
still validates the SERVER_ROOT_PATH preservation that is the actual
contract under test.
|
||
|
yuneng-jiang
|
f0ebfb2a1b |
fix(ui): break logout redirect loop across origins (#29360)
When the user has visited both the dev UI (e.g. localhost:3000) and the
proxy UI (e.g. localhost:4000) in the same tab, logging out from the dev
origin produced an infinite logout/login redirect.
The proxy-side LoginPage's "is the user still authenticated?" check
was reading getCookie("token"), which falls back to sessionStorage when
document.cookie has no token. The cross-origin clearTokenCookies() call
from the dev origin can clear cookies on the shared hostname, but cannot
reach sessionStorage on the proxy origin (sessionStorage is per-origin),
so the fallback returned a stale token and LoginPage interpreted the
user as logged in, redirecting back to the dev origin. Dev origin then
saw no cookie and redirected to LoginPage, repeating ~20x per second.
This change introduces getCookieFromDocument(), a cookie-only read with
no sessionStorage fallback, and uses it in LoginPage's already-logged-in
check. The HttpOnly-reverse-proxy defense from PR #23532 is unaffected:
storeLoginToken still writes both the JS cookie at /ui and the
sessionStorage backup, and getCookie still falls back for callers that
want the full read path.
|
||
|
ryan-crabbe-berri
|
a9cc6ed68c |
test(e2e): cover PROXY_LOGOUT_URL redirect on Logout (#29080)
* test(e2e): cover PROXY_LOGOUT_URL redirect on Logout Env-gated spec mirroring the existing serverRootPathRedirect pattern: when the proxy is booted with PROXY_LOGOUT_URL set, clicking Logout in the navbar must navigate to that external URL. The standard run_e2e.sh exports an empty value so the rest of the suite is unaffected; this spec self-skips unless the env var is populated. * test(e2e): run PROXY_LOGOUT_URL spec in the suite + harden logout assertions Boot the e2e proxy with PROXY_LOGOUT_URL set (job-level env in CircleCI and run_e2e.sh) so proxyLogoutUrl.spec.ts actually runs instead of self-skipping. Nothing else in the suite performs a logout, so this only affects the behavior under test. Harden the spec to verify the logout flow rather than a URL substring: - wait for /sso/get/ui_settings before clicking so logoutUrl is populated (otherwise window.location.href = "" silently reloads same-origin) - assert a token cookie exists first, and is cleared after logout - locate the dropdown via getByRole instead of internal antd CSS classes - stub the external destination and assert on URL origin + path prefix * test(e2e): assert exact PROXY_LOGOUT_URL on logout redirect Replace the origin + startsWith(pathname) checks with a single normalized href comparison. With PROXY_LOGOUT_URL=https://www.example.com the path was "/", so startsWith("/") matched any path and left path/query/hash unchecked. Comparing normalized hrefs pins scheme, host, port, path, query and hash while still tolerating the browser's trailing-slash/default-port normalization. |
||
|
ryan-crabbe-berri
|
90b5104475 |
feat(mcp/auth): additive key access-group grants + opt-in member assignment (#29313)
* fix(mcp): make key.access_group_ids grants additive over team ceiling A key whose unified access_group_ids grant a private MCP server was having that grant intersected against its team's MCP ceiling, so a key in a team scoped to other servers (or with no own scope) lost the granted server entirely. Resolve access_group_ids once as ungated additive grants and union them on top of the key/team ceiling instead of folding them into the key scope that gets intersected. * test(mcp): align key access-group tests with additive-grant model The previous commit moved key.access_group_ids resolution out of the intersected key ceiling (_get_allowed_mcp_servers_for_key) and into the ungated additive grant path (_get_key_access_group_mcp_server_extras), unioned on top of the team ceiling. Five tests from #28890/#29195 still asserted the old gated / in-key-scope contract and failed: - _get_allowed_mcp_servers_for_key now returns the object_permission ceiling only and never resolves access_group_ids; two tests now assert the group resolver is not called from that path (with and without an object_permission present). - The extras path is ungated, so a group whose assigned_team_ids / assigned_key_ids exclude the caller still contributes its servers. - The end-to-end test asserts the grant surfaces via the extras path rather than the base key path. - Dropped test_key_access_group_ids_empty_returns_no_extras; the empty case is already covered by the extras family's no-groups test. * feat(auth): gate member access-group assignment on keys behind opt-in Non-admin team members could attach access_group_ids to keys they create or update, letting them self-grant resources (MCP servers/models) the team admin never intended. Add an opt-in KEY_ACCESS_GROUP_ASSIGNMENT team-member permission (default-deny) enforced at /key/generate and /key/update; proxy and team admins bypass. Surfaces automatically as a checkbox in the team Member Permissions UI. * fix(auth): gate access-group assignment on /key/regenerate too RegenerateKeyRequest inherits access_group_ids and prepare_key_update_data persists it, so a non-admin key owner could self-grant access groups by regenerating. Apply the same opt-in member gate using the existing key's team. * test(auth): cover member access-group gate and additive MCP grants Add unit tests for enforce_member_can_assign_access_groups (deny without opt-in, allow with opt-in, and proxy-admin / team-admin / non-team-key bypasses) and for _get_key_access_group_mcp_server_extras (no-auth and no-resolved-servers return empty, resolved ids are expanded, errors degrade to no grants). |
||
|
ryan-crabbe-berri
|
37e6e2da1c |
test(e2e): assert internal-user navbar identity is scoped to that user (#29077)
* test(e2e): assert internal-user navbar identity is scoped to that user The existing login.spec.ts only checks the admin's navbar identity. This adds the symmetric check for the internal user — verifying the account button + dropdown surface the internal user's email, id, and role, and that no admin-scoped values leak through. * test(e2e): harden navbar identity test per review feedback Locate the user dropdown panel by a data-testid on the popupRender div instead of Ant Design internal + Tailwind class names, so styling refactors no longer risk breaking the identity-scoping assertions. Source the seeded user emails/ids from shared constants (match seed.sql) instead of hardcoding them inline. |
||
|
ryan-crabbe-berri
|
892838963c |
test(e2e): cover Internal User create-key flow when in no teams (#29083)
* test(e2e): cover Internal User create-key flow when in no teams The seeded e2e-internal-user is in two teams, so the "no team" branch of the Create Key modal — where the team dropdown must render empty — was unreachable. Seeds a noteam@test.local user and adds a spec that logs in fresh, opens the modal, and asserts the dropdown has zero options. * test(e2e): harden no-team dropdown assertion + add with-teams counterpart Replace the one-shot count() check with a settled-empty assertion: wait for the dropdown's loaded "No teams found" state before asserting zero options, so the test can't pass on a transient empty frame while the team-options request is still in flight. Add internalUserWithTeams.spec.ts as the differential partner; it logs in as the seeded e2e-internal-user (two team memberships) and asserts the dropdown lists exactly those teams. Without it, the no-team spec's zero-options assertion would still pass against a regression that empties the dropdown for every user. |
||
|
Sameer Kankute
|
af17400c38 |
feat(a2a): well-known agent-card discovery + LangGraph Platform mode (#28860)
* feat(a2a): well-known agent-card discovery + LangGraph Platform mode Adds a registration-time discovery flow so admins can paste an upstream agent URL, see its skills/capabilities, pick what to expose, and have the proxy front it with a LiteLLM-shaped agent card. Backend (new litellm/proxy/a2a/ module): - fetch_well_known_card walks /.well-known/agent-card.json, /.well-known/agent.json, /agent.json by default. langgraph_platform mode hits the canonical path with ?assistant_id=<id> (LangGraph serves one shared endpoint per deployment). - merge_agent_card overlays LiteLLM overrides on the upstream card: drops upstream url, forces protocolVersion=1.0, replaces securitySchemes with LiteLLMKey bearer, emits supportedInterfaces pointing at the proxy, filters capabilities to a small allowlist, strips non-v1.0 fields. - POST /v1/a2a/discover returns the raw upstream card (admin-only) so the UI can render skills/capabilities for selection. - create/update/patch agent endpoints pre-generate the agent_id and run merge_agent_card before storing, so DB.agent_card_params already embeds the proxy-fronted URL. UI (ui/litellm-dashboard): - New AgentCardDiscovery component with a parent-driven plan: discovery_mode + params + display URL. For LangGraph the parent composes (api_base, assistant_id); for pure A2A it uses the url field. Component hides the manual URL input when the parent drives. - add_agent_form wires discovery for every non-custom agent type and overlays the user's selections onto agent_card_params at submit, fixing the bug where dynamic agent forms ignored discovery picks. Completion-bridge fixes (paired): - Add kind: "message" to A2A response messages and unwrap result so it's a Message directly per spec (matches a2a SDK SendMessageResponse validation). - Forward A2A metadata to LangGraph runs via extra_body.metadata. * fix(a2a): preserve agent url, fix streaming chunk envelope, and protect forwarded metadata - Streaming chunk: move final out of the message object into the result envelope per the A2A spec. - Agent card merge: keep upstream url on the stored card so the runtime invocation path can locate the upstream backend; the public well-known endpoint already rewrites this field to the proxy URL before exposing it to clients. - Completion bridge: apply A2A forward metadata after merging litellm_params so an agent-configured extra_body cannot overwrite the forwarded metadata. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(a2a): fix legacy streaming chunk, agent card test, and metadata merge - providers/litellm_completion: move 'final' out of the message object into the result envelope per the A2A spec (matches the bridge fix). - agent endpoints test: the runtime invocation path now preserves the top-level 'url' on the stored card, so update the assertion to match. - completion bridge metadata: when forwarding A2A metadata via extra_body.metadata, merge into any existing extra_body.metadata instead of replacing it, so an agent-configured metadata block is preserved (forward metadata still wins on key conflicts). Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(a2a): remove dead duplicate transformation dir; drop SSRF-prone headers field from /v1/a2a/discover Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(a2a): revert accidental html→index.html rename from |
||
|
ryan-crabbe-berri
|
5eafe1c1fc |
test(e2e): cover navbar Logout flow as proxy admin (#29076)
* test(e2e): cover navbar Logout flow as proxy admin
The Logout button under the navbar User dropdown was an uncovered
manual-QA step. This test signs in as admin, opens the dropdown,
clicks Logout, then navigates to a protected page and asserts the
redirect to /ui/login — proving the session was cleared.
* test(e2e): fix logout dropdown trigger and account-menu selector
The button never rendered the literal text "User" (it shows initials +
display name), and the antd Dropdown uses trigger={["click"]}, so the
synthetic mouseover/mouseenter never opened the popup. Open it with a real
click on the button's aria-label ("Account menu — ...").
|
||
|
ryan-crabbe-berri
|
fcd5760891 |
test(e2e): cover Internal User key modal, team info, key page (#29074)
* test(e2e): cover Internal User key modal, team info, key page Three previously-uncovered manual-QA paths for the Internal User role: - Create Key modal — confirm the team dropdown is populated with the user's teams (verifying the role-scoped UI flow exists). - Team info page — confirm the Settings/Members tabs are hidden for a regular team member; only the read-only tabs render. - Virtual Keys page — confirm the proxy's internal litellm-dashboard team keys never leak into an internal user's table. * test(e2e): share clickTeamId helper, strengthen key-filter assertion Address review feedback on the Internal User e2e spec: - Extract clickTeamId into helpers/navigation.ts; import in both internalUser and teams specs instead of duplicating it. - Anchor the litellm-dashboard absence check on the user's own seeded key so it cannot pass vacuously against an empty table. - Drop redundant dismissFeedbackPopup calls (navigateToPage already dismisses internally). |
||
|
ryan-crabbe-berri
|
10bda4456a |
test(e2e): cover Internal Viewer nav, key, and team-info gating (#29075)
* test(e2e): cover Internal Viewer nav, key, and team-info gating Three previously-uncovered manual-QA paths for the Internal Viewer role: - Nav only renders the read-only sections; admin-only items (Internal Users, Organizations, Models + Endpoints) stay hidden. - Virtual Keys page hides Create New Key, and the key detail view hides Regenerate / Reset Spend / Delete actions. - Team info page hides Members and Settings tabs for the viewer. * test(e2e): scope viewer nav to sidebar, strengthen tab assertions Address review feedback on the Internal Viewer e2e spec: - Scope the nav test to the sidebar complementary landmark and match items by link role + accessible name. The prior CSS nav, aside selector grabbed the top bar (the sidebar is a complementary landmark, not a <nav> tag), so the assertions never hit the real nav links. - Land via navigateToPage so the networkidle wait settles the role-gated nav before asserting. - Assert the Virtual Keys tab is visible (was only commented). - Use toHaveCount(0) for hidden team tabs to match the nav block; tabs are conditionally rendered, not CSS-hidden. - Drop redundant dismissFeedbackPopup calls (navigateToPage already dismisses internally). |
||
|
michelligabriele
|
68852ef165 | fix(teams): expose keys_count on /v2/team/list and wire UI Resources badge (#28502) | ||
|
Mateo Wang
|
f27df8d516 |
docs: hand-written CLAUDE.md; point GEMINI.md and AGENTS.md at it (#29252)
* docs: replace generated CLAUDE.md with hand-written guidance, remove AGENTS.md Swap the auto-generated CLAUDE.md for a concise hand-written version that captures how we actually want agents to work in this repo: minimal comments, simplicity first, meaningful tests with a high mutation kill rate, PRs based off litellm_internal_staging rather than main, and curl against a live proxy as proof of fix instead of pasted pytest output. Remove AGENTS.md so there is one source of truth for agent guidance. The customer and company name confidentiality policy, along with the MCP available_on_public_internet note, are carried over from the previous CLAUDE.md. * fix: further clarify communication guidelines * docs: point GEMINI.md at CLAUDE.md instead of duplicating guidance Replace the standalone GEMINI.md copy, which had already drifted from the new CLAUDE.md, with a one-line pointer so Gemini reads the same single source of truth. * docs: simplify PR template test checklist item Replace the rigid "at least 1 test is a hard requirement" checklist line with "I have added meaningful tests", which matches the testing guidance in CLAUDE.md, and tidy a comma into a semicolon in the scope-isolation item. * docs: point AGENTS.md at CLAUDE.md instead of deleting it Keep AGENTS.md so tools that read it still resolve guidance, but collapse it to the same one-line pointer to CLAUDE.md used by GEMINI.md, keeping a single source of truth. * fix: make AI-generated rules more concise * fix: spelling Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: make the .env usage more careful * docs: restore MCP available_on_public_internet note to CLAUDE.md The PR description states this note was carried over verbatim from the previous CLAUDE.md, but it was dropped in the rewrite. Restore it so the file matches the description and the team guidance is not lost. * docs: restore browser storage and CI supply-chain safety notes to CLAUDE.md These security-relevant rules were dropped in the rewrite. Restore the sessionStorage-over-localStorage (XSS) guidance and the CI supply-chain rules (no curl|bash, pin versions, verify checksums) so agents editing UI or CI code are still steered away from those pitfalls. * docs: move area-specific guidance into nested CLAUDE.md files The MCP, browser-storage, and CI supply-chain notes are scoped to particular parts of the tree, so move each into a nested CLAUDE.md that Claude Code loads on demand when those files are touched: the MCP note under the mcp_server gateway, the browser-storage rule under the UI dashboard, and the CI supply-chain rules under .circleci. Keeps the root CLAUDE.md focused on general guidance while the area notes surface where they are relevant. * docs: keep CI supply-chain note in root CLAUDE.md CI guidance applies beyond .circleci (it also covers downloads in GitHub workflows and any CI script), and CI work does not reliably touch a single subtree, so a nested file under .circleci would not surface it dependably. Keep it in the always-loaded root instead. The MCP and browser-storage notes stay nested where they map cleanly to one area of the tree. * fix: make it clear we prefer httpOnly * chore: make ci rule more concise * chore: make concise Fix formatting and punctuation in MCP note. * fix: don't include Claude attribution --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
2bfbf14882 |
test(e2e): cover Team Admin view + member + key flows (#29072)
* test(e2e): cover Team Admin view + member + key flows
Adds a new spec exercising the previously-uncovered team-admin manual-QA
items: viewing all team keys (including other members'), adding a member,
removing a member, and creating a team key with All Team Models. Also
seeds a dedicated invitee user so the add-member test can run in parallel
with the proxy-admin invite test without colliding on the team roster.
* test(e2e): harden team-admin member specs per review feedback
Address Greptile feedback on the Team Admin spec:
- locate the delete action via getByTestId("delete-member") instead of
the fragile svg/img .last() selector
- match the seeded removable member by user_id (members_with_roles stores
no email, so the roster renders user_id)
- assert exact success-toast strings rather than broad regexes that could
match unrelated "success" text
|
||
|
ryan-crabbe-berri
|
9918a9c78c |
fix(guardrails): persist disable_global_guardrails on keys (#29233)
* fix(guardrails): restore disable_global_guardrails persistence for keys The per-key/team "Disable Global Guardrails" toggle silently stopped working after #17042, which removed `disable_global_guardrails` from the key/team request models and from the premium metadata allowlist. Without those, the UI's top-level field was dropped by pydantic and never folded into key `metadata`, so the runtime gate always read False and global default_on guardrails kept running. Restore the request-model fields (KeyRequestBase, NewTeamRequest, UpdateTeamRequest) and the `LiteLLM_ManagementEndpoint_MetadataFields_Premium` entry so the flag is promoted into metadata again. Because the key edit form always submits the flag (false by default), guard the UI so it is only sent when it actually changed (edit) or is enabled (create) — this keeps the premium gate on enabling intact while not 403-ing non-premium users who edit unrelated key fields, mirroring how guardrails/tags are already stripped. * test(guardrails): cover disable_global_guardrails toggle-off + clarify premium field comment Add a prepare_metadata_fields case asserting `disable_global_guardrails: False` overwrites an existing `True`, and rewrite the PREMIUM_METADATA_FIELDS comment to explain why boolean premium fields are excluded from the empty-value strip loop. |
||
|
ryan-crabbe-berri
|
5699a06413 |
test(e2e): cover AI Hub make-public flow and public model_hub_table (#29071)
* test(e2e): cover AI Hub make-public flow and public model_hub_table Three previously-uncovered manual-QA paths land in one spec: - Admin opens "Select Models to Make Public", advances through the multi-step modal, and verifies the success toast. - AI Hub tab strip exposes Model Hub / Agent Hub / MCP Hub / Skill Hub — note the manual-QA "Claude Code Plugin Marketplace" label was renamed to Skill Hub; the test pins the current name. - Anonymous /ui/model_hub_table loads with the master key as `?key=` and renders the Model Hub tab. Agent Hub / MCP Hub tabs are conditional on public data and are not asserted here. * test(e2e): harden AI Hub make-public + public hub assertions Address Greptile review: - Make-public test now asserts "Select All (N)" with N>=1 before clicking, so a missing-seed-data run surfaces immediately instead of timing out on the disabled Next button or the success toast. - Public model_hub_table test dismisses the feedback popup before the tab visibility assertion, matching the ordering used by navigateToPage so a popup race can't mask the tab mid-evaluation. * docs(e2e): explain admin vs public AI Hub tab asymmetry Greptile flagged the all-4-tabs assertion as a potential CI flake, inferring from the public-page comment that Agent Hub / MCP Hub might be data-conditional in the admin view too. They aren't — ModelHubTable renders all four tabs unconditionally for admins. Document the asymmetry inline so future readers (and future review passes) don't re-derive it. |
||
|
ryan-crabbe-berri
|
bc31c570f0 |
test(e2e): cover add-MCP-server flow via discovery → custom form (#29070)
* test(e2e): cover add-MCP-server flow via discovery → custom form
The "Add MCP server" manual-QA step was uncovered. This adds a test
that opens the discovery modal, jumps into the custom-server form,
fills name + Streamable HTTP transport + a placeholder URL + None
auth, submits, and verifies both the success toast and the new row.
* test(e2e): apply greptile fixes to MCP add-server test
- Anchor the auth-type Select via its enclosing Collapse panel
("Authentication") instead of the placeholder text. The Form.Item has
no label prop, so the previous `hasText: /auth type/i` filter was
matching via "Select auth type" placeholder copy — fragile.
- Document the intentional lack of teardown, matching the pattern used
in addModel.spec.ts: the e2e runner discards the DB per invocation.
Addresses Greptile P2s on PR #29070.
* test(e2e): scope MCP row assertion to the servers table
Scope the post-create row lookup to `table tbody` so the form modal's
`server_name` input — which still holds the timestamped value during
its close animation — can't satisfy the assertion before the server
actually lands in the list.
* docs(e2e): note MCP coverage scope and link to tracker
This spec only smoke-tests the happy-path Streamable HTTP + None auth
flow. Add a top-of-file comment pointing at E2E_COVERAGE.md so future
contributors can see what's still uncovered (other transports, all
auth types, edit/delete, BYOK, tool list/call, access groups).
|
||
|
ryan-crabbe-berri
|
9cac0471ae |
test(e2e): cover Team-BYOK add-model flow as proxy admin (#29068)
* test(e2e): cover Team-BYOK add-model flow as proxy admin The team-only model + team assignment was an uncovered manual-QA path. This adds a premium-gated test that toggles Team-BYOK, picks the seeded E2E Team CRUD, submits, and verifies the model lands in All Models with the team alias attached. * test(e2e): apply greptile fixes to Team-BYOK test - Add the 2s networkidle settle that the sibling addModel tests use — networkidle fires before the All Models table finishes re-rendering, so the search input was racing with the render. - Assert on `models-results-count` before inspecting the table body so an empty search result fails with a clear "expected results count" message instead of timing out on a missing row. Addresses Greptile P2s on PR #29068. * test(e2e): harden Team-BYOK test against flake and stale state - Add before/after cleanup that deletes any Cohere model already scoped to e2e-team-crud via /v2/model/info + /model/delete, so Playwright retries and local reruns don't accumulate rows. - Pick the team from the dropdown by role/option name instead of a global getByText match — avoids matching a previously-rendered tag elsewhere in the form. - Scope the "created successfully" assertion to .ant-notification so a stale toast from an earlier test in the same browser context can't vacuously satisfy it. - Tighten the All Models assertion: require a single row that contains BOTH the cohere model name AND the e2e-team-crud alias, so the team-less wildcard from the sibling "Add wildcard route" test can't satisfy the check. |
||
|
ryan-crabbe-berri
|
b0ea013042 |
test(e2e): cover add-fallback flow in Router Settings (#29069)
* test(e2e): cover add-fallback flow in Router Settings as proxy admin The Router Settings → Fallbacks → Add Fallbacks flow was an uncovered manual-QA path. This adds a test that opens the modal, picks a primary + fallback from the seeded mock models, saves, and verifies both render in the fallback table. * fix(e2e): make router-fallback test idempotent and pick antd options by text - Match `.ant-select-item-option` by text instead of `getByTitle(...)` — FallbackGroupConfig uses `options=` (not <Select.Option> children), so no `title` attribute is emitted and the title-based selector hangs. - Add before/after hooks that wipe any fallback for fake-openai-gpt-4 via /config/update so retries and local reruns don't trip on leftover state. - Tighten the success assertion to a single tbody row containing BOTH the primary and the fallback names — pre-existing rows can no longer vacuously satisfy the check. - Fix the stale "Three tabs" comment to "Four tabs". Addresses Greptile P2s on PR #29069. * fix(e2e): keyboard-select fallback models + correct cleanup endpoint - Replace mouse-based option clicks with click-to-focus + type + Enter. FallbackGroupConfig's Selects use `options=` and a custom getPopupContainer, so locating options via `.ant-select-dropdown` hit several races: DOM-clicks left antd's popup state stale (the primary popup then intercepted the fallback click), `getByRole` matched always-mounted hidden options, and pointer stability fought the open animation. Typing into the showSearch input narrows the listbox to one option and Enter selects it cleanly. - Assert on dialog-side state changes (the active tab adopts the primary model name; the chain helper shows "1/10 used") instead of popup contents — these reflect the actual selection landing. - Cleanup helper now hits /get/config/callbacks (the real endpoint; /get/callbacks returns 404), so the before/after reset actually clears prior router_settings.fallbacks state. |