get_end_user_object raises litellm.BudgetExceededError internally
when the end-user is over budget. The previous _safe_fetch in the
centralized gate swallowed it and returned None, which caused
common_checks to see end_user_object=None and skip the budget check
entirely — silently bypassing end-user budget enforcement.
Add BudgetExceededError to the re-raise list alongside HTTPException
and ProxyException (reported by Veria AI).
- Narrow _team_obj_from_token to require non-None team_id so mypy
passes.
- Preserve the no-auth dev-mode contract for deployments with
master_key unset AND no JWT/OAuth2 configured — the gate
short-circuits only in that specific combination. JWT or OAuth2
deployments without master_key still run the centralized authz.
- is_database_connection_error now enumerates data-layer PrismaError
subclasses (DataError, UniqueViolationError, ForeignKeyViolationError,
MissingRequiredValueError, RawQueryError, TableNotFoundError,
RecordNotFoundError) as False, and maps everything else (bare
PrismaError, connectivity subclasses, DB_CONNECTION_ERROR_TYPES) to
True. Known-safe-to-propagate errors don't trigger HA fallback;
unknown / generic PrismaError still falls back to preserve legacy
503 behavior.
- Update test_handle_authentication_error_db_unavailable_connectivity
to include PrismaError in the fallback list.
Multiple paths through _user_api_key_auth_builder returned a
UserAPIKeyAuth without running common_checks(): OAuth2 token validation,
OAuth2 proxy header hook, JWT admin shortcut, master_key path,
pass-through custom headers, the /user/auth route, and the
allow_requests_on_db_unavailable fallback. An operator-configured key
model-access list, max_budget, team_blocked flag, or team model scope
was therefore silently skipped on those paths. The HA-fallback token
was worse: it was a full proxy-admin synthetic, so a DB outage granted
full admin to every caller.
Fix three root causes (VERIA-18):
1. Centralize common_checks in the user_api_key_auth wrapper. The
builder paths no longer call it; the wrapper runs it once after the
builder returns, for every path. Introduces _run_centralized_common_checks
which gathers team/user/project/end_user/global_spend context in
parallel via asyncio.gather. Preserves the existing
custom_auth_run_common_checks opt-out for custom-auth deployments.
2. Narrow is_database_connection_error — drop the blanket PrismaError
catch that routed data-layer errors (UniqueViolationError, etc.)
into the HA fallback. Only real connectivity failures plus the
no_db_connection marker now qualify.
3. DB-unavailable fallback issues an INTERNAL_USER token with user_id
DB_UNAVAILABLE_FALLBACK_USER_ID instead of proxy-admin. An outage
can no longer escalate an anonymous caller.
JWT admin / master_key tokens still grant admin via a synthesized
admin user_object (so non_proxy_admin_allowed_routes_check in
common_checks recognizes them); other common_checks branches
(team_blocked, team_model_access) now apply uniformly.
Follow-up cleanup after an independent review pass surfaced a few
loose ends:
- Delete a 6x-duplicated filter block in litellm_mapped_tests_proxy_part2
(same kind of copy-paste residue we fixed earlier in
langfuse_logging_unit_tests).
- Delete the empty "Install Semgrep" run step in the semgrep job — the
command body was empty because semgrep is installed on-demand via
uv tool run in the next step.
- Standardize machine-executor image: one job was on ubuntu-2204:2023.10.1
while build_docker_database_image was already on ubuntu-2204:2024.04.1.
Bumped everything to 2024.04.1.
- Remove the legacy "version: 2" inside the workflows: block — CircleCI
2.1 top-level already declares the version.
- Drop `{{ checksum ".circleci/config.yml" }}` from cache keys (13 sites).
It was busting the cache on every unrelated config edit; the uv.lock
checksum alone is the right dependency cache key.
- Add partial-restore fallbacks to every restore_cache with a single
templated key (10 sites). Jobs now fall back to the latest cache with
a matching prefix if the exact uv.lock hash isn't cached yet.
Net: -14 lines.
Docker-executor jobs:
- Consolidate base images on cimg/python:3.12. Jobs previously on
3.11 (26 jobs), 3.9 (1 historical: upload-coverage), and an
incidental 3.13.1 (litellm_assistants_api_testing) now use 3.12.
- installing_litellm_on_python_3_13 keeps cimg/python:3.13.1 as its
explicit "latest Python supported" install-check matrix job.
Machine-executor jobs:
- Delete the miniconda install step from 10 jobs. uv now manages
Python directly: uv sync --python 3.12 auto-downloads a
python-build-standalone interpreter if the ubuntu-2204 base
image's default python doesn't match.
- Remove 37 "if [ -f conda.sh ]; then conda activate myenv" wrappers
and 2 unconditional conda activate blocks left behind from the
conda days.
- proxy_build_from_pip_tests keeps its 3.13 target (it was
conda create -n myenv python=3.13) via uv sync --python 3.13.
Net: -301 lines.
Add a start_postgres command parameterized on db_name (default
circle_test) that runs the postgres-db container and waits for port
5432 to accept connections. Replace all 11 inline docker run /
wait_for_service blocks with a single - start_postgres call.
The helm chart test overrides db_name to litellm_test; everything
else uses the default.
One of the 11 sites previously used a bespoke pg_isready loop instead
of wait_for_service; it now goes through the same TCP-probe path
everyone else uses, which is sufficient for test ordering purposes.
Net: -112 lines.
Add a single install_uv command in the commands: section that encodes
the uv version (0.10.9) and its SHA256 in one place, then replace all
42 inline curl|sha256|install blocks across every job that needs uv.
setup_litellm_test_deps now calls install_uv too, so the shared
test-dep bootstrap goes through the same path.
Bumping uv version or SHA is now a one-line change instead of 43.
Net: -203 lines.
- Remove mypy_linting job (GHA test-linting.yml already runs this)
- Remove three redundant "Install curl" apt-get steps (curl is
already present on the ubuntu-2204 machine image and used
successfully earlier in each affected job)
- Dedupe langfuse_logging_unit_tests filter block (6x copy of the
same two branch filters collapsed to 1)
- Pin all docker image references by @sha256 digest so builds stay
reproducible when upstream tags are updated:
cimg/python:3.9, 3.11, 3.12, 3.12-browsers, 3.13.1, cimg/node:20.19,
cimg/postgres:16.0, and postgres:14 used via docker run
Net: -62 lines, 49 image references pinned.
* add moonshot/kimi-k2.6 to model registry
* add moonshot/kimi-k2.6 to backup model registry
* add tests for moonshot/kimi-k2.6 model registry
* fix moonshot/kimi-k2.6 pricing and add reasoning support
* fix moonshot/kimi-k2.6 pricing and add reasoning support in backup
* update kimi-k2.6 tests: fix pricing, add tool_choice and reasoning checks
* fix: load kimi-k2.6 registry tests from local backup instead of remote cost map
P1 review: adaptive_router.py had a top-level import of
AdaptiveRouterUpdateQueue from litellm.proxy.db, which broke the
SDK/proxy boundary that every other router strategy respects. No
other router_strategy module imports from litellm.proxy at module
level.
The queue only depends on litellm._logging — it never needed to
live under litellm.proxy. Moved:
litellm/proxy/db/db_transaction_queue/adaptive_router_update_queue.py
→ litellm/router_strategy/adaptive_router/update_queue.py
tests/test_litellm/proxy/db/db_transaction_queue/
test_adaptive_router_update_queue.py
→ tests/test_litellm/router_strategy/adaptive_router/test_update_queue.py
Also switched the queue's logger from verbose_proxy_logger to
verbose_router_logger to match the new module's ownership.
P2 review: drop unused constant STAGNATION_JACCARD_EXACT from
config.py — it was defined but never referenced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously, members added to a team without an explicit per-member budget were
all linked to the same `litellm_budgettable` row referenced by the team's
`metadata.team_member_budget_id`. Updating one member's budget via
`/team/member_update` mutated the shared row and silently changed every other
member's budget too.
Now both write paths produce a private, per-member budget:
- `add_new_member` clones the team's default budget into a fresh row when a
member is added without `max_budget_in_team`/`allowed_models`. If no team
default exists, the membership is created with no budget.
- `_upsert_budget_and_membership` detects when an existing membership still
points at the team's default budget id and clones-on-write, relinking the
membership to the new private budget before applying the update.
- `team_member_update` reads `team_member_budget_id` from team metadata and
passes it through so the helper can make this distinction.
Adds unit tests for clone-on-write, in-place update of a private budget, and
the no-default-no-budget add path.
Made-with: Cursor
The prevent_key_leaks_in_exceptions CI check forbids '{args}' in
f-strings because it's a common shape for accidental API key leaks
in exception messages. _signature() uses an entirely local variable
named 'args' for tool-call arguments (loop-detection signatures, no
exception path), but the grep is substring-based. Rename to 'call_args'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
P1: start the adaptive-router flusher loop unconditionally at proxy boot
instead of gating on 'adaptive_routers is non-empty'. Adaptive routers
added via /config/reload after boot now have their queues drained.
State is lazy-loaded per router on first flush tick (new _state_loaded
flag on AdaptiveRouter) so hot-reloaded routers still get their
persisted priors.
P2: _finalize_adaptive_router_if_configured now prunes stale
AdaptiveRouterPostCallHook callbacks from every litellm callback list
before registering new ones. Without this, every Router replacement
left the old hooks wired up in litellm.callbacks and double-fired
signal recording for every request. Uses
logging_callback_manager.remove_callbacks_by_type (same pattern as the
semantic tool filter).
CI fixes:
- black --check failure: reformatted litellm/router.py
- schema migration diff: aligned @@index with the explicit index name
('idx_adaptive_router_session_activity') from the original migration
by adding 'map:' to all three schema.prisma copies. No new migration
needed.
Tests: 1 new covering the prune-on-hot-reload path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Mark last_updated_at (AdaptiveRouterState) and last_activity_at
(AdaptiveRouterSession) with @updatedAt so Prisma refreshes the
timestamps on every write. Without this the fields stayed frozen at
INSERT time and the last_activity_at index was misleading for any
future TTL/eviction logic. Applied to all three schema.prisma copies;
no migration SQL change needed (Prisma @updatedAt is a client-side
annotation that doesn't touch DDL).
- get_state_snapshot: report cell.total_samples instead of alpha+beta
for the 'samples' field. The previous value inflated every cell by
the COLD_START_MASS prior (e.g. showed 10.0 before any real traffic
arrived), which confused operators reading /adaptive_router/.../state.
Updated docs + the snapshot test to match.
Also fixes two pre-existing merge-break syntax errors in router.py
(missing ')' on the AdaptiveRouter TYPE_CHECKING import; truncated
async_pre_routing_hook dispatch call for the adaptive router branch)
that were masking the rest of the file from the interpreter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses two further Greptile findings:
- `_warn_if_db_ahead_of_head` only caught `psycopg.OperationalError`.
Non-connection DB errors (e.g. `InsufficientPrivilege` / 42501 if the
runtime DB user lacks SELECT on `_prisma_migrations`) would propagate
uncaught and crash startup — contradicting the docstring's
"informational only, never blocks" guarantee. Widen the catch to
`psycopg.DatabaseError` so all DB-layer errors are swallowed.
- In the P3009 and P3018 idempotent-recovery paths, the call to
`_resolve_specific_migration(name)` was not wrapped in its own
try/except. Being inside an active `except CalledProcessError`
handler, a new `CalledProcessError` from the resolve call would NOT
re-enter the same handler — it would propagate out as
`CalledProcessError`, past `proxy_cli.py`'s `except RuntimeError`,
crashing startup with an unhandled traceback instead of the intended
clean `sys.exit(2)`. Wrap both call sites to convert to RuntimeError.
Adds unit tests for both behaviors.
Follow-up on review feedback: the previous commit had the builder
download the query engine into /app/.cache, then threw it away in
the runtime stage and re-downloaded into /root/.cache. That doubled
the build-time network fetch.
Remove PRISMA_BINARY_CACHE_DIR and XDG_CACHE_HOME from the builder
stage as well, so its prisma generate lands in /root/.cache with the
correct path layout on its own. Drop the runtime-stage prisma generate
and instead COPY --from=builder /root/.cache /root/.cache. Single
download, smaller image.
- Open the psycopg connection in `_warn_if_db_ahead_of_head` with
autocommit=True. Without it, psycopg3's `with conn` calls COMMIT on
clean exit, which fails after the `UndefinedTable` (fresh-DB) branch
left the transaction in an aborted state — crashing first-run startups.
- Wrap the v2 `prisma db push` path in try/except and raise RuntimeError
on CalledProcessError/TimeoutExpired. Otherwise these propagate past
proxy_cli.py's `except RuntimeError` as unhandled tracebacks.
- Reword the loop-exhaustion error to cover the non-timeout exit path
(repeated P3005/P3009/P3018 idempotent-recovery `continue`s), not
just persistent timeouts.
Adds a unit test for the db_push error wrapping.
The uv migration added PRISMA_BINARY_CACHE_DIR=/app/.cache/... and
XDG_CACHE_HOME=/app/.cache to the runtime stages of Dockerfile and
Dockerfile.database. BINARY_PATHS in the generated prisma client was
baked to point into /app/.cache, so any deployment that mounts a volume
there (common with securityContext.readOnlyRootFilesystem: true and an
emptyDir/tmpfs for a writable cache) wipes the pre-downloaded query
engine at pod startup, producing BinaryNotFoundError during connect().
Before the uv migration, prisma-python defaulted to $HOME/.cache =
/root/.cache (runtime stage runs as root), which was unaffected by any
/app/* volume mounts. Restore that behaviour: drop the env vars from
the runtime stage, re-run prisma generate there so the query engine
AND the baked BINARY_PATHS both land in /root/.cache, and remove the
stale builder-stage /app/.cache (~800 MB).
Dockerfile.non_root is intentionally left alone — its /app/.cache
location is by design for the hardened offline-install flow.
Existing tests pinned exact kwargs on `PrismaManager.setup_database`,
but the opt-in v2 resolver added `use_v2_resolver=False` to every call.
Update the three assertions to reflect the new signature.
Fixes:
- TestHealthAppFactory::test_use_prisma_db_push_flag_behavior
- TestHealthAppFactory::test_startup_fails_when_db_setup_fails
Adds end-to-end CI coverage for `--use_v2_migration_resolver` via a new
job `installing_litellm_on_python_v2_migration_resolver`:
- Clones the pytest smoke path from `installing_litellm_on_python` but
uses a local Postgres sidecar instead of the shared DB to prevent
collisions with the v1 variant.
- Runs only the new `test_litellm_proxy_server_config_no_general_settings_v2_resolver`
which spawns the proxy with `--use_v2_migration_resolver` and smoke-tests
`/health/liveliness` and `/chat/completions`.
Refactors `test_basic_python_version.py`:
- Extracts the proxy spawn + smoke-test body into `_run_proxy_server_smoke_test`
so the v1 and v2 tests share the same code path.
- The existing `test_litellm_proxy_server_config_no_general_settings` is
now a thin wrapper that passes no extra args (v1 default, unchanged).
- Adds `..._v2_resolver` variant that passes `--use_v2_migration_resolver`.
The existing `installing_litellm_on_python` / `installing_litellm_on_python_3_13`
jobs filter out the v2 variant via `-k "not v2_resolver"` so they keep
running only against their shared DB, unchanged behavior.
Default behavior (v1) is unchanged. Users who have seen schema thrashing
during rolling deploys can opt into the v2 resolver with
`--use_v2_migration_resolver`.
Why v2 is safer:
- Runs `prisma migrate deploy` only.
- Recovers from P3005 (baseline) and idempotent P3009/P3018 errors, same
as v1.
- Never calls `_resolve_all_migrations`, which generates a schema diff
between the live DB and the shipped schema.prisma and applies it via
`prisma db execute`. That path bypassed every migration's SQL and was
the root cause of thrashing when two LiteLLM versions contended for
the same DB.
- Logs a non-blocking warning when the DB has migrations applied that
are newer than anything this build ships (ahead-of-HEAD). It does not
refuse to start — many users have unusual ledger state from past
thrashing, and blocking startup would be a breaking change.
Also prints a message on startup when the default (v1) resolver is in
use, pointing operators at the opt-in flag.
Adds unit tests covering the v2 fail-fast paths, the stripping of
Prisma-specific query params from DATABASE_URL (needed for psycopg),
the timestamp helpers, and pins the default: v1 still invokes
`_resolve_all_migrations`, v2 must not.
- _find_destructive_statements: add DROP INDEX to the docstring (the
regex already detects it; only the docstring lagged).
- create_migration: correct the base_branch default documented in the
docstring from "main" to "litellm_internal_staging".
Generating a migration from a stale branch could silently emit DROP
COLUMN for columns the stale branch did not know about, and the
script would write that SQL to a new migration file with no warning.
Adds two guards to ci_cd/run_migration.py:
- Branch freshness check: fetches origin/<base-branch> and exits 3 if
HEAD is behind. Default base is litellm_internal_staging. New
flags: --base-branch, --skip-freshness-check.
- Destructive guard: refuses (exit 2) if the generated diff contains
DROP COLUMN / DROP TABLE / DROP INDEX, unless --allow-destructive
is passed.
Refusal banners include guidance and an explicit callout instructing
AI agents not to auto-bypass the flags. Also treats Prisma's
"-- This is an empty migration." output as a no-op rather than
writing an empty file.
Updates litellm-proxy-extras/migration_runbook.md with the new
workflow, flag documentation, and agent warnings.
Two fail-safes for the /v1/messages → Bedrock Invoke pass-through so new
Anthropic-only extensions Claude Code starts sending can't reach Bedrock
and trigger a 400 "Extra inputs are not permitted":
1. Top-level body fields are filtered to a typed allowlist. New
`BedrockInvokeAnthropicMessagesRequest` TypedDict (in
`litellm/types/llms/bedrock.py`) captures the Bedrock Invoke Anthropic
Messages body schema; the runtime allowlist is derived from its
`__annotations__` so the type and the filter can't drift. Anchored to
the AWS reference page in docstrings + transform comment. An
exact-set test pins the resolved allowlist so any future edit forces
conscious review.
Drops context_management, output_config, speed, mcp_servers,
container, inference_geo, internal litellm_metadata, and any future
Anthropic addition. output_format stays as an active inline-schema
conversion (not just a strip).
2. The anthropic-beta header list is filtered + transformed against the
bedrock mapping for ALL betas, not just auto-injected ones. The
previous code union'd user-provided betas back in unfiltered, so a
client on a new Anthropic-direct beta (e.g. advisor-tool-…,
context-management-…) could still pin the request to fail. In a proxy
context the client can't know the backend is Bedrock; the provider
mapping is authoritative. User-provided drops are logged at WARNING
so intentional overrides leave a breadcrumb.
Updates one existing test that happened to assert on the old buggy
pass-through (it used output-128k-2025-02-19, which is null in the
bedrock mapping and would 400 at runtime); rewrote it against a
bedrock-supported beta.
Scope: messages/invoke only. The same user-beta bypass exists in
chat/invoke but that's a different code path with different
user-expectation trade-offs — follow-up.
user