The daily queue parameter types on _restore_spend_updates_to_in_memory_queues
were narrowed to specific subtypes (DailyUserSpendTransaction, etc), but
the caller passes Dict[str, BaseDailySpendTransaction] — the return type
of flush_and_get_aggregated_daily_spend_update_transactions. Widen the
parameters to the base type.
Also replace dynamic TypedDict key lookup (which returned object) with
explicit literal-keyed get() calls so mypy can type-narrow each field.
Replace recursive `_walk` helper with a stack-based traversal so the
recursive_detector CI check passes without adding to the ignore list,
and avoid Python recursion limits on deeply nested payloads.
Made-with: Cursor
store_in_memory_spend_updates_in_redis drained the in-memory queues
into local variables before the rpush pipeline. If rpush raised (cloud
Redis hiccup, timeout, connection blip), those already-drained
transactions were garbage-collected with the scheduler job, silently
losing all spend aggregated during that tick.
Wrap the rpush in try/except. On failure, re-enqueue the aggregated
transactions into their respective in-memory queues so the next
scheduler tick retries.
Add a unit test that seeds real queues, simulates an rpush failure,
and asserts the transactions land back in-memory.
Replace the calibration step (one request + 10-minute poll) with an
independent ground truth computed from response usage via
litellm.cost_per_token. All N requests are made up front, so a single
dropped Redis write no longer kills the test.
Add /health/readiness checks at test start and on poll timeout so the
failure message surfaces proxy state (db, cache) instead of "calibration
timed out".
Set PROXY_BATCH_WRITE_AT=2 in the spend tracking CI job to shorten the
scheduler flush window.
Principle: GHA handles work that doesn't need external API keys; CCI
stays for integration tests that hit real API endpoints.
Four CCI jobs moved to new or extended GHA workflows:
1. check_code_and_doc_quality (was 25 runs: ruff + import-safety +
21 code_coverage_tests + 3 documentation_tests + circular-imports).
- The 21 tests/code_coverage_tests/*.py scripts and the 3
tests/documentation_tests/*.py scripts run in the new
.github/workflows/test-code-quality.yml workflow.
- ruff, import-safety, and circular-imports were already run by
.github/workflows/test-linting.yml — no new migration needed.
- The 3 documentation_tests scripts read
docs/my-website/docs/proxy/config_settings.md. Since docs have
moved to BerriAI/litellm-docs, the GHA workflow checks out that
repo and symlinks docs/my-website -> the checkout so the
existing hardcoded paths resolve without touching the scripts.
The stale local docs/my-website/ copy in this repo will be
removed in a separate PR.
2. semgrep (custom-rule SAST against .semgrep/rules).
- New .github/workflows/test-semgrep.yml.
3. installing_litellm_on_python + installing_litellm_on_python_3_13
(pip install compat checks on Python 3.12 and 3.13).
- New .github/workflows/test-install-litellm.yml as a matrix job.
- 3.12 run also verifies litellm_enterprise import; 3.13 run
skips that check (matches previous CCI behavior).
- installing_litellm_on_python_v2_migration_resolver stays in CCI
because it requires a postgres service.
CCI .circleci/config.yml: -112 lines, 4 jobs and their workflow refs
removed.
Restore guardrail spend/UI event_type wiring, request_data on streaming
OUTPUT paths, and centralized match redaction after the upstream revert.
Made-with: Cursor
Follow-up cleanup after an independent review pass surfaced a few
loose ends:
- Delete a 6x-duplicated filter block in litellm_mapped_tests_proxy_part2
(same kind of copy-paste residue we fixed earlier in
langfuse_logging_unit_tests).
- Delete the empty "Install Semgrep" run step in the semgrep job — the
command body was empty because semgrep is installed on-demand via
uv tool run in the next step.
- Standardize machine-executor image: one job was on ubuntu-2204:2023.10.1
while build_docker_database_image was already on ubuntu-2204:2024.04.1.
Bumped everything to 2024.04.1.
- Remove the legacy "version: 2" inside the workflows: block — CircleCI
2.1 top-level already declares the version.
- Drop `{{ checksum ".circleci/config.yml" }}` from cache keys (13 sites).
It was busting the cache on every unrelated config edit; the uv.lock
checksum alone is the right dependency cache key.
- Add partial-restore fallbacks to every restore_cache with a single
templated key (10 sites). Jobs now fall back to the latest cache with
a matching prefix if the exact uv.lock hash isn't cached yet.
Net: -14 lines.
Docker-executor jobs:
- Consolidate base images on cimg/python:3.12. Jobs previously on
3.11 (26 jobs), 3.9 (1 historical: upload-coverage), and an
incidental 3.13.1 (litellm_assistants_api_testing) now use 3.12.
- installing_litellm_on_python_3_13 keeps cimg/python:3.13.1 as its
explicit "latest Python supported" install-check matrix job.
Machine-executor jobs:
- Delete the miniconda install step from 10 jobs. uv now manages
Python directly: uv sync --python 3.12 auto-downloads a
python-build-standalone interpreter if the ubuntu-2204 base
image's default python doesn't match.
- Remove 37 "if [ -f conda.sh ]; then conda activate myenv" wrappers
and 2 unconditional conda activate blocks left behind from the
conda days.
- proxy_build_from_pip_tests keeps its 3.13 target (it was
conda create -n myenv python=3.13) via uv sync --python 3.13.
Net: -301 lines.
Add a start_postgres command parameterized on db_name (default
circle_test) that runs the postgres-db container and waits for port
5432 to accept connections. Replace all 11 inline docker run /
wait_for_service blocks with a single - start_postgres call.
The helm chart test overrides db_name to litellm_test; everything
else uses the default.
One of the 11 sites previously used a bespoke pg_isready loop instead
of wait_for_service; it now goes through the same TCP-probe path
everyone else uses, which is sufficient for test ordering purposes.
Net: -112 lines.
Add a single install_uv command in the commands: section that encodes
the uv version (0.10.9) and its SHA256 in one place, then replace all
42 inline curl|sha256|install blocks across every job that needs uv.
setup_litellm_test_deps now calls install_uv too, so the shared
test-dep bootstrap goes through the same path.
Bumping uv version or SHA is now a one-line change instead of 43.
Net: -203 lines.
- Remove mypy_linting job (GHA test-linting.yml already runs this)
- Remove three redundant "Install curl" apt-get steps (curl is
already present on the ubuntu-2204 machine image and used
successfully earlier in each affected job)
- Dedupe langfuse_logging_unit_tests filter block (6x copy of the
same two branch filters collapsed to 1)
- Pin all docker image references by @sha256 digest so builds stay
reproducible when upstream tags are updated:
cimg/python:3.9, 3.11, 3.12, 3.12-browsers, 3.13.1, cimg/node:20.19,
cimg/postgres:16.0, and postgres:14 used via docker run
Net: -62 lines, 49 image references pinned.
* add moonshot/kimi-k2.6 to model registry
* add moonshot/kimi-k2.6 to backup model registry
* add tests for moonshot/kimi-k2.6 model registry
* fix moonshot/kimi-k2.6 pricing and add reasoning support
* fix moonshot/kimi-k2.6 pricing and add reasoning support in backup
* update kimi-k2.6 tests: fix pricing, add tool_choice and reasoning checks
* fix: load kimi-k2.6 registry tests from local backup instead of remote cost map
P1 review: adaptive_router.py had a top-level import of
AdaptiveRouterUpdateQueue from litellm.proxy.db, which broke the
SDK/proxy boundary that every other router strategy respects. No
other router_strategy module imports from litellm.proxy at module
level.
The queue only depends on litellm._logging — it never needed to
live under litellm.proxy. Moved:
litellm/proxy/db/db_transaction_queue/adaptive_router_update_queue.py
→ litellm/router_strategy/adaptive_router/update_queue.py
tests/test_litellm/proxy/db/db_transaction_queue/
test_adaptive_router_update_queue.py
→ tests/test_litellm/router_strategy/adaptive_router/test_update_queue.py
Also switched the queue's logger from verbose_proxy_logger to
verbose_router_logger to match the new module's ownership.
P2 review: drop unused constant STAGNATION_JACCARD_EXACT from
config.py — it was defined but never referenced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously, members added to a team without an explicit per-member budget were
all linked to the same `litellm_budgettable` row referenced by the team's
`metadata.team_member_budget_id`. Updating one member's budget via
`/team/member_update` mutated the shared row and silently changed every other
member's budget too.
Now both write paths produce a private, per-member budget:
- `add_new_member` clones the team's default budget into a fresh row when a
member is added without `max_budget_in_team`/`allowed_models`. If no team
default exists, the membership is created with no budget.
- `_upsert_budget_and_membership` detects when an existing membership still
points at the team's default budget id and clones-on-write, relinking the
membership to the new private budget before applying the update.
- `team_member_update` reads `team_member_budget_id` from team metadata and
passes it through so the helper can make this distinction.
Adds unit tests for clone-on-write, in-place update of a private budget, and
the no-default-no-budget add path.
Made-with: Cursor
The prevent_key_leaks_in_exceptions CI check forbids '{args}' in
f-strings because it's a common shape for accidental API key leaks
in exception messages. _signature() uses an entirely local variable
named 'args' for tool-call arguments (loop-detection signatures, no
exception path), but the grep is substring-based. Rename to 'call_args'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
P1: start the adaptive-router flusher loop unconditionally at proxy boot
instead of gating on 'adaptive_routers is non-empty'. Adaptive routers
added via /config/reload after boot now have their queues drained.
State is lazy-loaded per router on first flush tick (new _state_loaded
flag on AdaptiveRouter) so hot-reloaded routers still get their
persisted priors.
P2: _finalize_adaptive_router_if_configured now prunes stale
AdaptiveRouterPostCallHook callbacks from every litellm callback list
before registering new ones. Without this, every Router replacement
left the old hooks wired up in litellm.callbacks and double-fired
signal recording for every request. Uses
logging_callback_manager.remove_callbacks_by_type (same pattern as the
semantic tool filter).
CI fixes:
- black --check failure: reformatted litellm/router.py
- schema migration diff: aligned @@index with the explicit index name
('idx_adaptive_router_session_activity') from the original migration
by adding 'map:' to all three schema.prisma copies. No new migration
needed.
Tests: 1 new covering the prune-on-hot-reload path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Mark last_updated_at (AdaptiveRouterState) and last_activity_at
(AdaptiveRouterSession) with @updatedAt so Prisma refreshes the
timestamps on every write. Without this the fields stayed frozen at
INSERT time and the last_activity_at index was misleading for any
future TTL/eviction logic. Applied to all three schema.prisma copies;
no migration SQL change needed (Prisma @updatedAt is a client-side
annotation that doesn't touch DDL).
- get_state_snapshot: report cell.total_samples instead of alpha+beta
for the 'samples' field. The previous value inflated every cell by
the COLD_START_MASS prior (e.g. showed 10.0 before any real traffic
arrived), which confused operators reading /adaptive_router/.../state.
Updated docs + the snapshot test to match.
Also fixes two pre-existing merge-break syntax errors in router.py
(missing ')' on the AdaptiveRouter TYPE_CHECKING import; truncated
async_pre_routing_hook dispatch call for the adaptive router branch)
that were masking the rest of the file from the interpreter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses two further Greptile findings:
- `_warn_if_db_ahead_of_head` only caught `psycopg.OperationalError`.
Non-connection DB errors (e.g. `InsufficientPrivilege` / 42501 if the
runtime DB user lacks SELECT on `_prisma_migrations`) would propagate
uncaught and crash startup — contradicting the docstring's
"informational only, never blocks" guarantee. Widen the catch to
`psycopg.DatabaseError` so all DB-layer errors are swallowed.
- In the P3009 and P3018 idempotent-recovery paths, the call to
`_resolve_specific_migration(name)` was not wrapped in its own
try/except. Being inside an active `except CalledProcessError`
handler, a new `CalledProcessError` from the resolve call would NOT
re-enter the same handler — it would propagate out as
`CalledProcessError`, past `proxy_cli.py`'s `except RuntimeError`,
crashing startup with an unhandled traceback instead of the intended
clean `sys.exit(2)`. Wrap both call sites to convert to RuntimeError.
Adds unit tests for both behaviors.
Follow-up on review feedback: the previous commit had the builder
download the query engine into /app/.cache, then threw it away in
the runtime stage and re-downloaded into /root/.cache. That doubled
the build-time network fetch.
Remove PRISMA_BINARY_CACHE_DIR and XDG_CACHE_HOME from the builder
stage as well, so its prisma generate lands in /root/.cache with the
correct path layout on its own. Drop the runtime-stage prisma generate
and instead COPY --from=builder /root/.cache /root/.cache. Single
download, smaller image.
- Open the psycopg connection in `_warn_if_db_ahead_of_head` with
autocommit=True. Without it, psycopg3's `with conn` calls COMMIT on
clean exit, which fails after the `UndefinedTable` (fresh-DB) branch
left the transaction in an aborted state — crashing first-run startups.
- Wrap the v2 `prisma db push` path in try/except and raise RuntimeError
on CalledProcessError/TimeoutExpired. Otherwise these propagate past
proxy_cli.py's `except RuntimeError` as unhandled tracebacks.
- Reword the loop-exhaustion error to cover the non-timeout exit path
(repeated P3005/P3009/P3018 idempotent-recovery `continue`s), not
just persistent timeouts.
Adds a unit test for the db_push error wrapping.
The uv migration added PRISMA_BINARY_CACHE_DIR=/app/.cache/... and
XDG_CACHE_HOME=/app/.cache to the runtime stages of Dockerfile and
Dockerfile.database. BINARY_PATHS in the generated prisma client was
baked to point into /app/.cache, so any deployment that mounts a volume
there (common with securityContext.readOnlyRootFilesystem: true and an
emptyDir/tmpfs for a writable cache) wipes the pre-downloaded query
engine at pod startup, producing BinaryNotFoundError during connect().
Before the uv migration, prisma-python defaulted to $HOME/.cache =
/root/.cache (runtime stage runs as root), which was unaffected by any
/app/* volume mounts. Restore that behaviour: drop the env vars from
the runtime stage, re-run prisma generate there so the query engine
AND the baked BINARY_PATHS both land in /root/.cache, and remove the
stale builder-stage /app/.cache (~800 MB).
Dockerfile.non_root is intentionally left alone — its /app/.cache
location is by design for the hardened offline-install flow.
Existing tests pinned exact kwargs on `PrismaManager.setup_database`,
but the opt-in v2 resolver added `use_v2_resolver=False` to every call.
Update the three assertions to reflect the new signature.
Fixes:
- TestHealthAppFactory::test_use_prisma_db_push_flag_behavior
- TestHealthAppFactory::test_startup_fails_when_db_setup_fails
Adds end-to-end CI coverage for `--use_v2_migration_resolver` via a new
job `installing_litellm_on_python_v2_migration_resolver`:
- Clones the pytest smoke path from `installing_litellm_on_python` but
uses a local Postgres sidecar instead of the shared DB to prevent
collisions with the v1 variant.
- Runs only the new `test_litellm_proxy_server_config_no_general_settings_v2_resolver`
which spawns the proxy with `--use_v2_migration_resolver` and smoke-tests
`/health/liveliness` and `/chat/completions`.
Refactors `test_basic_python_version.py`:
- Extracts the proxy spawn + smoke-test body into `_run_proxy_server_smoke_test`
so the v1 and v2 tests share the same code path.
- The existing `test_litellm_proxy_server_config_no_general_settings` is
now a thin wrapper that passes no extra args (v1 default, unchanged).
- Adds `..._v2_resolver` variant that passes `--use_v2_migration_resolver`.
The existing `installing_litellm_on_python` / `installing_litellm_on_python_3_13`
jobs filter out the v2 variant via `-k "not v2_resolver"` so they keep
running only against their shared DB, unchanged behavior.
Default behavior (v1) is unchanged. Users who have seen schema thrashing
during rolling deploys can opt into the v2 resolver with
`--use_v2_migration_resolver`.
Why v2 is safer:
- Runs `prisma migrate deploy` only.
- Recovers from P3005 (baseline) and idempotent P3009/P3018 errors, same
as v1.
- Never calls `_resolve_all_migrations`, which generates a schema diff
between the live DB and the shipped schema.prisma and applies it via
`prisma db execute`. That path bypassed every migration's SQL and was
the root cause of thrashing when two LiteLLM versions contended for
the same DB.
- Logs a non-blocking warning when the DB has migrations applied that
are newer than anything this build ships (ahead-of-HEAD). It does not
refuse to start — many users have unusual ledger state from past
thrashing, and blocking startup would be a breaking change.
Also prints a message on startup when the default (v1) resolver is in
use, pointing operators at the opt-in flag.
Adds unit tests covering the v2 fail-fast paths, the stripping of
Prisma-specific query params from DATABASE_URL (needed for psycopg),
the timestamp helpers, and pins the default: v1 still invokes
`_resolve_all_migrations`, v2 must not.
