* test: add 24hr Redis-backed VCR cache to additional test suites
Extracts the existing llm_translation VCR plumbing into a reusable helper
(tests/_vcr_conftest_common.py) and wires it into the conftest.py files
of the test directories listed in LIT-2787:
audio_tests, batches_tests, guardrails_tests, image_gen_tests,
litellm_utils_tests, local_testing, logging_callback_tests,
pass_through_unit_tests, router_unit_tests, unified_google_tests
The same helper is also adopted by the pre-existing llm_translation and
llm_responses_api_testing conftests to remove the copy-pasted VCR setup.
Each consuming conftest:
- registers the Redis persister via pytest_recording_configure
- auto-marks collected tests with pytest.mark.vcr (skipping respx-using
files where applicable, since respx and vcrpy both patch httpx)
- gates cassette writes on test success via _vcr_outcome_gate
The cache is opt-in via CASSETTE_REDIS_URL; when unset, VCR is disabled
and tests hit live providers as before. LITELLM_VCR_DISABLE=1 still
forces a bypass for ad-hoc local runs.
Test directories that run LiteLLM proxy in Docker (build_and_test,
proxy_logging_guardrails_model_info_tests, proxy_store_model_in_db_tests)
are intentionally not included: VCR.py patches the in-process httpx
transport and cannot intercept calls made from inside a Docker container.
The installing_litellm_on_python* jobs make no LLM calls and don't
benefit from caching.
https://linear.app/litellm-ai/issue/LIT-2787/add-24hr-caching-to-additional-test-suites
* test(vcr): add safe-body matcher to handle JSONL and binary request bodies
vcrpy's stock body matcher inspects Content-Type and unconditionally
runs json.loads on application/json bodies. JSON Lines payloads (used
by the Bedrock batch S3 PUT and other upload paths) crash that with
json.JSONDecodeError: Extra data, before the matcher can return
'not a match'.
This was the root cause of the batches_testing CI job failing on
test_async_create_file once VCR auto-marking was applied to the
batches_tests directory.
Add a conservative byte-equality body matcher and use it in place of
'body' in the shared match_on tuple. The matcher is strictly more
conservative than vcrpy's default — the only thing it gives up is
'different JSON key order is treated as the same body', which doesn't
apply to deterministic litellm-built request payloads. It can never
produce a false positive that the default would have rejected, so
there is no cross-contamination risk.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): exclude tests that VCR replay actively breaks
A few tests are incompatible with cassette replay and were failing on
the latest CI run after VCR auto-marking was extended to local_testing
and logging_callback_tests:
- test_amazing_s3_logs.py (logging_callback_tests): the test asserts on
a per-run response_id that should round-trip through a real S3
PUT/LIST. vcrpy's boto3 stub intercepts the PUT and the LIST replays
stale keys, so the freshly-generated id is never found.
- test_async_embedding_azure (logging_callback_tests) and
test_amazing_sync_embedding (local_testing): the failure branches
deliberately pass api_key='my-bad-key' to assert that the failure
callback fires. We scrub auth headers from cassettes (so the bad-key
request matches the prior good-key request), and vcrpy replays the
recorded 200 — the failure callback never fires.
- test_assistants.py (local_testing): the OpenAI Assistants polling
APIs mint fresh thread/run IDs every recording session and then poll
until status=='completed'. Replays of those polled GETs can never
match a freshly-generated run id, so every CI run effectively
re-records and the suite blows past the 15m no_output_timeout.
Skip these from VCR auto-marking so they continue to hit live providers
as they did before this change. The remaining tests in each directory
still get cached.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): expand skip lists for second batch of incompatible tests
Followup to the previous commit. After re-running CI on the rebuilt
branch, three more tests surfaced as VCR-replay-incompatible:
- litellm_utils_testing :: test_get_valid_models_from_dynamic_api_key
Calls GET /v1/models with api_key='123' to assert the result is empty.
We scrub auth headers, so the bad-key request matches the prior
good-key cassette and replays the recorded model list.
- litellm_utils_testing :: test_litellm_overhead.py
Measures litellm_overhead_time_ms as a percentage of total wall-clock
time. With cached responses the upstream 'network' time collapses to
microseconds, blowing past the 40%% threshold the test asserts on.
Skip the whole file (every parametrization is at risk).
- local_testing_part1 :: test_async_custom_handler_completion and
test_async_custom_handler_embedding
Same bad-key failure-callback pattern as the already-skipped
test_amazing_sync_embedding.
- litellm_router_testing :: test_router_caching.py
Asserts on litellm's own router-level response cache by comparing
response1.id to response2.id across repeat upstream calls (test
bypasses litellm cache via ttl=0 and expects upstream to return a
*new* id). With VCR replay both upstream calls return the same
cassette body, so the ids are identical. Skip the whole file.
- logging_callback_tests :: test_async_chat_azure (preemptive)
Same shape as already-skipped test_async_embedding_azure; was masked
by upstream OpenAI rate-limit failures on baseline.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): use item.path and tighten matcher docstring
- Replace pytest's deprecated item.fspath with item.path in
apply_vcr_auto_marker_to_items so we don't emit deprecation
warnings under pytest 8.
- Clarify _safe_body_matcher docstring to reflect actual behavior
(direct == first, then UTF-8 bytes comparison, no repr fallback).
Addresses Greptile review feedback on PR #27159.
* test(vcr): swallow all RedisError on cassette save/load
Cassette persistence is strictly best-effort: any Redis-side failure
(connection blip, timeout, OutOfMemoryError when the maxmemory cap is
hit, READONLY replicas, etc.) should degrade to 'test passed but
cassette not cached' rather than fail the test on teardown.
Previously the persister only caught ConnectionError and TimeoutError,
so OutOfMemoryError — which Redis Cloud raises when the cassette cache
hits its memory cap and there are no evictable keys — propagated out of
vcrpy's autouse fixture and ERRORed otherwise-passing tests on
teardown. This caused the litellm_utils_testing CircleCI job to fail on
the latest commit's run, even though the underlying test was a unit
test that used mock_response and produced no real upstream traffic
(the cassette was dirtied by a background langfuse callback). The
rerun only succeeded because Redis evictions happened to free enough
room before the SET — i.e. it was timing-dependent flakiness.
Catch redis.exceptions.RedisError (the common base of all server- and
client-side Redis exceptions) on both save and load, and parametrize
the regression tests across ConnectionError, TimeoutError, and
OutOfMemoryError to pin the new behavior.
* test(vcr): surface cassette-cache failures with warnings + session banner
When the persister silently swallows a Redis OOM (or any RedisError) on
save/load there is otherwise no visible signal that the cache is
degraded — tests pass, the cassette just isn't persisted, and the next
session still hits the same Redis at the same near-cap memory.
Add three layers of observability so that failure mode is loud:
1. Per-process health counters ("save_failures", "load_failures", and
the last error string for each), exposed via cassette_cache_health()
and reset via reset_cassette_cache_health(). The persister
increments these in addition to logging.
2. VCRCassetteCacheWarning (UserWarning subclass) emitted via
warnings.warn() inside the persister's except block. Pytest's
built-in warnings summary at session end automatically lists every
such warning, so the failure is visible in CI logs without any
conftest-level wiring.
3. Session-end banner via emit_cassette_cache_session_banner() and a
stderr-fallback atexit handler registered from
register_persister_if_enabled(). Two states:
- red "VCR CASSETTE CACHE DEGRADED" when save_failures or
load_failures > 0
- yellow "VCR CASSETTE CACHE NEAR CAPACITY" (no failures, but
used_memory >= 85% of maxmemory) so the next session knows
the Redis is approaching OOM before any SET actually fails
Capacity comes from a best-effort INFO memory probe
(cassette_cache_capacity_snapshot) that returns None on any failure or
when maxmemory is uncapped. The atexit handler skips xdist workers so
only the controller emits.
Tests: parametrize the existing save/load swallow-error tests across
ConnectionError/TimeoutError/OutOfMemoryError, add direct tests for
the health counters and warning emission, and a new
test_vcr_conftest_common_banner.py covering banner output for every
state (silent/red/yellow/disabled/xdist-worker).
* test(vcr): bucket cassettes by API key fingerprint, drop bad-key skips
Tests that deliberately call an LLM API with a bad key (e.g. to assert
that the failure callback fires, or that check_valid_key returns False)
were being silently served the prior good-key cassette: we scrub the
real Authorization / x-api-key header from the cassette before storing
it, so a follow-up bad-key call is byte-identical to the good-key call
under the existing match_on tuple.
Add a 'key_fingerprint' custom matcher that distinguishes requests by
the SHA-256 of their API-key headers. The fingerprint is stamped into
a synthetic 'x-litellm-key-fp' header by a new before_record_request
hook, which then strips the real auth headers (we have to do the
scrubbing here instead of via vcrpy's filter_headers knob, because
filter_headers runs *first* and would erase the value we want to hash).
Bad-key requests now get a different cassette bucket than good-key
requests, so vcrpy will not replay a recorded 200 in place of the
expected 401. The fingerprint is a one-way hash of the secret, so
cassettes never contain the key.
This permanently removes the 'bad-key' category of skips:
- tests/local_testing: dropped ::test_amazing_sync_embedding,
::test_async_custom_handler_completion,
::test_async_custom_handler_embedding
- tests/logging_callback_tests: dropped ::test_async_chat_azure,
::test_async_embedding_azure
- tests/litellm_utils_tests: dropped
::test_get_valid_models_from_dynamic_api_key
Coverage: 7 new unit tests in tests/test_litellm/test_vcr_safe_body_matcher.py
covering header stripping, fingerprint determinism, no-auth bucketing,
good-vs-bad key discrimination, x-api-key (Anthropic/Azure) discrimination,
and idempotence under replay.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): drop redundant comments and docstrings
Trim narration of code that is already self-evident from function and
variable names. Keep the two genuinely non-obvious bits:
- ordering constraint between filter_headers and before_record_request,
which would invite a maintainer to re-introduce the bug if removed
- the per-directory _VCR_INCOMPATIBLE_FILES rationale, since 'why
exactly is this skipped' is not knowable from the test name alone
Also drop the 40-line commented-out drop-in conftest snippet at the
bottom of _vcr_conftest_common.py — the consuming conftests are the
canonical reference.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): make _before_record_request idempotent
vcrpy invokes before_record_request more than once per request:
can_play_response_for calls it, then __contains__ /
_responses (reached via play_response) call it again on the
result. The second invocation sees a request whose auth headers we
already stripped, so a naive recompute yields "no-key" and
overwrites the real fingerprint stored in the header.
This makes can_play_response_for and play_response disagree on
matchability — the former says "yes, we have a stored response for
this" (matching no-key to no-key) and the latter throws
UnhandledHTTPRequestError because it computes a fresh real
fingerprint that doesn't match the stored no-key.
In CI this manifested as ~30 failing tests across guardrails_testing,
audio_testing, batches_testing, image_gen_testing, llm_responses_api,
litellm_router_unit_testing, etc. Skip the recompute when the header
is already set, so re-applying the hook is a no-op.
Adds a regression test that fires the hook twice on the same dict and
asserts the fingerprint stays put.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): drop more redundant docstrings and headers
* test(vcr): enable 24hr cache for ocr_tests and search_tests
These two directories were the only non-dockerized test suites in the
build_and_test workflow that make live LLM/provider API calls but were
not VCR-enabled by this PR. Together they account for 96 tests:
- tests/ocr_tests/ (31): Mistral OCR, Azure AI OCR, Azure Document
Intelligence, Vertex AI OCR. Pure-unit tests inside the same files
(e.g. TestAzureDocumentIntelligencePagesParam) make no HTTP calls
and become benign VCR NOOPs.
- tests/search_tests/ (65): Brave, DataForSEO, DuckDuckGo, Exa,
Firecrawl, Google PSE, Linkup, Parallel.ai, Perplexity, SearchAPI,
Searxng, Serper, Tavily.
Both directories use the canonical minimal conftest pattern from
tests/audio_tests/conftest.py with no skip lists. None of the test
files use respx, none assert on per-call upstream non-determinism
(no response1.id != response2.id, no overhead-as-fraction-of-total,
no live polling), so the default match_on tuple should cache cleanly.
If a flake surfaces during the first cassette-recording CI run, we
can add a targeted skip the same way we did for the other dirs.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The Azure o-series tests were excluded from the conftest's VCR auto-marker
because of a respx/vcrpy transport-patching conflict, but the only respx
reference in the file was an unused `MockRouter` import. Drop the dead
import and remove the file from the conflict set so cassettes record on
first run and replay thereafter, eliminating the 60-95s live Azure latency
that was crashing xdist workers under --timeout=120 thread-mode timeouts.
The /otel-spans endpoint returns process-wide spans and tags
most_recent_parent by max start_time. After tightening that route to
proxy_admin (sk-1234), the GET /otel-spans request itself emits auth
spans that beat the chat-completion spans on start_time, so
most_recent_parent now points at the request's own auth trace
(['postgres', 'postgres']) and the >=5-span assertion fails.
Pick the chat-completion trace by content: it is the only trace whose
span list is a superset of {postgres, redis, raw_gen_ai_request,
batch_write_to_db}. Verified locally end-to-end against
otel_test_config.yaml + OTEL_EXPORTER=in_memory: 3/3 runs green.
/otel-spans now requires proxy admin (returns 401 'Only proxy admin
can be used to generate, delete, update info for new keys/users/teams.
Route=/otel-spans' for non-admin callers). Switch the GET call to use
the master key sk-1234 while keeping the generated key for the
chat-completion request that produces the spans.
- Add image_generation/http_utils.azure_deployment_image_generation_json_body; call
from azure.py (keeps AzureChatCompletion focused on chat).
- Rename finalize_image_edit_multipart_data to finalize_image_edit_request_data with
docstring covering multipart and JSON POST payloads (review feedback).
Co-authored-by: Cursor <cursoragent@cursor.com>
- Move _user_has_admin_view to litellm.proxy._types as
user_api_key_has_admin_view (single source of truth). common_utils.py
and isolation.py both import from there now, removing the duplicated
role-check that could silently diverge if new admin roles are added.
- Add pytest.importorskip("litellm_enterprise") to the two regression
tests that assert managed_files / managed_vector_stores are registered;
those keys come from ENTERPRISE_PROXY_HOOKS so the tests would fail
unconditionally in a checkout without the enterprise extra installed.
The Python 3.13 CCI smoke matrix surfaces a partially-initialized-module
ImportError when loading the managed files hook chain:
litellm.proxy.hooks/__init__ (mid-import)
-> enterprise.enterprise_hooks
-> litellm_enterprise.proxy.hooks.managed_files
-> litellm.llms.base_llm.managed_resources.isolation
-> litellm.proxy.management_endpoints.common_utils
-> litellm.proxy.utils (re-enters litellm.proxy.hooks)
The except ImportError block in hooks/__init__.py silently swallowed the
failure, leaving managed_files unregistered and POST /files returning
500 "Managed files hook not found".
Two-layer fix:
- Inline the 3-line _user_has_admin_view check in isolation.py instead
of importing it from litellm.proxy.management_endpoints.common_utils.
litellm.llms.* should not depend on litellm.proxy.* — removing this
layering violation breaks the cycle at its root.
- Define PROXY_HOOKS and get_proxy_hook before the conditional
enterprise import in litellm/proxy/hooks/__init__.py, so any future
re-entry resolves the public names instead of hitting an
ImportError on a partially-initialized module.
Also fold in two unrelated CCI repairs surfaced in the same staging run:
- tests/otel_tests/test_key_logging_callbacks.py: per-key
gcs_bucket_name / gcs_path_service_account are now stripped by
initialize_dynamic_callback_params, so the GCS client falls through
to the env-only branch. Update the assertion to match the new
"GCS_BUCKET_NAME is not set" message.
- .circleci/config.yml: tests/pass_through_tests now resolves
google-auth-library@10.x via the @google-cloud/vertexai 1.12.0 bump,
which uses dynamic ESM imports Jest 29 cannot load without
--experimental-vm-modules. Pass that flag in the Vertex JS test step.
Adds tests/test_litellm/proxy/hooks/test_proxy_hooks_init.py as a
regression guard: managed_files / managed_vector_stores must register,
and isolation.py must not transitively import litellm.proxy.utils.
secret_fields (containing raw HTTP headers including Authorization
Bearer tokens) was being included in proxy_server_request['body']
because the body snapshot was a copy.copy(data) of the full request
dict. This body gets serialized and persisted in the LiteLLM_SpendLogs
table, exposing user credentials in the database.
Root cause: data['secret_fields'] was set before the body snapshot at
data['proxy_server_request']['body'] = copy.copy(data), so the full
raw headers (including auth tokens) ended up in the snapshot.
Fix (defense in depth):
1. Exclude 'secret_fields' when creating the body snapshot in
litellm_pre_call_utils.py (primary fix)
2. Strip 'secret_fields' in _sanitize_request_body_for_spend_logs_payload
as a secondary safeguard
secret_fields remains available on the live data dict for legitimate
downstream consumers (MCP, Responses API).
Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>
OpenRouter has dropped active endpoints for anthropic/claude-3.7-sonnet,
causing test_reasoning_content_completion to fail with a 404 "No endpoints
found" error. Switch to anthropic/claude-sonnet-4.5, which is current and
supports reasoning streaming.
/simplify follow-ups:
* Replace the two-``pop`` reach into ``cache_dict``/``ttl_dict`` with
the existing public ``InMemoryCache.delete_cache(key)`` — the same
idiom used elsewhere in the proxy. Bonus: ``delete_cache`` calls
``_remove_key`` which also handles ``expiration_heap`` consistency
the direct pops were silently leaking.
* JSON-encode the sorted scope list for the cache key instead of
``"|".join``. ``user_id`` / ``team_id`` / ``org_id`` / ``api_key``
are free-form strings and could contain a literal ``|`` — JSON
quoting escapes any in-string separator unambiguously.
* Extract ``_allowed_container_ids_cache_key()`` so the read and
invalidation sites compute the key the same way.
* Fix a placeholder-then-overwrite test construction: the
``__module__.split(".")[0] and "proxy_admin"`` line evaluated to a
literal string that was immediately overwritten with the real enum
value. Hoist the import and construct directly.
Address Greptile P2 follow-ups from the prior round:
* Cache ``_get_allowed_container_ids`` (60s LRU/TTL keyed by sorted
owner-scope tuple) so ``GET /v1/containers`` doesn't issue a fresh
``find_many`` against ``litellm_managedobjecttable`` on every list
call. Invalidate the caller's own cache entry when they record a
new owner so the just-created container shows up on their next list.
* Tighten the admin early-return in ``record_container_owner`` to skip
ONLY when there's literally no container ID to stamp. An admin with
identity (the master-key path populates ``user_id`` + ``api_key``)
flows through the normal record path so admin-created containers are
tracked like any other caller's. The truly-identity-less admin case
still falls through to the 403 below — correct fail-secure default.
Skill-cache invalidation gap (also flagged by Greptile) is moot: there
is no skill update endpoint exposed; ownership-affecting mutations are
only delete (already invalidates) and create (new ID, no cache entry
to update).
Substantial reduction (~765 LOC) without changing the security
boundary:
* Drop ContainerOwnershipStore and LiteLLMSkillsStore — both were
one-method-per-Prisma-call wrappers. Inline the calls instead,
matching the established pattern in vector_store_endpoints,
agent_endpoints, and mcp_server/db.py.
* Drop the prisma_client is None in-memory fallback. Production
deploys always have Prisma; running ownership-critical paths on a
process-local dict is a security footgun in the dev-mode case it
was meant to support, and complicates every code path with a
branch. Fail-secure: skip recording if Prisma is unavailable, and
treat reads as "not found" (admin-only).
* Drop the hand-rolled module-level cache. Replace with the existing
litellm.caching.in_memory_cache.InMemoryCache, which already has
TTL + max-size + eviction tested in its own module. Sentinel string
for negative caching since InMemoryCache can't disambiguate "miss"
from "cached as None".
* Tests: drop coverage for removed code paths (in-memory fallback,
hand-rolled cache internals). Keep tests for actual behavior (cache
hit-rate, negative caching, owner check, list filtering,
identity-less reject, admin bypass).
Two cleanups:
* ``LiteLLMSkillsHandler.create_skill`` raised ``HTTPException`` for
identity-less callers, importing FastAPI from a ``litellm/llms/``
module — that violates the project rule that FastAPI lives only
under ``proxy/``. Switch to ``ValueError`` (the same shape the rest
of the handler uses for not-found/forbidden) and update the test.
* The proxy-auth body bouncer derived its observability ban list from
``_supported_callback_params`` only, missing
``_request_blocked_callback_params`` (where ``gcs_bucket_name`` and
``gcs_path_service_account`` live). Two recently-merged sibling PRs
(#27019 added the deny list, #27081 added the test asserting these
are rejected at the request body root) crossed without folding them
together. Union the GCS deny list into the bouncer's derivation so
the single source of truth covers both code paths.
UNSCOPED_RESOURCE_OWNER_SCOPE collapsed every caller without an
identity field (no user_id / team_id / org_id / api_key / token) into
a single shared owner — a cross-tenant access primitive: any two such
callers could see and delete each other's containers and skills.
Drop the sentinel. ``get_primary_resource_owner_scope`` returns
``None`` and ``get_resource_owner_scopes`` returns ``[]`` for
identity-less callers. ``record_container_owner`` and
``LiteLLMSkillsHandler.create_skill`` now reject creates from
identity-less callers with a 403 instead of stamping the placeholder.
Read paths already deny ``owner is None`` correctly so legacy rows
(if any) are admin-only.
LITELLM_ALLOW_UNTRACKED_CONTAINER_ACCESS and
LITELLM_ALLOW_UNOWNED_SKILL_ACCESS were operator-toggleable opt-outs
for the cross-tenant access primitive this PR closes — flipping either
on re-enabled exactly the VERIA-20 read path. Default-secure with no
escape hatch matches sibling fixes (vector-store cred isolation, semantic
cache key isolation, user_config strip): all rejected the
opt-out-of-security pattern.
Untracked containers and unowned skills (rows that pre-date this
enforcement) are admin-only. Non-admin owners need to either re-create
via the now-tracked flow or have an admin assign ``created_by`` on the
existing row. Update tests to assert the strict-only behaviour.
Two cleanups from the /simplify pass:
* ``_CONTAINER_OWNER_CACHE`` and ``_SKILL_CACHE`` now LRU-evict via
``OrderedDict.popitem(last=False)`` instead of full ``clear()`` at
capacity. Full clears converted a steady-state cached workload into a
periodic full-DB-load oscillation as the cache repopulated from zero
and cleared again. Reads now ``move_to_end`` so the just-touched
entry survives the next eviction. Mirrors the pre-existing LRU
pattern in ``_remember_container_owner``.
* ``LiteLLM_ManagedObjectTable.file_purpose`` Literal now includes
``"container"`` so Pydantic validation accepts rows written by the
ownership store.
The flag was an opt-in escape hatch for the cross-tenant leak the rest
of the patch closes — flipping it on (env var or constructor param)
re-enables exactly the VERIA-54 primitive on either backend. There is
no operational need that the secure path doesn't already meet:
- Qdrant: legacy points without ``litellm_cache_key`` payload are
excluded by the must-clause filter and treated as misses; new sets
populate the cache key, so cold-start lasts only as long as the
natural cache rebuild.
- Redis: existing unscoped index can't carry the new schema; the init
path falls back to ``{name}_isolated`` (and recreates it on stale
schema), leaving the legacy index untouched.
Drop the constructor param, env-var fallback, ``_using_legacy_unscoped_index``
flag, the legacy-reuse branch in ``_init_semantic_cache``, and the
matching guards in set/get paths. Update tests to drop the legacy-mode
cases and assert the secure-only behaviour.
