- Promote _fetch_managed_vector_stores_by_uuids from @staticmethod to a module-level
async helper get_managed_vector_store_rows_by_uuids, following the same standalone
helper pattern as get_team_object / get_key_object so the hot-path DB read is a
named importable function rather than an inline prisma_client.db.* call
- Pass no-log=True to both inner _call_aresponses sub-calls so they do not fire
independent billing/monitoring callbacks; cost is accumulated in the synthesized
response's _hidden_params for the outer responses() call
- Add test_H11b covering the primary queries (plural array) function-tool schema,
complementing H11 which exercises only the backward-compat singular query path
Made-with: Cursor
- Re-add should_use_emulated_file_search() to emulated_handler.py so H5/H6/H7/H13 tests don't fail with ImportError
- Remove per-file-id deduplication from _build_search_results_for_include so all chunks are returned (matching OpenAI native file_search behaviour); update test_H14 to assert 2 results
- Extract raw prisma DB query in check_vector_store_ids_access into a static _fetch_managed_vector_stores_by_uuids helper so the hot request path uses a named, testable function instead of an inline prisma_client.db.* call
- Remove developer-local path from test module docstring
Made-with: Cursor
Strip internal logging ids from emulated sub-calls, dedupe included search_results by file_id, clean unused imports, and add unit coverage for dedupe behavior.
Made-with: Cursor
Include all function_call items when building emulated follow-up input and update tests to assert real emulated routing + Responses-format function tool structure.
Made-with: Cursor
* fix(test): add missing mocks for test_streamable_http_mcp_handler_mock
The test was missing mocks for extract_mcp_auth_context and set_auth_context,
causing the handler to fail silently in the except block instead of reaching
session_manager.handle_request. This mirrors the fix already applied to the
sibling test_sse_mcp_handler_mock.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): route OpenAI models through chat completions in pass-through tests
The test_anthropic_messages_openai_model_streaming_cost_injection test fails
because the OpenAI Responses API returns 400 for requests routed through the
Anthropic Messages endpoint. Setting LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES=true
routes OpenAI models through the stable chat completions path instead.
Cost injection still works since it happens at the proxy level.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): fix assemblyai custom auth and router wildcard test flakiness
1. custom_auth_basic.py: Add user_role='proxy_admin' so the custom auth
user can access management endpoints like /key/generate. The test
test_assemblyai_transcribe_with_non_admin_key was hidden behind an
earlier -x failure and was never reached before.
2. test_router_utils.py: Add flaky(retries=3) and increase sleep from 1s
to 2s for test_router_get_model_group_usage_wildcard_routes. The async
callback needs time to write usage to cache, and 1s is insufficient on
slower CI hardware.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* ci: retrigger CI pipeline
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(mypy): use LitellmUserRoles enum instead of raw string in custom_auth_basic
Fixes mypy error: Argument 'user_role' has incompatible type 'str'; expected 'LitellmUserRoles | None'
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: don't close HTTP/SDK clients on LLMClientCache eviction (#22926)
* fix: don't close HTTP/SDK clients on LLMClientCache eviction
Removing the _remove_key override that eagerly called aclose()/close()
on evicted clients. Evicted clients may still be held by in-flight
streaming requests; closing them causes:
RuntimeError: Cannot send a request, as the client has been closed.
This is a regression from commit fb72979432. Clients that are no longer
referenced will be garbage-collected naturally. Explicit shutdown cleanup
happens via close_litellm_async_clients().
Fixes production crashes after the 1-hour cache TTL expires.
* test: update LLMClientCache unit tests for no-close-on-eviction behavior
Flip the assertions: evicted clients must NOT be closed. Replace
test_remove_key_closes_async_client → test_remove_key_does_not_close_async_client
and equivalents for sync/eviction paths.
Add test_remove_key_removes_plain_values for non-client cache entries.
Remove test_background_tasks_cleaned_up_after_completion (no more _background_tasks).
Remove test_remove_key_no_event_loop variant that depended on old behavior.
* test: add e2e tests for OpenAI SDK client surviving cache eviction
Add two new e2e tests using real AsyncOpenAI clients:
- test_evicted_openai_sdk_client_stays_usable: verifies size-based eviction
doesn't close the client
- test_ttl_expired_openai_sdk_client_stays_usable: verifies TTL expiry
eviction doesn't close the client
Both tests sleep after eviction so any create_task()-based close would
have time to run, making the regression detectable.
Also expand the module docstring to explain why the sleep is required.
* docs(AGENTS.md): add rule — never close HTTP/SDK clients on cache eviction
* docs(CLAUDE.md): add HTTP client cache safety guideline
* [Fix] Install bsdmainutils for column command in security scans
The security_scans.sh script uses `column` to format vulnerability
output, but the package wasn't installed in the CI environment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle string callback values in prometheus multiproc setup
When callbacks are configured as a plain string (e.g., `callbacks: "my_callback"`)
instead of a list, the proxy crashes on startup with:
TypeError: can only concatenate str (not "list") to str
Normalize each callback setting to a list before concatenating.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* bump: version 1.82.2 → 1.82.3
* fix(test): update test_startup_fails_when_db_setup_fails for opt-in enforcement
The --enforce_prisma_migration_check flag is now required to trigger
sys.exit(1) on DB migration failure, after #23675 flipped the default
behavior to warn-and-continue.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(cost_calculator): use model name for per-request custom pricing when router_model_id has no pricing
When custom pricing is passed as per-request kwargs (input_cost_per_token/output_cost_per_token),
completion() registers pricing under the model name, but _select_model_name_for_cost_calc was
selecting the router deployment hash (which has no pricing data), causing response_cost to be 0.0.
Now checks whether the router_model_id entry actually has pricing before preferring it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Add avideo_create_character, avideo_get_character, avideo_edit, and avideo_extension
to the skip condition since Azure video calls don't use initialize_azure_sdk_client.
Tests now properly skip with expected behavior instead of failing:
- test_ensure_initialize_azure_sdk_client_always_used[avideo_create_character] ✓
- test_ensure_initialize_azure_sdk_client_always_used[avideo_get_character] ✓
- test_ensure_initialize_azure_sdk_client_always_used[avideo_edit] ✓
- test_ensure_initialize_azure_sdk_client_always_used[avideo_extension] ✓
Made-with: Cursor
The xdist-conditional reload (manual reset in xdist mode) was missing
attributes that importlib.reload resets, causing Azure connection errors.
The original conftest used importlib.reload unconditionally (even under
xdist) and that worked on main. Restore that behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The old conftest never flushed HTTP client cache. Adding flush_cache() before
every test forces new TCP connections to external APIs, causing transient
connection failures under xdist parallelism. Global state isolation is already
handled by _SCALAR_DEFAULTS reset.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_rerank.py sets litellm.api_base = "http://localhost:4000" which leaked
to all subsequent tests on the same xdist worker, causing connection failures
across every provider (Cohere, Azure, OpenAI, etc.).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The revert of 9711e3adfe left xdist tests without proper state isolation.
Module-level assignments like `litellm.num_retries = 3` in 12+ test files
pollute shared globals, and the fixture was saving/restoring contaminated
values instead of resetting to true defaults.
- Capture true litellm defaults at conftest import time and reset before
each test (local_testing + llm_translation)
- Make llm_translation/conftest.py xdist-safe (skip reload under xdist,
add state isolation)
- Replace asyncio.sleep(2) with polling in cooldown handler tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: Fixes https://github.com/BerriAI/litellm/issues/23185
* fix(responses/main.py): ensure litellm metadata custom cost works
* refactor: move all logging updates to a common function, to have just 1 place to update logging kwarg updates
The conftest fixtures were saving/restoring the current (potentially
contaminated) values of litellm globals like num_retries instead of
resetting to true defaults. Under xdist, module-level assignments
(e.g. `litellm.num_retries = 3` in 12+ test files) pollute the
shared module state and leak across tests in the same worker.
- Capture true litellm defaults at conftest import time and reset
before each test (local_testing + llm_translation)
- Make llm_translation/conftest.py xdist-safe (skip reload, add
state isolation)
- Replace asyncio.sleep(2) with polling in cooldown handler tests
- Add @pytest.mark.flaky to tests making real API calls under xdist
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Router tests: expand conftest save/restore to cover all globals mutated
by router tests (default_fallbacks, tag_budget_config, request_timeout,
enable_azure_ad_token_refresh, num_retries_per_request, model_cost,
token_counter). These were leaking across xdist workers.
Proxy tests: move test_proxy_utils.py (169 parametrized) and
test_proxy_server.py (72 parametrized) from part2 to part1, balancing
~370 vs ~360 tests (was ~129 vs ~600).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test intermittently fails in CI due to Redis cache write propagation
delays, causing the second call to miss the cache and hit OpenAI directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
LLM responses are non-deterministic and ClaudeAgentOptions doesn't expose
temperature control. The model occasionally returns unexpected short responses
(e.g. just "!") instead of the expected greeting. Add up to 3 retries with a
more explicit prompt to make the test deterministic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test waited a fixed 3s for async callbacks to populate log_queue.
Under xdist -n 4, CPU contention can delay the GLOBAL_LOGGING_WORKER
background task beyond 3s. Replace fixed sleeps with polling loops
(up to 10s) that break as soon as the expected condition is met.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Module-level mutations (litellm.num_retries=3 in test_langfuse_e2e_test.py
and test_amazing_s3_logs.py, litellm.success_callback=['langfuse']) run
at import time, BEFORE any function fixture. The save/restore pattern
captured these polluted values as 'originals' and kept restoring them.
Fix: capture litellm defaults when conftest.py is first imported (before
test modules), then reset to those true defaults before each test instead
of saving/restoring the current (potentially polluted) state.
ThrottlingException is a transient AWS rate-limit error unrelated to code
correctness. Skip the test instead of failing the CI pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Clear _in_memory_loggers before/after each test to prevent cached logger
instances (LangsmithLogger, SlackAlerting, etc.) from leaking stale state
- Add pre_call_rules, post_call_rules to list attrs save/restore
- Add vector_store_registry to scalar attrs save/restore
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_avertex_batch_prediction: Add google.auth.default mock and env vars
so the test doesn't depend on real GCP credentials (was already a unit
test with mocked HTTP, just missing auth mock)
- test_async_create_batch[openai]: Add DNS pre-check that skips gracefully
when api.openai.com is unreachable instead of failing after 4 retries
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The logging tests mutate many more litellm globals than guardrails tests
(turn_off_message_logging, s3_callback_params, datadog_params, service_callback,
etc.). The initial save/restore list only covered callbacks and a few basics,
causing state leaks like redaction settings bleeding across tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add LITELLM_LOG=WARNING to suppress verbose DEBUG log output
- Remove -s flag to stop capturing all stdout
- Bump xdist workers from -n 2 to -n 4
- Add --timeout=120 for safety
- Rewrite conftest.py to use save/restore pattern (matching guardrails_tests)
instead of per-function importlib.reload + event loop creation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The retrieve_batch endpoint sets batch status to "complete" but never set
batch_processed=True, permanently blocking file deletion. CheckBatchCost
(the safety net) also excluded completed batches from its primary query,
so batch_processed was never set by either path.
Three fixes:
1. update_batch_in_database sets batch_processed=True when status reaches
"complete", with old-schema fallback retry
2. CheckBatchCost primary query no longer excludes complete/completed
(batch_processed=False filter prevents reprocessing)
3. retrieve_batch early-return now includes "complete" (DB-normalized
spelling) to avoid unnecessary provider re-polls
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_parallel_function_call_anthropic_error_msg was flaky because other
tests set litellm.modify_params=True without resetting it. When True,
the validation adds a dummy tool instead of raising UnsupportedParamsError.
Fix: save/restore modify_params around the test.
test_vertex_ai_llama_tool_calling failed on intermittent 404 from the
Llama model endpoint in us-east5. Fix: skip on NotFoundError like
the existing RateLimitError handling.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test used a fixed 40s sleep before checking spend logs, but async
spend logging in CI sometimes takes longer to flush. Replace with a
polling loop (10s interval, 120s max) that exits early on success.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The eager tiktoken tests were clearing all litellm modules from
sys.modules and re-importing, creating new module objects with different
class identities. This broke unittest.mock.patch for all subsequent
tests on the same xdist worker. Running these tests in subprocesses
provides perfect isolation.
Fixes: test_metadata_passed_to_custom_callback_codex_models,
test_oidc_github_success, test_oidc_google_cached,
test_oidc_google_failure,
test_encrypted_content_affinity_bypasses_rpm_limits, and 5 others.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test asserts that a ttl=0 cached entry expires immediately, so the
second call should not return cached content. Both calls used the same
mock_response text, making the content != assertion always fail. Use
different mock_response values so a cache hit is distinguishable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The isolate_litellm_state conftest fixture saved/restored litellm.callbacks
but never cleared it before each test, unlike the other callback lists. It
also didn't handle litellm.model_fallbacks. Leaked callbacks and fallback
config caused mocked tests to route through Router/fallback paths, hitting
real APIs with mock keys.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The check_batch_cost_job runs on a 10-40s interval and sets
batch_processed=True after sending the S3 callback. 30s (6×5s) wasn't
enough margin; 50s (10×5s) covers the worst-case poll interval plus
processing time, while still being 3.6x faster than the original 180s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 4 integration tests were flaky in CI because the AsyncHTTPHandler.post
mock was bypassed when aiohttp transport is used. Mock at the higher
BaseLLMHTTPHandler.async_response_api_handler level instead, which
bypasses the HTTP layer entirely while still exercising router deployment
selection, pre-call checks, and response post-processing (item ID rewriting).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace real OpenAI/Anthropic/Bedrock API calls with mock_response in
~20 cache tests to eliminate network-dependent flakiness
- Remove -x (fail-fast) from caching_unit_tests so all failures are reported
- Add parallelism: 2 with circleci tests run --split-by=timings
- Improve pip dependency cache key (v2-caching-deps) with fallback key
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests failed intermittently in CI (-n 8 workers) because cached
AsyncHTTPHandler instances from other tests bypassed the class-level
mock on AsyncHTTPHandler.post, causing real requests to OpenAI with
mock API keys. Router retries (default 2) masked the root cause.
- Add autouse fixture to flush litellm.in_memory_llm_clients_cache
before/after each test so mocks always apply to fresh clients
- Set num_retries=0 on all Router instances to surface mock failures
immediately instead of silently retrying
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a completed batch is served from the DB cache, _hidden_params was empty,
causing the managed files hook to skip output_file_id translation from raw
provider IDs to unified IDs. This fix populates unified_batch_id and model_id
on the early-return path, with a guard against double-encoding when the DB
already stores unified IDs.
Also reduces file deletion retry delay (20s→5s), reruns (5→2), and CI timeout
(30m→15m) to cut worst-case runtime from ~16min to ~4min.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Llama-3.2-3B-Instruct-Turbo is no longer available as a serverless model
on Together AI. Switch to Llama-3.3-70B-Instruct-Turbo which is still
available and has cost data in the model prices map.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add num_retries=0 to the async acompletion call to prevent retries
when the mock returns invalid response data. The test only validates
request payload format, not retry behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_router_context_window_check_pre_call_check_out_group: replace deprecated
gpt-3.5-turbo-1106 (removed from model_cost, returns max_input_tokens=0) with
gpt-4.1-mini + mock_response
- test_async_fallbacks: filter "Task was destroyed but it is pending" messages
that leak from parallel test execution in CI
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sameer Kankute