- test_avertex_batch_prediction: Add google.auth.default mock and env vars
so the test doesn't depend on real GCP credentials (was already a unit
test with mocked HTTP, just missing auth mock)
- test_async_create_batch[openai]: Add DNS pre-check that skips gracefully
when api.openai.com is unreachable instead of failing after 4 retries
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The logging tests mutate many more litellm globals than guardrails tests
(turn_off_message_logging, s3_callback_params, datadog_params, service_callback,
etc.). The initial save/restore list only covered callbacks and a few basics,
causing state leaks like redaction settings bleeding across tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add LITELLM_LOG=WARNING to suppress verbose DEBUG log output
- Remove -s flag to stop capturing all stdout
- Bump xdist workers from -n 2 to -n 4
- Add --timeout=120 for safety
- Rewrite conftest.py to use save/restore pattern (matching guardrails_tests)
instead of per-function importlib.reload + event loop creation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The retrieve_batch endpoint sets batch status to "complete" but never set
batch_processed=True, permanently blocking file deletion. CheckBatchCost
(the safety net) also excluded completed batches from its primary query,
so batch_processed was never set by either path.
Three fixes:
1. update_batch_in_database sets batch_processed=True when status reaches
"complete", with old-schema fallback retry
2. CheckBatchCost primary query no longer excludes complete/completed
(batch_processed=False filter prevents reprocessing)
3. retrieve_batch early-return now includes "complete" (DB-normalized
spelling) to avoid unnecessary provider re-polls
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_parallel_function_call_anthropic_error_msg was flaky because other
tests set litellm.modify_params=True without resetting it. When True,
the validation adds a dummy tool instead of raising UnsupportedParamsError.
Fix: save/restore modify_params around the test.
test_vertex_ai_llama_tool_calling failed on intermittent 404 from the
Llama model endpoint in us-east5. Fix: skip on NotFoundError like
the existing RateLimitError handling.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test used a fixed 40s sleep before checking spend logs, but async
spend logging in CI sometimes takes longer to flush. Replace with a
polling loop (10s interval, 120s max) that exits early on success.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The eager tiktoken tests were clearing all litellm modules from
sys.modules and re-importing, creating new module objects with different
class identities. This broke unittest.mock.patch for all subsequent
tests on the same xdist worker. Running these tests in subprocesses
provides perfect isolation.
Fixes: test_metadata_passed_to_custom_callback_codex_models,
test_oidc_github_success, test_oidc_google_cached,
test_oidc_google_failure,
test_encrypted_content_affinity_bypasses_rpm_limits, and 5 others.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test asserts that a ttl=0 cached entry expires immediately, so the
second call should not return cached content. Both calls used the same
mock_response text, making the content != assertion always fail. Use
different mock_response values so a cache hit is distinguishable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The isolate_litellm_state conftest fixture saved/restored litellm.callbacks
but never cleared it before each test, unlike the other callback lists. It
also didn't handle litellm.model_fallbacks. Leaked callbacks and fallback
config caused mocked tests to route through Router/fallback paths, hitting
real APIs with mock keys.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The check_batch_cost_job runs on a 10-40s interval and sets
batch_processed=True after sending the S3 callback. 30s (6×5s) wasn't
enough margin; 50s (10×5s) covers the worst-case poll interval plus
processing time, while still being 3.6x faster than the original 180s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 4 integration tests were flaky in CI because the AsyncHTTPHandler.post
mock was bypassed when aiohttp transport is used. Mock at the higher
BaseLLMHTTPHandler.async_response_api_handler level instead, which
bypasses the HTTP layer entirely while still exercising router deployment
selection, pre-call checks, and response post-processing (item ID rewriting).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace real OpenAI/Anthropic/Bedrock API calls with mock_response in
~20 cache tests to eliminate network-dependent flakiness
- Remove -x (fail-fast) from caching_unit_tests so all failures are reported
- Add parallelism: 2 with circleci tests run --split-by=timings
- Improve pip dependency cache key (v2-caching-deps) with fallback key
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests failed intermittently in CI (-n 8 workers) because cached
AsyncHTTPHandler instances from other tests bypassed the class-level
mock on AsyncHTTPHandler.post, causing real requests to OpenAI with
mock API keys. Router retries (default 2) masked the root cause.
- Add autouse fixture to flush litellm.in_memory_llm_clients_cache
before/after each test so mocks always apply to fresh clients
- Set num_retries=0 on all Router instances to surface mock failures
immediately instead of silently retrying
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a completed batch is served from the DB cache, _hidden_params was empty,
causing the managed files hook to skip output_file_id translation from raw
provider IDs to unified IDs. This fix populates unified_batch_id and model_id
on the early-return path, with a guard against double-encoding when the DB
already stores unified IDs.
Also reduces file deletion retry delay (20s→5s), reruns (5→2), and CI timeout
(30m→15m) to cut worst-case runtime from ~16min to ~4min.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Llama-3.2-3B-Instruct-Turbo is no longer available as a serverless model
on Together AI. Switch to Llama-3.3-70B-Instruct-Turbo which is still
available and has cost data in the model prices map.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add num_retries=0 to the async acompletion call to prevent retries
when the mock returns invalid response data. The test only validates
request payload format, not retry behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_router_context_window_check_pre_call_check_out_group: replace deprecated
gpt-3.5-turbo-1106 (removed from model_cost, returns max_input_tokens=0) with
gpt-4.1-mini + mock_response
- test_async_fallbacks: filter "Task was destroyed but it is pending" messages
that leak from parallel test execution in CI
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_post_call_rule_streaming in test_rules.py sets
litellm.post_call_rules but never cleans up. Since
pytest_collection_modifyitems sorts tests by name across modules,
the leaked rule causes failures in test_streaming.py,
test_register_model.py, and test_sagemaker.py.
Add pre_call_rules and post_call_rules to the isolate_litellm_state
fixture's save/restore and clear lists.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test relied on a global side effect (customLogger initialization in
litellm_logging.py) from prior tests' success callbacks to dispatch
failure callbacks. When tests run in parallel by file, no prior test
initializes customLogger, so the Router's deployment_callback_on_failure
was never invoked and cooldowns were never set.
Rewrite to directly call deployment_callback_on_failure with a proper
RateLimitError containing retry-after headers, testing the cooldown
logic without depending on the logging callback chain.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_hanging_request_azure: mock httpx.AsyncClient.send to simulate slow
response instead of racing real network latency against a 10ms timeout.
The old non-existent deployment (gpt-4o-new-test) returned 404 faster
than the timeout, causing NotFoundError instead of APITimeoutError.
- test_completion_together_ai_llama: update model from deprecated
Meta-Llama-3.1-8B-Instruct-Turbo to Llama-3.2-3B-Instruct-Turbo
(Together AI removed the old model from serverless).
- conftest.py: clear litellm.callbacks list before each test to prevent
proxy hooks (SkillsInjectionHook, VirtualKeyModelMaxBudgetLimiter)
from leaking across tests via Router initialization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test was failing because it depended on real API calls to deprecated
models. Now uses mock_response to validate streaming through the router
without external dependencies.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_async_fallbacks_streaming: replace deprecated gpt-3.5-turbo fallback
with gpt-4o-mini, fix use of module-level kwargs variable
- test_ausage_based_routing_fallbacks: remove Redis dependency to prevent
shared state across parallel CI containers (test already uses mock_response)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Router testing: add CircleCI parallelism=4 with timing-based test splitting
- Guardrails testing: add pytest-xdist -n 4, suppress DEBUG logs with LITELLM_LOG=WARNING
- Rewrite conftest.py in both test dirs for xdist compatibility (save/restore pattern)
- Fix module-level Router instances in test_router_fallback_handlers, test_router_custom_routing, test_acooldowns_router
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add sagemaker_nova provider for Nova models on SageMaker
Add support for custom/fine-tuned Amazon Nova models (Nova Micro, Nova Lite,
Nova 2 Lite) deployed on SageMaker Inference real-time endpoints.
Nova uses OpenAI-compatible request/response format with additional
Nova-specific parameters (top_k, reasoning_effort, allowed_token_ids,
truncate_prompt_tokens) and requires stream:true in the request body.
Nova endpoints also reject 'model' in the request body.
Changes:
- New provider: sagemaker_nova/<endpoint-name>
- SagemakerNovaConfig inherits from SagemakerChatConfig
- Override transform_request to strip 'model' from request body
- Override supports_stream_param_in_request_body (True for Nova)
- Extend get_supported_openai_params with Nova-specific params
- Refactored SagemakerChatConfig to use custom_llm_provider param
instead of hardcoded strings (backwards-compatible)
- Consolidated main.py routing for sagemaker_chat and sagemaker_nova
- 22 unit tests + 9 integration tests (skip-gated)
- Documentation with SDK, streaming, multimodal, and proxy examples
- All tests verified against live SageMaker Nova endpoint
* fix: move integration tests to tests/local_testing/ per test directory policy
* fix: remove unused module-level SagemakerNovaConfig instance
The sagemaker_nova_config singleton was never imported or used — the
ProviderConfigManager creates its own instance via the lambda registered
in utils.py. Removing this leftover boilerplate.
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Map provider finish_reason "content_filtered" to the OpenAI-compatible "content_filter" and extend core_helpers tests to cover this case.
Made-with: Cursor
* fix: restore offline tiktoken cache for non-root envs
Made-with: Cursor
* chore: mkdir for custom tiktoken cache dir
Made-with: Cursor
* test: patch tiktoken.get_encoding in custom-dir test to avoid network
Made-with: Cursor
* test: clear CUSTOM_TIKTOKEN_CACHE_DIR in helper for test isolation
Made-with: Cursor
* test: restore default_encoding module state after custom-dir test
Made-with: Cursor
* fix(bedrock): respect s3_region_name for batch file uploads (#23569)
* fix(bedrock): respect s3_region_name for batch file uploads (GovCloud fix)
* fix: s3_region_name always wins over aws_region_name for S3 signing (Greptile feedback)
* fix: _filter_headers_for_aws_signature - Bedrock KB (#23571)
* fix: _filter_headers_for_aws_signature
* fix: filter None header values in all post-signing re-merge paths
Addresses Greptile feedback: None-valued headers were being filtered
during SigV4 signing but re-merged back into the final headers dict
afterward, which would cause downstream HTTP client failures.
Made-with: Cursor
* feat(router): tag_regex routing — route by User-Agent regex without per-developer tag config (#23594)
* feat(router): add tag_regex support for header-based routing
Adds a new `tag_regex` field to litellm_params that lets operators route
requests based on regex patterns matched against request headers — primarily
User-Agent — without requiring per-developer tag configuration.
Use case: route all Claude Code traffic (User-Agent: claude-code/x.y.z) to
a dedicated deployment by setting:
tag_regex:
- "^User-Agent: claude-code\\/"
in the deployment's litellm_params. Works alongside existing `tags` routing;
exact tag match takes precedence over regex match. Unmatched requests fall
through to deployments tagged `default`.
The matched deployment, pattern, and user_agent are recorded in
`metadata["tag_routing"]` so they flow through to SpendLogs automatically.
* fix(tag_regex): address backwards-compat, metadata overwrite, and warning noise
Three issues from code review:
1. Backwards-compat: `has_tag_filter` was widened to activate on any non-empty
User-Agent, which would raise ValueError for existing deployments using plain
tags without a `default` fallback. Fix: only activate header-based regex
filtering when at least one candidate deployment has `tag_regex` configured.
2. Metadata overwrite: `metadata["tag_routing"]` was overwritten for every
matching deployment in the loop, leaving inaccurate provenance when multiple
deployments match. Fix: write only for the first match.
3. Warning noise: an invalid regex pattern logged one warning per header string
rather than once per pattern. Fix: compile first (catching re.error once),
then iterate over header strings.
Also adds two new tests covering these cases, and adds docs page for
tag_regex routing with a Claude Code walk-through.
* refactor(tag_regex): remove unnecessary _healthy_list copy
* docs: merge tag_regex section into tag_routing.md, remove standalone page
- Add ## Regex-based tag routing (tag_regex) section to existing
tag_routing.md instead of a separate page
- Remove tag_regex_routing.md standalone doc (odd UX to have a separate
page for a sub-feature)
- Remove proxy/tag_regex_routing from sidebars.js
- Add match_any=False debug warning in tag_based_routing.py when regex
routing fires under strict mode (regex always uses OR semantics)
* fix(tag_regex): address greptile review - security docs, strict-mode enforcement, validation order
- Strengthen security note in tag_routing.md: explicitly state User-Agent
is client-supplied and can be set to any value; frame tag_regex as a
traffic classification hint, not an access-control mechanism
- Move tag_regex startup validation before _add_deployment() so an invalid
pattern never leaves partial router state
- Enforce match_any=False strict-tag policy: when a deployment has both
tags and tag_regex and the strict tag check fails, skip the regex fallback
rather than silently bypassing the operator's intent
- Extract per-deployment match logic into _match_deployment() helper to
keep get_deployments_for_tag() readable
- Add two new tests: strict-mode blocks regex fallback, regex-only
deployment still matches under match_any=False
* fix(ci): apply Black formatting to 14 files and stabilize flaky caplog tests
- Run Black formatter on 14 files that were failing the lint check
- Replace caplog-based assertions in TestAliasConflicts with
unittest.mock.patch on verbose_logger.warning for xdist compatibility
- The caplog fixture can produce empty text in pytest-xdist workers
in certain CI environments, causing flaky test failures
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Rewrite Default Team Settings UI from Tremor to antd with hardcoded fields,
fix default team params not applying during team creation or persisting
across proxy restarts, remove dead code, and add comprehensive tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Handle response.failed, response.incomplete, and response.cancelled terminal events in background streaming
Previously the background streaming task only handled response.completed and
hardcoded the final status to "completed". This missed three other terminal
event types from the OpenAI streaming spec, causing failed/incomplete/cancelled
responses to be incorrectly marked as completed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Committed-By-Agent: claude
* Remove unused terminal_response_data variable
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Committed-By-Agent: claude
* Address code review: derive fallback status from event type, rewrite tests as integration tests
1. Replace hardcoded "completed" fallback in response_data.get("status")
with _event_to_status lookup so that response.incomplete and
response.cancelled events get the correct fallback if the response
body ever omits the status field.
2. Replace duplicated-logic unit tests with integration tests that
exercise background_streaming_task directly using mocked streaming
responses and assert on the final update_state call arguments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Committed-By-Agent: claude
* Remove dead mock_processor and unused mock_response parameter from test helper
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Committed-By-Agent: claude
* Remove FastAPI and UserAPIKeyAuth imports from test file
These types were only used as Mock(spec=...) arguments. Drop the spec
constraints and remove the top-level imports to avoid pulling FastAPI
into test files outside litellm/proxy/.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Committed-By-Agent: claude
* Log warning when streaming response has no body_iterator
If base_process_llm_request returns a non-streaming response (no
body_iterator), log a warning since this likely indicates a
misconfiguration or provider error rather than a successful completion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Committed-By-Agent: claude
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* PR #22867 added _remove_scope_from_cache_control for Bedrock and Azure AI but omitted Vertex AI. This applies the same pattern to VertexAIPartnerModelsAnthropicMessagesConfig."
* PR #22867 added _remove_scope_from_cache_control for Bedrock and Azure AI but omitted Vertex AI. This applies the same pattern to VertexAIPartnerModelsAnthropicMessagesConfig."
* PR #22867 added _remove_scope_from_cache_control to AzureAnthropicMessagesConfig
but missed VertexAIPartnerModelsAnthropicMessagesConfi Rather than duplicating the method again, moved it up to the base AnthropicMessagesConfig so all providers
inherit it, and removed the now-redundant copy from the Azure AI subclass.
* PR #22867 added _remove_scope_from_cache_control to AzureAnthropicMessagesConfig
but missed VertexAIPartnerModelsAnthropicMessagesConfi Rather than duplicating the method again, moved it up to the base AnthropicMessagesConfig so all providers
inherit it, and removed the now-redundant copy from the Azure AI subclass.
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Use `min`/`max` instead of `_min`/`_max` for Prisma group_by input
parameters. The underscore-prefixed names are output keys, not input
kwargs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Verifies that validate_key_team_change does not call can_team_access_model
for the "all-team-models" sentinel, allowing keys with that value to be
moved between teams without model validation failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests were outdated after #23472 added pagination (take/order) to find_many
and stale-row cleanup via update_many. Updated assertions to match new call
signatures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The self-exclusion filter compared raw key strings against SHA-256
hashed tokens from the DB, so keys were never excluded and
double-counting persisted. Now hash data.key before comparison.
Also add tpm_limit_type/rpm_limit_type to _throughput_fields_changed
guard, fall back to existing_key_row.team_id for team limit checks
(matching the org pattern), and add team self-exclusion test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Only run org validation (get_org_object + _check_org_key_limits) when
the update actually touches throughput-related fields (tpm_limit,
rpm_limit, or organization_id). Previously, any update to a key
belonging to an org would trigger the check, which would fail with a
400 if the org had been deleted — blocking unrelated field changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When updating a key, _check_org_key_limits and _check_team_key_limits
would include the key being updated in the find_many results, causing
its current limits to be counted twice (once from the DB query, once
from the new requested limits). This caused false 400 errors on valid
limit adjustments.
Fix: exclude the key being updated (by matching token) from the
allocated totals before checking limits.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of hardcoding SPEND_PER_REQUEST (which broke when the model
changed from gpt-3.5-turbo-0301 to gpt-3.5-turbo), make a single
calibration request first, poll for its spend, and use that as the
per-request cost. Fails fast with pytest.fail() after 5 retries if
calibration cannot determine the cost.
Also fixes a bug in test_basic_spend_accuracy where the user spend
assertion error message referenced user_info['info'] instead of
user_info['user_info'].
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Increase wait timeout to 90s and pytest.fail() instead of silently
continuing, so the failure message points at the real cause.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
gpt-3.5-turbo-0301 was removed from the model cost map, so every call
had response_cost=0 and team member spend never increased. The wait
helper also returned True after 3s regardless of whether spend updated.
- Switch fake-openai-endpoint to gpt-3.5-turbo (has pricing in cost map)
- Remove premature early-return in wait_for_team_member_spend_update
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Increase max_retries from 6 to 9 and retry_delay from 10s to 20s
(180s total wait, up from 60s) to give batch cost tracking more time
to finish before cleanup attempts file deletion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
