* fix: patch Host-header auth bypass in get_request_route
Starlette reconstructs request.url from the Host header. A malformed
Host like `localhost/?x=1` causes Starlette to build the full URL as
`http://localhost/?x=1/health`, which url-parses to path="/". Since "/"
is in LiteLLMRoutes.public_routes, all protected routes became reachable
without authentication.
Fix: read scope["path"] (set by uvicorn from the HTTP request line,
not derivable from headers) instead of request.url.path. Sub-path
deployments are handled via scope["app_root_path"] / scope["root_path"],
mirroring Starlette's own base_url construction logic.
Affected variants confirmed fixed:
Host: localhost/?x=1
Host: localhost:4000/?x=1
Host: localhost/#test
Host: localhost:4000/#test
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* style: reduce comments in route fix
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: block credential fields in RAG ingest vector_store options
Credential fields (vertex_credentials, aws_access_key_id, api_key, etc.)
in ingest_options.vector_store are now rejected at the API boundary with
a 400 error. Credentials must be configured server-side.
Previously any authenticated user could supply a vertex_credentials dict
with type=external_account pointing credential_source.file at an
arbitrary path (e.g. /proc/1/environ) and token_url at an
attacker-controlled server. google-auth's identity_pool.Credentials
refresh() would read the file and POST its contents to the attacker.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: block /key/update self-escalation by assigned users
Non-admin users who were assigned a key (created_by != caller) could
update any non-budget field — models, rpm_limit, guardrails, etc. —
without admin authorization, allowing privilege self-escalation.
Gate: only the key creator (created_by == caller) may edit their own
key without admin check; budget changes always require admin regardless
of creator status. All other callers must pass _check_key_admin_access.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: block user-controlled api_base in RAG ingest vector_store options
A user-supplied api_base in ingest_options.vector_store caused the server
to forward its configured provider credentials (Gemini, OpenAI) to an
attacker-controlled endpoint via SSRF.
Add api_base to the blocked credential params set alongside api_key and
the existing credential fields.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: restrict /utils/transform_request to PROXY_ADMIN and apply body safety check
Any authenticated internal_user could POST arbitrary provider config
(aws_sts_endpoint, api_base, etc.) to /utils/transform_request and have
the server forward its credentials to an attacker-controlled endpoint.
- Gate the endpoint on PROXY_ADMIN role (403 for all other roles)
- Call is_request_body_safe() to reject banned params even for admins
- Convert ValueError from safety check to HTTP 400
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: apply banned-param check to /utils/transform_request
Without is_request_body_safe(), any authenticated user could pass
aws_sts_endpoint, api_base, or aws_web_identity_token to
/utils/transform_request and have the server forward its configured
provider credentials to an attacker-controlled endpoint during SDK
credential resolution.
Applies the same banned-param blocklist already used by LLM endpoints.
Endpoint remains accessible to all authenticated users.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: block SSRF via api_base in /prompts/test dotprompt YAML frontmatter
Any frontmatter key not in ["model","input","output"] flowed into
optional_params and was merged into the LLM call data dict, bypassing
is_request_body_safe. An attacker with any bearer key could set
api_base in YAML to redirect the outbound LLM request — including the
provider API key — to an attacker-controlled host.
Fix: call is_request_body_safe on the constructed data dict after
optional_params are merged, before invoking ProxyBaseLLMRequestProcessing.
ValueError from the banned-param check is surfaced as HTTP 400.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Update litellm/proxy/rag_endpoints/endpoints.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix: coerce nested config strings before banned-param check
_NESTED_CONFIG_KEYS descent used isinstance(nested, dict) which silently
skipped litellm_embedding_config when delivered as a JSON string via
multipart/form-data. Banned params (api_base, aws_sts_endpoint, etc.)
nested inside the stringified value were invisible to is_request_body_safe.
_NESTED_METADATA_KEYS already used _coerce_metadata_to_dict which parses
JSON strings before checking. Apply the same coercion to _NESTED_CONFIG_KEYS.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: replace substring match with prefix match in is_llm_api_route
mapped_pass_through_routes used `_llm_passthrough_route in route` (substring)
so any admin-only path whose URL contained a provider name (openai, anthropic,
azure, bedrock, etc.) was misclassified as an LLM API route and bypassed the
admin gate in non_proxy_admin_allowed_routes_check.
Confirmed live: non-admin key could GET /credentials/by_name/openai (read
masked provider API key) and DELETE /credentials/openai (delete credential).
Fix: use exact match or startswith(prefix + "/") — the same pattern used
everywhere else in RouteChecks — so only routes that actually start with a
passthrough prefix are allowed through.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: stabilize PR #27878 test failures
- key_management_endpoints: extend can_skip_admin_check to team keys so
team members with /key/update permission can update non-budget fields.
can_team_member_execute_key_management_endpoint already validates team
membership + permission and raises if unauthorized; reaching the admin
check on a team key means the caller was authorized.
- test: set created_by on mock key in
test_update_key_non_budget_fields_allowed_for_internal_user so
caller_is_creator resolves correctly (MagicMock default ≠ user_id).
- auth_utils.get_request_route: guard against non-dict request.scope
(e.g. MagicMock in unit tests) to prevent a MagicMock leaking into
UserAPIKeyAuth.request_route and failing Pydantic validation.
- ci: assign test_multipart_bypass_repro.py to the proxy-runtime shard
in test-unit-proxy-db.yml to satisfy the shard-coverage check.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(lint): add explicit str() cast in get_request_route for MyPy
scope.get() returns Any|None which MyPy cannot coerce to str implicitly.
Wrap both scope.get() calls in str() to satisfy the type checker.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: guard bare-/ root_path strip + make total_spend migration idempotent
auth_utils.get_request_route: when Starlette sets scope["app_root_path"]
to "/" (e.g. behind some middleware), the old stripping logic would
remove the leading slash from every path ("/team/new" → "team/new"),
breaking route matching and causing auth to misclassify protected routes.
Skip stripping when root_path is bare "/".
migration: add IF NOT EXISTS to total_spend ALTER TABLE so the migration
is safe to replay when a prior partial run already created the column.
Without this guard, prisma migrate deploy fails on CI DBs that were
partially migrated, causing all subsequent DB operations (including
/team/new) to 500.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: require creator still owns key for personal-key bypass in /key/update
caller_is_creator now requires both created_by == caller AND user_id ==
caller. Previously checking only created_by let a demoted admin who
originally created a key for another user continue editing non-budget
fields on it after reassignment, bypassing _check_key_admin_access.
Adds regression test: creator whose key was reassigned is blocked (403).
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: extract auth checks to fix PLR0915 + broaden max_budget assertion
internal_user_endpoints._update_single_user_helper exceeded 50 statements
(PLR0915). Extract authorization checks into _check_user_update_authz helper
to bring statement count under the limit.
test_validate_max_budget: assert "negative" (substring of both the local
"cannot be negative" and the CI "non-negative finite number" messages) so
the test is stable regardless of which exact wording the function uses.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* feat(proxy): skip disable_background_health_check models on GET /health when flag set
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix comment
* fix greptile comments
* Fix health check fallback kwargs
* Format health endpoint
* Harden direct health check kwargs compatibility for monkeypatched perform_health_check
Replace substring-based TypeError detection with unexpected-keyword checks
and a short retry chain (full kwargs, instrumentation only, filter only,
minimal) so partial stubs work regardless of which optional kwarg fails first.
Add proxy unit tests for legacy three-arg stubs and single-kwarg variants.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix black
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
The pre-existing s3:// / gcs:// custom-logger tests called
``get_instance_fn`` without ``config_file_path``, which means the
new runtime gate (refuse remote URLs unless invoked from a
config-file load) now raises ``ValueError`` before reaching the
mocked download paths. Each test was exercising the documented
startup config-file load scenario; pass ``config_file_path="/any/path"``
to make that intent explicit and route past the gate.
Affected: test_s3_download_success, test_gcs_download_success,
test_invalid_url_format, test_download_failure_handling,
test_file_cleanup.
* fix(proxy): resolve cache handling issues in _lookup_deprecated_key
- Updated the in-memory cache for deprecated key lookups to store a 3-tuple (active_token_id, cache_expires_at_ts, revoke_at_ts) instead of a 2-tuple, ensuring proper unpacking and backward compatibility.
- Removed duplicate cache reads and added logic to handle legacy cache entries gracefully.
- Enhanced unit tests to cover scenarios for cache hits, DB misses, and respect for revoke_at timestamps, ensuring robust handling of the grace-period key-rotation feature.
* refactor(proxy): streamline cache handling in _lookup_deprecated_key
- Simplified the cache retrieval logic by directly unpacking the 3-tuple cache entries, removing the need for backward compatibility checks for 2-tuple entries.
- Updated unit tests to ensure that pre-warmed 3-tuple cache entries are served correctly without unnecessary database lookups.
* chore(ci): add new unit test for deprecated key grace period
- Included `test_deprecated_key_grace_period.py` in the CI workflow to enhance coverage for deprecated key handling scenarios.
* fix(proxy): remove unnecessary check for revoke_at in _lookup_deprecated_key
- Eliminated the redundant check for None on revoke_at, streamlining the logic for handling deprecated keys in the cache. This change enhances the efficiency of the key lookup process.
* test(proxy): add end-to-end tests for deprecated key lookup behavior
- Introduced a new test class `TestDeprecatedKeyLookupDbE2E` to validate the behavior of deprecated key lookups against a real Prisma-backed database.
- The test ensures that old key hashes resolve correctly and that repeated lookups utilize the in-memory cache without errors.
- Cleaned up the `_lookup_deprecated_key` function by removing an unnecessary check for `revoke_at`, enhancing the efficiency of the key lookup process.
Caller-supplied tags (`x-litellm-tags` header, body `tags`, `metadata.tags`)
were silently dropped unless the key/team had
`metadata.allow_client_tags: true` set. Restore the documented behavior:
tags from the request always flow into `metadata.tags` and union with any
admin-configured static tags from key/team/project metadata.
Removes the `allow_client_tags` opt-in flag from the pre-call pipeline.
The flag was only ever read here; it has no schema or endpoint footprint,
so leftover values in existing key metadata are inert.
Test cleanup mirrors the simplification: drop the three tests that
verified the strip-when-not-opted-in path, drop the `allow_client_tags`
fixture lines from the merge/union tests.
Assert _push_in_memory_increments_to_redis runs after async_log_success_event when dual_cache.redis_cache is set, and is skipped when Redis is not configured.
Co-authored-by: Cursor <cursoragent@cursor.com>
Several tests parametrized over (model, api_key, ...) tuples or raw
token strings, causing pytest to embed those values in the test ID
and print them in CI logs. Refactored each affected test to keep the
same coverage without putting key material into parametrize.
- audio_tests/test_audio_speech.py: split env-var keys into separate
azure/openai test functions sharing a helper; sync_mode parametrize
preserved.
- audio_tests/test_whisper.py: split into openai_whisper /
azure_whisper functions sharing a helper; response_format parametrize
preserved.
- local_testing/test_embedding.py: single-case parametrize inlined.
- proxy_unit_tests/test_user_api_key_auth.py: 5 header parametrize
cases split into 5 named tests sharing an _assert helper.
- proxy_unit_tests/test_proxy_utils.py: 4 api_key_value cases split
into 4 named tests.
- test_litellm/proxy/auth/test_user_api_key_auth.py: 5 key-prefix
cases (Bearer / Basic / lowercase bearer / raw / AWS SigV4) split
into 5 named tests.
Verified: black clean; 14 refactored unit tests pass; pytest collects
audio/embedding tests with safe IDs (no key material in test IDs).
Replaces the previous intersect-with-team.access_group_ids check, which
made the override unreachable in practice (the team-gate fallback already
covered every case the intersection allowed). The override now resolves
each of the key's access_group_ids via get_access_object and accepts the
group only if its assigned_team_ids includes the key's team_id, or its
assigned_key_ids includes the key's token. This fulfills the original ask
(a key can extend a team's allow-list via a group the admin granted to
that team or that specific key) while still rejecting foreign groups
referenced by team members of other teams.
A team member could set any access_group_ids on their key (e.g. a group
assigned only to a different team) and override the team's model
restriction. Intersect the key's access_group_ids with team_object.access_group_ids
in _key_access_group_grants_model so foreign groups are dropped before
model expansion. Adds a regression test that asserts expansion is never
called for foreign groups.
Three review items addressed:
* **Veria (Medium): SSRF via redirect.** ``fetch_validated_image_bytes``
was calling ``validate_url(url)`` once and then fetching with the
default httpx client, so a 3xx to an internal IP would have been
followed unvalidated. Switched to ``async_safe_get`` (the existing
SSRF primitive used elsewhere in the codebase) which walks each
redirect hop, re-validates, and rejects redirects to blocked
networks. Default ``litellm.user_url_validation`` is True so
protection is on out of the box.
* **Greptile (P2): SVG can embed JS.** Removed ``image/svg+xml`` from
the allowed-Content-Type set. The hardcoded response media type
(``image/jpeg`` / ``image/x-icon``) means a real SVG body wouldn't
render as SVG anyway in modern browsers — the allowlist entry was
giving up XSS surface for no actual SVG-rendering benefit. If real
SVG support is wanted later, that's a deliberate feature PR with CSP
/ nosniff bundled.
* **Greptile (P2): cache-write OSError drops validated bytes.** When
the upstream fetch succeeded but ``open(cache_path, "wb")`` raised
(read-only assets dir), the bytes were discarded and the default
logo was served — a silent regression for that deployment. Now
serve the validated bytes inline via ``Response(...)`` as a fallback
before falling back to default.
Tests:
- Replaced low-level mocks of ``validate_url`` with mocks of
``async_safe_get`` directly, exercising the helper's contract
rather than the SSRF primitive's internals.
- New ``test_rejects_svg_content_type`` confirms SVG is blocked.
- ``test_get_image_cache_logic`` fixture now sets
``mock_response.is_redirect = False`` so ``async_safe_get`` doesn't
treat the Mock's truthy attribute as a redirect.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three CI failures from the previous push, all addressed:
* ``lint`` (mypy): ``async_client.get(url, **request_kwargs)`` confused
mypy because ``AsyncHTTPHandler.get``'s second positional arg is typed
``bool | None``. Switched to an explicit branch:
``await async_client.get(rewritten_url, headers={"host": host_header})``
for the HTTP-rewritten case, plain ``get(rewritten_url)`` otherwise.
* ``proxy-infra`` /
``test_get_image_custom_local_logo_bypasses_cache``: the existing
test set ``UI_LOGO_PATH=/app/custom_logo.jpg`` with no
``LITELLM_ASSETS_PATH``, asserting the path was served verbatim. That
was the LFI behaviour the new path-containment guard closes. Updated
the test to set ``LITELLM_ASSETS_PATH=/app`` so the path is inside an
allowed root, and patched the helper's ``realpath`` / ``isfile`` to
go along with the mocked filesystem. Test intent (bypass cache when
``UI_LOGO_PATH`` is local) is preserved.
* ``auth-and-jwt`` / ``test_get_image_cache_logic``: existing test
built a ``Mock`` response without ``headers``, so the new
Content-Type check tripped on ``Mock().split(";")[0]``. Two fixes:
1. Set ``mock_response.headers = {"content-type": "image/jpeg"}``
on the test (matches the real upstream contract — a logo CDN
always sets a Content-Type).
2. Make ``fetch_validated_image_bytes`` defensive: if the
Content-Type header is missing or non-string, treat as non-image
and fall back to default. Closes a subtle hole — pre-fix, an
upstream that omits Content-Type entirely would have served
arbitrary bytes under the ``image/jpeg`` wrapper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI surfaced two issues from the previous commit:
1. ``general_settings`` and ``master_key`` were still imported at the top
of ``get_logging_payload`` but had no remaining users after the
master-key hash-detection blocks were removed. Drop the import.
2. ``tests/proxy_unit_tests/test_user_api_key_auth.py::test_x_litellm_api_key``
and ``tests/proxy_unit_tests/test_key_generate_prisma.py::test_master_key_hashing``
asserted ``valid_token.token == hash_token(master_key)`` — the
pre-alias behavior. The new contract is
``valid_token.token == LITELLM_PROXY_MASTER_KEY_ALIAS`` (and !=
``hash_token(master_key)``), since the master key (and its hash)
must not propagate to the verification-token column or any other
downstream consumer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The endpoint loaded the full merged YAML+DB config and re-saved every
top-level section to LiteLLM_Config rows via save_config(), so a UI toggle
of one field persisted unrelated YAML state to DB as a side effect. It
also rejected every request when store_model_in_db was False — including
the request that would flip the flag to True (chicken-and-egg).
Replace save_config with targeted per-section upserts: read the existing
litellm_config row, merge in the request, upsert just that row. Sections
the caller did not send are not touched. Drop the blanket
store_model_in_db guard — the endpoint already requires prisma_client,
and the startup-side override at proxy_server.py:6491 picks up
general_settings.store_model_in_db=True from the DB on next restart.
- test_prepare_key_update_data: replace bare MagicMock with
MagicMock(spec=LiteLLM_VerificationToken) and explicitly set
existing_key_row.metadata = {}, so reserved-field reads return real
values instead of MagicMock-returning-MagicMock. Fixes a regression
surfaced by the new reserved-metadata preservation logic.
- test_key_management_endpoints.py: black-format-only changes from
recent edits.
The routes in `global_spend_tracking_routes` (e.g. /global/spend/report,
/global/spend/teams, /global/spend/keys) return spend aggregated across
every team, customer, and api_key in the proxy. They were included in
`internal_user_routes` and `internal_user_view_only_routes`, so non-admin
roles could read proxy-wide spend.
Drop them from both non-admin route lists. PROXY_ADMIN and
PROXY_ADMIN_VIEW_ONLY access is preserved through their existing branches
in route_checks.py, and the `get_spend_routes` permission opt-in
continues to grant access for keys that need it.
Updates two pre-existing test parametrizations whose expected results
flip from True to False, and adds parametrized coverage over every
route in `global_spend_tracking_routes` for: PROXY_ADMIN_VIEW_ONLY
allowed, INTERNAL_USER blocked, INTERNAL_USER_VIEW_ONLY blocked,
INTERNAL_USER + get_spend_routes permission allowed.
Multiple paths through _user_api_key_auth_builder returned a
UserAPIKeyAuth without running common_checks(): OAuth2 token validation,
OAuth2 proxy header hook, JWT admin shortcut, master_key path,
pass-through custom headers, the /user/auth route, and the
allow_requests_on_db_unavailable fallback. An operator-configured key
model-access list, max_budget, team_blocked flag, or team model scope
was therefore silently skipped on those paths. The HA-fallback token
was worse: it was a full proxy-admin synthetic, so a DB outage granted
full admin to every caller.
Fix three root causes (VERIA-18):
1. Centralize common_checks in the user_api_key_auth wrapper. The
builder paths no longer call it; the wrapper runs it once after the
builder returns, for every path. Introduces _run_centralized_common_checks
which gathers team/user/project/end_user/global_spend context in
parallel via asyncio.gather. Preserves the existing
custom_auth_run_common_checks opt-out for custom-auth deployments.
2. Narrow is_database_connection_error — drop the blanket PrismaError
catch that routed data-layer errors (UniqueViolationError, etc.)
into the HA fallback. Only real connectivity failures plus the
no_db_connection marker now qualify.
3. DB-unavailable fallback issues an INTERNAL_USER token with user_id
DB_UNAVAILABLE_FALLBACK_USER_ID instead of proxy-admin. An outage
can no longer escalate an anonymous caller.
JWT admin / master_key tokens still grant admin via a synthesized
admin user_object (so non_proxy_admin_allowed_routes_check in
common_checks recognizes them); other common_checks branches
(team_blocked, team_model_access) now apply uniformly.
Two model-access gates run per request in `common_checks` and they're
asymmetric: `can_key_call_model` falls back to the key's
`access_group_ids`, but `can_team_access_model` only looks at
`team.models` + `team.access_group_ids`. A key granted a model via its
own access group on a model-restricted team is silently denied at the
team gate.
Wrap `can_team_access_model` in try/except in `common_checks`: on
`team_model_access_denied`, consult a new `_key_access_group_grants_model`
helper that expands `valid_token.access_group_ids` via the existing
`_get_models_from_access_groups` and checks via `_can_object_call_model`.
Re-raise if the key's access groups don't grant the model. Any other
exception propagates unchanged.
Effect: request allowed if `team allows X` OR `key's access group
grants X`, making the two gates symmetric.
Test: add three unit tests for `_key_access_group_grants_model`
covering: group covers model, key has no groups, group resolves but
does not cover model.
Addresses review feedback on the snapshot approach:
1. Class-instance mutable state
The snapshot only covers primitives + collections + None. Class
instances (DualCache, LLMClientCache) weren't reset between tests,
so in-place cache mutations could leak. Can't deepcopy these — they
hold thread locks — but they expose flush_cache(). Collect every
module attribute whose value implements flush_cache() at conftest
import, and invoke it per-test alongside the snapshot restore.
2. Silent skips are now warnings
_snapshot_mutable_state and _restore_mutable_state previously
swallowed exceptions, so if a future attr gained a property without
a setter (or other non-round-trippable state), an isolation gap
would have no signal. Emit warnings.warn on each failure path.
3. Docstring
Explicitly documents what IS and IS NOT reset, and tells authors to
use monkeypatch.setattr() for in-place mutations of instances
without flush_cache() (ProxyLogging, JWTHandler, etc.).
The previous snapshot only tracked list/dict/set values. Tests mutate
scalar module attrs too — master_key, premium_user, prisma_client — and
importlib.reload used to reset those implicitly. Under the snapshot
approach they were leaking between tests, so test_active_callbacks
failed in CI with "No api key passed in." once an earlier test left
master_key set to sk-1234.
Expand the snapshot to cover primitives (str/int/float/bool/bytes/tuple)
and None-valued attributes. Complex object instances are still skipped
to avoid deepcopy issues.
tests/proxy_unit_tests/conftest.py was calling importlib.reload(litellm) in an
autouse function-scoped fixture, which cost ~17s per test because it re-ran
the full litellm __init__ import chain. With 400+ proxy unit tests, this was
the single biggest driver of CI wall time — 18 of the top 20 slowest durations
in a typical run were just the 17s fixture setup.
Replace the reload with a snapshot-and-restore approach: snapshot the mutable
lists/dicts/sets on litellm and litellm.proxy.proxy_server once at conftest
import, then deep-copy that snapshot back before each test. Callback lists,
caches, router state, etc. still get reset between tests, but the expensive
import chain only runs once per worker.
Local measurement on test_proxy_utils.py: 188 tests in 3.50s (previously took
~15 minutes of CI wall time on a single worker).
