Files
litellm/tests/proxy_behavior/management/test_key_team_change.py
T
yuneng-jiangandGitHub 5e16f20962 test(proxy): phase-4 payload behavior pinning for tier-2/3 key + team management endpoints (#28681)
* test(proxy): phase-4 payload behavior pinning for tier-2/3 key + team management endpoints

Extends the Phase 1–3 behavior-pin suite at tests/proxy_behavior/management/
with a second axis: payload-shape pinning. Phase 1–3 held payload minimal
and pinned (actor, target) → status across 37 routes; Phase 4 holds the
caller fixed at an authorized actor, varies the payload shape, and asserts
the observable DB effect (on accept) or the named guard / row-unchanged
(on reject). Faithfulness contract from Phase 1–3 is unchanged.

Six families + one gap-closer (59 new scenarios, 620 → 679 total):

  * F1 — key budget / rate-limit (test_key_budget_limits.py, 18)
  * F2 — key↔team reassignment   (test_key_team_change.py, 6)
  * F3 — team budget / rate-limit (test_team_budget_limits.py, 15)
  * F4 — member-info validation   (test_team_member_info_validation.py, 5)
  * F5 — permission batching      (test_team_permissions_bulk_update.py, 6)
  * F6 — org-scoped team access   (+2 detail-string pins in existing files)
  * F7 — coverage gap-closer      (test_f7_coverage_closeout.py, 7)

Harness extensions in conftest.py (additive only):
  * create_scratch_org() seeder with its own scratch-prefixed budget row
  * budget / limit fields on create_scratch_team()
  * scratch teardown also sweeps litellm_organizationtable

Coverage telemetry (behavior-suite-only):
  * key_management_endpoints.py  60 % → 65 % (+82 lines)
  * team_endpoints.py            62 % → 72 % (+137 lines, crosses 70 % stretch)

Key lands under 70 % per plan §7 escape hatch — the gap is dominated by
routes outside F1–F6 scope (key list/info v2 internals) and structurally
dead org-budget guards (call sites at lines 889 + 2310 + 985 + 1751 load
the org without include_budget_table=True, so org.litellm_budget_table is
None at guard time and the aggregate guard no-ops). Pinned as observed
no-op behavior so a future fix that flips the flag turns these into reds.

Zero source-code changes; pyproject.toml diff is empty;
test_route_coverage.py stays green untouched; G3 grep guards still green;
local wall-time 14 s for the full suite (no coverage), 22 s with coverage.

G4 regression-replay protocol executed against three representative
fix-PR parents (410ce761dc, 0bd49ecb8b, 8bbc61e03c): all Phase 4 tests
PASS at pre-fix SHAs — confirming the F1–F7 layer is a helper-body pin,
not a regression-replay layer for those specific historical bypass
shapes. Targeted RED-bait scenarios for each fix are left for a
follow-up PR.

* test(proxy): push key_management_endpoints.py past the 70% stretch (F7-extension)

Adds 24 more payload-pin scenarios in test_f7_key_coverage_push.py
following the same accepted-effect / rejected-guard pattern. Each
scenario cites the file:line range it pins; same anti-snapshot rules
apply.

Target ranges (all reachable via HTTP-boundary payload variation):
  * 5942-6063  /key/health with metadata.logging → test_key_logging body
  * 4565-4692  /key/reset_spend happy + 404 + non-admin gate + value validation
  * 4421-4533  /key/regenerate ghost-404 + happy + new_key + grace_period
  * 4168-4202  _insert_deprecated_key body via grace_period
  * 6118-6133  _enforce_unique_key_alias duplicate-alias rejection
  * 6148-6169  validate_model_max_budget malformed-payload rejection
  * 4708-4789  validate_key_list_check user/team/org/key_hash branches
  * 2622-2733  /key/bulk_update mixed success/failure + admin gate + size limits
  * 2797-2950  /team/key/bulk_update all-keys path + explicit-keys dedupe + 404
  * 5108-5207  /key/aliases admin + scoped + search-filter branches
  * 3253-3303  /key/info ghost + explicit-key + no-key-uses-auth-header
  * 3427-3436  generate_key_helper_fn budget_limits initialization
  * 1794-1815  prepare_key_update_data duration + budget_duration paths
  * 5280-5388  _build_filter_conditions across include_created_by_keys/team/sort/alias

Coverage telemetry — full PR4 dataset:

  key_management_endpoints.py: 60 % → 71 %  (+11 pts, +194 lines)
  team_endpoints.py:           62 % → 72 %  (+10 pts, +137 lines)

Both files now over the plan §7 PR4.M4 70 % stretch as a side effect of
pinning real payload behavior. 721 tests pass in 19 s local (full suite,
no coverage); 27 s with coverage. Zero source-code changes; pyproject.toml
diff still empty; test_route_coverage.py + G3 grep guards still green.

Honest finding (kept from the prior commit's body): four structurally-dead
org-budget guards remain pinned as observed no-op behavior — they fire
only when get_org_object is called with include_budget_table=True, which
none of the four management-endpoint call sites currently do. Pinned so
a future change that flips the flag turns these into reds.

Two helper guards are honest-ceiling: _validate_reset_spend_value's
isinstance check at line 4568 is unreachable from HTTP because Pydantic
422s non-float before the helper runs; same shape for /team/key/bulk_update's
missing team_id / no-selector pre-handler guards.

* test(proxy): address PR review — try/finally cleanup + loosen 500 envelope pins + Optional annotations

Greptile review feedback on PR #28681:

1. Wrap manual budget-row cleanup in try/finally so an assertion failure
   doesn't leave non-scratch-prefixed budget rows orphaned across CI re-runs
   (test_team_new_with_team_member_budget_creates_budget_row and
   test_team_update_team_member_budget_upserts).
2. Loosen the two 500-status pins to in (400, 422, 500) — the named-guard
   substring is the real pin; the outer ValueError-wrap envelope is an
   implementation detail that a future improvement should be free to fix
   to a proper 400/422 without flipping these tests red.
3. Add missing Optional annotations on _seed_token's max_budget / metadata
   / team_id keyword args (they default to None).

Greptile's typo flag on 'read-world' in the conftest comment is declined —
'read-world' is the project's established term for the immutable seeded
world fixture (see other usages in conftest.py and actors.py).

721 tests still pass in 17 s.
2026-05-23 12:16:29 -07:00

230 lines
7.7 KiB
Python

"""Phase 4 F2 — payload-level pins for key↔team reassignment.
Pins `validate_key_team_change` (key_management_endpoints.py:2953), reached
via /key/update when the request changes `team_id`.
The handler runs four guards in order; each scenario isolates one path and
asserts the rejected row's `team_id` is UNCHANGED on a fresh DB re-read —
the regression shape that matters for cross-team / IDOR-class bugs is
"row mutated despite the helper raising", and the only way to catch it
is to compare the persisted state, not the response body.
The accepted scenario pins the happy path: re-read confirms team_id
flipped to the new team and the row is otherwise intact.
"""
import uuid
from typing import Any, Dict, Optional
import pytest
from prisma import Json
from litellm.proxy.utils import hash_token
from .actors import TEAM_ALPHA, Actor
from .conftest import create_scratch_team
pytestmark = pytest.mark.asyncio(loop_scope="session")
KEY_MODEL = "phase4-f2-key-model"
async def _seed_key_with_limits(
prisma,
scratch_prefix: str,
*,
user_id: str,
team_id: str,
models: Optional[list] = None,
tpm_limit: Optional[int] = None,
rpm_limit: Optional[int] = None,
) -> str:
"""Raw-seed a scratch key with explicit models / tpm / rpm — /key/generate
can't set these against a non-throughput team without firing F1's guards.
Returns the cleartext key for /key/update calls."""
cleartext = "sk-" + uuid.uuid4().hex
data: Dict[str, Any] = {
"token": hash_token(cleartext),
"key_alias": f"{scratch_prefix}-key",
"key_name": f"{scratch_prefix}-key",
"user_id": user_id,
"team_id": team_id,
"models": models or [],
}
if tpm_limit is not None:
data["tpm_limit"] = tpm_limit
if rpm_limit is not None:
data["rpm_limit"] = rpm_limit
await prisma.db.litellm_verificationtoken.create(data=data)
return cleartext
# ---------------------------------------------------------------------------
# Accepted — proxy admin moves a key into a scratch team that has the model,
# accommodates the limits, and has the key owner as a member.
# ---------------------------------------------------------------------------
async def test_key_team_change_accepted(proxy_client, prisma, scratch, world):
owner_id = world.keys[Actor.OWNER].user_id
target_team = await create_scratch_team(
prisma,
team_id=scratch.tag("target"),
member_user_ids=[owner_id],
models=[KEY_MODEL],
tpm_limit=10_000,
rpm_limit=1_000,
)
key_cleartext = await _seed_key_with_limits(
prisma,
scratch.prefix,
user_id=owner_id,
team_id=TEAM_ALPHA,
models=[KEY_MODEL],
tpm_limit=500,
rpm_limit=50,
)
seeder = world.keys[Actor.PROXY_ADMIN].cleartext
resp = await proxy_client.post(
"/key/update",
headers={"Authorization": f"Bearer {seeder}"},
json={"key": key_cleartext, "team_id": target_team},
)
assert resp.status_code == 200, resp.text
row = await prisma.db.litellm_verificationtoken.find_unique(
where={"token": hash_token(key_cleartext)}
)
assert row is not None
assert row.team_id == target_team, "key did not move to target team"
# ---------------------------------------------------------------------------
# Rejected — each path verifies row.team_id is UNCHANGED on DB re-read.
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
"scenario",
[
"team_lacks_model",
"key_tpm_exceeds_team_tpm",
"key_rpm_exceeds_team_rpm",
"key_owner_not_team_member",
],
)
async def test_key_team_change_rejected_guards(
scenario: str, proxy_client, prisma, scratch, world
):
owner_id = world.keys[Actor.OWNER].user_id
team_kwargs: Dict[str, Any] = {
"member_user_ids": [owner_id],
"models": [KEY_MODEL],
"tpm_limit": 10_000,
"rpm_limit": 1_000,
}
key_kwargs: Dict[str, Any] = {
"models": [KEY_MODEL],
"tpm_limit": 500,
"rpm_limit": 50,
}
if scenario == "team_lacks_model":
team_kwargs["models"] = ["something-else"]
elif scenario == "key_tpm_exceeds_team_tpm":
team_kwargs["tpm_limit"] = 100 # < 500
elif scenario == "key_rpm_exceeds_team_rpm":
team_kwargs["rpm_limit"] = 5 # < 50
elif scenario == "key_owner_not_team_member":
team_kwargs["member_user_ids"] = []
target_team = await create_scratch_team(
prisma,
team_id=scratch.tag("target"),
**team_kwargs,
)
key_cleartext = await _seed_key_with_limits(
prisma,
scratch.prefix,
user_id=owner_id,
team_id=TEAM_ALPHA,
**key_kwargs,
)
seeder = world.keys[Actor.PROXY_ADMIN].cleartext
resp = await proxy_client.post(
"/key/update",
headers={"Authorization": f"Bearer {seeder}"},
json={"key": key_cleartext, "team_id": target_team},
)
# The model-mismatch path lands as 400 (ProxyException from
# _can_object_call_model); the limit + membership paths land as 403
# (validate_key_team_change's own HTTPException). Both shapes count as
# "rejected" for the pin; what matters is the row stayed put.
assert resp.status_code in (
400,
403,
), f"{scenario}: expected 400/403, got {resp.status_code}: {resp.text}"
row = await prisma.db.litellm_verificationtoken.find_unique(
where={"token": hash_token(key_cleartext)}
)
assert row is not None
assert row.team_id == TEAM_ALPHA, (
f"{scenario}: row team_id mutated despite rejection — " f"got {row.team_id!r}"
)
# ---------------------------------------------------------------------------
# Rejected — initiator is neither proxy admin, team admin, nor permission-
# granted. Pinned separately because the source team here must also have the
# initiator listed as a non-admin member (otherwise the earlier user_id
# membership guard fires first).
# ---------------------------------------------------------------------------
async def test_key_team_change_rejected_initiator_not_admin(
proxy_client, prisma, scratch, world
):
owner_id = world.keys[Actor.OWNER].user_id
target_team = await create_scratch_team(
prisma,
team_id=scratch.tag("target"),
# Owner is the key's user_id; member of the team to clear membership
# guard. Internal-user role on world.keys[INTERNAL_USER] is also a
# member of TEAM_ALPHA but never an admin → role-gate path fires.
member_user_ids=[owner_id, world.keys[Actor.INTERNAL_USER].user_id],
models=[KEY_MODEL],
tpm_limit=10_000,
rpm_limit=1_000,
)
key_cleartext = await _seed_key_with_limits(
prisma,
scratch.prefix,
user_id=owner_id,
team_id=TEAM_ALPHA,
models=[KEY_MODEL],
tpm_limit=500,
rpm_limit=50,
)
# The internal_user actor is the initiator: a TEAM_ALPHA member, but not
# an admin anywhere, and has no team_member_permissions for /key/update.
initiator = world.keys[Actor.INTERNAL_USER].cleartext
resp = await proxy_client.post(
"/key/update",
headers={"Authorization": f"Bearer {initiator}"},
json={"key": key_cleartext, "team_id": target_team},
)
assert resp.status_code in (
401,
403,
), f"expected 401/403, got {resp.status_code}: {resp.text}"
row = await prisma.db.litellm_verificationtoken.find_unique(
where={"token": hash_token(key_cleartext)}
)
assert row is not None
assert row.team_id == TEAM_ALPHA, "row team_id mutated despite rejection"