mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-16 00:23:01 +00:00
* test(proxy): phase-4 payload behavior pinning for tier-2/3 key + team management endpoints Extends the Phase 1–3 behavior-pin suite at tests/proxy_behavior/management/ with a second axis: payload-shape pinning. Phase 1–3 held payload minimal and pinned (actor, target) → status across 37 routes; Phase 4 holds the caller fixed at an authorized actor, varies the payload shape, and asserts the observable DB effect (on accept) or the named guard / row-unchanged (on reject). Faithfulness contract from Phase 1–3 is unchanged. Six families + one gap-closer (59 new scenarios, 620 → 679 total): * F1 — key budget / rate-limit (test_key_budget_limits.py, 18) * F2 — key↔team reassignment (test_key_team_change.py, 6) * F3 — team budget / rate-limit (test_team_budget_limits.py, 15) * F4 — member-info validation (test_team_member_info_validation.py, 5) * F5 — permission batching (test_team_permissions_bulk_update.py, 6) * F6 — org-scoped team access (+2 detail-string pins in existing files) * F7 — coverage gap-closer (test_f7_coverage_closeout.py, 7) Harness extensions in conftest.py (additive only): * create_scratch_org() seeder with its own scratch-prefixed budget row * budget / limit fields on create_scratch_team() * scratch teardown also sweeps litellm_organizationtable Coverage telemetry (behavior-suite-only): * key_management_endpoints.py 60 % → 65 % (+82 lines) * team_endpoints.py 62 % → 72 % (+137 lines, crosses 70 % stretch) Key lands under 70 % per plan §7 escape hatch — the gap is dominated by routes outside F1–F6 scope (key list/info v2 internals) and structurally dead org-budget guards (call sites at lines 889 + 2310 + 985 + 1751 load the org without include_budget_table=True, so org.litellm_budget_table is None at guard time and the aggregate guard no-ops). Pinned as observed no-op behavior so a future fix that flips the flag turns these into reds. Zero source-code changes; pyproject.toml diff is empty; test_route_coverage.py stays green untouched; G3 grep guards still green; local wall-time 14 s for the full suite (no coverage), 22 s with coverage. G4 regression-replay protocol executed against three representative fix-PR parents (410ce761dc,0bd49ecb8b,8bbc61e03c): all Phase 4 tests PASS at pre-fix SHAs — confirming the F1–F7 layer is a helper-body pin, not a regression-replay layer for those specific historical bypass shapes. Targeted RED-bait scenarios for each fix are left for a follow-up PR. * test(proxy): push key_management_endpoints.py past the 70% stretch (F7-extension) Adds 24 more payload-pin scenarios in test_f7_key_coverage_push.py following the same accepted-effect / rejected-guard pattern. Each scenario cites the file:line range it pins; same anti-snapshot rules apply. Target ranges (all reachable via HTTP-boundary payload variation): * 5942-6063 /key/health with metadata.logging → test_key_logging body * 4565-4692 /key/reset_spend happy + 404 + non-admin gate + value validation * 4421-4533 /key/regenerate ghost-404 + happy + new_key + grace_period * 4168-4202 _insert_deprecated_key body via grace_period * 6118-6133 _enforce_unique_key_alias duplicate-alias rejection * 6148-6169 validate_model_max_budget malformed-payload rejection * 4708-4789 validate_key_list_check user/team/org/key_hash branches * 2622-2733 /key/bulk_update mixed success/failure + admin gate + size limits * 2797-2950 /team/key/bulk_update all-keys path + explicit-keys dedupe + 404 * 5108-5207 /key/aliases admin + scoped + search-filter branches * 3253-3303 /key/info ghost + explicit-key + no-key-uses-auth-header * 3427-3436 generate_key_helper_fn budget_limits initialization * 1794-1815 prepare_key_update_data duration + budget_duration paths * 5280-5388 _build_filter_conditions across include_created_by_keys/team/sort/alias Coverage telemetry — full PR4 dataset: key_management_endpoints.py: 60 % → 71 % (+11 pts, +194 lines) team_endpoints.py: 62 % → 72 % (+10 pts, +137 lines) Both files now over the plan §7 PR4.M4 70 % stretch as a side effect of pinning real payload behavior. 721 tests pass in 19 s local (full suite, no coverage); 27 s with coverage. Zero source-code changes; pyproject.toml diff still empty; test_route_coverage.py + G3 grep guards still green. Honest finding (kept from the prior commit's body): four structurally-dead org-budget guards remain pinned as observed no-op behavior — they fire only when get_org_object is called with include_budget_table=True, which none of the four management-endpoint call sites currently do. Pinned so a future change that flips the flag turns these into reds. Two helper guards are honest-ceiling: _validate_reset_spend_value's isinstance check at line 4568 is unreachable from HTTP because Pydantic 422s non-float before the helper runs; same shape for /team/key/bulk_update's missing team_id / no-selector pre-handler guards. * test(proxy): address PR review — try/finally cleanup + loosen 500 envelope pins + Optional annotations Greptile review feedback on PR #28681: 1. Wrap manual budget-row cleanup in try/finally so an assertion failure doesn't leave non-scratch-prefixed budget rows orphaned across CI re-runs (test_team_new_with_team_member_budget_creates_budget_row and test_team_update_team_member_budget_upserts). 2. Loosen the two 500-status pins to in (400, 422, 500) — the named-guard substring is the real pin; the outer ValueError-wrap envelope is an implementation detail that a future improvement should be free to fix to a proper 400/422 without flipping these tests red. 3. Add missing Optional annotations on _seed_token's max_budget / metadata / team_id keyword args (they default to None). Greptile's typo flag on 'read-world' in the conftest comment is declined — 'read-world' is the project's established term for the immutable seeded world fixture (see other usages in conftest.py and actors.py). 721 tests still pass in 17 s.
230 lines
7.7 KiB
Python
230 lines
7.7 KiB
Python
"""Phase 4 F2 — payload-level pins for key↔team reassignment.
|
|
|
|
Pins `validate_key_team_change` (key_management_endpoints.py:2953), reached
|
|
via /key/update when the request changes `team_id`.
|
|
|
|
The handler runs four guards in order; each scenario isolates one path and
|
|
asserts the rejected row's `team_id` is UNCHANGED on a fresh DB re-read —
|
|
the regression shape that matters for cross-team / IDOR-class bugs is
|
|
"row mutated despite the helper raising", and the only way to catch it
|
|
is to compare the persisted state, not the response body.
|
|
|
|
The accepted scenario pins the happy path: re-read confirms team_id
|
|
flipped to the new team and the row is otherwise intact.
|
|
"""
|
|
|
|
import uuid
|
|
from typing import Any, Dict, Optional
|
|
|
|
import pytest
|
|
from prisma import Json
|
|
|
|
from litellm.proxy.utils import hash_token
|
|
|
|
from .actors import TEAM_ALPHA, Actor
|
|
from .conftest import create_scratch_team
|
|
|
|
pytestmark = pytest.mark.asyncio(loop_scope="session")
|
|
|
|
KEY_MODEL = "phase4-f2-key-model"
|
|
|
|
|
|
async def _seed_key_with_limits(
|
|
prisma,
|
|
scratch_prefix: str,
|
|
*,
|
|
user_id: str,
|
|
team_id: str,
|
|
models: Optional[list] = None,
|
|
tpm_limit: Optional[int] = None,
|
|
rpm_limit: Optional[int] = None,
|
|
) -> str:
|
|
"""Raw-seed a scratch key with explicit models / tpm / rpm — /key/generate
|
|
can't set these against a non-throughput team without firing F1's guards.
|
|
Returns the cleartext key for /key/update calls."""
|
|
cleartext = "sk-" + uuid.uuid4().hex
|
|
data: Dict[str, Any] = {
|
|
"token": hash_token(cleartext),
|
|
"key_alias": f"{scratch_prefix}-key",
|
|
"key_name": f"{scratch_prefix}-key",
|
|
"user_id": user_id,
|
|
"team_id": team_id,
|
|
"models": models or [],
|
|
}
|
|
if tpm_limit is not None:
|
|
data["tpm_limit"] = tpm_limit
|
|
if rpm_limit is not None:
|
|
data["rpm_limit"] = rpm_limit
|
|
await prisma.db.litellm_verificationtoken.create(data=data)
|
|
return cleartext
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Accepted — proxy admin moves a key into a scratch team that has the model,
|
|
# accommodates the limits, and has the key owner as a member.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
async def test_key_team_change_accepted(proxy_client, prisma, scratch, world):
|
|
owner_id = world.keys[Actor.OWNER].user_id
|
|
target_team = await create_scratch_team(
|
|
prisma,
|
|
team_id=scratch.tag("target"),
|
|
member_user_ids=[owner_id],
|
|
models=[KEY_MODEL],
|
|
tpm_limit=10_000,
|
|
rpm_limit=1_000,
|
|
)
|
|
key_cleartext = await _seed_key_with_limits(
|
|
prisma,
|
|
scratch.prefix,
|
|
user_id=owner_id,
|
|
team_id=TEAM_ALPHA,
|
|
models=[KEY_MODEL],
|
|
tpm_limit=500,
|
|
rpm_limit=50,
|
|
)
|
|
seeder = world.keys[Actor.PROXY_ADMIN].cleartext
|
|
|
|
resp = await proxy_client.post(
|
|
"/key/update",
|
|
headers={"Authorization": f"Bearer {seeder}"},
|
|
json={"key": key_cleartext, "team_id": target_team},
|
|
)
|
|
assert resp.status_code == 200, resp.text
|
|
|
|
row = await prisma.db.litellm_verificationtoken.find_unique(
|
|
where={"token": hash_token(key_cleartext)}
|
|
)
|
|
assert row is not None
|
|
assert row.team_id == target_team, "key did not move to target team"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Rejected — each path verifies row.team_id is UNCHANGED on DB re-read.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"scenario",
|
|
[
|
|
"team_lacks_model",
|
|
"key_tpm_exceeds_team_tpm",
|
|
"key_rpm_exceeds_team_rpm",
|
|
"key_owner_not_team_member",
|
|
],
|
|
)
|
|
async def test_key_team_change_rejected_guards(
|
|
scenario: str, proxy_client, prisma, scratch, world
|
|
):
|
|
owner_id = world.keys[Actor.OWNER].user_id
|
|
|
|
team_kwargs: Dict[str, Any] = {
|
|
"member_user_ids": [owner_id],
|
|
"models": [KEY_MODEL],
|
|
"tpm_limit": 10_000,
|
|
"rpm_limit": 1_000,
|
|
}
|
|
key_kwargs: Dict[str, Any] = {
|
|
"models": [KEY_MODEL],
|
|
"tpm_limit": 500,
|
|
"rpm_limit": 50,
|
|
}
|
|
|
|
if scenario == "team_lacks_model":
|
|
team_kwargs["models"] = ["something-else"]
|
|
elif scenario == "key_tpm_exceeds_team_tpm":
|
|
team_kwargs["tpm_limit"] = 100 # < 500
|
|
elif scenario == "key_rpm_exceeds_team_rpm":
|
|
team_kwargs["rpm_limit"] = 5 # < 50
|
|
elif scenario == "key_owner_not_team_member":
|
|
team_kwargs["member_user_ids"] = []
|
|
|
|
target_team = await create_scratch_team(
|
|
prisma,
|
|
team_id=scratch.tag("target"),
|
|
**team_kwargs,
|
|
)
|
|
key_cleartext = await _seed_key_with_limits(
|
|
prisma,
|
|
scratch.prefix,
|
|
user_id=owner_id,
|
|
team_id=TEAM_ALPHA,
|
|
**key_kwargs,
|
|
)
|
|
seeder = world.keys[Actor.PROXY_ADMIN].cleartext
|
|
|
|
resp = await proxy_client.post(
|
|
"/key/update",
|
|
headers={"Authorization": f"Bearer {seeder}"},
|
|
json={"key": key_cleartext, "team_id": target_team},
|
|
)
|
|
# The model-mismatch path lands as 400 (ProxyException from
|
|
# _can_object_call_model); the limit + membership paths land as 403
|
|
# (validate_key_team_change's own HTTPException). Both shapes count as
|
|
# "rejected" for the pin; what matters is the row stayed put.
|
|
assert resp.status_code in (
|
|
400,
|
|
403,
|
|
), f"{scenario}: expected 400/403, got {resp.status_code}: {resp.text}"
|
|
|
|
row = await prisma.db.litellm_verificationtoken.find_unique(
|
|
where={"token": hash_token(key_cleartext)}
|
|
)
|
|
assert row is not None
|
|
assert row.team_id == TEAM_ALPHA, (
|
|
f"{scenario}: row team_id mutated despite rejection — " f"got {row.team_id!r}"
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Rejected — initiator is neither proxy admin, team admin, nor permission-
|
|
# granted. Pinned separately because the source team here must also have the
|
|
# initiator listed as a non-admin member (otherwise the earlier user_id
|
|
# membership guard fires first).
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
async def test_key_team_change_rejected_initiator_not_admin(
|
|
proxy_client, prisma, scratch, world
|
|
):
|
|
owner_id = world.keys[Actor.OWNER].user_id
|
|
target_team = await create_scratch_team(
|
|
prisma,
|
|
team_id=scratch.tag("target"),
|
|
# Owner is the key's user_id; member of the team to clear membership
|
|
# guard. Internal-user role on world.keys[INTERNAL_USER] is also a
|
|
# member of TEAM_ALPHA but never an admin → role-gate path fires.
|
|
member_user_ids=[owner_id, world.keys[Actor.INTERNAL_USER].user_id],
|
|
models=[KEY_MODEL],
|
|
tpm_limit=10_000,
|
|
rpm_limit=1_000,
|
|
)
|
|
key_cleartext = await _seed_key_with_limits(
|
|
prisma,
|
|
scratch.prefix,
|
|
user_id=owner_id,
|
|
team_id=TEAM_ALPHA,
|
|
models=[KEY_MODEL],
|
|
tpm_limit=500,
|
|
rpm_limit=50,
|
|
)
|
|
# The internal_user actor is the initiator: a TEAM_ALPHA member, but not
|
|
# an admin anywhere, and has no team_member_permissions for /key/update.
|
|
initiator = world.keys[Actor.INTERNAL_USER].cleartext
|
|
|
|
resp = await proxy_client.post(
|
|
"/key/update",
|
|
headers={"Authorization": f"Bearer {initiator}"},
|
|
json={"key": key_cleartext, "team_id": target_team},
|
|
)
|
|
assert resp.status_code in (
|
|
401,
|
|
403,
|
|
), f"expected 401/403, got {resp.status_code}: {resp.text}"
|
|
row = await prisma.db.litellm_verificationtoken.find_unique(
|
|
where={"token": hash_token(key_cleartext)}
|
|
)
|
|
assert row is not None
|
|
assert row.team_id == TEAM_ALPHA, "row team_id mutated despite rejection"
|