mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-13 03:05:31 +00:00
5e16f20962
* test(proxy): phase-4 payload behavior pinning for tier-2/3 key + team management endpoints Extends the Phase 1–3 behavior-pin suite at tests/proxy_behavior/management/ with a second axis: payload-shape pinning. Phase 1–3 held payload minimal and pinned (actor, target) → status across 37 routes; Phase 4 holds the caller fixed at an authorized actor, varies the payload shape, and asserts the observable DB effect (on accept) or the named guard / row-unchanged (on reject). Faithfulness contract from Phase 1–3 is unchanged. Six families + one gap-closer (59 new scenarios, 620 → 679 total): * F1 — key budget / rate-limit (test_key_budget_limits.py, 18) * F2 — key↔team reassignment (test_key_team_change.py, 6) * F3 — team budget / rate-limit (test_team_budget_limits.py, 15) * F4 — member-info validation (test_team_member_info_validation.py, 5) * F5 — permission batching (test_team_permissions_bulk_update.py, 6) * F6 — org-scoped team access (+2 detail-string pins in existing files) * F7 — coverage gap-closer (test_f7_coverage_closeout.py, 7) Harness extensions in conftest.py (additive only): * create_scratch_org() seeder with its own scratch-prefixed budget row * budget / limit fields on create_scratch_team() * scratch teardown also sweeps litellm_organizationtable Coverage telemetry (behavior-suite-only): * key_management_endpoints.py 60 % → 65 % (+82 lines) * team_endpoints.py 62 % → 72 % (+137 lines, crosses 70 % stretch) Key lands under 70 % per plan §7 escape hatch — the gap is dominated by routes outside F1–F6 scope (key list/info v2 internals) and structurally dead org-budget guards (call sites at lines 889 + 2310 + 985 + 1751 load the org without include_budget_table=True, so org.litellm_budget_table is None at guard time and the aggregate guard no-ops). Pinned as observed no-op behavior so a future fix that flips the flag turns these into reds. Zero source-code changes; pyproject.toml diff is empty; test_route_coverage.py stays green untouched; G3 grep guards still green; local wall-time 14 s for the full suite (no coverage), 22 s with coverage. G4 regression-replay protocol executed against three representative fix-PR parents (410ce761dc,0bd49ecb8b,8bbc61e03c): all Phase 4 tests PASS at pre-fix SHAs — confirming the F1–F7 layer is a helper-body pin, not a regression-replay layer for those specific historical bypass shapes. Targeted RED-bait scenarios for each fix are left for a follow-up PR. * test(proxy): push key_management_endpoints.py past the 70% stretch (F7-extension) Adds 24 more payload-pin scenarios in test_f7_key_coverage_push.py following the same accepted-effect / rejected-guard pattern. Each scenario cites the file:line range it pins; same anti-snapshot rules apply. Target ranges (all reachable via HTTP-boundary payload variation): * 5942-6063 /key/health with metadata.logging → test_key_logging body * 4565-4692 /key/reset_spend happy + 404 + non-admin gate + value validation * 4421-4533 /key/regenerate ghost-404 + happy + new_key + grace_period * 4168-4202 _insert_deprecated_key body via grace_period * 6118-6133 _enforce_unique_key_alias duplicate-alias rejection * 6148-6169 validate_model_max_budget malformed-payload rejection * 4708-4789 validate_key_list_check user/team/org/key_hash branches * 2622-2733 /key/bulk_update mixed success/failure + admin gate + size limits * 2797-2950 /team/key/bulk_update all-keys path + explicit-keys dedupe + 404 * 5108-5207 /key/aliases admin + scoped + search-filter branches * 3253-3303 /key/info ghost + explicit-key + no-key-uses-auth-header * 3427-3436 generate_key_helper_fn budget_limits initialization * 1794-1815 prepare_key_update_data duration + budget_duration paths * 5280-5388 _build_filter_conditions across include_created_by_keys/team/sort/alias Coverage telemetry — full PR4 dataset: key_management_endpoints.py: 60 % → 71 % (+11 pts, +194 lines) team_endpoints.py: 62 % → 72 % (+10 pts, +137 lines) Both files now over the plan §7 PR4.M4 70 % stretch as a side effect of pinning real payload behavior. 721 tests pass in 19 s local (full suite, no coverage); 27 s with coverage. Zero source-code changes; pyproject.toml diff still empty; test_route_coverage.py + G3 grep guards still green. Honest finding (kept from the prior commit's body): four structurally-dead org-budget guards remain pinned as observed no-op behavior — they fire only when get_org_object is called with include_budget_table=True, which none of the four management-endpoint call sites currently do. Pinned so a future change that flips the flag turns these into reds. Two helper guards are honest-ceiling: _validate_reset_spend_value's isinstance check at line 4568 is unreachable from HTTP because Pydantic 422s non-float before the helper runs; same shape for /team/key/bulk_update's missing team_id / no-selector pre-handler guards. * test(proxy): address PR review — try/finally cleanup + loosen 500 envelope pins + Optional annotations Greptile review feedback on PR #28681: 1. Wrap manual budget-row cleanup in try/finally so an assertion failure doesn't leave non-scratch-prefixed budget rows orphaned across CI re-runs (test_team_new_with_team_member_budget_creates_budget_row and test_team_update_team_member_budget_upserts). 2. Loosen the two 500-status pins to in (400, 422, 500) — the named-guard substring is the real pin; the outer ValueError-wrap envelope is an implementation detail that a future improvement should be free to fix to a proper 400/422 without flipping these tests red. 3. Add missing Optional annotations on _seed_token's max_budget / metadata / team_id keyword args (they default to None). Greptile's typo flag on 'read-world' in the conftest comment is declined — 'read-world' is the project's established term for the immutable seeded world fixture (see other usages in conftest.py and actors.py). 721 tests still pass in 17 s.
228 lines
8.4 KiB
Python
228 lines
8.4 KiB
Python
import pytest
|
|
|
|
from litellm.proxy._types import LitellmUserRoles
|
|
|
|
from .actors import Actor
|
|
from .conftest import create_scratch_actor, create_scratch_team
|
|
|
|
pytestmark = pytest.mark.asyncio(loop_scope="session")
|
|
|
|
|
|
# POST /team/update — actor x team-shape matrix (shapes built by _seed_target).
|
|
# Each request carries the team's own organization_id so a non-proxy-admin can
|
|
# reach the org-scoped branch of the route-permission gate (401 on denial),
|
|
# which fronts the handler's _verify_team_access. Only PROXY_ADMIN and an
|
|
# ORG_ADMIN of the team's org pass: an internal_user team admin is filtered by
|
|
# the route gate before _verify_team_access's team-admin branch is reached.
|
|
MARKER_ALIAS = "behavior-pin-update-marker-alias"
|
|
|
|
_MATRIX = [
|
|
("alpha/proxy_admin", Actor.PROXY_ADMIN, "alpha", 200),
|
|
("alpha/org_admin", Actor.ORG_ADMIN, "alpha", 200),
|
|
("alpha/team_admin", Actor.TEAM_ADMIN, "alpha", 401),
|
|
("alpha/internal_user", Actor.INTERNAL_USER, "alpha", 401),
|
|
("alpha/owner", Actor.OWNER, "alpha", 401),
|
|
("alpha/unrelated_same_org", Actor.UNRELATED_SAME_ORG, "alpha", 401),
|
|
("alpha/cross_org_user", Actor.CROSS_ORG_USER, "alpha", 401),
|
|
("alpha/service_account", Actor.SERVICE_ACCOUNT, "alpha", 401),
|
|
("alpha/org_b_admin", Actor.ORG_B_ADMIN, "alpha", 401),
|
|
("beta/proxy_admin", Actor.PROXY_ADMIN, "beta", 200),
|
|
("beta/org_admin", Actor.ORG_ADMIN, "beta", 401),
|
|
("beta/team_admin", Actor.TEAM_ADMIN, "beta", 401),
|
|
("beta/internal_user", Actor.INTERNAL_USER, "beta", 401),
|
|
("beta/owner", Actor.OWNER, "beta", 401),
|
|
("beta/unrelated_same_org", Actor.UNRELATED_SAME_ORG, "beta", 401),
|
|
("beta/cross_org_user", Actor.CROSS_ORG_USER, "beta", 401),
|
|
("beta/service_account", Actor.SERVICE_ACCOUNT, "beta", 401),
|
|
("beta/org_b_admin", Actor.ORG_B_ADMIN, "beta", 200),
|
|
]
|
|
|
|
|
|
async def _seed_target(prisma, world, shape: str, team_id: str) -> str:
|
|
"""Raw-seed the scratch target team; returns its organization_id."""
|
|
if shape == "alpha":
|
|
await create_scratch_team(
|
|
prisma,
|
|
team_id,
|
|
organization_id=world.org_a_id,
|
|
admin_user_ids=[world.keys[Actor.TEAM_ADMIN].user_id],
|
|
member_user_ids=[
|
|
world.keys[Actor.INTERNAL_USER].user_id,
|
|
world.keys[Actor.OWNER].user_id,
|
|
world.keys[Actor.UNRELATED_SAME_ORG].user_id,
|
|
world.keys[Actor.SERVICE_ACCOUNT].user_id,
|
|
],
|
|
)
|
|
return world.org_a_id
|
|
if shape == "beta":
|
|
await create_scratch_team(
|
|
prisma,
|
|
team_id,
|
|
organization_id=world.org_b_id,
|
|
member_user_ids=[world.keys[Actor.CROSS_ORG_USER].user_id],
|
|
)
|
|
return world.org_b_id
|
|
pytest.fail(f"unknown shape={shape}") # pragma: no cover
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"actor,shape,expected_status",
|
|
[(a, sh, s) for (_id, a, sh, s) in _MATRIX],
|
|
ids=[s[0] for s in _MATRIX],
|
|
)
|
|
async def test_team_update_authz_matrix(
|
|
actor: Actor,
|
|
shape: str,
|
|
expected_status: int,
|
|
proxy_client,
|
|
prisma,
|
|
scratch,
|
|
world,
|
|
):
|
|
org_id = await _seed_target(prisma, world, shape, scratch.prefix)
|
|
caller = world.keys[actor]
|
|
|
|
resp = await proxy_client.post(
|
|
"/team/update",
|
|
headers={"Authorization": f"Bearer {caller.cleartext}"},
|
|
json={
|
|
"team_id": scratch.prefix,
|
|
"team_alias": MARKER_ALIAS,
|
|
"organization_id": org_id,
|
|
},
|
|
)
|
|
assert (
|
|
resp.status_code == expected_status
|
|
), f"{actor.value} {shape}: {resp.status_code} {resp.text}"
|
|
|
|
row = await prisma.db.litellm_teamtable.find_unique(
|
|
where={"team_id": scratch.prefix}
|
|
)
|
|
assert row is not None
|
|
if expected_status == 200:
|
|
assert row.team_alias == MARKER_ALIAS
|
|
else:
|
|
assert row.team_alias != MARKER_ALIAS, "denied but team mutated"
|
|
|
|
|
|
async def test_team_update_requires_proxy_admin_without_org_context(
|
|
proxy_client, prisma, scratch, world
|
|
):
|
|
"""With no organization_id in the body the route gate has no org context
|
|
and falls back to proxy-admin-only: an org admin of the team's own org
|
|
is 401, PROXY_ADMIN is 200."""
|
|
await _seed_target(prisma, world, "alpha", scratch.prefix)
|
|
|
|
denied = await proxy_client.post(
|
|
"/team/update",
|
|
headers={"Authorization": f"Bearer {world.keys[Actor.ORG_ADMIN].cleartext}"},
|
|
json={"team_id": scratch.prefix, "team_alias": MARKER_ALIAS},
|
|
)
|
|
assert denied.status_code == 401, denied.text
|
|
|
|
allowed = await proxy_client.post(
|
|
"/team/update",
|
|
headers={"Authorization": f"Bearer {world.keys[Actor.PROXY_ADMIN].cleartext}"},
|
|
json={"team_id": scratch.prefix, "team_alias": MARKER_ALIAS},
|
|
)
|
|
assert allowed.status_code == 200, allowed.text
|
|
|
|
|
|
# Relocation gate — moving a team to a different org. The scratch team starts
|
|
# in ORG_A; each scenario relocates it to ORG_B. PROXY_ADMIN bypasses;
|
|
# ORG_B_ADMIN clears the route gate (dest-org admin) but fails
|
|
# _verify_team_access on the source team (403); the rest fail the route gate
|
|
# (401). The relocation-*allowed* branch (caller is org admin of both orgs) is
|
|
# covered by test_team_update_org_relocation_allowed_for_dual_org_admin below.
|
|
_RELOCATION = [
|
|
("proxy_admin", Actor.PROXY_ADMIN, 200),
|
|
("org_b_admin", Actor.ORG_B_ADMIN, 403),
|
|
("org_admin", Actor.ORG_ADMIN, 401),
|
|
("team_admin", Actor.TEAM_ADMIN, 401),
|
|
("internal_user", Actor.INTERNAL_USER, 401),
|
|
]
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"actor,expected_status",
|
|
[(a, s) for (_id, a, s) in _RELOCATION],
|
|
ids=[s[0] for s in _RELOCATION],
|
|
)
|
|
async def test_team_update_org_relocation_gate(
|
|
actor: Actor,
|
|
expected_status: int,
|
|
proxy_client,
|
|
prisma,
|
|
scratch,
|
|
world,
|
|
):
|
|
await _seed_target(prisma, world, "alpha", scratch.prefix)
|
|
caller = world.keys[actor]
|
|
|
|
resp = await proxy_client.post(
|
|
"/team/update",
|
|
headers={"Authorization": f"Bearer {caller.cleartext}"},
|
|
json={"team_id": scratch.prefix, "organization_id": world.org_b_id},
|
|
)
|
|
assert (
|
|
resp.status_code == expected_status
|
|
), f"{actor.value}: {resp.status_code} {resp.text}"
|
|
|
|
row = await prisma.db.litellm_teamtable.find_unique(
|
|
where={"team_id": scratch.prefix}
|
|
)
|
|
assert row is not None
|
|
if expected_status == 200:
|
|
assert row.organization_id == world.org_b_id
|
|
else:
|
|
assert row.organization_id == world.org_a_id, "denied but team relocated"
|
|
|
|
|
|
# Phase 4 F6 — explicit pin on the `_verify_team_access` 403 detail string
|
|
# when an org_admin clears the destination route gate but fails the source
|
|
# team's org-membership check. The relocation matrix above covers the
|
|
# status; this guard turns a silent rename of the helper's exception detail
|
|
# into a CI red.
|
|
async def test_team_update_org_b_admin_relocation_rejection_detail(
|
|
proxy_client, prisma, scratch, world
|
|
):
|
|
await _seed_target(prisma, world, "alpha", scratch.prefix)
|
|
resp = await proxy_client.post(
|
|
"/team/update",
|
|
headers={"Authorization": f"Bearer {world.keys[Actor.ORG_B_ADMIN].cleartext}"},
|
|
json={"team_id": scratch.prefix, "organization_id": world.org_b_id},
|
|
)
|
|
assert resp.status_code == 403, resp.text
|
|
assert "do not have access to this team" in resp.text, resp.text
|
|
|
|
|
|
async def test_team_update_org_relocation_allowed_for_dual_org_admin(
|
|
proxy_client, prisma, scratch, world
|
|
):
|
|
"""Relocation-allowed branch: a caller who is org admin of BOTH the source
|
|
and destination org may relocate a team between them. Completes the
|
|
_RELOCATION matrix, whose allowed branch PR2 left open — no seeded actor is
|
|
a dual-org admin, so one is minted with create_scratch_actor."""
|
|
actor = await create_scratch_actor(
|
|
prisma,
|
|
scratch.prefix,
|
|
user_role=LitellmUserRoles.ORG_ADMIN.value,
|
|
org_admin_of=(world.org_a_id, world.org_b_id),
|
|
)
|
|
team_id = await create_scratch_team(
|
|
prisma, scratch.tag("team"), organization_id=world.org_a_id
|
|
)
|
|
|
|
resp = await proxy_client.post(
|
|
"/team/update",
|
|
headers={"Authorization": f"Bearer {actor.cleartext}"},
|
|
json={"team_id": team_id, "organization_id": world.org_b_id},
|
|
)
|
|
assert resp.status_code == 200, resp.text
|
|
|
|
row = await prisma.db.litellm_teamtable.find_unique(where={"team_id": team_id})
|
|
assert row is not None
|
|
assert (
|
|
row.organization_id == world.org_b_id
|
|
), "dual-org admin relocation not applied"
|