4 Commits

Include renames

Author	SHA1	Message	Date
4 people	074455c138	fix(auth): expand all-team-models sentinel in can_key_call_model for batch validation (#29746) ... * fix(auth): expand all-team-models sentinel in can_key_call_model Keys with models=["all-team-models"] were denied during batch JSONL model validation because can_key_call_model matched the literal string against the model name. Add _resolve_key_models_for_auth_check to expand the sentinel to team_models before the check, consistent with get_key_models in model_checks.py and the completion-route bypass. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(auth): document empty team_models unrestricted access behavior; add regression test Adds a docstring note to _resolve_key_models_for_auth_check explaining that when team_models is empty, all-team-models resolves to [] which is treated as unrestricted access (consistent with get_key_models behavior on other auth paths). Adds a test to lock in this behavior. * fix(auth): deny all-team-models access when key has no team_id A key configured with models=["all-team-models"] but no team_id could previously resolve to an empty allowlist, which _check_model_access_helper treats as unrestricted access. Now the sentinel is only expanded when team_id is set; otherwise the unresolved sentinel stays in the model list and causes a deny (no real model name matches it). Same fix applied to get_key_models in model_checks.py for consistency across batch and non-batch auth paths. * style: black format model_checks.py * Fix batch all-team-models auth * style: black format batch_rate_limiter.py * fix(test): add tool_use_system_prompt_tokens to model prices schema validator * fix(batch): catch get_team_object errors to avoid 404 escaping batch auth * fix(batch): apply per-member model scope check after team auth in batch validation * Fail closed on batch team auth fetch errors * test(batch): cover team_object grant and member-scope denial in batch auth --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-05 09:04:45 -07:00
5 people	c233cbbc2a	fix(batches): skip unnecessary batch input file reads (#29114) ... * fix(batches): skip unnecessary batch input file reads Skip expensive pre-read of batch input files when no batch limits apply and model allowlist checks are not required, and decode model-embedded file IDs before file-content fetches to prevent upstream 404s. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(batch-rate-limiter): prevent user metadata flag from bypassing model allowlist The skip_batch_input_file_rate_limiting flag in litellm_metadata is user-controllable for batch requests (request-body metadata lands in litellm_metadata via LITELLM_METADATA_ROUTES). Honoring it unconditionally also skipped _enforce_batch_file_model_access, letting a restricted key submit a JSONL referencing models outside its allowlist. Only honor the metadata-based skip when the key has no model allowlist to enforce. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(batch_rate_limiter): enforce model access check before honoring skip paths Admin-configured skips (disable_batch_input_file_rate_limiting, skip_batch_input_file_rate_limiting_for_models/_for_providers) and the no-applicable-rate-limits short-circuit previously bypassed _enforce_batch_file_model_access. A key with a restricted model allowlist could therefore submit a batch JSONL referencing models outside its allowlist whenever any of these skip paths fired, and the provider-skip path was attacker-controllable via the request body's custom_llm_provider field. Hoist the model-access guard to the top so restricted keys always have their JSONL validated regardless of which skip would otherwise apply. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(batch_rate_limiter): wildcard model bypass + fail-open embedded model creds - _key_requires_batch_model_access_check: check '*' / all-proxy-models before access_group_ids so wildcard keys skip the JSONL download. - _resolve_batch_input_file_fetch_params: wrap embedded-model get_credentials_for_model in try/except HTTPException, mirroring the request-model fallback path, and always decode the file id. Co-authored-by: Yassin Kortam <yassin@berri.ai> * perf(batch_rate_limiter): reuse rate-limit descriptors across skip check and counter increment * test(batch_rate_limiter): cover skip-path and file-fetch helpers Add unit tests for the batch rate limiter's new skip/routing helpers so the diff's patch coverage no longer depends on the CircleCI batches job, whose coverage upload is blocked when an unrelated Bedrock integration test aborts the run. Covers _get_batch_routing_model, _matches_skip_list, _key_requires_batch_model_access_check, _has_applicable_batch_rate_limits, _should_skip_batch_input_file_processing, _resolve_batch_input_file_fetch_params, the descriptor-reuse path of _check_and_increment_batch_counters, and the non-bytes file content guard in count_input_file_usage. * fix(batch_rate_limiter): resolve provider skip from trusted deployment creds Resolve the batch provider from router deployment credentials instead of the user-supplied custom_llm_provider request field, so an unrestricted key cannot spoof a skip-listed provider to bypass batch rate limiting. Strengthen the provider-skip test to assert the file download and descriptor work were short-circuited, and add a test that a spoofed provider still falls through to rate-limit evaluation. * fix(batch_rate_limiter): guard model-embedded credential lookup on llm_router presence * test(batch_rate_limiter): drive real no-skip fetch path and pin wildcard+access-group predicate The spoofed-provider test configured empty descriptors, so the no-limits shortcut skipped the file fetch and the assertion only proved the provider allow-list did not short-circuit before descriptor evaluation. Give the key an applicable rate limit so the only thing that can prevent the fetch is the provider skip, then assert afile_content is awaited and the counters are incremented; the spoofed custom_llm_provider must not skip processing. Also cover the wildcard / all-proxy-models plus access_group_ids combination in the model-access predicate so the wildcard-wins behavior is locked down. * fix(batch_rate_limiter): drop client-controlled skip flag to close quota bypass The litellm_metadata.skip_batch_input_file_rate_limiting flag was read straight from the request body, so any caller whose key had unrestricted model access could send it and skip the input-file download, token count, and RPM/TPM reservation, bypassing their batch rate limits. Skip decisions now derive only from server-controlled general_settings. * fix(batch_rate_limiter): match per-model skip on file-bound model only The per-model skip resolved its model from _get_batch_routing_model, which prefers the client-supplied top-level model field. That field only selects routing credentials; the models a batch actually runs are the body.model entries in the input JSONL. An unrestricted key could therefore name a skip-listed deployment at the top level while routing a different, same-provider model through the file, skipping the download, token count and rate-limit reservation to bypass batch RPM/TPM limits. Match the per-model skip against the file-bound model only (model-embedded file id or unified managed file target), which is fixed when the file is created and reflects the model the batch runs. The provider skip keeps using the routing model since an admin opting out of a whole provider already accepts any of that provider's models. * fix(batch_rate_limiter): drop forgeable per-model skip to close quota bypass The per-model skip matched skip_batch_input_file_rate_limiting_for_models against the model bound to the input file id. That model comes from decode_model_from_file_id / the unified file id, both unsigned base64 the caller fully controls, so a caller could re-encode an accessible provider file id with a skip-listed model while the JSONL still routes rate-limited body.model entries and bypass the batch RPM/TPM counters. The models a batch actually runs are its JSONL body.model entries, which cannot be known without reading the file, so no caller-influenced model identifier can safely gate a skip. Remove the per-model skip entirely. The provider skip stays because the provider is resolved from trusted deployment credentials and the batch is constrained to run on that provider; the global disable and no-applicable-limits skips stay because they do not depend on caller input. * fix(batch_rate_limiter): warn when no-op per-model skip key is configured * test(batch_rate_limiter): patch llm_router so model-embedded credential-error test hits fallback * fix(batch_rate_limiter): resolve provider skip from file-bound model create_batch routes a model-embedded or unified file id on the model bound to that file and ignores the top-level model, so deriving the provider skip from the top-level model first let a caller point model at a skip-listed provider while the file routed a rate-limited one, skipping counter enforcement. Resolve the routing model from the file binding first, matching the batch endpoint. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-01 20:03:19 -07:00
4 people	70d2748d80	fix(proxy): map stripped batch body.model to proxy alias for auth (#29264) ... * fix(proxy): map stripped batch body.model to proxy alias for auth replace_model_in_jsonl rewrites JSONL body.model to the provider id before upload; batch file access checks must resolve that id back to model_name so keys granted the proxy alias are not rejected with 403. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): surface resolved proxy alias in batch file 403 detail --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-05-29 19:58:04 -07:00
3 people	b80246971b	fix(batches): count non-chat tokens, validate batch-file model access (VERIA-39) (#27015) ... * fix(batches): count non-chat tokens and validate every model in batch file Two security control bypasses on POST /v1/batches: 1. `_get_batch_job_input_file_usage` only summed tokens for `body.messages` (chat completions). Embedding (`input`) and text completion (`prompt`) batches reported zero, letting massive non-chat workloads slip past TPM rate limits. Extend the counter to handle string and list shapes for both fields. 2. The batch input file was forwarded to the upstream provider without inspecting the models named inside the JSONL — only the outer `model` query parameter was checked against the caller's allowlist. A caller restricted to gpt-3.5 could submit a batch targeting gpt-4o and the upstream would execute it under the proxy's shared API key. Add `_get_models_from_batch_input_file_content` (returns the distinct `body.model` values) and call it from `_enforce_batch_file_model_access` in the pre-call hook, which runs each model through `can_key_call_model` so the same allowlist semantics (wildcards, access groups, all-proxy-models, team aliases) the proxy enforces on `/chat/completions` apply here too. Any unauthorized model raises a 403 before the file is forwarded. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(batches): count pre-tokenized prompt/input shapes, classify 403 logs Two follow-ups from the Greptile review on the batch validation PR: 1. P1 TPM bypass via integer token arrays. The OpenAI batch schema accepts ``prompt`` and ``input`` as ``list[int]`` (a single pre-tokenized prompt) or ``list[list[int]]`` (multiple) in addition to the string and ``list[str]`` shapes. Pre-fix only the string shapes were counted, so a caller could submit a batch with hundreds of millions of pre-tokenized tokens and the rate limiter would record zero. Extract the per-field logic into ``_count_prompt_or_input_tokens`` and count each int as one token. 2. P2 access-denial logs were indistinguishable from I/O failures. ``count_input_file_usage`` caught every exception under a generic "Error counting input file usage" message, so an intentional 403 from ``_enforce_batch_file_model_access`` looked the same in the logs as a missing file or a Prisma timeout. Catch ``HTTPException`` separately and log 403s at WARNING level with a security-relevant message before re-raising. Tests cover the new shapes: single ``list[int]``, ``list[list[int]]`` (the worst-case bypass vector), and embeddings ``input`` with pre-tokenized arrays. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-01 17:36:12 -07:00