mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-21 08:26:34 +00:00
[Feat] - Ishaan main merge branch (#23596)
* fix(bedrock): respect s3_region_name for batch file uploads (#23569) * fix(bedrock): respect s3_region_name for batch file uploads (GovCloud fix) * fix: s3_region_name always wins over aws_region_name for S3 signing (Greptile feedback) * fix: _filter_headers_for_aws_signature - Bedrock KB (#23571) * fix: _filter_headers_for_aws_signature * fix: filter None header values in all post-signing re-merge paths Addresses Greptile feedback: None-valued headers were being filtered during SigV4 signing but re-merged back into the final headers dict afterward, which would cause downstream HTTP client failures. Made-with: Cursor * feat(router): tag_regex routing — route by User-Agent regex without per-developer tag config (#23594) * feat(router): add tag_regex support for header-based routing Adds a new `tag_regex` field to litellm_params that lets operators route requests based on regex patterns matched against request headers — primarily User-Agent — without requiring per-developer tag configuration. Use case: route all Claude Code traffic (User-Agent: claude-code/x.y.z) to a dedicated deployment by setting: tag_regex: - "^User-Agent: claude-code\\/" in the deployment's litellm_params. Works alongside existing `tags` routing; exact tag match takes precedence over regex match. Unmatched requests fall through to deployments tagged `default`. The matched deployment, pattern, and user_agent are recorded in `metadata["tag_routing"]` so they flow through to SpendLogs automatically. * fix(tag_regex): address backwards-compat, metadata overwrite, and warning noise Three issues from code review: 1. Backwards-compat: `has_tag_filter` was widened to activate on any non-empty User-Agent, which would raise ValueError for existing deployments using plain tags without a `default` fallback. Fix: only activate header-based regex filtering when at least one candidate deployment has `tag_regex` configured. 2. Metadata overwrite: `metadata["tag_routing"]` was overwritten for every matching deployment in the loop, leaving inaccurate provenance when multiple deployments match. Fix: write only for the first match. 3. Warning noise: an invalid regex pattern logged one warning per header string rather than once per pattern. Fix: compile first (catching re.error once), then iterate over header strings. Also adds two new tests covering these cases, and adds docs page for tag_regex routing with a Claude Code walk-through. * refactor(tag_regex): remove unnecessary _healthy_list copy * docs: merge tag_regex section into tag_routing.md, remove standalone page - Add ## Regex-based tag routing (tag_regex) section to existing tag_routing.md instead of a separate page - Remove tag_regex_routing.md standalone doc (odd UX to have a separate page for a sub-feature) - Remove proxy/tag_regex_routing from sidebars.js - Add match_any=False debug warning in tag_based_routing.py when regex routing fires under strict mode (regex always uses OR semantics) * fix(tag_regex): address greptile review - security docs, strict-mode enforcement, validation order - Strengthen security note in tag_routing.md: explicitly state User-Agent is client-supplied and can be set to any value; frame tag_regex as a traffic classification hint, not an access-control mechanism - Move tag_regex startup validation before _add_deployment() so an invalid pattern never leaves partial router state - Enforce match_any=False strict-tag policy: when a deployment has both tags and tag_regex and the strict tag check fails, skip the regex fallback rather than silently bypassing the operator's intent - Extract per-deployment match logic into _match_deployment() helper to keep get_deployments_for_tag() readable - Add two new tests: strict-mode blocks regex fallback, regex-only deployment still matches under match_any=False * fix(ci): apply Black formatting to 14 files and stabilize flaky caplog tests - Run Black formatter on 14 files that were failing the lint check - Replace caplog-based assertions in TestAliasConflicts with unittest.mock.patch on verbose_logger.warning for xdist compatibility - The caplog fixture can produce empty text in pytest-xdist workers in certain CI environments, causing flaky test failures Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
This commit is contained in:
Ishaan Jaff
co-authored by
Cursor Agent
Ishaan Jaff
Cursor Agent
Ishaan Jaff
parent
25ee2fb3f9
commit
b87d1f8dad
@@ -209,6 +209,106 @@ Expect to see the following response header when this works
|
||||
x-litellm-model-id: default-model
|
||||
```
|
||||
|
||||
## Regex-based tag routing (`tag_regex`)
|
||||
|
||||
Use `tag_regex` to route requests based on regex patterns matched against request headers, without requiring clients to pass a tag explicitly. This is useful when clients already send a recognisable header, such as `User-Agent`.
|
||||
|
||||
**Use case: route all Claude Code traffic to dedicated AWS accounts**
|
||||
|
||||
Claude Code always sends `User-Agent: claude-code/<version>`. With `tag_regex` you can route that traffic to a dedicated deployment automatically — no per-developer configuration needed.
|
||||
|
||||
### 1. Config
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
# Claude Code traffic → dedicated deployment, matched by User-Agent
|
||||
- model_name: claude-sonnet
|
||||
litellm_params:
|
||||
model: bedrock/converse/anthropic-claude-sonnet-4-6
|
||||
aws_region_name: us-east-1
|
||||
aws_role_name: arn:aws:iam::111122223333:role/LiteLLMClaudeCode
|
||||
tag_regex:
|
||||
- "^User-Agent: claude-code\\/" # matches claude-code/1.x, 2.x, etc.
|
||||
model_info:
|
||||
id: claude-code-deployment
|
||||
|
||||
# All other traffic falls back to the default deployment
|
||||
- model_name: claude-sonnet
|
||||
litellm_params:
|
||||
model: bedrock/converse/anthropic-claude-sonnet-4-6
|
||||
aws_region_name: us-east-1
|
||||
aws_role_name: arn:aws:iam::444455556666:role/LiteLLMDefault
|
||||
tags:
|
||||
- default
|
||||
model_info:
|
||||
id: regular-deployment
|
||||
|
||||
router_settings:
|
||||
enable_tag_filtering: true
|
||||
tag_filtering_match_any: true
|
||||
|
||||
general_settings:
|
||||
master_key: sk-1234
|
||||
```
|
||||
|
||||
### 2. Verify routing
|
||||
|
||||
Claude Code sets `User-Agent: claude-code/<version>` automatically — no client config needed:
|
||||
|
||||
```shell
|
||||
# Claude Code request (User-Agent set automatically by Claude Code)
|
||||
curl http://localhost:4000/v1/chat/completions \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-H "User-Agent: claude-code/1.2.3" \
|
||||
-d '{"model": "claude-sonnet", "messages": [{"role": "user", "content": "hi"}]}'
|
||||
# → x-litellm-model-id: claude-code-deployment
|
||||
|
||||
# Any other client (no matching User-Agent) → default deployment
|
||||
curl http://localhost:4000/v1/chat/completions \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-d '{"model": "claude-sonnet", "messages": [{"role": "user", "content": "hi"}]}'
|
||||
# → x-litellm-model-id: regular-deployment
|
||||
```
|
||||
|
||||
### How matching works
|
||||
|
||||
| Priority | Condition | Result |
|
||||
|----------|-----------|--------|
|
||||
| 1 | Request has `tags` AND deployment has `tags` | Exact tag match (respects `match_any` setting) |
|
||||
| 2 | Deployment has `tag_regex` AND request has a `User-Agent` | Regex match (always OR logic — any pattern match suffices) |
|
||||
| 3 | Deployment has `tags: [default]` | Default fallback |
|
||||
| 4 | No default set | All healthy deployments returned |
|
||||
|
||||
`tag_regex` always uses OR semantics — `tag_filtering_match_any=False` applies only to exact tag matching, not to regex patterns.
|
||||
|
||||
### Observability
|
||||
|
||||
When a regex matches, `tag_routing` is written into request metadata and flows to SpendLogs:
|
||||
|
||||
```json
|
||||
{
|
||||
"tag_routing": {
|
||||
"matched_via": "tag_regex",
|
||||
"matched_value": "^User-Agent: claude-code\\/",
|
||||
"user_agent": "claude-code/1.2.3",
|
||||
"request_tags": []
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Security note
|
||||
|
||||
:::caution
|
||||
|
||||
**`User-Agent` is a client-supplied header and can be set to any value.** Any API consumer can send `User-Agent: claude-code/1.0` regardless of whether they are actually using Claude Code.
|
||||
|
||||
Do not rely on `tag_regex` routing to enforce access controls or spend limits — use [team/key-based routing](./users) for that. `tag_regex` is a **traffic classification hint** (useful for billing visibility, capacity planning, and routing convenience), not a security boundary.
|
||||
|
||||
:::
|
||||
|
||||
|
||||
---
|
||||
|
||||
## ✨ Team based tag routing (Enterprise)
|
||||
|
||||
LiteLLM Proxy supports team-based tag routing, allowing you to associate specific tags with teams and route requests accordingly. Example **Team A can access gpt-4 deployment A, Team B can access gpt-4 deployment B** (LLM Access Control For Teams)
|
||||
|
||||
Reference in New Issue
Block a user