mirror of
https://github.com/tiennm99/litellm.git
synced 2026-06-18 00:48:01 +00:00
feat(proxy): add credential overrides per team/project via model_config metadata (#24438)
This commit is contained in:
michelligabriele
@@ -0,0 +1,274 @@
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# Per-Team/Project Credential Routing
|
||||
|
||||
Route the same model to different LLM provider endpoints (e.g. different Azure instances) based on which team or project makes the request.
|
||||
|
||||
## Overview
|
||||
|
||||
In multi-tenant deployments, different teams often need the same model name (e.g., `gpt-4`) to hit different provider endpoints — for example, separate Azure OpenAI instances per business unit for cost isolation, data residency, or rate limit separation.
|
||||
|
||||
**Credential routing** lets you configure this in team/project metadata using the existing [credentials table](./ui_credentials.md), without duplicating model definitions or creating separate model groups per team.
|
||||
|
||||
```
|
||||
Hotel Team → gpt-4 → https://hotel-eastus.openai.azure.com/
|
||||
Flight Team → gpt-4 → https://flight-centralus.openai.azure.com/
|
||||
```
|
||||
|
||||
### Precedence Chain
|
||||
|
||||
When a request comes in, the system walks this precedence chain (first match wins):
|
||||
|
||||
1. **Clientside credentials** — `api_base`/`api_key` passed in the request body ([docs](./clientside_auth.md))
|
||||
2. **Project model-specific** — override for this exact model in the project's `model_config`
|
||||
3. **Project default** — `defaultconfig` in the project's `model_config`
|
||||
4. **Team model-specific** — override for this exact model in the team's `model_config`
|
||||
5. **Team default** — `defaultconfig` in the team's `model_config`
|
||||
6. **Deployment default** — the model's `litellm_params` as configured in `config.yaml`
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Step 1: Create Credentials
|
||||
|
||||
Store your Azure endpoint credentials in the credentials table. You can do this via the [UI](./ui_credentials.md) or API:
|
||||
|
||||
```bash showLineNumbers
|
||||
# Create credential for Hotel team's Azure endpoint
|
||||
curl -X POST 'http://0.0.0.0:4000/credentials' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"credential_name": "hotel-azure-eastus",
|
||||
"credential_values": {
|
||||
"api_base": "https://hotel-eastus.openai.azure.com/",
|
||||
"api_key": "sk-azure-hotel-key-xxx"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
```bash showLineNumbers
|
||||
# Create credential for Flight team's Azure endpoint
|
||||
curl -X POST 'http://0.0.0.0:4000/credentials' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"credential_name": "flight-azure-centralus",
|
||||
"credential_values": {
|
||||
"api_base": "https://flight-centralus.openai.azure.com/",
|
||||
"api_key": "sk-azure-flight-key-xxx"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Step 2: Set `model_config` on Teams
|
||||
|
||||
Add a `model_config` key to the team's metadata referencing the credential by name:
|
||||
|
||||
```bash showLineNumbers
|
||||
# Hotel team — default Azure endpoint for all models
|
||||
curl -X PATCH 'http://0.0.0.0:4000/team/update' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"team_id": "hotel-team-id",
|
||||
"metadata": {
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {
|
||||
"litellm_credentials": "hotel-azure-eastus"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
```bash showLineNumbers
|
||||
# Flight team — default Azure endpoint for all models
|
||||
curl -X PATCH 'http://0.0.0.0:4000/team/update' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"team_id": "flight-team-id",
|
||||
"metadata": {
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {
|
||||
"litellm_credentials": "flight-azure-centralus"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Step 3: Make Requests
|
||||
|
||||
Requests are automatically routed to the correct Azure endpoint based on the API key's team:
|
||||
|
||||
```bash showLineNumbers
|
||||
# Request using Hotel team's API key → routes to hotel-eastus.openai.azure.com
|
||||
curl http://localhost:4000/v1/chat/completions \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'Authorization: Bearer sk-hotel-team-key' \
|
||||
-d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}'
|
||||
|
||||
# Request using Flight team's API key → routes to flight-centralus.openai.azure.com
|
||||
curl http://localhost:4000/v1/chat/completions \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'Authorization: Bearer sk-flight-team-key' \
|
||||
-d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}'
|
||||
```
|
||||
|
||||
## Per-Model Overrides
|
||||
|
||||
You can set different credentials for specific models while keeping a default for everything else:
|
||||
|
||||
```bash showLineNumbers
|
||||
curl -X PATCH 'http://0.0.0.0:4000/team/update' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"team_id": "hotel-team-id",
|
||||
"metadata": {
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {
|
||||
"litellm_credentials": "hotel-azure-eastus"
|
||||
}
|
||||
},
|
||||
"gpt-4": {
|
||||
"azure": {
|
||||
"litellm_credentials": "hotel-azure-westus"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
With this config:
|
||||
- `gpt-4` requests → `hotel-azure-westus` credential (model-specific)
|
||||
- All other models → `hotel-azure-eastus` credential (default)
|
||||
|
||||
## Project-Level Overrides
|
||||
|
||||
Projects inherit their team's `model_config` but can override at the project level. Project overrides take precedence over team overrides.
|
||||
|
||||
```bash showLineNumbers
|
||||
# Project overrides the team default for all models
|
||||
curl -X PATCH 'http://0.0.0.0:4000/project/update' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"project_id": "hotel-rec-app-id",
|
||||
"metadata": {
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {
|
||||
"litellm_credentials": "hotel-rec-azure"
|
||||
}
|
||||
},
|
||||
"gpt-4-vision": {
|
||||
"azure": {
|
||||
"litellm_credentials": "hotel-rec-vision"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Full Example: Hotel Team with Two Projects
|
||||
|
||||
**Setup:**
|
||||
- **Hotel Team**: default `hotel-azure-eastus`, GPT-4 override to `hotel-azure-westus`
|
||||
- **Hotel Rec App** (project): default `hotel-rec-azure`, GPT-4-Vision override to `hotel-rec-vision`
|
||||
- **Hotel Review App** (project): no overrides — inherits team config
|
||||
|
||||
**Resolution:**
|
||||
|
||||
| Request | Resolved Credential | Why |
|
||||
|---|---|---|
|
||||
| Hotel Rec App → `gpt-4` | `hotel-rec-azure` | Project default (no project model-specific match for gpt-4) |
|
||||
| Hotel Rec App → `gpt-4-vision` | `hotel-rec-vision` | Project model-specific |
|
||||
| Hotel Review App → `gpt-3.5` | `hotel-azure-eastus` | Team default (no project config) |
|
||||
| Hotel Review App → `gpt-4` | `hotel-azure-westus` | Team model-specific |
|
||||
|
||||
## `model_config` Schema
|
||||
|
||||
The `model_config` key is a JSON object in team/project `metadata`:
|
||||
|
||||
```json
|
||||
{
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"<provider>": {
|
||||
"litellm_credentials": "<credential-name>"
|
||||
}
|
||||
},
|
||||
"<model-name>": {
|
||||
"<provider>": {
|
||||
"litellm_credentials": "<credential-name>"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
| `defaultconfig` | Fallback credential for any model not explicitly listed |
|
||||
| `<model-name>` | Model-specific override — must match the LiteLLM model group name |
|
||||
| `<provider>` | Provider key (e.g. `azure`, `openai`, `bedrock`). When the model name includes a provider prefix (e.g. `azure/gpt-4`), the system prefers the matching provider key |
|
||||
| `litellm_credentials` | Name of a credential in the [credentials table](./ui_credentials.md) |
|
||||
|
||||
### Credential Values
|
||||
|
||||
The referenced credential can contain any combination of:
|
||||
|
||||
| Key | Description |
|
||||
|---|---|
|
||||
| `api_base` | Provider endpoint URL |
|
||||
| `api_key` | API key for the provider |
|
||||
| `api_version` | API version (e.g. for Azure) |
|
||||
|
||||
Only keys present in the credential are applied. Keys already in the request (e.g. clientside `api_version`) are never overwritten.
|
||||
|
||||
## Enabling the Feature
|
||||
|
||||
This feature is **disabled by default** and must be explicitly enabled. To enable it:
|
||||
|
||||
<Tabs>
|
||||
|
||||
<TabItem value="config" label="config.yaml">
|
||||
|
||||
```yaml
|
||||
litellm_settings:
|
||||
enable_model_config_credential_overrides: true
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="env" label="Environment Variable">
|
||||
|
||||
```bash
|
||||
export LITELLM_ENABLE_MODEL_CONFIG_CREDENTIAL_OVERRIDES=true
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
</Tabs>
|
||||
|
||||
:::info
|
||||
The feature flag must be enabled before `model_config` entries in team/project metadata take effect. Without it, credential routing is completely inert — no metadata is read, no credentials are resolved.
|
||||
:::
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Adding LLM Credentials](./ui_credentials.md) — Create and manage reusable credentials
|
||||
- [Project Management](./project_management.md) — Project hierarchy and API
|
||||
- [Team Budgets](./team_budgets.md) — Team-level budget management
|
||||
- [Clientside LLM Credentials](./clientside_auth.md) — Passing credentials in the request body
|
||||
- [Credential Usage Tracking](./credential_usage_tracking.md) — Track spend by credential
|
||||
@@ -563,7 +563,8 @@ const sidebars = {
|
||||
"proxy/model_access",
|
||||
"proxy/model_access_groups",
|
||||
"proxy/access_groups",
|
||||
"proxy/team_model_add"
|
||||
"proxy/team_model_add",
|
||||
"proxy/credential_routing"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -318,6 +318,7 @@ return_response_headers: bool = (
|
||||
False # get response headers from LLM Api providers - example x-remaining-requests,
|
||||
)
|
||||
enable_json_schema_validation: bool = False
|
||||
enable_model_config_credential_overrides: bool = False
|
||||
enable_key_alias_format_validation: bool = (
|
||||
False # opt-in validation of key_alias format on /key/generate and /key/update
|
||||
)
|
||||
|
||||
@@ -10,6 +10,7 @@ from starlette.datastructures import Headers
|
||||
import litellm
|
||||
from litellm._logging import verbose_logger, verbose_proxy_logger
|
||||
from litellm._service_logger import ServiceLogging
|
||||
from litellm.litellm_core_utils.credential_accessor import CredentialAccessor
|
||||
from litellm.litellm_core_utils.safe_json_loads import safe_json_loads
|
||||
from litellm.proxy._types import (
|
||||
AddTeamCallback,
|
||||
@@ -1264,6 +1265,9 @@ async def add_litellm_data_to_request( # noqa: PLR0915
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
)
|
||||
|
||||
# Save pre-alias model name for credential override lookup
|
||||
_pre_alias_model = data.get("model")
|
||||
|
||||
# Team Model Aliases
|
||||
_update_model_if_team_alias_exists(
|
||||
data=data,
|
||||
@@ -1280,6 +1284,14 @@ async def add_litellm_data_to_request( # noqa: PLR0915
|
||||
"[PROXY] returned data from litellm_pre_call_utils: %s", data
|
||||
)
|
||||
|
||||
# Team/Project credential overrides from model_config
|
||||
# Placed after the debug log to avoid leaking credential secrets in logs
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data,
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
pre_alias_model_name=_pre_alias_model,
|
||||
)
|
||||
|
||||
## ENFORCED PARAMS CHECK
|
||||
# loop through each enforced param
|
||||
# example enforced_params ['user', 'metadata', 'metadata.generation_name']
|
||||
@@ -1407,6 +1419,175 @@ def _update_model_if_key_alias_exists(
|
||||
return
|
||||
|
||||
|
||||
def _apply_credential_overrides_from_model_config(
|
||||
data: dict,
|
||||
user_api_key_dict: UserAPIKeyAuth,
|
||||
pre_alias_model_name: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Walk the model_config precedence chain in team/project metadata.
|
||||
If a matching credential is found, set api_base/api_key/api_version on data
|
||||
so they override deployment defaults in the router.
|
||||
|
||||
Precedence (highest to lowest):
|
||||
1. Clientside credentials (already in data — skip if present)
|
||||
2. Project model-specific override
|
||||
3. Project default override (defaultconfig)
|
||||
4. Team model-specific override
|
||||
5. Team default override (defaultconfig)
|
||||
6. Deployment default (no action needed)
|
||||
"""
|
||||
# Feature flag gate — disabled by default, opt in with litellm.enable_model_config_credential_overrides = True
|
||||
if not litellm.enable_model_config_credential_overrides:
|
||||
return
|
||||
|
||||
# Respect clientside credentials — highest precedence
|
||||
if data.get("api_base") is not None or data.get("api_key") is not None:
|
||||
return
|
||||
|
||||
model_name = data.get("model")
|
||||
if not model_name:
|
||||
return
|
||||
|
||||
project_metadata = user_api_key_dict.project_metadata or {}
|
||||
team_metadata = user_api_key_dict.team_metadata or {}
|
||||
|
||||
project_model_config = project_metadata.get("model_config")
|
||||
team_model_config = team_metadata.get("model_config")
|
||||
|
||||
if not project_model_config and not team_model_config:
|
||||
return
|
||||
|
||||
# Extract provider hint from model name (e.g. "azure/gpt-4" -> "azure")
|
||||
provider: Optional[str] = None
|
||||
if "/" in model_name:
|
||||
provider = model_name.split("/", 1)[0]
|
||||
|
||||
credential_name = _resolve_credential_from_model_config(
|
||||
model_name=model_name,
|
||||
project_model_config=project_model_config,
|
||||
team_model_config=team_model_config,
|
||||
pre_alias_model_name=pre_alias_model_name,
|
||||
provider=provider,
|
||||
)
|
||||
|
||||
if not credential_name:
|
||||
return
|
||||
|
||||
credential_values = CredentialAccessor.get_credential_values(credential_name)
|
||||
if not credential_values:
|
||||
_safe_cred = str(credential_name).replace("\n", "").replace("\r", "")
|
||||
verbose_proxy_logger.warning(
|
||||
"model_config references credential '%s' but it was not found or has no values",
|
||||
_safe_cred,
|
||||
)
|
||||
return
|
||||
|
||||
# Apply credential overrides only for keys not already in the request
|
||||
for key in ("api_base", "api_key", "api_version"):
|
||||
if key in credential_values and key not in data:
|
||||
data[key] = credential_values[key]
|
||||
|
||||
_safe_model = str(model_name).replace("\n", "").replace("\r", "")
|
||||
_safe_cred = str(credential_name).replace("\n", "").replace("\r", "")
|
||||
verbose_proxy_logger.debug(
|
||||
"Applied credential override '%s' for model '%s'",
|
||||
_safe_cred,
|
||||
_safe_model,
|
||||
)
|
||||
|
||||
|
||||
def _resolve_credential_from_model_config(
|
||||
model_name: str,
|
||||
project_model_config: Optional[dict],
|
||||
team_model_config: Optional[dict],
|
||||
pre_alias_model_name: Optional[str] = None,
|
||||
provider: Optional[str] = None,
|
||||
) -> Optional[str]:
|
||||
"""
|
||||
Walk the precedence chain and return the first matching credential name.
|
||||
|
||||
Checks (in order):
|
||||
1. project_model_config[model_name][provider] — project model-specific
|
||||
2. project_model_config[pre_alias_model_name][provider] — project pre-alias
|
||||
3. project_model_config["defaultconfig"][provider] — project default
|
||||
4. team_model_config[model_name][provider] — team model-specific
|
||||
5. team_model_config[pre_alias_model_name][provider] — team pre-alias
|
||||
6. team_model_config["defaultconfig"][provider] — team default
|
||||
|
||||
When a model-specific entry exists but contains no litellm_credentials,
|
||||
the function falls through to defaultconfig. This is intentional —
|
||||
an entry without litellm_credentials is treated as incomplete config,
|
||||
not as an explicit "no override" signal.
|
||||
"""
|
||||
# Build the list of model names to try (post-alias first, then pre-alias)
|
||||
model_names_to_try = [model_name]
|
||||
if pre_alias_model_name and pre_alias_model_name != model_name:
|
||||
model_names_to_try.append(pre_alias_model_name)
|
||||
|
||||
for model_config in (project_model_config, team_model_config):
|
||||
if not model_config or not isinstance(model_config, dict):
|
||||
continue
|
||||
|
||||
# Model-specific check (try resolved name, then pre-alias name)
|
||||
for name in model_names_to_try:
|
||||
model_entry = model_config.get(name)
|
||||
if model_entry:
|
||||
credential_name = _extract_credential_from_entry(
|
||||
model_entry, provider=provider
|
||||
)
|
||||
if credential_name:
|
||||
return credential_name
|
||||
_safe_name = str(name).replace("\n", "").replace("\r", "")
|
||||
verbose_proxy_logger.debug(
|
||||
"model_config entry '%s' found but has no litellm_credentials, "
|
||||
"trying next candidate",
|
||||
_safe_name,
|
||||
)
|
||||
|
||||
# Default check
|
||||
default_entry = model_config.get("defaultconfig")
|
||||
if default_entry:
|
||||
credential_name = _extract_credential_from_entry(
|
||||
default_entry, provider=provider
|
||||
)
|
||||
if credential_name:
|
||||
return credential_name
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _extract_credential_from_entry(
|
||||
entry: dict, provider: Optional[str] = None
|
||||
) -> Optional[str]:
|
||||
"""
|
||||
Extract litellm_credentials from a model_config entry.
|
||||
|
||||
Entry structure: {"azure": {"litellm_credentials": "name"}, ...}
|
||||
|
||||
When provider is given (e.g. "azure"), tries an exact provider match first.
|
||||
Falls back to the first credential found across all provider keys.
|
||||
"""
|
||||
if not isinstance(entry, dict):
|
||||
return None
|
||||
|
||||
# Prefer exact provider match when provider hint is available
|
||||
if provider and provider in entry:
|
||||
provider_config = entry[provider]
|
||||
if isinstance(provider_config, dict):
|
||||
credential_name = provider_config.get("litellm_credentials")
|
||||
if credential_name:
|
||||
return credential_name
|
||||
|
||||
# Fall back to first available provider
|
||||
for provider_config in entry.values():
|
||||
if isinstance(provider_config, dict):
|
||||
credential_name = provider_config.get("litellm_credentials")
|
||||
if credential_name:
|
||||
return credential_name
|
||||
return None
|
||||
|
||||
|
||||
def _get_enforced_params(
|
||||
general_settings: Optional[dict], user_api_key_dict: UserAPIKeyAuth
|
||||
) -> Optional[list]:
|
||||
|
||||
@@ -13,14 +13,18 @@ from litellm.proxy._types import TeamCallbackMetadata, UserAPIKeyAuth
|
||||
from litellm.proxy.litellm_pre_call_utils import (
|
||||
KeyAndTeamLoggingSettings,
|
||||
LiteLLMProxyRequestSetup,
|
||||
_apply_credential_overrides_from_model_config,
|
||||
_extract_credential_from_entry,
|
||||
_get_dynamic_logging_metadata,
|
||||
_get_enforced_params,
|
||||
_get_metadata_variable_name,
|
||||
_resolve_credential_from_model_config,
|
||||
_update_model_if_key_alias_exists,
|
||||
add_guardrails_from_policy_engine,
|
||||
add_litellm_data_to_request,
|
||||
check_if_token_is_service_account,
|
||||
)
|
||||
from litellm.types.utils import CredentialItem
|
||||
|
||||
sys.path.insert(
|
||||
0, os.path.abspath("../../..")
|
||||
@@ -1912,3 +1916,542 @@ async def test_bearer_token_not_in_debug_logs():
|
||||
f"Bearer token leaked in debug logs. "
|
||||
f"Found token in log output:\n{log_output[:500]}"
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Tests for credential overrides from model_config (team/project metadata)
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def setup_test_credentials():
|
||||
"""Populate litellm.credential_list with test credentials and enable feature flag, clean up after."""
|
||||
original = litellm.credential_list[:]
|
||||
original_flag = litellm.enable_model_config_credential_overrides
|
||||
litellm.enable_model_config_credential_overrides = True
|
||||
litellm.credential_list.extend(
|
||||
[
|
||||
CredentialItem(
|
||||
credential_name="hotel-azure-eastus",
|
||||
credential_info={},
|
||||
credential_values={
|
||||
"api_base": "https://hotel-eastus.openai.azure.com/",
|
||||
"api_key": "key-hotel-eastus",
|
||||
},
|
||||
),
|
||||
CredentialItem(
|
||||
credential_name="hotel-azure-westus",
|
||||
credential_info={},
|
||||
credential_values={
|
||||
"api_base": "https://hotel-westus.openai.azure.com/",
|
||||
"api_key": "key-hotel-westus",
|
||||
},
|
||||
),
|
||||
CredentialItem(
|
||||
credential_name="hotel-rec-azure",
|
||||
credential_info={},
|
||||
credential_values={
|
||||
"api_base": "https://hotel-rec-app.openai.azure.com/",
|
||||
"api_key": "key-hotel-rec",
|
||||
},
|
||||
),
|
||||
CredentialItem(
|
||||
credential_name="hotel-rec-vision",
|
||||
credential_info={},
|
||||
credential_values={
|
||||
"api_base": "https://hotel-rec-vision.openai.azure.com/",
|
||||
"api_key": "key-hotel-rec-vision",
|
||||
"api_version": "2024-06-01",
|
||||
},
|
||||
),
|
||||
CredentialItem(
|
||||
credential_name="flight-azure-centralus",
|
||||
credential_info={},
|
||||
credential_values={
|
||||
"api_base": "https://flight-centralus.openai.azure.com/",
|
||||
"api_key": "key-flight-centralus",
|
||||
},
|
||||
),
|
||||
]
|
||||
)
|
||||
yield
|
||||
litellm.credential_list[:] = original
|
||||
litellm.enable_model_config_credential_overrides = original_flag
|
||||
|
||||
|
||||
# --- Unit tests for _extract_credential_from_entry ---
|
||||
|
||||
|
||||
def test_extract_credential_from_entry_azure():
|
||||
entry = {"azure": {"litellm_credentials": "my-cred"}}
|
||||
assert _extract_credential_from_entry(entry) == "my-cred"
|
||||
|
||||
|
||||
def test_extract_credential_from_entry_no_credential():
|
||||
entry = {"azure": {"some_other_key": "value"}}
|
||||
assert _extract_credential_from_entry(entry) is None
|
||||
|
||||
|
||||
def test_extract_credential_from_entry_empty():
|
||||
assert _extract_credential_from_entry({}) is None
|
||||
|
||||
|
||||
def test_extract_credential_from_entry_non_dict_value():
|
||||
entry = {"azure": "not-a-dict"}
|
||||
assert _extract_credential_from_entry(entry) is None
|
||||
|
||||
|
||||
def test_extract_credential_from_entry_non_dict_entry():
|
||||
"""Non-dict entry (e.g. string) should return None, not crash."""
|
||||
assert _extract_credential_from_entry("my-cred-name") is None
|
||||
assert _extract_credential_from_entry(["a", "list"]) is None
|
||||
assert _extract_credential_from_entry(42) is None
|
||||
|
||||
|
||||
# --- Unit tests for _resolve_credential_from_model_config ---
|
||||
|
||||
|
||||
def test_resolve_project_model_specific_wins():
|
||||
project_config = {
|
||||
"gpt-4": {"azure": {"litellm_credentials": "proj-gpt4"}},
|
||||
"defaultconfig": {"azure": {"litellm_credentials": "proj-default"}},
|
||||
}
|
||||
team_config = {
|
||||
"gpt-4": {"azure": {"litellm_credentials": "team-gpt4"}},
|
||||
"defaultconfig": {"azure": {"litellm_credentials": "team-default"}},
|
||||
}
|
||||
result = _resolve_credential_from_model_config(
|
||||
"gpt-4", project_config, team_config
|
||||
)
|
||||
assert result == "proj-gpt4"
|
||||
|
||||
|
||||
def test_resolve_project_default_wins_over_team():
|
||||
project_config = {
|
||||
"defaultconfig": {"azure": {"litellm_credentials": "proj-default"}},
|
||||
}
|
||||
team_config = {
|
||||
"gpt-4": {"azure": {"litellm_credentials": "team-gpt4"}},
|
||||
"defaultconfig": {"azure": {"litellm_credentials": "team-default"}},
|
||||
}
|
||||
result = _resolve_credential_from_model_config(
|
||||
"gpt-4", project_config, team_config
|
||||
)
|
||||
assert result == "proj-default"
|
||||
|
||||
|
||||
def test_resolve_team_model_specific_wins_over_team_default():
|
||||
team_config = {
|
||||
"gpt-4": {"azure": {"litellm_credentials": "team-gpt4"}},
|
||||
"defaultconfig": {"azure": {"litellm_credentials": "team-default"}},
|
||||
}
|
||||
result = _resolve_credential_from_model_config("gpt-4", None, team_config)
|
||||
assert result == "team-gpt4"
|
||||
|
||||
|
||||
def test_resolve_team_default_used_as_fallback():
|
||||
team_config = {
|
||||
"defaultconfig": {"azure": {"litellm_credentials": "team-default"}},
|
||||
}
|
||||
result = _resolve_credential_from_model_config("gpt-3.5", None, team_config)
|
||||
assert result == "team-default"
|
||||
|
||||
|
||||
def test_resolve_no_match_returns_none():
|
||||
result = _resolve_credential_from_model_config("gpt-4", None, None)
|
||||
assert result is None
|
||||
|
||||
|
||||
def test_resolve_empty_configs_returns_none():
|
||||
result = _resolve_credential_from_model_config("gpt-4", {}, {})
|
||||
assert result is None
|
||||
|
||||
|
||||
def test_resolve_model_not_in_any_config():
|
||||
project_config = {"gpt-4": {"azure": {"litellm_credentials": "x"}}}
|
||||
result = _resolve_credential_from_model_config("gpt-3.5", project_config, None)
|
||||
assert result is None
|
||||
|
||||
|
||||
# --- Integration tests for _apply_credential_overrides_from_model_config ---
|
||||
|
||||
|
||||
def test_apply_overrides_project_model_specific(setup_test_credentials):
|
||||
"""Scenario 2: Hotel Rec App -> gpt-4-vision -> project model-specific."""
|
||||
data = {"model": "gpt-4-vision"}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata={
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {"litellm_credentials": "hotel-azure-eastus"}
|
||||
},
|
||||
"gpt-4": {"azure": {"litellm_credentials": "hotel-azure-westus"}},
|
||||
}
|
||||
},
|
||||
project_metadata={
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {"litellm_credentials": "hotel-rec-azure"}
|
||||
},
|
||||
"gpt-4-vision": {
|
||||
"azure": {"litellm_credentials": "hotel-rec-vision"}
|
||||
},
|
||||
}
|
||||
},
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
assert data["api_base"] == "https://hotel-rec-vision.openai.azure.com/"
|
||||
assert data["api_key"] == "key-hotel-rec-vision"
|
||||
assert data["api_version"] == "2024-06-01"
|
||||
|
||||
|
||||
def test_apply_overrides_project_default(setup_test_credentials):
|
||||
"""Scenario 1: Hotel Rec App -> gpt-4 -> project default."""
|
||||
data = {"model": "gpt-4"}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata={
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {"litellm_credentials": "hotel-azure-eastus"}
|
||||
},
|
||||
"gpt-4": {"azure": {"litellm_credentials": "hotel-azure-westus"}},
|
||||
}
|
||||
},
|
||||
project_metadata={
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {"litellm_credentials": "hotel-rec-azure"}
|
||||
},
|
||||
"gpt-4-vision": {
|
||||
"azure": {"litellm_credentials": "hotel-rec-vision"}
|
||||
},
|
||||
}
|
||||
},
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
assert data["api_base"] == "https://hotel-rec-app.openai.azure.com/"
|
||||
assert data["api_key"] == "key-hotel-rec"
|
||||
|
||||
|
||||
def test_apply_overrides_team_model_specific(setup_test_credentials):
|
||||
"""Scenario 4: Hotel Review App -> gpt-4 -> team model-specific."""
|
||||
data = {"model": "gpt-4"}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata={
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {"litellm_credentials": "hotel-azure-eastus"}
|
||||
},
|
||||
"gpt-4": {"azure": {"litellm_credentials": "hotel-azure-westus"}},
|
||||
}
|
||||
},
|
||||
project_metadata={},
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
assert data["api_base"] == "https://hotel-westus.openai.azure.com/"
|
||||
assert data["api_key"] == "key-hotel-westus"
|
||||
|
||||
|
||||
def test_apply_overrides_team_default(setup_test_credentials):
|
||||
"""Scenario 3: Hotel Review App -> gpt-3.5 -> team default."""
|
||||
data = {"model": "gpt-3.5"}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata={
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {"litellm_credentials": "hotel-azure-eastus"}
|
||||
},
|
||||
"gpt-4": {"azure": {"litellm_credentials": "hotel-azure-westus"}},
|
||||
}
|
||||
},
|
||||
project_metadata={},
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
assert data["api_base"] == "https://hotel-eastus.openai.azure.com/"
|
||||
assert data["api_key"] == "key-hotel-eastus"
|
||||
|
||||
|
||||
def test_apply_overrides_no_config(setup_test_credentials):
|
||||
"""Scenario 6: No model_config anywhere -> data unchanged."""
|
||||
data = {"model": "gpt-4"}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata={},
|
||||
project_metadata={},
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
assert "api_base" not in data
|
||||
assert "api_key" not in data
|
||||
|
||||
|
||||
def test_apply_overrides_clientside_credentials_take_precedence(
|
||||
setup_test_credentials,
|
||||
):
|
||||
"""Clientside api_base/api_key in data should block model_config override."""
|
||||
data = {
|
||||
"model": "gpt-4",
|
||||
"api_base": "https://my-custom-endpoint.openai.azure.com/",
|
||||
"api_key": "my-custom-key",
|
||||
}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata={
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {"litellm_credentials": "hotel-azure-eastus"}
|
||||
}
|
||||
}
|
||||
},
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
assert data["api_base"] == "https://my-custom-endpoint.openai.azure.com/"
|
||||
assert data["api_key"] == "my-custom-key"
|
||||
|
||||
|
||||
def test_apply_overrides_missing_credential_name(setup_test_credentials):
|
||||
"""model_config references a credential that doesn't exist -> no override."""
|
||||
data = {"model": "gpt-4"}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata={
|
||||
"model_config": {
|
||||
"gpt-4": {
|
||||
"azure": {"litellm_credentials": "nonexistent-credential"}
|
||||
}
|
||||
}
|
||||
},
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
assert "api_base" not in data
|
||||
assert "api_key" not in data
|
||||
|
||||
|
||||
def test_apply_overrides_api_version_only_if_present(setup_test_credentials):
|
||||
"""api_version should only be set if the credential contains it."""
|
||||
data = {"model": "gpt-3.5"}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata={
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {"litellm_credentials": "hotel-azure-eastus"}
|
||||
}
|
||||
}
|
||||
},
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
assert data["api_base"] == "https://hotel-eastus.openai.azure.com/"
|
||||
assert data["api_key"] == "key-hotel-eastus"
|
||||
assert "api_version" not in data
|
||||
|
||||
|
||||
def test_apply_overrides_no_model_in_data(setup_test_credentials):
|
||||
"""No model in request data -> skip override."""
|
||||
data = {"messages": [{"role": "user", "content": "hello"}]}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata={
|
||||
"model_config": {
|
||||
"defaultconfig": {
|
||||
"azure": {"litellm_credentials": "some-cred"}
|
||||
}
|
||||
}
|
||||
},
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
assert "api_base" not in data
|
||||
|
||||
|
||||
def test_apply_overrides_none_metadata(setup_test_credentials):
|
||||
"""None metadata on both team and project -> skip override."""
|
||||
data = {"model": "gpt-4"}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata=None,
|
||||
project_metadata=None,
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
assert "api_base" not in data
|
||||
|
||||
|
||||
def test_apply_overrides_clientside_api_version_preserved(setup_test_credentials):
|
||||
"""Clientside api_version should not be overwritten by credential."""
|
||||
data = {"model": "gpt-4-vision", "api_version": "2025-01-01"}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata={
|
||||
"model_config": {
|
||||
"gpt-4-vision": {
|
||||
"azure": {"litellm_credentials": "hotel-rec-vision"}
|
||||
}
|
||||
}
|
||||
},
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
# api_base and api_key should be set from credential
|
||||
assert data["api_base"] == "https://hotel-rec-vision.openai.azure.com/"
|
||||
assert data["api_key"] == "key-hotel-rec-vision"
|
||||
# api_version should be preserved from the request, not overwritten
|
||||
assert data["api_version"] == "2025-01-01"
|
||||
|
||||
|
||||
def test_resolve_non_dict_model_config_ignored():
|
||||
"""Non-dict model_config (e.g. string) should be safely skipped."""
|
||||
result = _resolve_credential_from_model_config("gpt-4", "not-a-dict", None)
|
||||
assert result is None
|
||||
|
||||
result = _resolve_credential_from_model_config(
|
||||
"gpt-4", None, ["also", "not", "a", "dict"]
|
||||
)
|
||||
assert result is None
|
||||
|
||||
# Valid config still works alongside invalid one
|
||||
result = _resolve_credential_from_model_config(
|
||||
"gpt-4",
|
||||
"invalid",
|
||||
{"gpt-4": {"azure": {"litellm_credentials": "valid-cred"}}},
|
||||
)
|
||||
assert result == "valid-cred"
|
||||
|
||||
|
||||
def test_resolve_pre_alias_model_name_fallback():
|
||||
"""model_config keyed on pre-alias name should match after alias resolution."""
|
||||
team_config = {
|
||||
"gpt-4": {"azure": {"litellm_credentials": "team-gpt4"}},
|
||||
}
|
||||
# Post-alias name doesn't match, but pre-alias does (team scope)
|
||||
result = _resolve_credential_from_model_config(
|
||||
"azure/gpt-4-0613", None, team_config, pre_alias_model_name="gpt-4"
|
||||
)
|
||||
assert result == "team-gpt4"
|
||||
|
||||
# Same test for project scope
|
||||
project_config = {
|
||||
"gpt-4": {"azure": {"litellm_credentials": "proj-gpt4"}},
|
||||
}
|
||||
result = _resolve_credential_from_model_config(
|
||||
"azure/gpt-4-0613", project_config, None, pre_alias_model_name="gpt-4"
|
||||
)
|
||||
assert result == "proj-gpt4"
|
||||
|
||||
|
||||
def test_resolve_post_alias_name_takes_priority():
|
||||
"""Post-alias (resolved) name should be tried before pre-alias name."""
|
||||
team_config = {
|
||||
"gpt-4": {"azure": {"litellm_credentials": "pre-alias-cred"}},
|
||||
"gpt-4o-team-1": {"azure": {"litellm_credentials": "post-alias-cred"}},
|
||||
}
|
||||
# Team scope
|
||||
result = _resolve_credential_from_model_config(
|
||||
"gpt-4o-team-1", None, team_config, pre_alias_model_name="gpt-4"
|
||||
)
|
||||
assert result == "post-alias-cred"
|
||||
|
||||
# Project scope
|
||||
result = _resolve_credential_from_model_config(
|
||||
"gpt-4o-team-1", team_config, None, pre_alias_model_name="gpt-4"
|
||||
)
|
||||
assert result == "post-alias-cred"
|
||||
|
||||
|
||||
def test_apply_overrides_with_alias(setup_test_credentials):
|
||||
"""Credential override should work when model name was changed by alias."""
|
||||
# Simulate: user called "my-gpt4", alias resolved to "azure/gpt-4-custom"
|
||||
# model_config is keyed on "my-gpt4" (the pre-alias name)
|
||||
data = {"model": "azure/gpt-4-custom"}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata={
|
||||
"model_config": {
|
||||
"my-gpt4": {"azure": {"litellm_credentials": "hotel-azure-eastus"}},
|
||||
}
|
||||
},
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data,
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
pre_alias_model_name="my-gpt4",
|
||||
)
|
||||
assert data["api_base"] == "https://hotel-eastus.openai.azure.com/"
|
||||
assert data["api_key"] == "key-hotel-eastus"
|
||||
|
||||
|
||||
def test_apply_overrides_feature_flag_disabled_by_default():
|
||||
"""Feature flag defaults to False — credential overrides are inert until explicitly enabled."""
|
||||
assert litellm.enable_model_config_credential_overrides is False
|
||||
data = {"model": "gpt-4"}
|
||||
user_api_key_dict = UserAPIKeyAuth(
|
||||
api_key="test-key",
|
||||
team_metadata={
|
||||
"model_config": {
|
||||
"gpt-4": {"azure": {"litellm_credentials": "hotel-azure-eastus"}}
|
||||
}
|
||||
},
|
||||
)
|
||||
_apply_credential_overrides_from_model_config(
|
||||
data=data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
assert "api_base" not in data
|
||||
assert "api_key" not in data
|
||||
|
||||
|
||||
def test_extract_credential_provider_hint_prefers_exact_match():
|
||||
"""Provider hint selects the correct provider in a multi-provider entry."""
|
||||
entry = {
|
||||
"openai": {"litellm_credentials": "openai-cred"},
|
||||
"azure": {"litellm_credentials": "azure-cred"},
|
||||
}
|
||||
# With provider hint, should pick the exact match
|
||||
assert _extract_credential_from_entry(entry, provider="azure") == "azure-cred"
|
||||
assert _extract_credential_from_entry(entry, provider="openai") == "openai-cred"
|
||||
|
||||
# Without provider hint, falls back to first key (insertion order)
|
||||
result = _extract_credential_from_entry(entry)
|
||||
assert result in ("openai-cred", "azure-cred")
|
||||
|
||||
# Unknown provider falls back to first available
|
||||
result = _extract_credential_from_entry(entry, provider="bedrock")
|
||||
assert result in ("openai-cred", "azure-cred")
|
||||
|
||||
|
||||
def test_resolve_provider_hint_from_model_name():
|
||||
"""Provider prefix in model name (e.g. azure/gpt-4) threads through to entry extraction."""
|
||||
config = {
|
||||
"gpt-4": {
|
||||
"openai": {"litellm_credentials": "openai-cred"},
|
||||
"azure": {"litellm_credentials": "azure-cred"},
|
||||
},
|
||||
}
|
||||
# Model name "azure/gpt-4" -> provider="azure" -> should prefer azure-cred
|
||||
# But _resolve_credential_from_model_config tries "azure/gpt-4" first (no match),
|
||||
# then falls to defaultconfig (no match). So we need to use pre_alias_model_name.
|
||||
result = _resolve_credential_from_model_config(
|
||||
"azure/gpt-4", config, None, pre_alias_model_name="gpt-4", provider="azure"
|
||||
)
|
||||
assert result == "azure-cred"
|
||||
|
||||
Reference in New Issue
Block a user