mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-21 08:26:34 +00:00
Merge branch 'main' into litellm_view_key_pagination_calls_fix
This commit is contained in:
Achintya Rajan
@@ -2606,6 +2606,8 @@ jobs:
|
||||
-e GEMINI_API_KEY=$GEMINI_API_KEY \
|
||||
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
|
||||
-e ASSEMBLYAI_API_KEY=$ASSEMBLYAI_API_KEY \
|
||||
-e AZURE_API_KEY_PASSHROUGH=$AZURE_API_KEY_PASSHROUGH \
|
||||
-e AZURE_API_BASE_PASSHROUGH=$AZURE_API_BASE_PASSHROUGH \
|
||||
-e USE_DDTRACE=True \
|
||||
-e DD_API_KEY=$DD_API_KEY \
|
||||
-e DD_SITE=$DD_SITE \
|
||||
|
||||
@@ -8,6 +8,10 @@ Track spend for keys, users, and teams across 100+ LLMs.
|
||||
|
||||
LiteLLM automatically tracks spend for all known models. See our [model cost map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)
|
||||
|
||||
:::tip Keep Pricing Data Updated
|
||||
[Sync model pricing data from GitHub](../sync_models_github.md) to ensure accurate cost tracking.
|
||||
:::
|
||||
|
||||
### How to Track Spend with LiteLLM
|
||||
|
||||
**Step 1**
|
||||
|
||||
@@ -19,6 +19,10 @@ model_list:
|
||||
|
||||
Retrieve detailed information about each model listed in the `/model/info` endpoint, including descriptions from the `config.yaml` file, and additional model info (e.g. max tokens, cost per input token, etc.) pulled from the model_info you set and the [litellm model cost map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json). Sensitive details like API keys are excluded for security purposes.
|
||||
|
||||
:::tip Sync Model Data
|
||||
Keep your model pricing data up to date by [syncing models from GitHub](../sync_models_github.md).
|
||||
:::
|
||||
|
||||
<Tabs
|
||||
defaultValue="curl"
|
||||
values={[
|
||||
|
||||
@@ -221,6 +221,8 @@ litellm_settings:
|
||||
|
||||
2. Make a request with the custom metadata labels
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="Curl" label="Curl Request">
|
||||
```bash
|
||||
curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
|
||||
-H 'Content-Type: application/json' \
|
||||
@@ -244,6 +246,34 @@ curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
|
||||
}
|
||||
}'
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="key" label="on Key">
|
||||
|
||||
```bash
|
||||
curl -L -X POST 'http://0.0.0.0:4000/key/generate' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"metadata": {
|
||||
"foo": "hello world"
|
||||
}
|
||||
}'
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="team" label="on Team">
|
||||
|
||||
```bash
|
||||
curl -L -X POST 'http://0.0.0.0:4000/team/new' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"metadata": {
|
||||
"foo": "hello world"
|
||||
}
|
||||
}'
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
3. Check your `/metrics` endpoint for the custom metrics
|
||||
|
||||
|
||||
@@ -0,0 +1,354 @@
|
||||
# LiteLLM Self-Hosted Security & Encryption FAQ
|
||||
|
||||
## Data in Transit Encryption
|
||||
|
||||
### Does the product encrypt data in transit?
|
||||
|
||||
**Yes**, LiteLLM encrypts data in transit using TLS/SSL.
|
||||
|
||||
### Available in both OSS and Enterprise?
|
||||
|
||||
**Yes**, TLS encryption is available in both Open Source and Enterprise versions.
|
||||
|
||||
### In transit between the calling client and the product?
|
||||
|
||||
**Yes**, HTTPS/TLS is supported through SSL certificate configuration.
|
||||
|
||||
**Configuration:**
|
||||
```bash
|
||||
# CLI
|
||||
litellm --ssl_keyfile_path /path/to/key.pem --ssl_certfile_path /path/to/cert.pem
|
||||
|
||||
# Environment Variables
|
||||
export SSL_KEYFILE_PATH="/path/to/key.pem"
|
||||
export SSL_CERTFILE_PATH="/path/to/cert.pem"
|
||||
```
|
||||
|
||||
**Documentation Reference:** `docs/my-website/docs/guides/security_settings.md`
|
||||
|
||||
### In transit between the product and the LLM providers?
|
||||
|
||||
**Yes**, all connections to LLM providers use TLS encryption by default.
|
||||
|
||||
**Implementation Details:**
|
||||
- Uses Python's `ssl.create_default_context()`
|
||||
- Leverages HTTPX and aiohttp libraries with SSL/TLS enabled
|
||||
- Uses certifi CA bundle by default for SSL verification
|
||||
|
||||
**Code Reference:** `litellm/llms/custom_httpx/http_handler.py` (lines 43-105)
|
||||
|
||||
### Are TCP sessions to the LLM providers shared?
|
||||
|
||||
**Yes**, TCP connections are pooled and reused.
|
||||
|
||||
**Details:**
|
||||
- Connection pooling is enabled by default
|
||||
- Default: 1000 max concurrent connections with keepalive
|
||||
- Sessions are maintained across requests to the same provider
|
||||
- Reduces overhead of TLS handshakes
|
||||
|
||||
**Code Reference:** `litellm/llms/custom_httpx/http_handler.py` (lines 704-712)
|
||||
|
||||
### Or does the product negotiate a new TLS session with the same LLM provider for every sequential call?
|
||||
|
||||
**No**, TLS sessions are reused through connection pooling. New TLS handshakes are not performed for every request.
|
||||
|
||||
### How is it encrypted?
|
||||
|
||||
**TLS 1.2 and TLS 1.3**
|
||||
|
||||
Uses Python's default SSL context which supports both TLS 1.2 and TLS 1.3. The specific version negotiated depends on:
|
||||
- Python version
|
||||
- System SSL library (typically OpenSSL)
|
||||
- Server capabilities
|
||||
|
||||
**Implementation:** `ssl.create_default_context()` in Python
|
||||
|
||||
### How are these added to the product's configuration?
|
||||
|
||||
#### x.509 Certificate
|
||||
|
||||
**Method 1: CLI Arguments**
|
||||
```bash
|
||||
litellm --ssl_certfile_path /path/to/certificate.pem
|
||||
```
|
||||
|
||||
**Method 2: Environment Variable**
|
||||
```bash
|
||||
export SSL_CERTFILE_PATH="/path/to/certificate.pem"
|
||||
```
|
||||
|
||||
#### Private Key
|
||||
|
||||
**Method 1: CLI Arguments**
|
||||
```bash
|
||||
litellm --ssl_keyfile_path /path/to/private_key.pem
|
||||
```
|
||||
|
||||
**Method 2: Environment Variable**
|
||||
```bash
|
||||
export SSL_KEYFILE_PATH="/path/to/private_key.pem"
|
||||
```
|
||||
|
||||
#### Certificate Bundle/Chain
|
||||
|
||||
**For client-to-proxy connections:**
|
||||
Use standard SSL certificate setup with intermediate certificates bundled in the certfile.
|
||||
|
||||
**For proxy-to-LLM provider connections:**
|
||||
|
||||
**Method 1: Config YAML**
|
||||
```yaml
|
||||
litellm_settings:
|
||||
ssl_verify: "/path/to/ca_bundle.pem"
|
||||
```
|
||||
|
||||
**Method 2: Environment Variable**
|
||||
```bash
|
||||
export SSL_CERT_FILE="/path/to/ca_bundle.pem"
|
||||
```
|
||||
|
||||
**Method 3: Client Certificate Authentication**
|
||||
```yaml
|
||||
litellm_settings:
|
||||
ssl_certificate: "/path/to/client_certificate.pem"
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```bash
|
||||
export SSL_CERTIFICATE="/path/to/client_certificate.pem"
|
||||
```
|
||||
|
||||
### Documentation Coverage
|
||||
|
||||
**Primary Documentation:**
|
||||
- `docs/my-website/docs/guides/security_settings.md` - SSL/TLS configuration guide
|
||||
|
||||
**Additional References:**
|
||||
- `litellm/proxy/proxy_cli.py` (lines 455-467) - CLI options
|
||||
- `docs/my-website/docs/completion/http_handler_config.md` - Custom HTTP handler configuration
|
||||
|
||||
---
|
||||
|
||||
## Data at Rest Encryption
|
||||
|
||||
### Does the product encrypt data at rest?
|
||||
|
||||
**Partially**. Only specific sensitive data is encrypted at rest.
|
||||
|
||||
### What data is stored in encrypted form?
|
||||
|
||||
#### Encrypted Data:
|
||||
1. **LLM API Keys** - Model credentials in `LiteLLM_ProxyModelTable.litellm_params`
|
||||
2. **Provider Credentials** - Stored in `LiteLLM_CredentialsTable.credential_values`
|
||||
3. **Configuration Secrets** - Sensitive config values in `LiteLLM_Config` table
|
||||
4. **Virtual Keys** - When using secret managers (optional feature)
|
||||
|
||||
#### NOT Encrypted:
|
||||
1. **Spend Logs** - Request/response data in `LiteLLM_SpendLogs`
|
||||
2. **Audit Logs** - Change history in `LiteLLM_AuditLog`
|
||||
3. **User/Team/Organization Data** - Metadata and configuration
|
||||
4. **Cached Prompts and Completions** - Cache data is stored in plaintext
|
||||
|
||||
### Cached prompts and completions?
|
||||
|
||||
**No**, cached prompts and completions are **NOT encrypted**.
|
||||
|
||||
Cache backends (Redis, S3, local disk) store data as plaintext JSON.
|
||||
|
||||
**Code References:**
|
||||
- `litellm/caching/redis_cache.py`
|
||||
- `litellm/caching/s3_cache.py`
|
||||
- `litellm/caching/caching.py`
|
||||
|
||||
### Configuration data?
|
||||
|
||||
**Partially encrypted**.
|
||||
|
||||
#### What IS Encrypted:
|
||||
- LLM API keys and credentials in model configurations
|
||||
- Sensitive values in `LiteLLM_Config` table
|
||||
- Credential values in `LiteLLM_CredentialsTable`
|
||||
|
||||
#### What is NOT Encrypted:
|
||||
- Model names and aliases
|
||||
- Rate limits and budget settings
|
||||
- User/team/organization metadata
|
||||
- Non-sensitive configuration parameters
|
||||
|
||||
**Code Reference:** `litellm/proxy/management_endpoints/model_management_endpoints.py` (lines 275-308)
|
||||
|
||||
### Log data?
|
||||
|
||||
**No**, log data is **NOT encrypted**.
|
||||
|
||||
Log data stored in database tables is in plaintext:
|
||||
- `LiteLLM_SpendLogs` - Contains request/response data, tokens, spend
|
||||
- `LiteLLM_ErrorLogs` - Error information
|
||||
- `LiteLLM_AuditLog` - Audit trail of changes
|
||||
|
||||
**Note:** You can disable logging to avoid storing sensitive data:
|
||||
|
||||
```yaml
|
||||
general_settings:
|
||||
disable_spend_logs: True # Disable writing spend logs to DB
|
||||
disable_error_logs: True # Disable writing error logs to DB
|
||||
```
|
||||
|
||||
**Documentation:** `docs/my-website/docs/proxy/db_info.md` (lines 52-60)
|
||||
|
||||
### Where is it stored?
|
||||
|
||||
#### In the DB?
|
||||
|
||||
**Yes**, encrypted data is stored in PostgreSQL database.
|
||||
|
||||
**Key Tables with Encrypted Data:**
|
||||
- `LiteLLM_ProxyModelTable` - Model configurations with encrypted API keys
|
||||
- `LiteLLM_CredentialsTable` - Credential values
|
||||
- `LiteLLM_Config` - Configuration secrets
|
||||
|
||||
**Schema Reference:** `schema.prisma`
|
||||
|
||||
#### In the filesystem?
|
||||
|
||||
**No**, encrypted data is not stored in the filesystem by default.
|
||||
|
||||
**Note:** If using disk cache (`disk_cache_dir`), cached data is stored unencrypted.
|
||||
|
||||
#### Somewhere else?
|
||||
|
||||
**Optional:** When using secret managers (AWS Secrets Manager, Azure Key Vault, HashiCorp Vault), encrypted data can be stored externally.
|
||||
|
||||
**Configuration:**
|
||||
```yaml
|
||||
general_settings:
|
||||
key_management_system: "aws_secret_manager" # or "azure_key_vault", "hashicorp_vault"
|
||||
```
|
||||
|
||||
**Documentation:** `docs/my-website/docs/secret.md`
|
||||
|
||||
### How is it encrypted?
|
||||
|
||||
**Algorithm:** NaCl SecretBox (XSalsa20-Poly1305 AEAD)
|
||||
|
||||
**NOT AES-256** - LiteLLM uses NaCl (Networking and Cryptography Library) which provides:
|
||||
- XSalsa20 stream cipher
|
||||
- Poly1305 MAC for authentication
|
||||
- Equivalent security to AES-256
|
||||
|
||||
**Key Derivation:**
|
||||
1. Takes `LITELLM_SALT_KEY` (or `LITELLM_MASTER_KEY` if salt key not set)
|
||||
2. Hashes with SHA-256 to derive 256-bit encryption key
|
||||
3. Uses NaCl SecretBox for authenticated encryption
|
||||
|
||||
**Code Reference:** `litellm/proxy/common_utils/encrypt_decrypt_utils.py` (lines 69-112)
|
||||
|
||||
**Implementation:**
|
||||
```python
|
||||
import hashlib
|
||||
import nacl.secret
|
||||
|
||||
# Derive 256-bit key from salt
|
||||
hash_object = hashlib.sha256(signing_key.encode())
|
||||
hash_bytes = hash_object.digest()
|
||||
|
||||
# Create SecretBox and encrypt
|
||||
box = nacl.secret.SecretBox(hash_bytes)
|
||||
encrypted = box.encrypt(value_bytes)
|
||||
```
|
||||
|
||||
### Setting the Encryption Key
|
||||
|
||||
**Required Environment Variable:**
|
||||
```bash
|
||||
export LITELLM_SALT_KEY="your-strong-random-key-here"
|
||||
```
|
||||
|
||||
**Important Notes:**
|
||||
- ⚠️ **Must be set before adding any models**
|
||||
- ⚠️ **Never change this key** - encrypted data becomes unrecoverable
|
||||
- ⚠️ Use a strong random key (recommended: https://1password.com/password-generator/)
|
||||
- If not set, falls back to `LITELLM_MASTER_KEY`
|
||||
|
||||
**Documentation:** `docs/my-website/docs/proxy/prod.md` (section 8, lines 184-196)
|
||||
|
||||
### Documentation Coverage
|
||||
|
||||
**Primary Documentation:**
|
||||
- `docs/my-website/docs/proxy/prod.md` (section 8) - LITELLM_SALT_KEY setup
|
||||
- `docs/my-website/docs/secret.md` - Secret management systems
|
||||
- `docs/my-website/docs/proxy/db_info.md` - Database information
|
||||
|
||||
**Additional References:**
|
||||
- `security.md` - General security measures
|
||||
- `docs/my-website/docs/data_security.md` - Data privacy overview
|
||||
- `schema.prisma` - Database schema with encrypted fields
|
||||
|
||||
---
|
||||
|
||||
## Summary of Security Features
|
||||
|
||||
### ✅ Provided Out of the Box
|
||||
|
||||
1. **TLS/SSL encryption** for client-to-proxy connections
|
||||
2. **TLS encryption** for proxy-to-LLM provider connections (with connection pooling)
|
||||
3. **Encrypted storage** of LLM API keys and credentials
|
||||
4. **Support for TLS 1.2 and TLS 1.3**
|
||||
5. **Connection pooling** to reduce TLS handshake overhead
|
||||
|
||||
### ⚠️ Important Limitations
|
||||
|
||||
1. **Cached data is NOT encrypted** (Redis, S3, disk cache)
|
||||
2. **Log data is NOT encrypted** (spend logs, audit logs)
|
||||
3. **Request/response payloads in logs are NOT encrypted**
|
||||
4. **Uses NaCl SecretBox, NOT AES-256** (equivalent security)
|
||||
5. **TLS version not explicitly configured** - uses Python/system defaults
|
||||
|
||||
### 🔧 Configuration Requirements
|
||||
|
||||
**For Production Deployments:**
|
||||
|
||||
1. **Set LITELLM_SALT_KEY** before adding any models
|
||||
2. **Configure SSL certificates** for HTTPS client connections
|
||||
3. **Consider disabling logs** if they contain sensitive data
|
||||
4. **Use secret managers** for enhanced security (optional)
|
||||
5. **Configure CA bundles** if using custom certificates
|
||||
|
||||
---
|
||||
|
||||
## Quick Start Security Checklist
|
||||
|
||||
```bash
|
||||
# 1. Generate a strong salt key
|
||||
export LITELLM_SALT_KEY="$(openssl rand -base64 32)"
|
||||
|
||||
# 2. Set up SSL certificates (for HTTPS)
|
||||
export SSL_KEYFILE_PATH="/path/to/private_key.pem"
|
||||
export SSL_CERTFILE_PATH="/path/to/certificate.pem"
|
||||
|
||||
# 3. Configure database
|
||||
export DATABASE_URL="postgresql://user:password@host:port/dbname"
|
||||
|
||||
# 4. (Optional) Disable logs if they contain sensitive data
|
||||
# Add to config.yaml:
|
||||
# general_settings:
|
||||
# disable_spend_logs: True
|
||||
# disable_error_logs: True
|
||||
|
||||
# 5. Start LiteLLM Proxy
|
||||
litellm --config config.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- **LiteLLM Documentation:** https://docs.litellm.ai/
|
||||
- **Security Settings Guide:** https://docs.litellm.ai/docs/guides/security_settings
|
||||
- **Production Deployment:** https://docs.litellm.ai/docs/proxy/prod
|
||||
- **Secret Management:** https://docs.litellm.ai/docs/secret
|
||||
|
||||
For security inquiries: support@berri.ai
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# Syncing Models to GitHub model_context_window
|
||||
|
||||
Sync model pricing data from GitHub's `model_prices_and_context_window.json` file outside of the LiteLLM UI.
|
||||
|
||||
> **📹 Video Tutorial**: [Watch how to sync models via the Admin UI](https://www.loom.com/share/ba41acc1882d41b284bbddbb0e9c27ce?sid=bdae351e-2026-4e39-932b-fcb185ff612c)
|
||||
|
||||
## Quick Start
|
||||
|
||||
**Manual sync:**
|
||||
```bash
|
||||
curl -X POST "https://your-proxy-url/reload/model_cost_map" \
|
||||
-H "Authorization: Bearer YOUR_ADMIN_TOKEN" \
|
||||
-H "Content-Type: application/json"
|
||||
```
|
||||
|
||||
**Automatic sync every 6 hours:**
|
||||
```bash
|
||||
curl -X POST "https://your-proxy-url/schedule/model_cost_map_reload?hours=6" \
|
||||
-H "Authorization: Bearer YOUR_ADMIN_TOKEN" \
|
||||
-H "Content-Type: application/json"
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/reload/model_cost_map` | POST | Manual sync |
|
||||
| `/schedule/model_cost_map_reload?hours={hours}` | POST | Schedule periodic sync |
|
||||
| `/schedule/model_cost_map_reload` | DELETE | Cancel scheduled sync |
|
||||
| `/schedule/model_cost_map_reload/status` | GET | Check sync status |
|
||||
|
||||
**Authentication:** Requires admin role or master key
|
||||
|
||||
## Python Example
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
def sync_models(proxy_url, admin_token):
|
||||
response = requests.post(
|
||||
f"{proxy_url}/reload/model_cost_map",
|
||||
headers={"Authorization": f"Bearer {admin_token}"}
|
||||
)
|
||||
return response.json()
|
||||
|
||||
# Usage
|
||||
result = sync_models("https://your-proxy-url", "your-admin-token")
|
||||
print(result['message'])
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
**Custom model cost map URL:**
|
||||
```bash
|
||||
export LITELLM_MODEL_COST_MAP_URL="https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json"
|
||||
```
|
||||
|
||||
**Use local model cost map:**
|
||||
```bash
|
||||
export LITELLM_LOCAL_MODEL_COST_MAP=True
|
||||
```
|
||||
@@ -54,6 +54,20 @@ Allow others to create/delete their own keys.
|
||||
|
||||
[**Go Here**](./self_serve.md)
|
||||
|
||||
## Model Management
|
||||
|
||||
The Admin UI provides comprehensive model management capabilities:
|
||||
|
||||
- **Add Models**: Add new models through the UI without restarting the proxy
|
||||
- **Model Hub**: Make models public for developers to discover available models
|
||||
- **Price Data Sync**: Keep model pricing data up to date by syncing from GitHub
|
||||
|
||||
For detailed information on model management, see [Model Management](./model_management.md).
|
||||
|
||||
:::tip Sync Model Pricing Data
|
||||
[Sync model pricing data from GitHub](./sync_models_github.md) to keep your model cost information current.
|
||||
:::
|
||||
|
||||
## Disable Admin UI
|
||||
|
||||
Set `DISABLE_ADMIN_UI="True"` in your environment to disable the Admin UI.
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 253 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 603 KiB |
@@ -50,30 +50,6 @@ pip install litellm==1.75.5.post2
|
||||
- **Oracle Cloud Infrastructure** - New LLM provider for calling models on Oracle Cloud Infrastructure.
|
||||
- **Digital Ocean's Gradient AI** - New LLM provider for calling models on Digital Ocean's Gradient AI platform.
|
||||
|
||||
|
||||
### 54% RPS Improvement
|
||||
|
||||
Throughput increased by 54% (1,040 → 1,602 RPS, aggregated) per instance while maintaining a 40 ms median overhead. The improvement comes from fixing major O(n²) inefficiencies in the router, primarily caused by repeated use of in statements inside loops over large arrays. Tests were run with a database-only setup (no cache hits). As a result, p95 latency improved by 30% (2,700 → 1,900 ms), enhancing overall stability and scalability under heavy load.
|
||||
|
||||
---
|
||||
|
||||
### Test Setup
|
||||
|
||||
All benchmarks were executed using Locust with 1,000 concurrent users and a ramp-up of 500. The environment was configured to stress the routing layer and eliminate caching as a variable.
|
||||
|
||||
**System Specs**
|
||||
|
||||
- **CPU:** 8 vCPUs
|
||||
- **Memory:** 32 GB RAM
|
||||
|
||||
**Configuration (config.yaml)**
|
||||
|
||||
View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/53f7d554a5d2afcf2c4edb5b6be68ff4)
|
||||
|
||||
**Load Script (no_cache_hits.py)**
|
||||
|
||||
View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/42c33d7a4dc7a57f56a78b560dee3a42)
|
||||
|
||||
---
|
||||
|
||||
### Risk of Upgrade
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: "[Preview] v1.77.5-stable - MCP OAuth 2.0 Support"
|
||||
title: "v1.77.5-stable - MCP OAuth 2.0 Support"
|
||||
slug: "v1-77-5"
|
||||
date: 2025-09-29T10:00:00
|
||||
authors:
|
||||
@@ -11,6 +11,10 @@ authors:
|
||||
title: CTO, LiteLLM
|
||||
url: https://www.linkedin.com/in/reffajnaahsi/
|
||||
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
|
||||
- name: Alexsander Hamir
|
||||
title: Backend Performance Engineer
|
||||
url: https://www.linkedin.com/in/alexsander-baptista/
|
||||
image_url: https://media.licdn.com/dms/image/v2/D5603AQGXnziu4kqNCQ/profile-displayphoto-crop_800_800/B56ZkxEcuOKEAI-/0/1757464874550?e=1762387200&v=beta&t=9SNXLsWhx8OnYPAMQ9fqAr02oevDYEAL2vMYg2f9ieg
|
||||
|
||||
hide_table_of_contents: false
|
||||
---
|
||||
@@ -28,7 +32,7 @@ import TabItem from '@theme/TabItem';
|
||||
docker run \
|
||||
-e STORE_MODEL_IN_DB=True \
|
||||
-p 4000:4000 \
|
||||
ghcr.io/berriai/litellm:v1.77.5.rc.1
|
||||
ghcr.io/berriai/litellm:v1.77.5-stable
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
@@ -49,7 +53,54 @@ pip install litellm==1.77.5
|
||||
- **MCP OAuth 2.0 Support** - Enhanced authentication for Model Context Protocol integrations
|
||||
- **Scheduled Key Rotations** - Automated key rotation capabilities for enhanced security
|
||||
- **New Gemini 2.5 Flash & Flash-lite Models** - Latest September 2025 preview models with improved pricing and features
|
||||
- **Performance Improvements** - Critical InMemoryCache unbounded growth resolution
|
||||
- **Performance Improvements** - 54% RPS improvement
|
||||
|
||||
---
|
||||
|
||||
### Scheduled Key Rotations
|
||||
|
||||
<Image img={require('../../img/release_notes/schedule_key_rotations.png')} style={{ width: '800px', height: 'auto' }} />
|
||||
|
||||
<br/>
|
||||
|
||||
This release brings support for scheduling virtual key rotations on LiteLLM AI Gateway.
|
||||
|
||||
This is great for Proxy Admins looking to enforce Enterprise Grade security for use cases going through LiteLLM AI Gateway.
|
||||
|
||||
From this release you can enforce Virtual Keys to rotate on a schedule of your choice e.g every 15 days/30 days/60 days etc.
|
||||
|
||||
---
|
||||
### Performance Improvements - 54% RPS Improvement
|
||||
|
||||
<Image img={require('../../img/release_notes/perf_77_5.png')} style={{ width: '800px', height: 'auto' }} />
|
||||
|
||||
<br/>
|
||||
|
||||
This release brings a 54% RPS improvement (1,040 → 1,602 RPS, aggregated) per instance.
|
||||
|
||||
The improvement comes from fixing O(n²) inefficiencies in the LiteLLM Router, primarily caused by repeated use of `in` statements inside loops over large arrays.
|
||||
|
||||
Tests were run with a database-only setup (no cache hits).
|
||||
|
||||
#### Test Setup
|
||||
|
||||
All benchmarks were executed using Locust with 1,000 concurrent users and a ramp-up of 500. The environment was configured to stress the routing layer and eliminate caching as a variable.
|
||||
|
||||
**System Specs**
|
||||
|
||||
- **CPU:** 8 vCPUs
|
||||
- **Memory:** 32 GB RAM
|
||||
|
||||
**Configuration (config.yaml)**
|
||||
|
||||
View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/53f7d554a5d2afcf2c4edb5b6be68ff4)
|
||||
|
||||
**Load Script (no_cache_hits.py)**
|
||||
|
||||
View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/42c33d7a4dc7a57f56a78b560dee3a42)
|
||||
|
||||
---
|
||||
|
||||
|
||||
## New Models / Updated Models
|
||||
|
||||
|
||||
@@ -0,0 +1,364 @@
|
||||
---
|
||||
title: "[Preview] v1.77.7-stable - Claude Sonnet 4.5"
|
||||
slug: "v1-77-7"
|
||||
date: 2025-10-04T10:00:00
|
||||
authors:
|
||||
- name: Krrish Dholakia
|
||||
title: CEO, LiteLLM
|
||||
url: https://www.linkedin.com/in/krish-d/
|
||||
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
|
||||
- name: Ishaan Jaff
|
||||
title: CTO, LiteLLM
|
||||
url: https://www.linkedin.com/in/reffajnaahsi/
|
||||
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
|
||||
- name: Alexsander Hamir
|
||||
title: Backend Performance Engineer
|
||||
url: https://www.linkedin.com/in/alexsander-baptista/
|
||||
image_url: https://media.licdn.com/dms/image/v2/D5603AQGXnziu4kqNCQ/profile-displayphoto-crop_800_800/B56ZkxEcuOKEAI-/0/1757464874550?e=1762387200&v=beta&t=9SNXLsWhx8OnYPAMQ9fqAr02oevDYEAL2vMYg2f9ieg
|
||||
- name: Achintya Srivastava
|
||||
title: Fullstack Engineer
|
||||
url: https://www.linkedin.com/in/achintya-rajan/
|
||||
image_url: https://media.licdn.com/dms/image/v2/D5603AQGdkEeyJTdljw/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1716271140869?e=1762387200&v=beta&t=9gOoLPeqR2E5z3KSX61EUj3HVZXmgo87vhVuSHeffjc
|
||||
- name: Sameer Kankute
|
||||
title: Backend Engineer (LLM Translation)
|
||||
url: https://www.linkedin.com/in/sameer-kankute/
|
||||
image_url: https://media.licdn.com/dms/image/v2/D4D03AQHB_loQYd5gjg/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1719137160975?e=1762387200&v=beta&t=0jbuX-f4eSnDxBY3olI6meuYr-LMbObhFmFbRcKF5mY
|
||||
|
||||
hide_table_of_contents: false
|
||||
---
|
||||
|
||||
import Image from '@theme/IdealImage';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
## Deploy this version
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="docker" label="Docker">
|
||||
|
||||
``` showLineNumbers title="docker run litellm"
|
||||
docker run \
|
||||
-e STORE_MODEL_IN_DB=True \
|
||||
-p 4000:4000 \
|
||||
ghcr.io/berriai/litellm:v1.77.7.rc.1
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="pip" label="Pip">
|
||||
|
||||
``` showLineNumbers title="pip install litellm"
|
||||
pip install litellm==1.77.7.rc.1
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
---
|
||||
|
||||
## Key Highlights
|
||||
|
||||
- **Dynamic Rate Limiter v3** - Automatically maximizes throughput when capacity is available (< 80% saturation) by allowing lower-priority requests to use unused capacity, then switches to fair priority-based allocation under high load (≥ 80%) to prevent blocking
|
||||
- **Major Performance Improvements** - 2.9x lower median latency at 1,000 concurrent users.
|
||||
- **Claude Sonnet 4.5** - Support for Anthropic's new Claude Sonnet 4.5 model family with 200K+ context and tiered pricing
|
||||
- **MCP Gateway Enhancements** - Fine-grained tool control, server permissions, and forwardable headers
|
||||
- **AMD Lemonade & Nvidia NIM** - New provider support for AMD Lemonade and Nvidia NIM Rerank
|
||||
- **GitLab Prompt Management** - GitLab-based prompt management integration
|
||||
|
||||
### Performance - 2.9x Lower Median Latency
|
||||
|
||||
<Image img={require('../../img/release_notes/perf_77_7.png')} style={{ width: '800px', height: 'auto' }} />
|
||||
|
||||
<br/>
|
||||
|
||||
This update removes LiteLLM router inefficiencies, reducing complexity from O(M×N) to O(1). Previously, it built a new array and ran repeated checks like data["model"] in llm_router.get_model_ids(). Now, a direct ID-to-deployment map eliminates redundant allocations and scans.
|
||||
|
||||
As a result, performance improved across all latency percentiles:
|
||||
|
||||
- **Median latency:** 320 ms → **110 ms** (−65.6%)
|
||||
- **p95 latency:** 850 ms → **440 ms** (−48.2%)
|
||||
- **p99 latency:** 1,400 ms → **810 ms** (−42.1%)
|
||||
- **Average latency:** 864 ms → **310 ms** (−64%)
|
||||
|
||||
|
||||
#### Test Setup
|
||||
|
||||
**Locust**
|
||||
|
||||
- **Concurrent users:** 1,000
|
||||
- **Ramp-up:** 500
|
||||
|
||||
**System Specs**
|
||||
|
||||
- **CPU:** 4 vCPUs
|
||||
- **Memory:** 8 GB RAM
|
||||
- **LiteLLM Workers:** 4
|
||||
- **Instances**: 4
|
||||
|
||||
**Configuration (config.yaml)**
|
||||
|
||||
View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/53f7d554a5d2afcf2c4edb5b6be68ff4)
|
||||
|
||||
**Load Script (no_cache_hits.py)**
|
||||
|
||||
View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/42c33d7a4dc7a57f56a78b560dee3a42)
|
||||
|
||||
## New Models / Updated Models
|
||||
|
||||
#### New Model Support
|
||||
|
||||
| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features |
|
||||
| -------- | ----- | -------------- | ------------------- | -------------------- | -------- |
|
||||
| Anthropic | `claude-sonnet-4-5` | 200K | $3.00 | $15.00 | Chat, reasoning, vision, function calling, prompt caching |
|
||||
| Anthropic | `claude-sonnet-4-5-20250929` | 200K | $3.00 | $15.00 | Chat, reasoning, vision, function calling, prompt caching |
|
||||
| Bedrock | `eu.anthropic.claude-sonnet-4-5-20250929-v1:0` | 200K | $3.00 | $15.00 | Chat, reasoning, vision, function calling, prompt caching |
|
||||
| Azure AI | `azure_ai/grok-4` | 131K | $5.50 | $27.50 | Chat, reasoning, function calling, web search |
|
||||
| Azure AI | `azure_ai/grok-4-fast-reasoning` | 131K | $0.43 | $1.73 | Chat, reasoning, function calling, web search |
|
||||
| Azure AI | `azure_ai/grok-4-fast-non-reasoning` | 131K | $0.43 | $1.73 | Chat, function calling, web search |
|
||||
| Azure AI | `azure_ai/grok-code-fast-1` | 131K | $3.50 | $17.50 | Chat, function calling, web search |
|
||||
| Groq | `groq/moonshotai/kimi-k2-instruct-0905` | Context varies | Pricing varies | Pricing varies | Chat, function calling |
|
||||
| Ollama | Ollama Cloud models | Varies | Free | Free | Self-hosted models via Ollama Cloud |
|
||||
|
||||
#### Features
|
||||
|
||||
- **[Anthropic](../../docs/providers/anthropic)**
|
||||
- Add new claude-sonnet-4-5 model family with tiered pricing above 200K tokens - [PR #15041](https://github.com/BerriAI/litellm/pull/15041)
|
||||
- Add anthropic/claude-sonnet-4-5 to model price json with prompt caching support - [PR #15049](https://github.com/BerriAI/litellm/pull/15049)
|
||||
- Add 200K prices for Sonnet 4.5 - [PR #15140](https://github.com/BerriAI/litellm/pull/15140)
|
||||
- Add cost tracking for /v1/messages in streaming response - [PR #15102](https://github.com/BerriAI/litellm/pull/15102)
|
||||
- Add /v1/messages/count_tokens to Anthropic routes for non-admin user access - [PR #15034](https://github.com/BerriAI/litellm/pull/15034)
|
||||
- **[Gemini](../../docs/providers/gemini)**
|
||||
- Ignore type param for gemini tools - [PR #15022](https://github.com/BerriAI/litellm/pull/15022)
|
||||
- **[Vertex AI](../../docs/providers/vertex)**
|
||||
- Add LiteLLM Overhead metric for VertexAI - [PR #15040](https://github.com/BerriAI/litellm/pull/15040)
|
||||
- Support googlemap grounding in vertex ai - [PR #15179](https://github.com/BerriAI/litellm/pull/15179)
|
||||
- **[Azure](../../docs/providers/azure)**
|
||||
- Add azure_ai grok-4 model family - [PR #15137](https://github.com/BerriAI/litellm/pull/15137)
|
||||
- Use the `extra_query` parameter for GET requests in Azure Batch - [PR #14997](https://github.com/BerriAI/litellm/pull/14997)
|
||||
- Use extra_query for download results (Batch API) - [PR #15025](https://github.com/BerriAI/litellm/pull/15025)
|
||||
- Add support for Azure AD token-based authorization - [PR #14813](https://github.com/BerriAI/litellm/pull/14813)
|
||||
- **[Ollama](../../docs/providers/ollama)**
|
||||
- Add ollama cloud models - [PR #15008](https://github.com/BerriAI/litellm/pull/15008)
|
||||
- **[Groq](../../docs/providers/groq)**
|
||||
- Add groq/moonshotai/kimi-k2-instruct-0905 - [PR #15079](https://github.com/BerriAI/litellm/pull/15079)
|
||||
- **[OpenAI](../../docs/providers/openai)**
|
||||
- Add support for GPT 5 codex models - [PR #14841](https://github.com/BerriAI/litellm/pull/14841)
|
||||
- **[DeepInfra](../../docs/providers/deepinfra)**
|
||||
- Update DeepInfra model data refresh with latest pricing - [PR #14939](https://github.com/BerriAI/litellm/pull/14939)
|
||||
- **[Bedrock](../../docs/providers/bedrock)**
|
||||
- Add JP Cross-Region Inference - [PR #15188](https://github.com/BerriAI/litellm/pull/15188)
|
||||
- Add "eu.anthropic.claude-sonnet-4-5-20250929-v1:0" - [PR #15181](https://github.com/BerriAI/litellm/pull/15181)
|
||||
- Add twelvelabs bedrock Async Invoke Support - [PR #14871](https://github.com/BerriAI/litellm/pull/14871)
|
||||
- **[Nvidia NIM](../../docs/providers/nvidia_nim)**
|
||||
- Add Nvidia NIM Rerank Support - [PR #15152](https://github.com/BerriAI/litellm/pull/15152)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- **[VLLM](../../docs/providers/vllm)**
|
||||
- Fix response_format bug in hosted vllm audio_transcription - [PR #15010](https://github.com/BerriAI/litellm/pull/15010)
|
||||
- Fix passthrough of atranscription into kwargs going to upstream provider - [PR #15005](https://github.com/BerriAI/litellm/pull/15005)
|
||||
- **[OCI](../../docs/providers/oci)**
|
||||
- Fix OCI Generative AI Integration when using Proxy - [PR #15072](https://github.com/BerriAI/litellm/pull/15072)
|
||||
- **General**
|
||||
- Fix: Authorization header to use correct "Bearer" capitalization - [PR #14764](https://github.com/BerriAI/litellm/pull/14764)
|
||||
- Bug fix: gpt-5-chat-latest has incorrect max_input_tokens value - [PR #15116](https://github.com/BerriAI/litellm/pull/15116)
|
||||
- Update request handling for original exceptions - [PR #15013](https://github.com/BerriAI/litellm/pull/15013)
|
||||
|
||||
#### New Provider Support
|
||||
|
||||
- **[AMD Lemonade](../../docs/providers/lemonade)**
|
||||
- Add AMD Lemonade provider support - [PR #14840](https://github.com/BerriAI/litellm/pull/14840)
|
||||
|
||||
---
|
||||
|
||||
## LLM API Endpoints
|
||||
|
||||
#### Features
|
||||
|
||||
- **[Responses API](../../docs/response_api)**
|
||||
- Return Cost for Responses API Streaming requests - [PR #15053](https://github.com/BerriAI/litellm/pull/15053)
|
||||
|
||||
- **[/generateContent](../../docs/providers/gemini)**
|
||||
- Add full support for native Gemini API translation - [PR #15029](https://github.com/BerriAI/litellm/pull/15029)
|
||||
|
||||
- **Passthrough Gemini Routes**
|
||||
- Add Gemini generateContent passthrough cost tracking - [PR #15014](https://github.com/BerriAI/litellm/pull/15014)
|
||||
- Add streamGenerateContent cost tracking in passthrough - [PR #15199](https://github.com/BerriAI/litellm/pull/15199)
|
||||
|
||||
- **Passthrough Vertex AI Routes**
|
||||
- Add cost tracking for Vertex AI Passthrough `/predict` endpoint - [PR #15019](https://github.com/BerriAI/litellm/pull/15019)
|
||||
- Add cost tracking for Vertex AI Live API WebSocket Passthrough - [PR #14956](https://github.com/BerriAI/litellm/pull/14956)
|
||||
|
||||
- **General**
|
||||
- Preserve Whitespace Characters in Model Response Streams - [PR #15160](https://github.com/BerriAI/litellm/pull/15160)
|
||||
- Add provider name to payload specification - [PR #15130](https://github.com/BerriAI/litellm/pull/15130)
|
||||
- Ensure query params are forwarded from origin url to downstream request - [PR #15087](https://github.com/BerriAI/litellm/pull/15087)
|
||||
|
||||
---
|
||||
|
||||
## Management Endpoints / UI
|
||||
|
||||
#### Features
|
||||
|
||||
- **Virtual Keys**
|
||||
- Ensure LLM_API_KEYs can access pass through routes - [PR #15115](https://github.com/BerriAI/litellm/pull/15115)
|
||||
- Support 'guaranteed_throughput' when setting limits on keys belonging to a team - [PR #15120](https://github.com/BerriAI/litellm/pull/15120)
|
||||
|
||||
- **Models + Endpoints**
|
||||
- Ensure OCI secret fields not shared on /models and /v1/models endpoints - [PR #15085](https://github.com/BerriAI/litellm/pull/15085)
|
||||
- Add snowflake on UI - [PR #15083](https://github.com/BerriAI/litellm/pull/15083)
|
||||
- Make UI theme settings publicly accessible for custom branding - [PR #15074](https://github.com/BerriAI/litellm/pull/15074)
|
||||
|
||||
- **Admin Settings**
|
||||
- Ensure OTEL settings are saved in DB after set on UI - [PR #15118](https://github.com/BerriAI/litellm/pull/15118)
|
||||
- Top api key tags - [PR #15151](https://github.com/BerriAI/litellm/pull/15151), [PR #15156](https://github.com/BerriAI/litellm/pull/15156)
|
||||
|
||||
- **MCP**
|
||||
- show health status of MCP servers - [PR #15185](https://github.com/BerriAI/litellm/pull/15185)
|
||||
- allow setting extra headers on the UI - [PR #15185](https://github.com/BerriAI/litellm/pull/15185)
|
||||
- allow editing allowed tools on the UI - [PR #15185](https://github.com/BerriAI/litellm/pull/15185)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- **Virtual Keys**
|
||||
- (security) prevent user key from updating other user keys - [PR #15201](https://github.com/BerriAI/litellm/pull/15201)
|
||||
- (security) don't return all keys with blank key alias on /v2/key/info - [PR #15201](https://github.com/BerriAI/litellm/pull/15201)
|
||||
- Fix Session Token Cookie Infinite Logout Loop - [PR #15146](https://github.com/BerriAI/litellm/pull/15146)
|
||||
|
||||
- **Models + Endpoints**
|
||||
- Make UI theme settings publicly accessible for custom branding - [PR #15074](https://github.com/BerriAI/litellm/pull/15074)
|
||||
|
||||
- **Teams**
|
||||
- fix failed copy to clipboard for http ui - [PR #15195](https://github.com/BerriAI/litellm/pull/15195)
|
||||
|
||||
- **Logs**
|
||||
- fix logs page render logs on filter lookup - [PR #15195](https://github.com/BerriAI/litellm/pull/15195)
|
||||
- fix lookup list of end users (migrate to more efficient /customers/list lookup) - [PR #15195](https://github.com/BerriAI/litellm/pull/15195)
|
||||
|
||||
- **Test key**
|
||||
- update selected model on key change - [PR #15197](https://github.com/BerriAI/litellm/pull/15197)
|
||||
|
||||
- **Dashboard**
|
||||
- Fix LiteLLM model name fallback in dashboard overview - [PR #14998](https://github.com/BerriAI/litellm/pull/14998)
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Logging / Guardrail / Prompt Management Integrations
|
||||
|
||||
#### Features
|
||||
|
||||
- **[OpenTelemetry](../../docs/observability/otel)**
|
||||
- Use generation_name for span naming in logging method - [PR #14799](https://github.com/BerriAI/litellm/pull/14799)
|
||||
- **[Langfuse](../../docs/proxy/logging#langfuse)**
|
||||
- Handle non-serializable objects in Langfuse logging - [PR #15148](https://github.com/BerriAI/litellm/pull/15148)
|
||||
- Set usage_details.total in langfuse integration - [PR #15015](https://github.com/BerriAI/litellm/pull/15015)
|
||||
- **[Prometheus](../../docs/proxy/prometheus)**
|
||||
- support custom metadata labels on key/team - [PR #15094](https://github.com/BerriAI/litellm/pull/15094)
|
||||
|
||||
|
||||
#### Guardrails
|
||||
|
||||
- **[Javelin](../../docs/proxy/guardrails)**
|
||||
- Add Javelin standalone guardrails integration for LiteLLM Proxy - [PR #14983](https://github.com/BerriAI/litellm/pull/14983)
|
||||
- Add logging for important status fields in guardrails - [PR #15090](https://github.com/BerriAI/litellm/pull/15090)
|
||||
- Don't run post_call guardrail if no text returned from Bedrock - [PR #15106](https://github.com/BerriAI/litellm/pull/15106)
|
||||
|
||||
#### Prompt Management
|
||||
|
||||
- **[GitLab](../../docs/proxy/prompt_management)**
|
||||
- GitLab based Prompt manager - [PR #14988](https://github.com/BerriAI/litellm/pull/14988)
|
||||
|
||||
---
|
||||
|
||||
## Spend Tracking, Budgets and Rate Limiting
|
||||
|
||||
- **Cost Tracking**
|
||||
- Proxy: end user cost tracking in the responses API - [PR #15124](https://github.com/BerriAI/litellm/pull/15124)
|
||||
- **Parallel Request Limiter v3**
|
||||
- Use well known redis cluster hashing algorithm - [PR #15052](https://github.com/BerriAI/litellm/pull/15052)
|
||||
- Fixes to dynamic rate limiter v3 - add saturation detection - [PR #15119](https://github.com/BerriAI/litellm/pull/15119)
|
||||
- Dynamic Rate Limiter v3 - fixes for detecting saturation + fixes for post saturation behavior - [PR #15192](https://github.com/BerriAI/litellm/pull/15192)
|
||||
- **Teams**
|
||||
- Add model specific tpm/rpm limits to teams on LiteLLM - [PR #15044](https://github.com/BerriAI/litellm/pull/15044)
|
||||
|
||||
---
|
||||
|
||||
## MCP Gateway
|
||||
|
||||
- **Server Configuration**
|
||||
- Specify forwardable headers, specify allowed/disallowed tools for MCP servers - [PR #15002](https://github.com/BerriAI/litellm/pull/15002)
|
||||
- Enforce server permissions on call tools - [PR #15044](https://github.com/BerriAI/litellm/pull/15044)
|
||||
- MCP Gateway Fine-grained Tools Addition - [PR #15153](https://github.com/BerriAI/litellm/pull/15153)
|
||||
- **Bug Fixes**
|
||||
- Remove servername prefix mcp tools tests - [PR #14986](https://github.com/BerriAI/litellm/pull/14986)
|
||||
- Resolve regression with duplicate Mcp-Protocol-Version header - [PR #15050](https://github.com/BerriAI/litellm/pull/15050)
|
||||
- Fix test_mcp_server.py - [PR #15183](https://github.com/BerriAI/litellm/pull/15183)
|
||||
|
||||
---
|
||||
|
||||
## Performance / Loadbalancing / Reliability improvements
|
||||
|
||||
- **Router Optimizations**
|
||||
- **+62.5% P99 Latency Improvement** - Remove router inefficiencies (from O(M*N) to O(1)) - [PR #15046](https://github.com/BerriAI/litellm/pull/15046)
|
||||
- Remove hasattr checks in Router - [PR #15082](https://github.com/BerriAI/litellm/pull/15082)
|
||||
- Remove Double Lookups - [PR #15084](https://github.com/BerriAI/litellm/pull/15084)
|
||||
- Optimize _filter_cooldown_deployments from O(n×m + k×n) to O(n) - [PR #15091](https://github.com/BerriAI/litellm/pull/15091)
|
||||
- Optimize unhealthy deployment filtering in retry path (O(n*m) → O(n+m)) - [PR #15110](https://github.com/BerriAI/litellm/pull/15110)
|
||||
- **Cache Optimizations**
|
||||
- Reduce complexity of InMemoryCache.evict_cache from O(n*log(n)) to O(log(n)) - [PR #15000](https://github.com/BerriAI/litellm/pull/15000)
|
||||
- Avoiding expensive operations when cache isn't available - [PR #15182](https://github.com/BerriAI/litellm/pull/15182)
|
||||
- **Worker Management**
|
||||
- Add proxy CLI option to recycle workers after N requests - [PR #15007](https://github.com/BerriAI/litellm/pull/15007)
|
||||
- **Metrics & Monitoring**
|
||||
- LiteLLM Overhead metric tracking - Add support for tracking litellm overhead on cache hits - [PR #15045](https://github.com/BerriAI/litellm/pull/15045)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Updates
|
||||
|
||||
- **Provider Documentation**
|
||||
- Update litellm docs from latest release - [PR #15004](https://github.com/BerriAI/litellm/pull/15004)
|
||||
- Add missing api_key parameter - [PR #15058](https://github.com/BerriAI/litellm/pull/15058)
|
||||
- **General Documentation**
|
||||
- Use docker compose instead of docker-compose - [PR #15024](https://github.com/BerriAI/litellm/pull/15024)
|
||||
- Add railtracks to projects that are using litellm - [PR #15144](https://github.com/BerriAI/litellm/pull/15144)
|
||||
- Perf: Last week improvement - [PR #15193](https://github.com/BerriAI/litellm/pull/15193)
|
||||
- Sync models GitHub documentation with Loom video and cross-reference - [PR #15191](https://github.com/BerriAI/litellm/pull/15191)
|
||||
|
||||
---
|
||||
|
||||
## Security Fixes
|
||||
|
||||
- **JWT Token Security** - Don't log JWT SSO token on .info() log - [PR #15145](https://github.com/BerriAI/litellm/pull/15145)
|
||||
|
||||
---
|
||||
|
||||
## New Contributors
|
||||
|
||||
* @herve-ves made their first contribution in [PR #14998](https://github.com/BerriAI/litellm/pull/14998)
|
||||
* @wenxi-onyx made their first contribution in [PR #15008](https://github.com/BerriAI/litellm/pull/15008)
|
||||
* @jpetrucciani made their first contribution in [PR #15005](https://github.com/BerriAI/litellm/pull/15005)
|
||||
* @abhijitjavelin made their first contribution in [PR #14983](https://github.com/BerriAI/litellm/pull/14983)
|
||||
* @ZeroClover made their first contribution in [PR #15039](https://github.com/BerriAI/litellm/pull/15039)
|
||||
* @cedarm made their first contribution in [PR #15043](https://github.com/BerriAI/litellm/pull/15043)
|
||||
* @Isydmr made their first contribution in [PR #15025](https://github.com/BerriAI/litellm/pull/15025)
|
||||
* @serializer made their first contribution in [PR #15013](https://github.com/BerriAI/litellm/pull/15013)
|
||||
* @eddierichter-amd made their first contribution in [PR #14840](https://github.com/BerriAI/litellm/pull/14840)
|
||||
* @malags made their first contribution in [PR #15000](https://github.com/BerriAI/litellm/pull/15000)
|
||||
* @henryhwang made their first contribution in [PR #15029](https://github.com/BerriAI/litellm/pull/15029)
|
||||
* @plafleur made their first contribution in [PR #15111](https://github.com/BerriAI/litellm/pull/15111)
|
||||
* @tyler-liner made their first contribution in [PR #14799](https://github.com/BerriAI/litellm/pull/14799)
|
||||
* @Amir-R25 made their first contribution in [PR #15144](https://github.com/BerriAI/litellm/pull/15144)
|
||||
* @georg-wolflein made their first contribution in [PR #15124](https://github.com/BerriAI/litellm/pull/15124)
|
||||
* @niharm made their first contribution in [PR #15140](https://github.com/BerriAI/litellm/pull/15140)
|
||||
* @anthony-liner made their first contribution in [PR #15015](https://github.com/BerriAI/litellm/pull/15015)
|
||||
* @rishiganesh2002 made their first contribution in [PR #15153](https://github.com/BerriAI/litellm/pull/15153)
|
||||
* @danielaskdd made their first contribution in [PR #15160](https://github.com/BerriAI/litellm/pull/15160)
|
||||
* @JVenberg made their first contribution in [PR #15146](https://github.com/BerriAI/litellm/pull/15146)
|
||||
* @speglich made their first contribution in [PR #15072](https://github.com/BerriAI/litellm/pull/15072)
|
||||
* @daily-kim made their first contribution in [PR #14764](https://github.com/BerriAI/litellm/pull/14764)
|
||||
|
||||
---
|
||||
|
||||
## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.77.5.rc.4...v1.77.7.rc.1)**
|
||||
@@ -674,6 +674,7 @@ const sidebars = {
|
||||
items: [
|
||||
"data_security",
|
||||
"data_retention",
|
||||
"proxy/security_encryption_faq",
|
||||
"migration_policy",
|
||||
{
|
||||
type: "category",
|
||||
|
||||
@@ -119,6 +119,7 @@ class PagerDutyAlerting(SlackAlerting):
|
||||
user_api_key_end_user_id=_meta.get("user_api_key_end_user_id"),
|
||||
user_api_key_user_email=_meta.get("user_api_key_user_email"),
|
||||
user_api_key_request_route=_meta.get("user_api_key_request_route"),
|
||||
user_api_key_auth_metadata=_meta.get("user_api_key_auth_metadata"),
|
||||
)
|
||||
)
|
||||
|
||||
@@ -196,7 +197,11 @@ class PagerDutyAlerting(SlackAlerting):
|
||||
user_api_key_alias=user_api_key_dict.key_alias,
|
||||
user_api_key_spend=user_api_key_dict.spend,
|
||||
user_api_key_max_budget=user_api_key_dict.max_budget,
|
||||
user_api_key_budget_reset_at=user_api_key_dict.budget_reset_at.isoformat() if user_api_key_dict.budget_reset_at else None,
|
||||
user_api_key_budget_reset_at=(
|
||||
user_api_key_dict.budget_reset_at.isoformat()
|
||||
if user_api_key_dict.budget_reset_at
|
||||
else None
|
||||
),
|
||||
user_api_key_org_id=user_api_key_dict.org_id,
|
||||
user_api_key_team_id=user_api_key_dict.team_id,
|
||||
user_api_key_user_id=user_api_key_dict.user_id,
|
||||
@@ -204,6 +209,7 @@ class PagerDutyAlerting(SlackAlerting):
|
||||
user_api_key_end_user_id=user_api_key_dict.end_user_id,
|
||||
user_api_key_user_email=user_api_key_dict.user_email,
|
||||
user_api_key_request_route=user_api_key_dict.request_route,
|
||||
user_api_key_auth_metadata=user_api_key_dict.metadata,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@@ -21,6 +21,7 @@ from litellm._logging import print_verbose, verbose_logger
|
||||
from litellm.integrations.custom_logger import CustomLogger
|
||||
from litellm.proxy._types import LiteLLM_TeamTable, UserAPIKeyAuth
|
||||
from litellm.types.integrations.prometheus import *
|
||||
from litellm.types.integrations.prometheus import _sanitize_prometheus_label_name
|
||||
from litellm.types.utils import StandardLoggingPayload
|
||||
from litellm.utils import get_end_user_id_for_cost_tracking
|
||||
|
||||
@@ -794,9 +795,16 @@ class PrometheusLogger(CustomLogger):
|
||||
output_tokens = standard_logging_payload["completion_tokens"]
|
||||
tokens_used = standard_logging_payload["total_tokens"]
|
||||
response_cost = standard_logging_payload["response_cost"]
|
||||
_requester_metadata = standard_logging_payload["metadata"].get(
|
||||
_requester_metadata: Optional[dict] = standard_logging_payload["metadata"].get(
|
||||
"requester_metadata"
|
||||
)
|
||||
user_api_key_auth_metadata: Optional[dict] = standard_logging_payload[
|
||||
"metadata"
|
||||
].get("user_api_key_auth_metadata")
|
||||
combined_metadata: Dict[str, Any] = {
|
||||
**(_requester_metadata if _requester_metadata else {}),
|
||||
**(user_api_key_auth_metadata if user_api_key_auth_metadata else {}),
|
||||
}
|
||||
if standard_logging_payload is not None and isinstance(
|
||||
standard_logging_payload, dict
|
||||
):
|
||||
@@ -828,8 +836,7 @@ class PrometheusLogger(CustomLogger):
|
||||
exception_status=None,
|
||||
exception_class=None,
|
||||
custom_metadata_labels=get_custom_labels_from_metadata(
|
||||
metadata=standard_logging_payload["metadata"].get("requester_metadata")
|
||||
or {}
|
||||
metadata=combined_metadata
|
||||
),
|
||||
route=standard_logging_payload["metadata"].get(
|
||||
"user_api_key_request_route"
|
||||
@@ -1649,9 +1656,22 @@ class PrometheusLogger(CustomLogger):
|
||||
api_base: Optional[str],
|
||||
api_provider: str,
|
||||
):
|
||||
self.litellm_deployment_state.labels(
|
||||
litellm_model_name, model_id, api_base, api_provider
|
||||
).set(state)
|
||||
"""
|
||||
Set the deployment state.
|
||||
"""
|
||||
### get labels
|
||||
_labels = prometheus_label_factory(
|
||||
supported_enum_labels=self.get_labels_for_metric(
|
||||
metric_name="litellm_deployment_state"
|
||||
),
|
||||
enum_values=UserAPIKeyLabelValues(
|
||||
litellm_model_name=litellm_model_name,
|
||||
model_id=model_id,
|
||||
api_base=api_base,
|
||||
api_provider=api_provider,
|
||||
),
|
||||
)
|
||||
self.litellm_deployment_state.labels(**_labels).set(state)
|
||||
|
||||
def set_deployment_healthy(
|
||||
self,
|
||||
@@ -2228,8 +2248,10 @@ def prometheus_label_factory(
|
||||
|
||||
if enum_values.custom_metadata_labels is not None:
|
||||
for key, value in enum_values.custom_metadata_labels.items():
|
||||
if key in supported_enum_labels:
|
||||
filtered_labels[key] = value
|
||||
# check sanitized key
|
||||
sanitized_key = _sanitize_prometheus_label_name(key)
|
||||
if sanitized_key in supported_enum_labels:
|
||||
filtered_labels[sanitized_key] = value
|
||||
|
||||
# Add custom tags if configured
|
||||
if enum_values.tags is not None:
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+3
@@ -0,0 +1,3 @@
|
||||
-- AlterTable
|
||||
ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "allowed_tools" TEXT[] DEFAULT ARRAY[]::TEXT[];
|
||||
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
-- AlterTable
|
||||
ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "extra_headers" TEXT[] DEFAULT ARRAY[]::TEXT[];
|
||||
|
||||
@@ -178,6 +178,8 @@ model LiteLLM_MCPServerTable {
|
||||
updated_by String?
|
||||
mcp_info Json? @default("{}")
|
||||
mcp_access_groups String[]
|
||||
allowed_tools String[] @default([])
|
||||
extra_headers String[] @default([])
|
||||
// Health check status
|
||||
status String? @default("unknown")
|
||||
last_health_check DateTime?
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "litellm-proxy-extras"
|
||||
version = "0.2.22"
|
||||
version = "0.2.25"
|
||||
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
|
||||
authors = ["BerriAI"]
|
||||
readme = "README.md"
|
||||
@@ -22,7 +22,7 @@ requires = ["poetry-core"]
|
||||
build-backend = "poetry.core.masonry.api"
|
||||
|
||||
[tool.commitizen]
|
||||
version = "0.2.22"
|
||||
version = "0.2.25"
|
||||
version_files = [
|
||||
"pyproject.toml:version",
|
||||
"../requirements.txt:litellm-proxy-extras==",
|
||||
|
||||
+1
-1
@@ -290,7 +290,7 @@ banned_keywords_list: Optional[Union[str, List]] = None
|
||||
llm_guard_mode: Literal["all", "key-specific", "request-specific"] = "all"
|
||||
guardrail_name_config_map: Dict[str, GuardrailItem] = {}
|
||||
include_cost_in_streaming_usage: bool = False
|
||||
### PROMPTS ###
|
||||
### PROMPTS ####
|
||||
from litellm.types.prompts.init_prompts import PromptSpec
|
||||
|
||||
prompt_name_config_map: Dict[str, PromptSpec] = {}
|
||||
|
||||
@@ -4040,6 +4040,7 @@ class StandardLoggingPayloadSetup:
|
||||
usage_object=usage_object,
|
||||
requester_custom_headers=None,
|
||||
cold_storage_object_key=None,
|
||||
user_api_key_auth_metadata=None,
|
||||
)
|
||||
if isinstance(metadata, dict):
|
||||
# Filter the metadata dictionary to include only the specified keys
|
||||
@@ -4755,6 +4756,7 @@ def get_standard_logging_metadata(
|
||||
requester_custom_headers=None,
|
||||
user_api_key_request_route=None,
|
||||
cold_storage_object_key=None,
|
||||
user_api_key_auth_metadata=None,
|
||||
)
|
||||
if isinstance(metadata, dict):
|
||||
# Update the clean_metadata with values from input metadata that match StandardLoggingMetadata fields
|
||||
|
||||
@@ -1,14 +1,15 @@
|
||||
"""
|
||||
Support for Snowflake REST API
|
||||
Support for Snowflake REST API
|
||||
"""
|
||||
|
||||
from typing import TYPE_CHECKING, Any, List, Optional, Tuple
|
||||
import json
|
||||
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
|
||||
|
||||
import httpx
|
||||
|
||||
from litellm.secret_managers.main import get_secret_str
|
||||
from litellm.types.llms.openai import AllMessageValues
|
||||
from litellm.types.utils import ModelResponse
|
||||
from litellm.types.utils import ChatCompletionMessageToolCall, Function, ModelResponse
|
||||
|
||||
from ...openai_like.chat.transformation import OpenAIGPTConfig
|
||||
|
||||
@@ -22,15 +23,25 @@ else:
|
||||
|
||||
class SnowflakeConfig(OpenAIGPTConfig):
|
||||
"""
|
||||
source: https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex
|
||||
Reference: https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-llm-rest-api
|
||||
|
||||
Snowflake Cortex LLM REST API supports function calling with specific models (e.g., Claude 3.5 Sonnet).
|
||||
This config handles transformation between OpenAI format and Snowflake's tool_spec format.
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def get_config(cls):
|
||||
return super().get_config()
|
||||
|
||||
def get_supported_openai_params(self, model: str) -> List:
|
||||
return ["temperature", "max_tokens", "top_p", "response_format"]
|
||||
def get_supported_openai_params(self, model: str) -> List[str]:
|
||||
return [
|
||||
"temperature",
|
||||
"max_tokens",
|
||||
"top_p",
|
||||
"response_format",
|
||||
"tools",
|
||||
"tool_choice",
|
||||
]
|
||||
|
||||
def map_openai_params(
|
||||
self,
|
||||
@@ -56,6 +67,57 @@ class SnowflakeConfig(OpenAIGPTConfig):
|
||||
optional_params[param] = value
|
||||
return optional_params
|
||||
|
||||
def _transform_tool_calls_from_snowflake_to_openai(
|
||||
self, content_list: List[Dict[str, Any]]
|
||||
) -> Tuple[str, Optional[List[ChatCompletionMessageToolCall]]]:
|
||||
"""
|
||||
Transform Snowflake tool calls to OpenAI format.
|
||||
|
||||
Args:
|
||||
content_list: Snowflake's content_list array containing text and tool_use items
|
||||
|
||||
Returns:
|
||||
Tuple of (text_content, tool_calls)
|
||||
|
||||
Snowflake format in content_list:
|
||||
{
|
||||
"type": "tool_use",
|
||||
"tool_use": {
|
||||
"tool_use_id": "tooluse_...",
|
||||
"name": "get_weather",
|
||||
"input": {"location": "Paris"}
|
||||
}
|
||||
}
|
||||
|
||||
OpenAI format (returned tool_calls):
|
||||
ChatCompletionMessageToolCall(
|
||||
id="tooluse_...",
|
||||
type="function",
|
||||
function=Function(name="get_weather", arguments='{"location": "Paris"}')
|
||||
)
|
||||
"""
|
||||
text_content = ""
|
||||
tool_calls: List[ChatCompletionMessageToolCall] = []
|
||||
|
||||
for idx, content_item in enumerate(content_list):
|
||||
if content_item.get("type") == "text":
|
||||
text_content += content_item.get("text", "")
|
||||
|
||||
## TOOL CALLING
|
||||
elif content_item.get("type") == "tool_use":
|
||||
tool_use_data = content_item.get("tool_use", {})
|
||||
tool_call = ChatCompletionMessageToolCall(
|
||||
id=tool_use_data.get("tool_use_id", ""),
|
||||
type="function",
|
||||
function=Function(
|
||||
name=tool_use_data.get("name", ""),
|
||||
arguments=json.dumps(tool_use_data.get("input", {})),
|
||||
),
|
||||
)
|
||||
tool_calls.append(tool_call)
|
||||
|
||||
return text_content, tool_calls if tool_calls else None
|
||||
|
||||
def transform_response(
|
||||
self,
|
||||
model: str,
|
||||
@@ -71,6 +133,7 @@ class SnowflakeConfig(OpenAIGPTConfig):
|
||||
json_mode: Optional[bool] = None,
|
||||
) -> ModelResponse:
|
||||
response_json = raw_response.json()
|
||||
|
||||
logging_obj.post_call(
|
||||
input=messages,
|
||||
api_key="",
|
||||
@@ -78,6 +141,26 @@ class SnowflakeConfig(OpenAIGPTConfig):
|
||||
additional_args={"complete_input_dict": request_data},
|
||||
)
|
||||
|
||||
## RESPONSE TRANSFORMATION
|
||||
# Snowflake returns content_list (not content) with tool_use objects
|
||||
# We need to transform this to OpenAI's format with content + tool_calls
|
||||
if "choices" in response_json and len(response_json["choices"]) > 0:
|
||||
choice = response_json["choices"][0]
|
||||
if "message" in choice and "content_list" in choice["message"]:
|
||||
content_list = choice["message"]["content_list"]
|
||||
(
|
||||
text_content,
|
||||
tool_calls,
|
||||
) = self._transform_tool_calls_from_snowflake_to_openai(content_list)
|
||||
|
||||
# Update the choice message with OpenAI format
|
||||
choice["message"]["content"] = text_content
|
||||
if tool_calls:
|
||||
choice["message"]["tool_calls"] = tool_calls
|
||||
|
||||
# Remove Snowflake-specific content_list
|
||||
del choice["message"]["content_list"]
|
||||
|
||||
returned_response = ModelResponse(**response_json)
|
||||
|
||||
returned_response.model = "snowflake/" + (returned_response.model or "")
|
||||
@@ -150,6 +233,95 @@ class SnowflakeConfig(OpenAIGPTConfig):
|
||||
|
||||
return api_base
|
||||
|
||||
def _transform_tools(self, tools: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Transform OpenAI tool format to Snowflake tool format.
|
||||
|
||||
Args:
|
||||
tools: List of tools in OpenAI format
|
||||
|
||||
Returns:
|
||||
List of tools in Snowflake format
|
||||
|
||||
OpenAI format:
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "...",
|
||||
"parameters": {...}
|
||||
}
|
||||
}
|
||||
|
||||
Snowflake format:
|
||||
{
|
||||
"tool_spec": {
|
||||
"type": "generic",
|
||||
"name": "get_weather",
|
||||
"description": "...",
|
||||
"input_schema": {...}
|
||||
}
|
||||
}
|
||||
"""
|
||||
snowflake_tools: List[Dict[str, Any]] = []
|
||||
for tool in tools:
|
||||
if tool.get("type") == "function":
|
||||
function = tool.get("function", {})
|
||||
snowflake_tool: Dict[str, Any] = {
|
||||
"tool_spec": {
|
||||
"type": "generic",
|
||||
"name": function.get("name"),
|
||||
"input_schema": function.get(
|
||||
"parameters",
|
||||
{"type": "object", "properties": {}},
|
||||
),
|
||||
}
|
||||
}
|
||||
# Add description if present
|
||||
if "description" in function:
|
||||
snowflake_tool["tool_spec"]["description"] = function[
|
||||
"description"
|
||||
]
|
||||
|
||||
snowflake_tools.append(snowflake_tool)
|
||||
|
||||
return snowflake_tools
|
||||
|
||||
def _transform_tool_choice(
|
||||
self, tool_choice: Union[str, Dict[str, Any]]
|
||||
) -> Union[str, Dict[str, Any]]:
|
||||
"""
|
||||
Transform OpenAI tool_choice format to Snowflake format.
|
||||
|
||||
Args:
|
||||
tool_choice: Tool choice in OpenAI format (str or dict)
|
||||
|
||||
Returns:
|
||||
Tool choice in Snowflake format
|
||||
|
||||
OpenAI format:
|
||||
{"type": "function", "function": {"name": "get_weather"}}
|
||||
|
||||
Snowflake format:
|
||||
{"type": "tool", "name": ["get_weather"]}
|
||||
|
||||
Note: String values ("auto", "required", "none") pass through unchanged.
|
||||
"""
|
||||
if isinstance(tool_choice, str):
|
||||
# "auto", "required", "none" pass through as-is
|
||||
return tool_choice
|
||||
|
||||
if isinstance(tool_choice, dict):
|
||||
if tool_choice.get("type") == "function":
|
||||
function_name = tool_choice.get("function", {}).get("name")
|
||||
if function_name:
|
||||
return {
|
||||
"type": "tool",
|
||||
"name": [function_name], # Snowflake expects array
|
||||
}
|
||||
|
||||
return tool_choice
|
||||
|
||||
def transform_request(
|
||||
self,
|
||||
model: str,
|
||||
@@ -160,6 +332,18 @@ class SnowflakeConfig(OpenAIGPTConfig):
|
||||
) -> dict:
|
||||
stream: bool = optional_params.pop("stream", None) or False
|
||||
extra_body = optional_params.pop("extra_body", {})
|
||||
|
||||
## TOOL CALLING
|
||||
# Transform tools from OpenAI format to Snowflake's tool_spec format
|
||||
tools = optional_params.pop("tools", None)
|
||||
if tools:
|
||||
optional_params["tools"] = self._transform_tools(tools)
|
||||
|
||||
# Transform tool_choice from OpenAI format to Snowflake's tool name array format
|
||||
tool_choice = optional_params.pop("tool_choice", None)
|
||||
if tool_choice:
|
||||
optional_params["tool_choice"] = self._transform_tool_choice(tool_choice)
|
||||
|
||||
return {
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
|
||||
@@ -415,8 +415,8 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
|
||||
googleSearchRetrieval = self.get_tool_value(tool, VertexToolName.GOOGLE_SEARCH_RETRIEVAL.value)
|
||||
elif tool_name and tool_name == VertexToolName.ENTERPRISE_WEB_SEARCH.value:
|
||||
enterpriseWebSearch = self.get_tool_value(tool, VertexToolName.ENTERPRISE_WEB_SEARCH.value)
|
||||
elif tool_name and tool_name == VertexToolName.URL_CONTEXT.value:
|
||||
urlContext = self.get_tool_value(tool, VertexToolName.URL_CONTEXT.value)
|
||||
elif tool_name and (tool_name == VertexToolName.URL_CONTEXT.value or tool_name == "urlContext"):
|
||||
urlContext = self.get_tool_value(tool, tool_name)
|
||||
elif tool_name and (
|
||||
tool_name == VertexToolName.GOOGLE_MAPS.value or tool_name == "google_maps"
|
||||
):
|
||||
@@ -448,9 +448,10 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
|
||||
"Invalid tool={}. Use `litellm.set_verbose` or `litellm --detailed_debug` to see raw request."
|
||||
)
|
||||
|
||||
_tools = Tools(
|
||||
function_declarations=gtool_func_declarations,
|
||||
)
|
||||
# Only include function_declarations if there are actual functions
|
||||
_tools = Tools()
|
||||
if gtool_func_declarations:
|
||||
_tools["function_declarations"] = gtool_func_declarations
|
||||
if googleSearch is not None:
|
||||
_tools[VertexToolName.GOOGLE_SEARCH.value] = googleSearch
|
||||
if googleSearchRetrieval is not None:
|
||||
|
||||
+2
-2
@@ -2872,7 +2872,7 @@ def completion( # type: ignore # noqa: PLR0915
|
||||
custom_llm_provider=custom_llm_provider, # type: ignore
|
||||
client=client,
|
||||
api_base=api_base,
|
||||
extra_headers=extra_headers,
|
||||
extra_headers=headers,
|
||||
)
|
||||
|
||||
elif custom_llm_provider == "vertex_ai":
|
||||
@@ -2941,7 +2941,7 @@ def completion( # type: ignore # noqa: PLR0915
|
||||
custom_llm_provider=custom_llm_provider, # type: ignore
|
||||
client=client,
|
||||
api_base=api_base,
|
||||
extra_headers=extra_headers,
|
||||
extra_headers=headers,
|
||||
)
|
||||
elif "openai" in model:
|
||||
# Vertex Model Garden - OpenAI compatible models
|
||||
|
||||
@@ -3324,28 +3324,27 @@
|
||||
"supports_web_search": true
|
||||
},
|
||||
"azure_ai/grok-4-fast-non-reasoning": {
|
||||
"input_cost_per_token": 5e-06,
|
||||
"input_cost_per_token": 0.43e-06,
|
||||
"output_cost_per_token": 1.73e-06,
|
||||
"litellm_provider": "azure_ai",
|
||||
"max_input_tokens": 131072,
|
||||
"max_output_tokens": 131072,
|
||||
"max_tokens": 131072,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 2.5e-03,
|
||||
"source": "https://azure.microsoft.com/en-us/blog/grok-4-is-now-available-in-azure-ai-foundry-unlock-frontier-intelligence-and-business-ready-capabilities/",
|
||||
"supports_function_calling": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_web_search": true
|
||||
},
|
||||
"azure_ai/grok-4-fast-reasoning": {
|
||||
"input_cost_per_token": 5.8e-06,
|
||||
"input_cost_per_token": 0.43e-06,
|
||||
"output_cost_per_token": 1.73e-06,
|
||||
"litellm_provider": "azure_ai",
|
||||
"max_input_tokens": 131072,
|
||||
"max_output_tokens": 131072,
|
||||
"max_tokens": 131072,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 2.9e-03,
|
||||
"source": "https://azure.microsoft.com/en-us/blog/grok-4-is-now-available-in-azure-ai-foundry-unlock-frontier-intelligence-and-business-ready-capabilities/",
|
||||
"source": "https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/announcing-the-grok-4-fast-models-from-xai-now-available-in-azure-ai-foundry/4456701",
|
||||
"supports_function_calling": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
@@ -22174,6 +22173,307 @@
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/bigscience/mt0-xxl-13b": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.0005,
|
||||
"output_cost_per_token": 0.002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/core42/jais-13b-chat": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.0005,
|
||||
"output_cost_per_token": 0.002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/google/flan-t5-xl-3b": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.0001,
|
||||
"output_cost_per_token": 0.00025,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-13b-chat-v2": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.0005,
|
||||
"output_cost_per_token": 0.002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-13b-instruct-v2": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.0005,
|
||||
"output_cost_per_token": 0.002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-3-3-8b-instruct": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.00025,
|
||||
"output_cost_per_token": 0.001,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-4-h-small": {
|
||||
"max_tokens": 20480,
|
||||
"max_input_tokens": 20480,
|
||||
"max_output_tokens": 20480,
|
||||
"input_cost_per_token": 0.000625,
|
||||
"output_cost_per_token": 0.0025,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-guardian-3-2-2b": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.00015,
|
||||
"output_cost_per_token": 0.0006,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-guardian-3-3-8b": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.00025,
|
||||
"output_cost_per_token": 0.001,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-ttm-1024-96-r2": {
|
||||
"max_tokens": 512,
|
||||
"max_input_tokens": 512,
|
||||
"max_output_tokens": 512,
|
||||
"input_cost_per_token": 0.000625,
|
||||
"output_cost_per_token": 0.000625,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-ttm-1536-96-r2": {
|
||||
"max_tokens": 512,
|
||||
"max_input_tokens": 512,
|
||||
"max_output_tokens": 512,
|
||||
"input_cost_per_token": 0.000625,
|
||||
"output_cost_per_token": 0.000625,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-ttm-512-96-r2": {
|
||||
"max_tokens": 512,
|
||||
"max_input_tokens": 512,
|
||||
"max_output_tokens": 512,
|
||||
"input_cost_per_token": 0.000625,
|
||||
"output_cost_per_token": 0.000625,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-vision-3-2-2b": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.00015,
|
||||
"output_cost_per_token": 0.0006,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": true
|
||||
},
|
||||
"watsonx/meta-llama/llama-3-2-11b-vision-instruct": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.00025,
|
||||
"output_cost_per_token": 0.001,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"watsonx/meta-llama/llama-3-2-1b-instruct": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.0001,
|
||||
"output_cost_per_token": 0.0002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/meta-llama/llama-3-2-3b-instruct": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.00015,
|
||||
"output_cost_per_token": 0.0006,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/meta-llama/llama-3-2-90b-vision-instruct": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.002,
|
||||
"output_cost_per_token": 0.008,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"watsonx/meta-llama/llama-3-3-70b-instruct": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.002,
|
||||
"output_cost_per_token": 0.006,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/meta-llama/llama-4-maverick-17b": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.0005,
|
||||
"output_cost_per_token": 0.002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/meta-llama/llama-guard-3-11b-vision": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.00025,
|
||||
"output_cost_per_token": 0.001,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": true
|
||||
},
|
||||
"watsonx/mistralai/mistral-medium-2505": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.00225,
|
||||
"output_cost_per_token": 0.00675,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/mistralai/mistral-small-2503": {
|
||||
"max_tokens": 32000,
|
||||
"max_input_tokens": 32000,
|
||||
"max_output_tokens": 32000,
|
||||
"input_cost_per_token": 0.0002,
|
||||
"output_cost_per_token": 0.0006,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/mistralai/pixtral-12b-2409": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.00015,
|
||||
"output_cost_per_token": 0.00015,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": true
|
||||
},
|
||||
"watsonx/openai/gpt-oss-120b": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.004,
|
||||
"output_cost_per_token": 0.016,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/sdaia/allam-1-13b-instruct": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.0005,
|
||||
"output_cost_per_token": 0.002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
|
||||
"whisper-1": {
|
||||
"input_cost_per_second": 0.0001,
|
||||
"litellm_provider": "openai",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
from litellm._uuid import uuid
|
||||
from typing import Any, Dict, Iterable, List, Optional, Set, Union
|
||||
|
||||
from litellm._logging import verbose_proxy_logger
|
||||
from litellm._uuid import uuid
|
||||
from litellm.proxy._types import (
|
||||
LiteLLM_MCPServerTable,
|
||||
LiteLLM_ObjectPermissionTable,
|
||||
@@ -30,7 +30,7 @@ def _prepare_mcp_server_data(
|
||||
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
|
||||
|
||||
# Convert model to dict
|
||||
data_dict = data.model_dump()
|
||||
data_dict = data.model_dump(exclude_none=True)
|
||||
# Ensure alias is always present in the dict (even if None)
|
||||
if "alias" not in data_dict:
|
||||
data_dict["alias"] = getattr(data, "alias", None)
|
||||
|
||||
@@ -10,7 +10,7 @@ import asyncio
|
||||
import datetime
|
||||
import hashlib
|
||||
import json
|
||||
from typing import Any, Dict, List, Optional, Union, cast
|
||||
from typing import Any, Dict, List, Optional, Set, Union, cast
|
||||
|
||||
from fastapi import HTTPException
|
||||
from mcp.types import CallToolRequestParams as MCPCallToolRequestParams
|
||||
@@ -240,50 +240,64 @@ class MCPServerManager:
|
||||
)
|
||||
|
||||
def add_update_server(self, mcp_server: LiteLLM_MCPServerTable):
|
||||
if mcp_server.server_id not in self.get_registry():
|
||||
_mcp_info: MCPInfo = mcp_server.mcp_info or {}
|
||||
# Use helper to deserialize environment dictionary
|
||||
# Safely access env field which may not exist on Prisma model objects
|
||||
env_data = getattr(mcp_server, "env", None)
|
||||
env_dict = _deserialize_env_dict(env_data)
|
||||
# Use alias for name if present, else server_name
|
||||
name_for_prefix = (
|
||||
mcp_server.alias or mcp_server.server_name or mcp_server.server_id
|
||||
)
|
||||
# Preserve all custom fields from database while setting defaults for core fields
|
||||
mcp_info: MCPInfo = _mcp_info.copy()
|
||||
# Set default values for core fields if not present
|
||||
if "server_name" not in mcp_info:
|
||||
mcp_info["server_name"] = mcp_server.server_name or mcp_server.server_id
|
||||
if "description" not in mcp_info and mcp_server.description:
|
||||
mcp_info["description"] = mcp_server.description
|
||||
try:
|
||||
if mcp_server.server_id not in self.get_registry():
|
||||
_mcp_info: MCPInfo = mcp_server.mcp_info or {}
|
||||
# Use helper to deserialize environment dictionary
|
||||
# Safely access env field which may not exist on Prisma model objects
|
||||
env_data = getattr(mcp_server, "env", None)
|
||||
env_dict = _deserialize_env_dict(env_data)
|
||||
# Use alias for name if present, else server_name
|
||||
name_for_prefix = (
|
||||
mcp_server.alias or mcp_server.server_name or mcp_server.server_id
|
||||
)
|
||||
# Preserve all custom fields from database while setting defaults for core fields
|
||||
mcp_info: MCPInfo = _mcp_info.copy()
|
||||
# Set default values for core fields if not present
|
||||
if "server_name" not in mcp_info:
|
||||
mcp_info["server_name"] = (
|
||||
mcp_server.server_name or mcp_server.server_id
|
||||
)
|
||||
if "description" not in mcp_info and mcp_server.description:
|
||||
mcp_info["description"] = mcp_server.description
|
||||
|
||||
new_server = MCPServer(
|
||||
server_id=mcp_server.server_id,
|
||||
name=name_for_prefix,
|
||||
alias=getattr(mcp_server, "alias", None),
|
||||
server_name=getattr(mcp_server, "server_name", None),
|
||||
url=mcp_server.url,
|
||||
transport=cast(MCPTransportType, mcp_server.transport),
|
||||
auth_type=cast(MCPAuthType, mcp_server.auth_type),
|
||||
mcp_info=mcp_info,
|
||||
extra_headers=getattr(mcp_server, "extra_headers", None),
|
||||
# oauth specific fields
|
||||
client_id=getattr(mcp_server, "client_id", None),
|
||||
client_secret=getattr(mcp_server, "client_secret", None),
|
||||
scopes=getattr(mcp_server, "scopes", None),
|
||||
authorization_url=getattr(mcp_server, "authorization_url", None),
|
||||
token_url=getattr(mcp_server, "token_url", None),
|
||||
# Stdio-specific fields
|
||||
command=getattr(mcp_server, "command", None),
|
||||
args=getattr(mcp_server, "args", None) or [],
|
||||
env=env_dict,
|
||||
access_groups=getattr(mcp_server, "mcp_access_groups", None),
|
||||
allowed_tools=getattr(mcp_server, "allowed_tools", None),
|
||||
disallowed_tools=getattr(mcp_server, "disallowed_tools", None),
|
||||
)
|
||||
self.registry[mcp_server.server_id] = new_server
|
||||
verbose_logger.debug(f"Added MCP Server: {name_for_prefix}")
|
||||
new_server = MCPServer(
|
||||
server_id=mcp_server.server_id,
|
||||
name=name_for_prefix,
|
||||
alias=getattr(mcp_server, "alias", None),
|
||||
server_name=getattr(mcp_server, "server_name", None),
|
||||
url=mcp_server.url,
|
||||
transport=cast(MCPTransportType, mcp_server.transport),
|
||||
auth_type=cast(MCPAuthType, mcp_server.auth_type),
|
||||
mcp_info=mcp_info,
|
||||
extra_headers=getattr(mcp_server, "extra_headers", None),
|
||||
# oauth specific fields
|
||||
client_id=getattr(mcp_server, "client_id", None),
|
||||
client_secret=getattr(mcp_server, "client_secret", None),
|
||||
scopes=getattr(mcp_server, "scopes", None),
|
||||
authorization_url=getattr(mcp_server, "authorization_url", None),
|
||||
token_url=getattr(mcp_server, "token_url", None),
|
||||
# Stdio-specific fields
|
||||
command=getattr(mcp_server, "command", None),
|
||||
args=getattr(mcp_server, "args", None) or [],
|
||||
env=env_dict,
|
||||
access_groups=getattr(mcp_server, "mcp_access_groups", None),
|
||||
allowed_tools=getattr(mcp_server, "allowed_tools", None),
|
||||
disallowed_tools=getattr(mcp_server, "disallowed_tools", None),
|
||||
)
|
||||
self.registry[mcp_server.server_id] = new_server
|
||||
verbose_logger.debug(f"Added MCP Server: {name_for_prefix}")
|
||||
|
||||
except Exception as e:
|
||||
verbose_logger.debug(f"Failed to add MCP server: {str(e)}")
|
||||
raise e
|
||||
|
||||
def get_all_mcp_server_ids(self) -> Set[str]:
|
||||
"""
|
||||
Get all MCP server IDs
|
||||
"""
|
||||
all_servers = list(self.get_registry().values())
|
||||
return {server.server_id for server in all_servers}
|
||||
|
||||
async def get_allowed_mcp_servers(
|
||||
self, user_api_key_auth: Optional[UserAPIKeyAuth] = None
|
||||
@@ -1118,25 +1132,23 @@ class MCPServerManager:
|
||||
if _server_id in allowed_server_ids:
|
||||
list_mcp_servers.append(
|
||||
LiteLLM_MCPServerTable(
|
||||
server_id=_server_id,
|
||||
server_name=_server_config.name,
|
||||
alias=_server_config.alias,
|
||||
url=_server_config.url,
|
||||
transport=_server_config.transport,
|
||||
auth_type=_server_config.auth_type,
|
||||
created_at=datetime.datetime.now(),
|
||||
updated_at=datetime.datetime.now(),
|
||||
description=(
|
||||
_server_config.mcp_info.get("description")
|
||||
if _server_config.mcp_info
|
||||
else None
|
||||
),
|
||||
mcp_info=_server_config.mcp_info,
|
||||
mcp_access_groups=_server_config.access_groups or [],
|
||||
# Stdio-specific fields
|
||||
command=getattr(_server_config, "command", None),
|
||||
args=getattr(_server_config, "args", None) or [],
|
||||
env=getattr(_server_config, "env", None) or {},
|
||||
**{
|
||||
**_server_config.model_dump(),
|
||||
"created_at": datetime.datetime.now(),
|
||||
"updated_at": datetime.datetime.now(),
|
||||
"description": (
|
||||
_server_config.mcp_info.get("description")
|
||||
if _server_config.mcp_info
|
||||
else None
|
||||
),
|
||||
"allowed_tools": _server_config.allowed_tools or [],
|
||||
"mcp_info": _server_config.mcp_info,
|
||||
"mcp_access_groups": _server_config.access_groups or [],
|
||||
"extra_headers": _server_config.extra_headers or [],
|
||||
"command": getattr(_server_config, "command", None),
|
||||
"args": getattr(_server_config, "args", None) or [],
|
||||
"env": getattr(_server_config, "env", None) or {},
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
@@ -1176,44 +1188,19 @@ class MCPServerManager:
|
||||
}
|
||||
)
|
||||
|
||||
# Map servers to their teams and return with health data
|
||||
from typing import cast
|
||||
## mark invalid servers w/ reason for being invalid
|
||||
valid_server_ids = self.get_all_mcp_server_ids()
|
||||
for server in list_mcp_servers:
|
||||
if server.server_id not in valid_server_ids:
|
||||
server.status = "unhealthy"
|
||||
## try adding server to registry to get error
|
||||
try:
|
||||
self.add_update_server(server)
|
||||
except Exception as e:
|
||||
server.health_check_error = str(e)
|
||||
server.health_check_error = "Server is not in in memory registry yet. This could be a temporary sync issue."
|
||||
|
||||
return [
|
||||
LiteLLM_MCPServerTable(
|
||||
server_id=server.server_id,
|
||||
server_name=server.server_name,
|
||||
alias=server.alias,
|
||||
description=server.description,
|
||||
url=server.url,
|
||||
transport=server.transport,
|
||||
auth_type=server.auth_type,
|
||||
created_at=server.created_at,
|
||||
created_by=server.created_by,
|
||||
updated_at=server.updated_at,
|
||||
updated_by=server.updated_by,
|
||||
mcp_access_groups=(
|
||||
server.mcp_access_groups
|
||||
if server.mcp_access_groups is not None
|
||||
else []
|
||||
),
|
||||
allowed_tools=(
|
||||
server.allowed_tools
|
||||
if server.allowed_tools is not None
|
||||
else []
|
||||
),
|
||||
mcp_info=server.mcp_info,
|
||||
teams=cast(
|
||||
List[Dict[str, str | None]],
|
||||
server_to_teams_map.get(server.server_id, []),
|
||||
),
|
||||
# Stdio-specific fields
|
||||
command=getattr(server, "command", None),
|
||||
args=getattr(server, "args", None) or [],
|
||||
env=getattr(server, "env", None) or {},
|
||||
)
|
||||
for server in list_mcp_servers
|
||||
]
|
||||
return list_mcp_servers
|
||||
|
||||
async def reload_servers_from_database(self):
|
||||
"""
|
||||
|
||||
File diff suppressed because one or more lines are too long
+1
-1
File diff suppressed because one or more lines are too long
+2
-2
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+1
-1
File diff suppressed because one or more lines are too long
+1
-1
@@ -1 +1 @@
|
||||
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[185],{77401:function(n,e,t){Promise.resolve().then(t.t.bind(t,39974,23)),Promise.resolve().then(t.t.bind(t,2778,23))},2778:function(){},39974:function(n){n.exports={style:{fontFamily:"'__Inter_1c856b', '__Inter_Fallback_1c856b'",fontStyle:"normal"},className:"__className_1c856b"}}},function(n){n.O(0,[919,986,971,117,744],function(){return n(n.s=77401)}),_N_E=n.O()}]);
|
||||
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[185],{85210:function(n,e,t){Promise.resolve().then(t.t.bind(t,39974,23)),Promise.resolve().then(t.t.bind(t,2778,23))},2778:function(){},39974:function(n){n.exports={style:{fontFamily:"'__Inter_1c856b', '__Inter_Fallback_1c856b'",fontStyle:"normal"},className:"__className_1c856b"}}},function(n){n.O(0,[919,986,971,117,744],function(){return n(n.s=85210)}),_N_E=n.O()}]);
|
||||
+1
-1
@@ -1 +1 @@
|
||||
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[418],{96422:function(e,n,t){Promise.resolve().then(t.bind(t,52829))},52829:function(e,n,t){"use strict";t.r(n),t.d(n,{default:function(){return f}});var u=t(57437),s=t(2265),c=t(99376),r=t(72162);function f(){let e=(0,c.useSearchParams)().get("key"),[n,t]=(0,s.useState)(null);return(0,s.useEffect)(()=>{e&&t(e)},[e]),(0,u.jsx)(r.Z,{accessToken:n})}}},function(e){e.O(0,[50,521,154,162,971,117,744],function(){return e(e.s=96422)}),_N_E=e.O()}]);
|
||||
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[418],{67355:function(e,n,t){Promise.resolve().then(t.bind(t,52829))},52829:function(e,n,t){"use strict";t.r(n),t.d(n,{default:function(){return f}});var u=t(57437),s=t(2265),c=t(99376),r=t(72162);function f(){let e=(0,c.useSearchParams)().get("key"),[n,t]=(0,s.useState)(null);return(0,s.useEffect)(()=>{e&&t(e)},[e]),(0,u.jsx)(r.Z,{accessToken:n})}}},function(e){e.O(0,[50,521,49,162,971,117,744],function(){return e(e.s=67355)}),_N_E=e.O()}]);
|
||||
+1
@@ -0,0 +1 @@
|
||||
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[25],{38520:function(e,n,u){Promise.resolve().then(u.bind(u,22775))},22775:function(e,n,u){"use strict";u.r(n),u.d(n,{default:function(){return f}});var t=u(57437),s=u(2265),r=u(99376),c=u(97851);function f(){let e=(0,r.useSearchParams)().get("key"),[n,u]=(0,s.useState)(null);return(0,s.useEffect)(()=>{e&&u(e)},[e]),(0,t.jsx)(c.Z,{accessToken:n,publicPage:!0,premiumUser:!1,userRole:null})}}},function(e){e.O(0,[50,521,866,49,162,851,971,117,744],function(){return e(e.s=38520)}),_N_E=e.O()}]);
|
||||
-1
@@ -1 +0,0 @@
|
||||
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[25],{9397:function(e,n,u){Promise.resolve().then(u.bind(u,22775))},22775:function(e,n,u){"use strict";u.r(n),u.d(n,{default:function(){return f}});var t=u(57437),s=u(2265),r=u(99376),c=u(97851);function f(){let e=(0,r.useSearchParams)().get("key"),[n,u]=(0,s.useState)(null);return(0,s.useEffect)(()=>{e&&u(e)},[e]),(0,t.jsx)(c.Z,{accessToken:n,publicPage:!0,premiumUser:!1,userRole:null})}}},function(e){e.O(0,[50,521,866,154,162,851,971,117,744],function(){return e(e.s=9397)}),_N_E=e.O()}]);
|
||||
-1
File diff suppressed because one or more lines are too long
+1
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+1
-1
@@ -1 +1 @@
|
||||
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[744],{60400:function(e,n,t){Promise.resolve().then(t.t.bind(t,12846,23)),Promise.resolve().then(t.t.bind(t,19107,23)),Promise.resolve().then(t.t.bind(t,61060,23)),Promise.resolve().then(t.t.bind(t,4707,23)),Promise.resolve().then(t.t.bind(t,80,23)),Promise.resolve().then(t.t.bind(t,36423,23))}},function(e){var n=function(n){return e(e.s=n)};e.O(0,[971,117],function(){return n(54278),n(60400)}),_N_E=e.O()}]);
|
||||
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[744],{78483:function(e,n,t){Promise.resolve().then(t.t.bind(t,12846,23)),Promise.resolve().then(t.t.bind(t,19107,23)),Promise.resolve().then(t.t.bind(t,61060,23)),Promise.resolve().then(t.t.bind(t,4707,23)),Promise.resolve().then(t.t.bind(t,80,23)),Promise.resolve().then(t.t.bind(t,36423,23))}},function(e){var n=function(n){return e(e.s=n)};e.O(0,[971,117],function(){return n(54278),n(78483)}),_N_E=e.O()}]);
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,9 @@
|
||||
<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 146.36 139.16" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
<defs>
|
||||
<style>
|
||||
.cls-1{fill:#29b5e8;fill-rule:evenodd;}
|
||||
</style>
|
||||
</defs>
|
||||
<path class="cls-1" d="M134.81,60.1l-16.47,9.49L134.81,79a8.65,8.65,0,1,1-8.67,15l-29.51-17a8.68,8.68,0,0,1-4.33-7.75,8.48,8.48,0,0,1,.31-2,8.68,8.68,0,0,1,4-5.19l29.51-16.94A8.69,8.69,0,0,1,138,48.31,8.58,8.58,0,0,1,134.81,60.1Zm-15.59,46L89.72,89.13a8.72,8.72,0,0,0-13.06,7.48v33.9a8.69,8.69,0,0,0,17.37,0v-19L110.54,121a8.66,8.66,0,1,0,8.68-15Zm-34-33.16L72.92,85.09a2.44,2.44,0,0,1-1.54.65H67.77a2.51,2.51,0,0,1-1.54-.65L54,72.9a2.45,2.45,0,0,1-.64-1.52v-3.6A2.5,2.5,0,0,1,54,66.25L66.23,54.06a2.5,2.5,0,0,1,1.54-.64h3.61a2.45,2.45,0,0,1,1.54.64L85.18,66.25a2.49,2.49,0,0,1,.63,1.53v3.6A2.44,2.44,0,0,1,85.18,72.9Zm-9.8-3.38A2.59,2.59,0,0,0,74.73,68l-3.55-3.51a2.51,2.51,0,0,0-1.54-.64h-.13a2.46,2.46,0,0,0-1.53.64L64.43,68a2.51,2.51,0,0,0-.63,1.55v.13a2.41,2.41,0,0,0,.63,1.52L68,74.7a2.48,2.48,0,0,0,1.53.64h.13a2.51,2.51,0,0,0,1.54-.64l3.55-3.53a2.49,2.49,0,0,0,.65-1.52ZM19.93,33.08,49.44,50a8.73,8.73,0,0,0,13.07-7.49V8.64a8.69,8.69,0,0,0-17.37,0v19l-16.53-9.5a8.65,8.65,0,1,0-8.68,15ZM84.69,51.16a8.64,8.64,0,0,0,5-1.13l29.5-17a8.65,8.65,0,1,0-8.68-15L94,27.61v-19a8.69,8.69,0,0,0-17.37,0v33.9A8.66,8.66,0,0,0,84.69,51.16ZM54.48,88a8.58,8.58,0,0,0-5,1.13L19.93,106.06a8.66,8.66,0,1,0,8.68,15l16.53-9.49v19a8.69,8.69,0,0,0,17.37,0V96.61A8.65,8.65,0,0,0,54.48,88Zm-8-15.87a8.61,8.61,0,0,0-4-10L13,45.14A8.69,8.69,0,0,0,1.17,48.31,8.59,8.59,0,0,0,4.35,60.1l16.47,9.49L4.35,79A8.65,8.65,0,1,0,13,94l29.48-17A8.59,8.59,0,0,0,46.47,72.13Zm93.15-56.22H138.3v1.63h1.32c.61,0,1-.28,1-.8S140.26,15.91,139.62,15.91Zm-2.94-1.5h3c1.62,0,2.7.89,2.7,2.27a2.16,2.16,0,0,1-1.08,1.9l1.17,1.68v.34h-1.69L139.62,19H138.3V20.6h-1.62Zm8.3,3.22a5.48,5.48,0,0,0-5.58-5.83c-3.31,0-5.51,2.39-5.51,5.83,0,3.28,2.2,5.82,5.51,5.82A5.47,5.47,0,0,0,145,17.63Zm1.38,0c0,3.89-2.6,7.14-7,7.14s-6.89-3.28-6.89-7.14,2.57-7.14,6.89-7.14S146.36,13.73,146.36,17.63Z">
|
||||
</path>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.0 KiB |
File diff suppressed because one or more lines are too long
@@ -1,7 +1,7 @@
|
||||
2:I[19107,[],"ClientPageRoot"]
|
||||
3:I[55139,["665","static/chunks/3014691f-b7b79b78e27792f3.js","990","static/chunks/13b76428-ebdf3012af0e4489.js","50","static/chunks/50-d0da2dd7acce2eb9.js","521","static/chunks/521-d97d355792d44830.js","866","static/chunks/866-3523e0e07cf314f6.js","313","static/chunks/313-27c820a98e9413e5.js","154","static/chunks/154-f87cf692dcea3018.js","162","static/chunks/162-dd6427ff1a4ad9f4.js","851","static/chunks/851-bbe6d02cf41bb87a.js","931","static/chunks/app/page-f400068ac45ce482.js"],"default",1]
|
||||
3:I[73148,["665","static/chunks/3014691f-b7b79b78e27792f3.js","990","static/chunks/13b76428-ebdf3012af0e4489.js","50","static/chunks/50-d0da2dd7acce2eb9.js","521","static/chunks/521-d97d355792d44830.js","866","static/chunks/866-9e1803a09e9ae8da.js","313","static/chunks/313-0025fb08e386c4b8.js","49","static/chunks/49-b6f167418ea8dbf4.js","162","static/chunks/162-ffcd7d9fbb033bdf.js","851","static/chunks/851-0a73701a3a0b0187.js","931","static/chunks/app/page-5b9ff2d173a47e2c.js"],"default",1]
|
||||
4:I[4707,[],""]
|
||||
5:I[36423,[],""]
|
||||
0:["WkpkdsewrdPMuTzVGS_5j",[[["",{"children":["__PAGE__",{}]},"$undefined","$undefined",true],["",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/349654da14372cd9.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/4103fa525703177b.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_1c856b","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
|
||||
0:["_S0i_Y-CCoYQWc9dIuLxF",[[["",{"children":["__PAGE__",{}]},"$undefined","$undefined",true],["",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/349654da14372cd9.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/c200be8dd8638678.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_1c856b","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
|
||||
6:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"LiteLLM Dashboard"}],["$","meta","3",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","4",{"rel":"icon","href":"/favicon.ico","type":"image/x-icon","sizes":"16x16"}],["$","link","5",{"rel":"icon","href":"./favicon.ico"}],["$","meta","6",{"name":"next-size-adjust"}]]
|
||||
1:null
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
2:I[19107,[],"ClientPageRoot"]
|
||||
3:I[52829,["50","static/chunks/50-d0da2dd7acce2eb9.js","521","static/chunks/521-d97d355792d44830.js","154","static/chunks/154-f87cf692dcea3018.js","162","static/chunks/162-dd6427ff1a4ad9f4.js","418","static/chunks/app/model_hub/page-237d2973f13202c4.js"],"default",1]
|
||||
3:I[52829,["50","static/chunks/50-d0da2dd7acce2eb9.js","521","static/chunks/521-d97d355792d44830.js","49","static/chunks/49-b6f167418ea8dbf4.js","162","static/chunks/162-ffcd7d9fbb033bdf.js","418","static/chunks/app/model_hub/page-d7915f579b770030.js"],"default",1]
|
||||
4:I[4707,[],""]
|
||||
5:I[36423,[],""]
|
||||
0:["WkpkdsewrdPMuTzVGS_5j",[[["",{"children":["model_hub",{"children":["__PAGE__",{}]}]},"$undefined","$undefined",true],["",{"children":["model_hub",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[null,["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children","model_hub","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/349654da14372cd9.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/4103fa525703177b.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_1c856b","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
|
||||
0:["_S0i_Y-CCoYQWc9dIuLxF",[[["",{"children":["model_hub",{"children":["__PAGE__",{}]}]},"$undefined","$undefined",true],["",{"children":["model_hub",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[null,["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children","model_hub","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/349654da14372cd9.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/c200be8dd8638678.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_1c856b","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
|
||||
6:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"LiteLLM Dashboard"}],["$","meta","3",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","4",{"rel":"icon","href":"/favicon.ico","type":"image/x-icon","sizes":"16x16"}],["$","link","5",{"rel":"icon","href":"./favicon.ico"}],["$","meta","6",{"name":"next-size-adjust"}]]
|
||||
1:null
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1,7 +1,7 @@
|
||||
2:I[19107,[],"ClientPageRoot"]
|
||||
3:I[22775,["50","static/chunks/50-d0da2dd7acce2eb9.js","521","static/chunks/521-d97d355792d44830.js","866","static/chunks/866-3523e0e07cf314f6.js","154","static/chunks/154-f87cf692dcea3018.js","162","static/chunks/162-dd6427ff1a4ad9f4.js","851","static/chunks/851-bbe6d02cf41bb87a.js","25","static/chunks/app/model_hub_table/page-5d1aa98a47f9e9fd.js"],"default",1]
|
||||
3:I[22775,["50","static/chunks/50-d0da2dd7acce2eb9.js","521","static/chunks/521-d97d355792d44830.js","866","static/chunks/866-9e1803a09e9ae8da.js","49","static/chunks/49-b6f167418ea8dbf4.js","162","static/chunks/162-ffcd7d9fbb033bdf.js","851","static/chunks/851-0a73701a3a0b0187.js","25","static/chunks/app/model_hub_table/page-2f23c22a47b20607.js"],"default",1]
|
||||
4:I[4707,[],""]
|
||||
5:I[36423,[],""]
|
||||
0:["WkpkdsewrdPMuTzVGS_5j",[[["",{"children":["model_hub_table",{"children":["__PAGE__",{}]}]},"$undefined","$undefined",true],["",{"children":["model_hub_table",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[null,["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children","model_hub_table","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/349654da14372cd9.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/4103fa525703177b.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_1c856b","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
|
||||
0:["_S0i_Y-CCoYQWc9dIuLxF",[[["",{"children":["model_hub_table",{"children":["__PAGE__",{}]}]},"$undefined","$undefined",true],["",{"children":["model_hub_table",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[null,["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children","model_hub_table","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/349654da14372cd9.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/c200be8dd8638678.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_1c856b","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
|
||||
6:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"LiteLLM Dashboard"}],["$","meta","3",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","4",{"rel":"icon","href":"/favicon.ico","type":"image/x-icon","sizes":"16x16"}],["$","link","5",{"rel":"icon","href":"./favicon.ico"}],["$","meta","6",{"name":"next-size-adjust"}]]
|
||||
1:null
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,7 +1,7 @@
|
||||
2:I[19107,[],"ClientPageRoot"]
|
||||
3:I[12011,["665","static/chunks/3014691f-b7b79b78e27792f3.js","50","static/chunks/50-d0da2dd7acce2eb9.js","154","static/chunks/154-f87cf692dcea3018.js","461","static/chunks/app/onboarding/page-099f7aa4c559d470.js"],"default",1]
|
||||
3:I[12011,["665","static/chunks/3014691f-b7b79b78e27792f3.js","50","static/chunks/50-d0da2dd7acce2eb9.js","49","static/chunks/49-b6f167418ea8dbf4.js","461","static/chunks/app/onboarding/page-1fd26064f407ad1d.js"],"default",1]
|
||||
4:I[4707,[],""]
|
||||
5:I[36423,[],""]
|
||||
0:["WkpkdsewrdPMuTzVGS_5j",[[["",{"children":["onboarding",{"children":["__PAGE__",{}]}]},"$undefined","$undefined",true],["",{"children":["onboarding",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[null,["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children","onboarding","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/349654da14372cd9.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/4103fa525703177b.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_1c856b","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
|
||||
0:["_S0i_Y-CCoYQWc9dIuLxF",[[["",{"children":["onboarding",{"children":["__PAGE__",{}]}]},"$undefined","$undefined",true],["",{"children":["onboarding",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[null,["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children","onboarding","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/349654da14372cd9.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/c200be8dd8638678.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_1c856b","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
|
||||
6:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"LiteLLM Dashboard"}],["$","meta","3",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","4",{"rel":"icon","href":"/favicon.ico","type":"image/x-icon","sizes":"16x16"}],["$","link","5",{"rel":"icon","href":"./favicon.ico"}],["$","meta","6",{"name":"next-size-adjust"}]]
|
||||
1:null
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
model_list:
|
||||
- model_name: byok-fixed-gpt-4o-mini
|
||||
- model_name: openai/gpt-4o
|
||||
litellm_params:
|
||||
model: openai/gpt-4o-mini
|
||||
api_base: "https://webhook.site/2f385e05-00aa-402b-86d1-efc9261471a5"
|
||||
@@ -16,15 +16,18 @@ model_list:
|
||||
api_base: "https://webhook.site/2f385e05-00aa-402b-86d1-efc9261471a5"
|
||||
api_key: dummy
|
||||
|
||||
mcp_servers:
|
||||
github_mcp:
|
||||
url: "https://api.githubcopilot.com/mcp"
|
||||
auth_type: oauth2
|
||||
authorization_url: https://github.com/login/oauth/authorize
|
||||
token_url: https://github.com/login/oauth/access_token
|
||||
client_id: os.environ/GITHUB_OAUTH_CLIENT_ID
|
||||
client_secret: os.environ/GITHUB_OAUTH_CLIENT_SECRET
|
||||
scopes: ["public_repo", "user:email"]
|
||||
allowed_tools: ["list_tools"]
|
||||
# disallowed_tools: ["repo_delete"]
|
||||
# mcp_servers:
|
||||
# github_mcp:
|
||||
# url: "https://api.githubcopilot.com/mcp"
|
||||
# auth_type: oauth2
|
||||
# authorization_url: https://github.com/login/oauth/authorize
|
||||
# token_url: https://github.com/login/oauth/access_token
|
||||
# client_id: os.environ/GITHUB_OAUTH_CLIENT_ID
|
||||
# client_secret: os.environ/GITHUB_OAUTH_CLIENT_SECRET
|
||||
# scopes: ["public_repo", "user:email"]
|
||||
# allowed_tools: ["list_tools"]
|
||||
# # disallowed_tools: ["repo_delete"]
|
||||
|
||||
litellm_settings:
|
||||
callbacks: ["prometheus"]
|
||||
custom_prometheus_metadata_labels: ["metadata.initiative", "metadata.business-unit"]
|
||||
+13
-1
@@ -731,6 +731,7 @@ class GenerateRequestBase(LiteLLMPydanticObjectBase):
|
||||
metadata: Optional[dict] = {}
|
||||
tpm_limit: Optional[int] = None
|
||||
rpm_limit: Optional[int] = None
|
||||
|
||||
budget_duration: Optional[str] = None
|
||||
allowed_cache_controls: Optional[list] = []
|
||||
config: Optional[dict] = {}
|
||||
@@ -755,6 +756,12 @@ class KeyRequestBase(GenerateRequestBase):
|
||||
tags: Optional[List[str]] = None
|
||||
enforced_params: Optional[List[str]] = None
|
||||
allowed_routes: Optional[list] = []
|
||||
rpm_limit_type: Optional[
|
||||
Literal["guaranteed_throughput", "best_effort_throughput"]
|
||||
] = None # raise an error if 'guaranteed_throughput' is set and we're overallocating rpm
|
||||
tpm_limit_type: Optional[
|
||||
Literal["guaranteed_throughput", "best_effort_throughput"]
|
||||
] = None # raise an error if 'guaranteed_throughput' is set and we're overallocating tpm
|
||||
|
||||
|
||||
class LiteLLMKeyType(str, enum.Enum):
|
||||
@@ -918,6 +925,7 @@ class NewMCPServerRequest(LiteLLMPydanticObjectBase):
|
||||
mcp_info: Optional[MCPInfo] = None
|
||||
mcp_access_groups: List[str] = Field(default_factory=list)
|
||||
allowed_tools: Optional[List[str]] = None
|
||||
extra_headers: Optional[List[str]] = None
|
||||
# Stdio-specific fields
|
||||
command: Optional[str] = None
|
||||
args: List[str] = Field(default_factory=list)
|
||||
@@ -987,9 +995,10 @@ class LiteLLM_MCPServerTable(LiteLLMPydanticObjectBase):
|
||||
teams: List[Dict[str, Optional[str]]] = Field(default_factory=list)
|
||||
mcp_access_groups: List[str] = Field(default_factory=list)
|
||||
allowed_tools: List[str] = Field(default_factory=list)
|
||||
extra_headers: List[str] = Field(default_factory=list)
|
||||
mcp_info: Optional[MCPInfo] = None
|
||||
# Health check status
|
||||
status: Optional[str] = Field(
|
||||
status: Optional[Literal["healthy", "unhealthy", "unknown"]] = Field(
|
||||
default="unknown",
|
||||
description="Health status: 'healthy', 'unhealthy', 'unknown'",
|
||||
)
|
||||
@@ -3056,6 +3065,8 @@ class PassThroughEndpointLoggingTypedDict(TypedDict):
|
||||
LiteLLM_ManagementEndpoint_MetadataFields = [
|
||||
"model_rpm_limit",
|
||||
"model_tpm_limit",
|
||||
"rpm_limit_type",
|
||||
"tpm_limit_type",
|
||||
"guardrails",
|
||||
"tags",
|
||||
"enforced_params",
|
||||
@@ -3068,6 +3079,7 @@ LiteLLM_ManagementEndpoint_MetadataFields_Premium = [
|
||||
"tags",
|
||||
"team_member_key_duration",
|
||||
"prompts",
|
||||
"logging",
|
||||
]
|
||||
|
||||
|
||||
|
||||
@@ -289,8 +289,8 @@ def initialize_callbacks_on_proxy( # noqa: PLR0915
|
||||
|
||||
def get_model_group_from_litellm_kwargs(kwargs: dict) -> Optional[str]:
|
||||
_litellm_params = kwargs.get("litellm_params", None) or {}
|
||||
_metadata = _litellm_params.get(get_metadata_variable_name_from_litellm_params(_litellm_params)) or {}
|
||||
_model_group = _metadata.get("model_group", None) or kwargs.get("model", None)
|
||||
_metadata = _litellm_params.get(get_metadata_variable_name_from_kwargs(kwargs)) or {}
|
||||
_model_group = _metadata.get("model_group", None)
|
||||
if _model_group is not None:
|
||||
return _model_group
|
||||
|
||||
@@ -367,8 +367,8 @@ def add_guardrail_to_applied_guardrails_header(
|
||||
_metadata["applied_guardrails"] = [guardrail_name]
|
||||
|
||||
|
||||
def get_metadata_variable_name_from_litellm_params(
|
||||
litellm_params: dict
|
||||
def get_metadata_variable_name_from_kwargs(
|
||||
kwargs: dict
|
||||
) -> Literal["metadata", "litellm_metadata"]:
|
||||
"""
|
||||
Helper to return what the "metadata" field should be called in the request data
|
||||
@@ -381,4 +381,4 @@ def get_metadata_variable_name_from_litellm_params(
|
||||
- OpenAI then started using this field for their metadata
|
||||
- LiteLLM is now moving to using `litellm_metadata` for our metadata
|
||||
"""
|
||||
return "litellm_metadata" if "litellm_metadata" in litellm_params else "metadata"
|
||||
return "litellm_metadata" if "litellm_metadata" in kwargs else "metadata"
|
||||
|
||||
@@ -39,6 +39,7 @@ def decrypt_value_helper(
|
||||
value: str,
|
||||
key: str, # this is just for debug purposes, showing the k,v pair that's invalid. not a signing key.
|
||||
exception_type: Literal["debug", "error"] = "error",
|
||||
return_original_value: bool = False,
|
||||
):
|
||||
signing_key = _get_salt_key()
|
||||
|
||||
@@ -55,14 +56,14 @@ def decrypt_value_helper(
|
||||
error_message = f"Error decrypting value for key: {key}, Did your master_key/salt key change recently? \nError: {str(e)}\nSet permanent salt key - https://docs.litellm.ai/docs/proxy/prod#5-set-litellm-salt-key"
|
||||
if exception_type == "debug":
|
||||
verbose_proxy_logger.debug(error_message)
|
||||
return None
|
||||
return value if return_original_value else None
|
||||
|
||||
verbose_proxy_logger.debug(
|
||||
f"Unable to decrypt value={value} for key: {key}, returning None"
|
||||
)
|
||||
verbose_proxy_logger.exception(error_message)
|
||||
# [Non-Blocking Exception. - this should not block decrypting other values]
|
||||
return None
|
||||
return value if return_original_value else None
|
||||
|
||||
|
||||
def encrypt_value(value: str, signing_key: str):
|
||||
|
||||
@@ -865,7 +865,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
|
||||
_get_parent_otel_span_from_kwargs,
|
||||
)
|
||||
from litellm.proxy.common_utils.callback_utils import (
|
||||
get_metadata_variable_name_from_litellm_params,
|
||||
get_metadata_variable_name_from_kwargs,
|
||||
get_model_group_from_litellm_kwargs,
|
||||
)
|
||||
from litellm.types.caching import RedisPipelineIncrementOperation
|
||||
@@ -883,7 +883,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
|
||||
|
||||
# Get metadata from kwargs
|
||||
litellm_metadata = kwargs["litellm_params"].get(
|
||||
get_metadata_variable_name_from_litellm_params(kwargs["litellm_params"]), {}
|
||||
get_metadata_variable_name_from_kwargs(kwargs), {}
|
||||
)
|
||||
if litellm_metadata is None:
|
||||
return
|
||||
|
||||
@@ -51,7 +51,11 @@ class _ProxyDBLogger(CustomLogger):
|
||||
user_api_key_alias=user_api_key_dict.key_alias,
|
||||
user_api_key_spend=user_api_key_dict.spend,
|
||||
user_api_key_max_budget=user_api_key_dict.max_budget,
|
||||
user_api_key_budget_reset_at=user_api_key_dict.budget_reset_at.isoformat() if user_api_key_dict.budget_reset_at else None,
|
||||
user_api_key_budget_reset_at=(
|
||||
user_api_key_dict.budget_reset_at.isoformat()
|
||||
if user_api_key_dict.budget_reset_at
|
||||
else None
|
||||
),
|
||||
user_api_key_user_email=user_api_key_dict.user_email,
|
||||
user_api_key_user_id=user_api_key_dict.user_id,
|
||||
user_api_key_team_id=user_api_key_dict.team_id,
|
||||
@@ -59,15 +63,16 @@ class _ProxyDBLogger(CustomLogger):
|
||||
user_api_key_team_alias=user_api_key_dict.team_alias,
|
||||
user_api_key_end_user_id=user_api_key_dict.end_user_id,
|
||||
user_api_key_request_route=user_api_key_dict.request_route,
|
||||
user_api_key_auth_metadata=user_api_key_dict.metadata,
|
||||
)
|
||||
)
|
||||
_metadata["user_api_key"] = user_api_key_dict.api_key
|
||||
_metadata["status"] = "failure"
|
||||
_metadata[
|
||||
"error_information"
|
||||
] = StandardLoggingPayloadSetup.get_error_information(
|
||||
original_exception=original_exception,
|
||||
traceback_str=traceback_str,
|
||||
_metadata["error_information"] = (
|
||||
StandardLoggingPayloadSetup.get_error_information(
|
||||
original_exception=original_exception,
|
||||
traceback_str=traceback_str,
|
||||
)
|
||||
)
|
||||
|
||||
existing_metadata: dict = request_data.get("metadata", None) or {}
|
||||
|
||||
@@ -579,7 +579,12 @@ class LiteLLMProxyRequestSetup:
|
||||
user_api_key_end_user_id=user_api_key_dict.end_user_id,
|
||||
user_api_key_user_email=user_api_key_dict.user_email,
|
||||
user_api_key_request_route=user_api_key_dict.request_route,
|
||||
user_api_key_budget_reset_at=user_api_key_dict.budget_reset_at.isoformat() if user_api_key_dict.budget_reset_at else None,
|
||||
user_api_key_budget_reset_at=(
|
||||
user_api_key_dict.budget_reset_at.isoformat()
|
||||
if user_api_key_dict.budget_reset_at
|
||||
else None
|
||||
),
|
||||
user_api_key_auth_metadata=None,
|
||||
)
|
||||
return user_api_key_logged_metadata
|
||||
|
||||
@@ -607,6 +612,39 @@ class LiteLLMProxyRequestSetup:
|
||||
)
|
||||
return data
|
||||
|
||||
@staticmethod
|
||||
def add_management_endpoint_metadata_to_request_metadata(
|
||||
data: dict,
|
||||
management_endpoint_metadata: dict,
|
||||
_metadata_variable_name: str,
|
||||
) -> dict:
|
||||
"""
|
||||
Adds the `UserAPIKeyAuth` metadata to the request metadata.
|
||||
|
||||
ignore any sensitive fields like logging, api_key, etc.
|
||||
"""
|
||||
if _metadata_variable_name not in data:
|
||||
return data
|
||||
from litellm.proxy._types import (
|
||||
LiteLLM_ManagementEndpoint_MetadataFields,
|
||||
LiteLLM_ManagementEndpoint_MetadataFields_Premium,
|
||||
)
|
||||
|
||||
# ignore any special fields
|
||||
added_metadata = {}
|
||||
for k, v in management_endpoint_metadata.items():
|
||||
if k not in (
|
||||
LiteLLM_ManagementEndpoint_MetadataFields_Premium
|
||||
+ LiteLLM_ManagementEndpoint_MetadataFields
|
||||
):
|
||||
added_metadata[k] = v
|
||||
if data[_metadata_variable_name].get("user_api_key_auth_metadata") is None:
|
||||
data[_metadata_variable_name]["user_api_key_auth_metadata"] = {}
|
||||
data[_metadata_variable_name]["user_api_key_auth_metadata"].update(
|
||||
added_metadata
|
||||
)
|
||||
return data
|
||||
|
||||
@staticmethod
|
||||
def add_key_level_controls(
|
||||
key_metadata: Optional[dict], data: dict, _metadata_variable_name: str
|
||||
@@ -651,6 +689,13 @@ class LiteLLMProxyRequestSetup:
|
||||
key_metadata["disable_fallbacks"], bool
|
||||
):
|
||||
data["disable_fallbacks"] = key_metadata["disable_fallbacks"]
|
||||
|
||||
## KEY-LEVEL METADATA
|
||||
data = LiteLLMProxyRequestSetup.add_management_endpoint_metadata_to_request_metadata(
|
||||
data=data,
|
||||
management_endpoint_metadata=key_metadata,
|
||||
_metadata_variable_name=_metadata_variable_name,
|
||||
)
|
||||
return data
|
||||
|
||||
@staticmethod
|
||||
@@ -889,6 +934,15 @@ async def add_litellm_data_to_request( # noqa: PLR0915
|
||||
"spend_logs_metadata"
|
||||
]
|
||||
|
||||
## TEAM-LEVEL METADATA
|
||||
data = (
|
||||
LiteLLMProxyRequestSetup.add_management_endpoint_metadata_to_request_metadata(
|
||||
data=data,
|
||||
management_endpoint_metadata=team_metadata,
|
||||
_metadata_variable_name=_metadata_variable_name,
|
||||
)
|
||||
)
|
||||
|
||||
# Team spend, budget - used by prometheus.py
|
||||
data[_metadata_variable_name][
|
||||
"user_api_key_team_max_budget"
|
||||
|
||||
@@ -43,7 +43,7 @@ def _set_object_metadata_field(
|
||||
value: Value to set for the field
|
||||
"""
|
||||
if field_name in LiteLLM_ManagementEndpoint_MetadataFields_Premium:
|
||||
_premium_user_check()
|
||||
_premium_user_check(field_name)
|
||||
object_data.metadata = object_data.metadata or {}
|
||||
object_data.metadata[field_name] = value
|
||||
|
||||
|
||||
@@ -27,6 +27,7 @@ from litellm.caching import DualCache
|
||||
from litellm.constants import LENGTH_OF_LITELLM_GENERATED_KEY, UI_SESSION_TOKEN_TEAM_ID
|
||||
from litellm.litellm_core_utils.duration_parser import duration_in_seconds
|
||||
from litellm.proxy._types import *
|
||||
from litellm.proxy._types import LiteLLM_VerificationToken
|
||||
from litellm.proxy.auth.auth_checks import (
|
||||
_cache_key_object,
|
||||
_delete_cache_key_object,
|
||||
@@ -90,10 +91,10 @@ def _get_user_in_team(
|
||||
def _calculate_key_rotation_time(rotation_interval: str) -> datetime:
|
||||
"""
|
||||
Helper function to calculate the next rotation time for a key based on the rotation interval.
|
||||
|
||||
|
||||
Args:
|
||||
rotation_interval: String representing the rotation interval (e.g., '30d', '90d', '1h')
|
||||
|
||||
|
||||
Returns:
|
||||
datetime: The calculated next rotation time in UTC
|
||||
"""
|
||||
@@ -102,28 +103,34 @@ def _calculate_key_rotation_time(rotation_interval: str) -> datetime:
|
||||
return now + timedelta(seconds=interval_seconds)
|
||||
|
||||
|
||||
def _set_key_rotation_fields(data: dict, auto_rotate: bool, rotation_interval: Optional[str]) -> None:
|
||||
def _set_key_rotation_fields(
|
||||
data: dict, auto_rotate: bool, rotation_interval: Optional[str]
|
||||
) -> None:
|
||||
"""
|
||||
Helper function to set rotation fields in key data if auto_rotate is enabled.
|
||||
|
||||
|
||||
Args:
|
||||
data: Dictionary to update with rotation fields
|
||||
auto_rotate: Whether auto rotation is enabled
|
||||
rotation_interval: The rotation interval string (required if auto_rotate is True)
|
||||
"""
|
||||
if auto_rotate and rotation_interval:
|
||||
data.update({
|
||||
"auto_rotate": auto_rotate,
|
||||
"rotation_interval": rotation_interval,
|
||||
"key_rotation_at": _calculate_key_rotation_time(rotation_interval)
|
||||
})
|
||||
data.update(
|
||||
{
|
||||
"auto_rotate": auto_rotate,
|
||||
"rotation_interval": rotation_interval,
|
||||
"key_rotation_at": _calculate_key_rotation_time(rotation_interval),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _is_allowed_to_make_key_request(
|
||||
user_api_key_dict: UserAPIKeyAuth, user_id: Optional[str], team_id: Optional[str]
|
||||
user_api_key_dict: UserAPIKeyAuth,
|
||||
user_id: Optional[str],
|
||||
team_id: Optional[str],
|
||||
) -> bool:
|
||||
"""
|
||||
Assert user only creates keys for themselves
|
||||
Assert user only creates/updates keys for themselves
|
||||
|
||||
Relevant issue: https://github.com/BerriAI/litellm/issues/7336
|
||||
"""
|
||||
@@ -332,6 +339,7 @@ def common_key_access_checks(
|
||||
data: Union[GenerateKeyRequest, UpdateKeyRequest],
|
||||
llm_router: Optional[Router],
|
||||
premium_user: bool,
|
||||
user_id: Optional[str] = None,
|
||||
) -> Literal[True]:
|
||||
"""
|
||||
Check if user is allowed to make a key request, for this key
|
||||
@@ -339,7 +347,7 @@ def common_key_access_checks(
|
||||
try:
|
||||
_is_allowed_to_make_key_request(
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
user_id=data.user_id,
|
||||
user_id=user_id or data.user_id,
|
||||
team_id=data.team_id,
|
||||
)
|
||||
except AssertionError as e:
|
||||
@@ -542,6 +550,15 @@ async def _common_key_generation_helper( # noqa: PLR0915
|
||||
value=getattr(data, field),
|
||||
)
|
||||
|
||||
for field in LiteLLM_ManagementEndpoint_MetadataFields:
|
||||
if getattr(data, field, None) is not None:
|
||||
_set_object_metadata_field(
|
||||
object_data=data,
|
||||
field_name=field,
|
||||
value=getattr(data, field),
|
||||
)
|
||||
delattr(data, field)
|
||||
|
||||
data_json = data.model_dump(exclude_unset=True, exclude_none=True) # type: ignore
|
||||
|
||||
data_json = handle_key_type(data, data_json)
|
||||
@@ -620,6 +637,153 @@ async def _common_key_generation_helper( # noqa: PLR0915
|
||||
return response
|
||||
|
||||
|
||||
def check_team_key_model_specific_limits(
|
||||
keys: List[LiteLLM_VerificationToken],
|
||||
team_table: LiteLLM_TeamTableCachedObj,
|
||||
data: Union[GenerateKeyRequest, UpdateKeyRequest],
|
||||
) -> None:
|
||||
"""
|
||||
Check if the team key is allocating model specific limits. If so, raise an error if we're overallocating.
|
||||
"""
|
||||
if data.model_rpm_limit is None and data.model_tpm_limit is None:
|
||||
return
|
||||
# get total model specific tpm/rpm limit
|
||||
model_specific_rpm_limit: Dict[str, int] = {}
|
||||
model_specific_tpm_limit: Dict[str, int] = {}
|
||||
|
||||
for key in keys:
|
||||
if key.metadata.get("model_rpm_limit", None) is not None:
|
||||
for model, rpm_limit in key.metadata.get("model_rpm_limit", {}).items():
|
||||
model_specific_rpm_limit[model] = (
|
||||
model_specific_rpm_limit.get(model, 0) + rpm_limit
|
||||
)
|
||||
if key.metadata.get("model_tpm_limit", None) is not None:
|
||||
for model, tpm_limit in key.metadata.get("model_tpm_limit", {}).items():
|
||||
model_specific_tpm_limit[model] = (
|
||||
model_specific_tpm_limit.get(model, 0) + tpm_limit
|
||||
)
|
||||
if data.model_rpm_limit is not None:
|
||||
for model, rpm_limit in data.model_rpm_limit.items():
|
||||
if (
|
||||
model_specific_rpm_limit.get(model, 0) + rpm_limit
|
||||
> team_table.rpm_limit
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Allocated RPM limit={model_specific_rpm_limit.get(model, 0)} + Key RPM limit={rpm_limit} is greater than team RPM limit={team_table.rpm_limit}",
|
||||
)
|
||||
elif team_table.metadata and team_table.metadata.get("model_rpm_limit"):
|
||||
team_model_specific_rpm_limit_dict = team_table.metadata.get(
|
||||
"model_rpm_limit", {}
|
||||
)
|
||||
team_model_specific_rpm_limit = team_model_specific_rpm_limit_dict.get(
|
||||
model
|
||||
)
|
||||
if (
|
||||
model_specific_rpm_limit.get(model, 0) + rpm_limit
|
||||
> team_model_specific_rpm_limit
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Allocated RPM limit={model_specific_rpm_limit.get(model, 0)} + Key RPM limit={rpm_limit} is greater than team RPM limit={team_model_specific_rpm_limit.get(model, 0)}",
|
||||
)
|
||||
if data.model_tpm_limit is not None:
|
||||
for model, tpm_limit in data.model_tpm_limit.items():
|
||||
if (
|
||||
team_table.tpm_limit is not None
|
||||
and model_specific_tpm_limit.get(model, 0) + tpm_limit
|
||||
> team_table.tpm_limit
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Allocated TPM limit={model_specific_tpm_limit.get(model, 0)} + Key TPM limit={tpm_limit} is greater than team TPM limit={team_table.tpm_limit}",
|
||||
)
|
||||
elif team_table.metadata and team_table.metadata.get("model_tpm_limit"):
|
||||
team_model_specific_tpm_limit_dict = team_table.metadata.get(
|
||||
"model_tpm_limit", {}
|
||||
)
|
||||
team_model_specific_tpm_limit = team_model_specific_tpm_limit_dict.get(
|
||||
model
|
||||
)
|
||||
if (
|
||||
team_model_specific_tpm_limit
|
||||
and model_specific_tpm_limit.get(model, 0) + tpm_limit
|
||||
> team_model_specific_tpm_limit
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Allocated TPM limit={model_specific_tpm_limit.get(model, 0)} + Key TPM limit={tpm_limit} is greater than team TPM limit={team_model_specific_tpm_limit}",
|
||||
)
|
||||
|
||||
|
||||
def check_team_key_rpm_tpm_limits(
|
||||
keys: List[LiteLLM_VerificationToken],
|
||||
team_table: LiteLLM_TeamTableCachedObj,
|
||||
data: Union[GenerateKeyRequest, UpdateKeyRequest],
|
||||
) -> None:
|
||||
"""
|
||||
Check if the team key is allocating rpm/tpm limits. If so, raise an error if we're overallocating.
|
||||
"""
|
||||
if keys is not None and len(keys) > 0:
|
||||
allocated_tpm = sum(key.tpm_limit for key in keys if key.tpm_limit is not None)
|
||||
allocated_rpm = sum(key.rpm_limit for key in keys if key.rpm_limit is not None)
|
||||
else:
|
||||
allocated_tpm = 0
|
||||
allocated_rpm = 0
|
||||
if (
|
||||
data.tpm_limit is not None
|
||||
and team_table.tpm_limit is not None
|
||||
and data.tpm_limit + allocated_tpm > team_table.tpm_limit
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Allocated TPM limit={allocated_tpm} + Key TPM limit={data.tpm_limit} is greater than team TPM limit={team_table.tpm_limit}",
|
||||
)
|
||||
if (
|
||||
data.rpm_limit is not None
|
||||
and team_table.rpm_limit is not None
|
||||
and data.rpm_limit + allocated_rpm > team_table.rpm_limit
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Allocated RPM limit={allocated_rpm} + Key RPM limit={data.rpm_limit} is greater than team RPM limit={team_table.rpm_limit}",
|
||||
)
|
||||
|
||||
|
||||
async def _check_team_key_limits(
|
||||
team_table: LiteLLM_TeamTableCachedObj,
|
||||
data: Union[GenerateKeyRequest, UpdateKeyRequest],
|
||||
prisma_client: PrismaClient,
|
||||
) -> None:
|
||||
"""
|
||||
Check if the team key is allocating guaranteed throughput limits. If so, raise an error if we're overallocating.
|
||||
|
||||
Only runs check if tpm_limit_type or rpm_limit_type is "guaranteed_throughput"
|
||||
"""
|
||||
if (
|
||||
data.tpm_limit_type != "guaranteed_throughput"
|
||||
and data.rpm_limit_type != "guaranteed_throughput"
|
||||
):
|
||||
return
|
||||
# get all team keys
|
||||
# calculate allocated tpm/rpm limit
|
||||
# check if specified tpm/rpm limit is greater than allocated tpm/rpm limit
|
||||
|
||||
keys = await prisma_client.db.litellm_verificationtoken.find_many(
|
||||
where={"team_id": team_table.team_id},
|
||||
)
|
||||
check_team_key_model_specific_limits(
|
||||
keys=keys,
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
)
|
||||
check_team_key_rpm_tpm_limits(
|
||||
keys=keys,
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/key/generate",
|
||||
tags=["key management"],
|
||||
@@ -661,6 +825,8 @@ async def generate_key_fn(
|
||||
- model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}}. IF null or {} then no model specific budget.
|
||||
- model_rpm_limit: Optional[dict] - key-specific model rpm limit. Example - {"text-davinci-002": 1000, "gpt-3.5-turbo": 1000}. IF null or {} then no model specific rpm limit.
|
||||
- model_tpm_limit: Optional[dict] - key-specific model tpm limit. Example - {"text-davinci-002": 1000, "gpt-3.5-turbo": 1000}. IF null or {} then no model specific tpm limit.
|
||||
- tpm_limit_type: Optional[str] - Type of tpm limit. Options: "best_effort_throughput" (no error if we're overallocating tpm), "guaranteed_throughput" (raise an error if we're overallocating tpm). Defaults to "best_effort_throughput".
|
||||
- rpm_limit_type: Optional[str] - Type of rpm limit. Options: "best_effort_throughput" (no error if we're overallocating rpm), "guaranteed_throughput" (raise an error if we're overallocating rpm). Defaults to "best_effort_throughput".
|
||||
- allowed_cache_controls: Optional[list] - List of allowed cache control values. Example - ["no-cache", "no-store"]. See all values - https://docs.litellm.ai/docs/proxy/caching#turn-on--off-caching-per-request
|
||||
- blocked: Optional[bool] - Whether the key is blocked.
|
||||
- rpm_limit: Optional[int] - Specify rpm limit for a given key (Requests per minute)
|
||||
@@ -696,12 +862,19 @@ async def generate_key_fn(
|
||||
- user_id: (str) Unique user id - used for tracking spend across multiple keys for same user id.
|
||||
"""
|
||||
try:
|
||||
from litellm.proxy._types import CommonProxyErrors
|
||||
from litellm.proxy.proxy_server import (
|
||||
prisma_client,
|
||||
user_api_key_cache,
|
||||
user_custom_key_generate,
|
||||
)
|
||||
|
||||
if prisma_client is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail={"error": CommonProxyErrors.db_not_connected_error.value},
|
||||
)
|
||||
|
||||
verbose_proxy_logger.debug("entered /key/generate")
|
||||
|
||||
if user_custom_key_generate is not None:
|
||||
@@ -729,7 +902,6 @@ async def generate_key_fn(
|
||||
verbose_proxy_logger.debug(
|
||||
f"Error getting team object in `/key/generate`: {e}"
|
||||
)
|
||||
team_table = None
|
||||
|
||||
key_generation_check(
|
||||
team_table=team_table,
|
||||
@@ -738,12 +910,20 @@ async def generate_key_fn(
|
||||
route=KeyManagementRoutes.KEY_GENERATE,
|
||||
)
|
||||
|
||||
if team_table is not None:
|
||||
await _check_team_key_limits(
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
prisma_client=prisma_client,
|
||||
)
|
||||
|
||||
return await _common_key_generation_helper(
|
||||
data=data,
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
litellm_changed_by=litellm_changed_by,
|
||||
team_table=team_table,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
verbose_proxy_logger.exception(
|
||||
"litellm.proxy.proxy_server.generate_key_fn(): Exception occured - {}".format(
|
||||
@@ -797,6 +977,8 @@ async def generate_service_account_key_fn(
|
||||
- model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}}. IF null or {} then no model specific budget.
|
||||
- model_rpm_limit: Optional[dict] - key-specific model rpm limit. Example - {"text-davinci-002": 1000, "gpt-3.5-turbo": 1000}. IF null or {} then no model specific rpm limit.
|
||||
- model_tpm_limit: Optional[dict] - key-specific model tpm limit. Example - {"text-davinci-002": 1000, "gpt-3.5-turbo": 1000}. IF null or {} then no model specific tpm limit.
|
||||
- tpm_limit_type: Optional[str] - TPM rate limit type - "best_effort_throughput" or "guaranteed_throughput"
|
||||
- rpm_limit_type: Optional[str] - RPM rate limit type - "best_effort_throughput" or "guaranteed_throughput"
|
||||
- allowed_cache_controls: Optional[list] - List of allowed cache control values. Example - ["no-cache", "no-store"]. See all values - https://docs.litellm.ai/docs/proxy/caching#turn-on--off-caching-per-request
|
||||
- blocked: Optional[bool] - Whether the key is blocked.
|
||||
- rpm_limit: Optional[int] - Specify rpm limit for a given key (Requests per minute)
|
||||
@@ -825,12 +1007,19 @@ async def generate_service_account_key_fn(
|
||||
- user_id: (str) Unique user id - used for tracking spend across multiple keys for same user id.
|
||||
|
||||
"""
|
||||
from litellm.proxy._types import CommonProxyErrors
|
||||
from litellm.proxy.proxy_server import (
|
||||
prisma_client,
|
||||
user_api_key_cache,
|
||||
user_custom_key_generate,
|
||||
)
|
||||
|
||||
if prisma_client is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail={"error": CommonProxyErrors.db_not_connected_error.value},
|
||||
)
|
||||
|
||||
await validate_team_id_used_in_service_account_request(
|
||||
team_id=data.team_id,
|
||||
prisma_client=prisma_client,
|
||||
@@ -863,6 +1052,13 @@ async def generate_service_account_key_fn(
|
||||
)
|
||||
team_table = None
|
||||
|
||||
if team_table is not None:
|
||||
await _check_team_key_limits(
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
prisma_client=prisma_client,
|
||||
)
|
||||
|
||||
key_generation_check(
|
||||
team_table=team_table,
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
@@ -903,7 +1099,7 @@ def prepare_metadata_fields(
|
||||
if k in LiteLLM_ManagementEndpoint_MetadataFields_Premium:
|
||||
from litellm.proxy.utils import _premium_user_check
|
||||
|
||||
_premium_user_check()
|
||||
_premium_user_check(k)
|
||||
casted_metadata[k] = v
|
||||
|
||||
except Exception as e:
|
||||
@@ -1089,6 +1285,8 @@ async def update_key_fn(
|
||||
- rpm_limit: Optional[int] - Requests per minute limit
|
||||
- model_rpm_limit: Optional[dict] - Model-specific RPM limits {"gpt-4": 100, "claude-v1": 200}
|
||||
- model_tpm_limit: Optional[dict] - Model-specific TPM limits {"gpt-4": 100000, "claude-v1": 200000}
|
||||
- tpm_limit_type: Optional[str] - TPM rate limit type - "best_effort_throughput" or "guaranteed_throughput"
|
||||
- rpm_limit_type: Optional[str] - RPM rate limit type - "best_effort_throughput" or "guaranteed_throughput"
|
||||
- allowed_cache_controls: Optional[list] - List of allowed cache control values
|
||||
- duration: Optional[str] - Key validity duration ("30d", "1h", etc.)
|
||||
- permissions: Optional[dict] - Key-specific permissions
|
||||
@@ -1136,13 +1334,6 @@ async def update_key_fn(
|
||||
if prisma_client is None:
|
||||
raise Exception("Not connected to DB!")
|
||||
|
||||
common_key_access_checks(
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
data=data,
|
||||
llm_router=llm_router,
|
||||
premium_user=premium_user,
|
||||
)
|
||||
|
||||
existing_key_row = await prisma_client.get_data(
|
||||
token=data.key, table_name="key", query_type="find_unique"
|
||||
)
|
||||
@@ -1153,6 +1344,25 @@ async def update_key_fn(
|
||||
detail={"error": f"Team not found, passed team_id={data.team_id}"},
|
||||
)
|
||||
|
||||
## sanity check - prevent non-proxy admin user from updating key to belong to a different user
|
||||
if (
|
||||
data.user_id is not None
|
||||
and data.user_id != existing_key_row.user_id
|
||||
and user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN.value
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail=f"User={data.user_id} is not allowed to update key={key} to belong to user={existing_key_row.user_id}",
|
||||
)
|
||||
|
||||
common_key_access_checks(
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
data=data,
|
||||
user_id=existing_key_row.user_id,
|
||||
llm_router=llm_router,
|
||||
premium_user=premium_user,
|
||||
)
|
||||
|
||||
# check if user has permission to update key
|
||||
await TeamMemberPermissionChecks.can_team_member_execute_key_management_endpoint(
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
@@ -1162,14 +1372,25 @@ async def update_key_fn(
|
||||
user_api_key_cache=user_api_key_cache,
|
||||
)
|
||||
|
||||
# if team change - check if this is possible
|
||||
if is_different_team(data=data, existing_key_row=existing_key_row):
|
||||
# Only check team limits if key has a team_id
|
||||
team_obj: Optional[LiteLLM_TeamTableCachedObj] = None
|
||||
if data.team_id is not None:
|
||||
team_obj = await get_team_object(
|
||||
team_id=cast(str, data.team_id),
|
||||
team_id=data.team_id,
|
||||
prisma_client=prisma_client,
|
||||
user_api_key_cache=user_api_key_cache,
|
||||
check_db_only=True,
|
||||
)
|
||||
|
||||
if team_obj is not None:
|
||||
await _check_team_key_limits(
|
||||
team_table=team_obj,
|
||||
data=data,
|
||||
prisma_client=prisma_client,
|
||||
)
|
||||
|
||||
# if team change - check if this is possible
|
||||
if is_different_team(data=data, existing_key_row=existing_key_row):
|
||||
if llm_router is None:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -1177,6 +1398,14 @@ async def update_key_fn(
|
||||
"error": "LLM router not found. Please set it up by passing in a valid config.yaml or adding models via the UI."
|
||||
},
|
||||
)
|
||||
# team_obj should be set since is_different_team() returns True only when data.team_id is not None
|
||||
if team_obj is None:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail={
|
||||
"error": "Team object not found for team change validation"
|
||||
},
|
||||
)
|
||||
validate_key_team_change(
|
||||
key=existing_key_row,
|
||||
team=team_obj,
|
||||
@@ -1198,9 +1427,9 @@ async def update_key_fn(
|
||||
|
||||
# Handle rotation fields if auto_rotate is being enabled
|
||||
_set_key_rotation_fields(
|
||||
non_default_values,
|
||||
non_default_values.get("auto_rotate", False),
|
||||
non_default_values.get("rotation_interval")
|
||||
non_default_values,
|
||||
non_default_values.get("auto_rotate", False),
|
||||
non_default_values.get("rotation_interval"),
|
||||
)
|
||||
|
||||
_data = {**non_default_values, "token": key}
|
||||
@@ -1602,8 +1831,6 @@ def _check_model_access_group(
|
||||
return True
|
||||
|
||||
|
||||
|
||||
|
||||
async def generate_key_helper_fn( # noqa: PLR0915
|
||||
request_type: Literal[
|
||||
"user", "key"
|
||||
@@ -1766,12 +1993,12 @@ async def generate_key_helper_fn( # noqa: PLR0915
|
||||
"allowed_routes": allowed_routes or [],
|
||||
"object_permission_id": object_permission_id,
|
||||
}
|
||||
|
||||
|
||||
# Add rotation fields if auto_rotate is enabled
|
||||
_set_key_rotation_fields(
|
||||
data=key_data,
|
||||
auto_rotate=auto_rotate or False,
|
||||
rotation_interval=rotation_interval
|
||||
rotation_interval=rotation_interval,
|
||||
)
|
||||
|
||||
if (
|
||||
|
||||
@@ -12,7 +12,6 @@ All /team management endpoints
|
||||
import asyncio
|
||||
import json
|
||||
import traceback
|
||||
from litellm._uuid import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Dict, List, Optional, Tuple, Union, cast
|
||||
|
||||
@@ -22,6 +21,7 @@ from pydantic import BaseModel
|
||||
|
||||
import litellm
|
||||
from litellm._logging import verbose_proxy_logger
|
||||
from litellm._uuid import uuid
|
||||
from litellm.proxy._types import (
|
||||
BlockTeamRequest,
|
||||
CommonProxyErrors,
|
||||
@@ -105,7 +105,7 @@ router = APIRouter()
|
||||
|
||||
class TeamMemberBudgetHandler:
|
||||
"""Helper class to handle team member budget, RPM, and TPM limit operations"""
|
||||
|
||||
|
||||
@staticmethod
|
||||
def should_create_budget(
|
||||
team_member_budget: Optional[float] = None,
|
||||
@@ -113,12 +113,14 @@ class TeamMemberBudgetHandler:
|
||||
team_member_tpm_limit: Optional[int] = None,
|
||||
) -> bool:
|
||||
"""Check if any team member limits are provided"""
|
||||
return any([
|
||||
team_member_budget is not None,
|
||||
team_member_rpm_limit is not None,
|
||||
team_member_tpm_limit is not None,
|
||||
])
|
||||
|
||||
return any(
|
||||
[
|
||||
team_member_budget is not None,
|
||||
team_member_rpm_limit is not None,
|
||||
team_member_tpm_limit is not None,
|
||||
]
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
async def create_team_member_budget_table(
|
||||
data: Union[NewTeamRequest, LiteLLM_TeamTable],
|
||||
@@ -146,7 +148,7 @@ class TeamMemberBudgetHandler:
|
||||
budget_id=budget_id,
|
||||
budget_duration=data.budget_duration,
|
||||
)
|
||||
|
||||
|
||||
if team_member_budget is not None:
|
||||
budget_request.max_budget = team_member_budget
|
||||
if team_member_rpm_limit is not None:
|
||||
@@ -165,12 +167,12 @@ class TeamMemberBudgetHandler:
|
||||
new_team_data_json["metadata"][
|
||||
"team_member_budget_id"
|
||||
] = team_member_budget_table.budget_id
|
||||
|
||||
|
||||
# Remove team member fields from new_team_data_json
|
||||
TeamMemberBudgetHandler._clean_team_member_fields(new_team_data_json)
|
||||
|
||||
return new_team_data_json
|
||||
|
||||
|
||||
@staticmethod
|
||||
async def upsert_team_member_budget_table(
|
||||
team_table: LiteLLM_TeamTable,
|
||||
@@ -193,14 +195,14 @@ class TeamMemberBudgetHandler:
|
||||
if team_member_budget_id is not None and isinstance(team_member_budget_id, str):
|
||||
# Budget exists - create update request with only provided values
|
||||
budget_request = BudgetNewRequest(budget_id=team_member_budget_id)
|
||||
|
||||
|
||||
if team_member_budget is not None:
|
||||
budget_request.max_budget = team_member_budget
|
||||
if team_member_rpm_limit is not None:
|
||||
budget_request.rpm_limit = team_member_rpm_limit
|
||||
if team_member_tpm_limit is not None:
|
||||
budget_request.tpm_limit = team_member_tpm_limit
|
||||
|
||||
|
||||
budget_row = await update_budget(
|
||||
budget_obj=budget_request,
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
@@ -221,11 +223,11 @@ class TeamMemberBudgetHandler:
|
||||
team_member_rpm_limit=team_member_rpm_limit,
|
||||
team_member_tpm_limit=team_member_tpm_limit,
|
||||
)
|
||||
|
||||
|
||||
# Remove team member fields from updated_kv
|
||||
TeamMemberBudgetHandler._clean_team_member_fields(updated_kv)
|
||||
return updated_kv
|
||||
|
||||
|
||||
@staticmethod
|
||||
def _clean_team_member_fields(data_dict: dict) -> None:
|
||||
"""Remove team member fields from data dictionary"""
|
||||
@@ -267,7 +269,6 @@ async def get_all_team_memberships(
|
||||
return returned_tm
|
||||
|
||||
|
||||
|
||||
#### TEAM MANAGEMENT ####
|
||||
@router.post(
|
||||
"/team/new",
|
||||
@@ -383,7 +384,7 @@ async def new_team( # noqa: PLR0915
|
||||
"error": f"Team id = {data.team_id} already exists. Please use a different team id."
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# If max_budget is not explicitly provided in the request,
|
||||
# check for a default value in the proxy configuration.
|
||||
if data.max_budget is None:
|
||||
@@ -503,7 +504,7 @@ async def new_team( # noqa: PLR0915
|
||||
|
||||
# Set Management Endpoint Metadata Fields
|
||||
for field in LiteLLM_ManagementEndpoint_MetadataFields_Premium:
|
||||
if getattr(data, field) is not None:
|
||||
if getattr(data, field, None) is not None:
|
||||
_set_object_metadata_field(
|
||||
object_data=complete_team_data,
|
||||
field_name=field,
|
||||
|
||||
@@ -96,9 +96,13 @@ async def set_env_variables_in_header(custom_headers: Optional[dict]) -> Optiona
|
||||
# langfuse requires b64 encoded headers - we construct that here
|
||||
_langfuse_public_key = custom_headers["LANGFUSE_PUBLIC_KEY"]
|
||||
_langfuse_secret_key = custom_headers["LANGFUSE_SECRET_KEY"]
|
||||
if isinstance(_langfuse_public_key, str) and _langfuse_public_key.startswith("os.environ/"):
|
||||
if isinstance(
|
||||
_langfuse_public_key, str
|
||||
) and _langfuse_public_key.startswith("os.environ/"):
|
||||
_langfuse_public_key = get_secret_str(_langfuse_public_key)
|
||||
if isinstance(_langfuse_secret_key, str) and _langfuse_secret_key.startswith("os.environ/"):
|
||||
if isinstance(
|
||||
_langfuse_secret_key, str
|
||||
) and _langfuse_secret_key.startswith("os.environ/"):
|
||||
_langfuse_secret_key = get_secret_str(_langfuse_secret_key)
|
||||
headers["Authorization"] = "Basic " + b64encode(
|
||||
f"{_langfuse_public_key}:{_langfuse_secret_key}".encode("utf-8")
|
||||
@@ -107,7 +111,9 @@ async def set_env_variables_in_header(custom_headers: Optional[dict]) -> Optiona
|
||||
# for all other headers
|
||||
headers[key] = value
|
||||
if isinstance(value, str) and "os.environ/" in value:
|
||||
verbose_proxy_logger.debug("pass through endpoint - looking up 'os.environ/' variable")
|
||||
verbose_proxy_logger.debug(
|
||||
"pass through endpoint - looking up 'os.environ/' variable"
|
||||
)
|
||||
# get string section that is os.environ/
|
||||
start_index = value.find("os.environ/")
|
||||
_variable_name = value[start_index:]
|
||||
@@ -200,7 +206,9 @@ async def chat_completion_pass_through_endpoint( # noqa: PLR0915
|
||||
# skip router if user passed their key
|
||||
if "api_key" in data:
|
||||
llm_response = asyncio.create_task(litellm.aadapter_completion(**data))
|
||||
elif llm_router is not None and data["model"] in router_model_names: # model in router model list
|
||||
elif (
|
||||
llm_router is not None and data["model"] in router_model_names
|
||||
): # model in router model list
|
||||
llm_response = asyncio.create_task(llm_router.aadapter_completion(**data))
|
||||
elif (
|
||||
llm_router is not None
|
||||
@@ -214,8 +222,8 @@ async def chat_completion_pass_through_endpoint( # noqa: PLR0915
|
||||
llm_response = asyncio.create_task(
|
||||
llm_router.aadapter_completion(**data, specific_deployment=True)
|
||||
)
|
||||
elif (
|
||||
llm_router is not None and llm_router.has_model_id(data["model"])
|
||||
elif llm_router is not None and llm_router.has_model_id(
|
||||
data["model"]
|
||||
): # model in router model list
|
||||
llm_response = asyncio.create_task(llm_router.aadapter_completion(**data))
|
||||
elif (
|
||||
@@ -229,7 +237,10 @@ async def chat_completion_pass_through_endpoint( # noqa: PLR0915
|
||||
else:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail={"error": "completion: Invalid model name passed in model=" + data.get("model", "")},
|
||||
detail={
|
||||
"error": "completion: Invalid model name passed in model="
|
||||
+ data.get("model", "")
|
||||
},
|
||||
)
|
||||
|
||||
# Await the llm_response task
|
||||
@@ -243,7 +254,9 @@ async def chat_completion_pass_through_endpoint( # noqa: PLR0915
|
||||
|
||||
### ALERTING ###
|
||||
asyncio.create_task(
|
||||
proxy_logging_obj.update_request_status(litellm_call_id=data.get("litellm_call_id", ""), status="success")
|
||||
proxy_logging_obj.update_request_status(
|
||||
litellm_call_id=data.get("litellm_call_id", ""), status="success"
|
||||
)
|
||||
)
|
||||
|
||||
verbose_proxy_logger.debug("final response: %s", response)
|
||||
@@ -265,7 +278,11 @@ async def chat_completion_pass_through_endpoint( # noqa: PLR0915
|
||||
await proxy_logging_obj.post_call_failure_hook(
|
||||
user_api_key_dict=user_api_key_dict, original_exception=e, request_data=data
|
||||
)
|
||||
verbose_proxy_logger.exception("litellm.proxy.proxy_server.completion(): Exception occured - {}".format(str(e)))
|
||||
verbose_proxy_logger.exception(
|
||||
"litellm.proxy.proxy_server.completion(): Exception occured - {}".format(
|
||||
str(e)
|
||||
)
|
||||
)
|
||||
error_msg = f"{str(e)}"
|
||||
raise ProxyException(
|
||||
message=getattr(e, "message", error_msg),
|
||||
@@ -284,7 +301,11 @@ class HttpPassThroughEndpointHelpers(BasePassthroughUtils):
|
||||
) -> dict:
|
||||
excluded_headers = {"transfer-encoding", "content-encoding"}
|
||||
|
||||
return_headers = {key: value for key, value in headers.items() if key.lower() not in excluded_headers}
|
||||
return_headers = {
|
||||
key: value
|
||||
for key, value in headers.items()
|
||||
if key.lower() not in excluded_headers
|
||||
}
|
||||
if litellm_call_id:
|
||||
return_headers["x-litellm-call-id"] = litellm_call_id
|
||||
if custom_headers:
|
||||
@@ -411,8 +432,10 @@ class HttpPassThroughEndpointHelpers(BasePassthroughUtils):
|
||||
|
||||
for field_name, field_value in form_data.items():
|
||||
if isinstance(field_value, (StarletteUploadFile, UploadFile)):
|
||||
files[field_name] = await HttpPassThroughEndpointHelpers._build_request_files_from_upload_file(
|
||||
upload_file=field_value
|
||||
files[field_name] = (
|
||||
await HttpPassThroughEndpointHelpers._build_request_files_from_upload_file(
|
||||
upload_file=field_value
|
||||
)
|
||||
)
|
||||
else:
|
||||
form_data_dict[field_name] = field_value
|
||||
@@ -462,8 +485,11 @@ class HttpPassThroughEndpointHelpers(BasePassthroughUtils):
|
||||
user_api_key_spend=user_api_key_dict.spend,
|
||||
user_api_key_max_budget=user_api_key_dict.max_budget,
|
||||
user_api_key_budget_reset_at=(
|
||||
user_api_key_dict.budget_reset_at.isoformat() if user_api_key_dict.budget_reset_at else None
|
||||
user_api_key_dict.budget_reset_at.isoformat()
|
||||
if user_api_key_dict.budget_reset_at
|
||||
else None
|
||||
),
|
||||
user_api_key_auth_metadata=user_api_key_dict.metadata,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -496,12 +522,16 @@ class HttpPassThroughEndpointHelpers(BasePassthroughUtils):
|
||||
"passthrough_logging_payload": passthrough_logging_payload,
|
||||
}
|
||||
|
||||
logging_obj.model_call_details["passthrough_logging_payload"] = passthrough_logging_payload
|
||||
logging_obj.model_call_details["passthrough_logging_payload"] = (
|
||||
passthrough_logging_payload
|
||||
)
|
||||
|
||||
return kwargs
|
||||
|
||||
@staticmethod
|
||||
def construct_target_url_with_subpath(base_target: str, subpath: str, include_subpath: Optional[bool]) -> str:
|
||||
def construct_target_url_with_subpath(
|
||||
base_target: str, subpath: str, include_subpath: Optional[bool]
|
||||
) -> str:
|
||||
"""
|
||||
Helper function to construct the full target URL with subpath handling.
|
||||
|
||||
@@ -604,7 +634,9 @@ async def pass_through_request( # noqa: PLR0915
|
||||
).encode("ascii")
|
||||
)
|
||||
|
||||
endpoint_type: EndpointType = HttpPassThroughEndpointHelpers.get_endpoint_type(str(url))
|
||||
endpoint_type: EndpointType = HttpPassThroughEndpointHelpers.get_endpoint_type(
|
||||
str(url)
|
||||
)
|
||||
|
||||
if custom_body:
|
||||
_parsed_body = custom_body
|
||||
@@ -665,13 +697,15 @@ async def pass_through_request( # noqa: PLR0915
|
||||
logging_obj.model_call_details["litellm_call_id"] = litellm_call_id
|
||||
|
||||
# combine url with query params for logging
|
||||
requested_query_params: Optional[dict] = (
|
||||
query_params or dict(request.query_params)
|
||||
requested_query_params: Optional[dict] = query_params or dict(
|
||||
request.query_params
|
||||
)
|
||||
|
||||
requested_query_params_str = None
|
||||
if requested_query_params:
|
||||
requested_query_params_str = "&".join(f"{k}={v}" for k, v in requested_query_params.items())
|
||||
requested_query_params_str = "&".join(
|
||||
f"{k}={v}" for k, v in requested_query_params.items()
|
||||
)
|
||||
|
||||
logging_url = str(url)
|
||||
if requested_query_params_str:
|
||||
@@ -689,9 +723,11 @@ async def pass_through_request( # noqa: PLR0915
|
||||
"headers": headers,
|
||||
},
|
||||
)
|
||||
stream = HttpPassThroughEndpointHelpers._update_stream_param_based_on_request_body(
|
||||
parsed_body=_parsed_body,
|
||||
stream=stream,
|
||||
stream = (
|
||||
HttpPassThroughEndpointHelpers._update_stream_param_based_on_request_body(
|
||||
parsed_body=_parsed_body,
|
||||
stream=stream,
|
||||
)
|
||||
)
|
||||
|
||||
if stream:
|
||||
@@ -708,7 +744,9 @@ async def pass_through_request( # noqa: PLR0915
|
||||
try:
|
||||
response.raise_for_status()
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise HTTPException(status_code=e.response.status_code, detail=await e.response.aread())
|
||||
raise HTTPException(
|
||||
status_code=e.response.status_code, detail=await e.response.aread()
|
||||
)
|
||||
|
||||
return StreamingResponse(
|
||||
PassThroughStreamingHandler.chunk_processor(
|
||||
@@ -730,16 +768,20 @@ async def pass_through_request( # noqa: PLR0915
|
||||
verbose_proxy_logger.debug("request method: {}".format(request.method))
|
||||
verbose_proxy_logger.debug("request url: {}".format(url))
|
||||
verbose_proxy_logger.debug("request headers: {}".format(headers))
|
||||
verbose_proxy_logger.debug("requested_query_params={}".format(requested_query_params))
|
||||
verbose_proxy_logger.debug(
|
||||
"requested_query_params={}".format(requested_query_params)
|
||||
)
|
||||
verbose_proxy_logger.debug("request body: {}".format(_parsed_body))
|
||||
|
||||
response = await HttpPassThroughEndpointHelpers.non_streaming_http_request_handler(
|
||||
request=request,
|
||||
async_client=async_client,
|
||||
url=url,
|
||||
headers=headers,
|
||||
requested_query_params=requested_query_params,
|
||||
_parsed_body=_parsed_body,
|
||||
response = (
|
||||
await HttpPassThroughEndpointHelpers.non_streaming_http_request_handler(
|
||||
request=request,
|
||||
async_client=async_client,
|
||||
url=url,
|
||||
headers=headers,
|
||||
requested_query_params=requested_query_params,
|
||||
_parsed_body=_parsed_body,
|
||||
)
|
||||
)
|
||||
verbose_proxy_logger.debug("response.headers= %s", response.headers)
|
||||
|
||||
@@ -747,7 +789,9 @@ async def pass_through_request( # noqa: PLR0915
|
||||
try:
|
||||
response.raise_for_status()
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise HTTPException(status_code=e.response.status_code, detail=await e.response.aread())
|
||||
raise HTTPException(
|
||||
status_code=e.response.status_code, detail=await e.response.aread()
|
||||
)
|
||||
|
||||
return StreamingResponse(
|
||||
PassThroughStreamingHandler.chunk_processor(
|
||||
@@ -769,7 +813,9 @@ async def pass_through_request( # noqa: PLR0915
|
||||
try:
|
||||
response.raise_for_status()
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise HTTPException(status_code=e.response.status_code, detail=e.response.text)
|
||||
raise HTTPException(
|
||||
status_code=e.response.status_code, detail=e.response.text
|
||||
)
|
||||
|
||||
if response.status_code >= 300:
|
||||
raise HTTPException(status_code=response.status_code, detail=response.text)
|
||||
@@ -822,7 +868,9 @@ async def pass_through_request( # noqa: PLR0915
|
||||
api_base=str(url._uri_reference) if url else None,
|
||||
)
|
||||
verbose_proxy_logger.exception(
|
||||
"litellm.proxy.proxy_server.pass_through_endpoint(): Exception occured - {}".format(str(e))
|
||||
"litellm.proxy.proxy_server.pass_through_endpoint(): Exception occured - {}".format(
|
||||
str(e)
|
||||
)
|
||||
)
|
||||
|
||||
#########################################################
|
||||
@@ -921,12 +969,16 @@ def create_pass_through_route(
|
||||
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
|
||||
query_params: Optional[dict] = None,
|
||||
custom_body: Optional[dict] = None,
|
||||
stream: Optional[bool] = None, # if pass-through endpoint is a streaming request
|
||||
stream: Optional[
|
||||
bool
|
||||
] = None, # if pass-through endpoint is a streaming request
|
||||
subpath: str = "", # captures sub-paths when include_subpath=True
|
||||
):
|
||||
# Construct the full target URL with subpath if needed
|
||||
full_target = HttpPassThroughEndpointHelpers.construct_target_url_with_subpath(
|
||||
base_target=target, subpath=subpath, include_subpath=include_subpath
|
||||
full_target = (
|
||||
HttpPassThroughEndpointHelpers.construct_target_url_with_subpath(
|
||||
base_target=target, subpath=subpath, include_subpath=include_subpath
|
||||
)
|
||||
)
|
||||
|
||||
return await pass_through_request( # type: ignore
|
||||
@@ -1078,7 +1130,9 @@ async def websocket_passthrough_request( # noqa: PLR0915
|
||||
# Create a dummy request object for WebSocket connections to maintain compatibility
|
||||
# with the existing _init_kwargs_for_pass_through_endpoint function
|
||||
class DummyRequest:
|
||||
def __init__(self, url: str, method: str = "WEBSOCKET", headers: Optional[dict] = None):
|
||||
def __init__(
|
||||
self, url: str, method: str = "WEBSOCKET", headers: Optional[dict] = None
|
||||
):
|
||||
self.url = url
|
||||
self.method = method
|
||||
self.headers = headers or {}
|
||||
@@ -1183,9 +1237,9 @@ async def websocket_passthrough_request( # noqa: PLR0915
|
||||
)
|
||||
if extracted_model:
|
||||
kwargs["model"] = extracted_model
|
||||
kwargs[
|
||||
"custom_llm_provider"
|
||||
] = "vertex_ai-language-models"
|
||||
kwargs["custom_llm_provider"] = (
|
||||
"vertex_ai-language-models"
|
||||
)
|
||||
# Update logging object with correct model
|
||||
logging_obj.model = extracted_model
|
||||
logging_obj.model_call_details[
|
||||
@@ -1251,9 +1305,9 @@ async def websocket_passthrough_request( # noqa: PLR0915
|
||||
# Update logging object with correct model
|
||||
logging_obj.model = extracted_model
|
||||
logging_obj.model_call_details["model"] = extracted_model
|
||||
logging_obj.model_call_details[
|
||||
"custom_llm_provider"
|
||||
] = "vertex_ai_language_models"
|
||||
logging_obj.model_call_details["custom_llm_provider"] = (
|
||||
"vertex_ai_language_models"
|
||||
)
|
||||
verbose_proxy_logger.debug(
|
||||
f"WebSocket passthrough ({endpoint}): Successfully extracted model '{extracted_model}' and set provider to 'vertex_ai' from server setup response"
|
||||
)
|
||||
@@ -1597,11 +1651,15 @@ class InitPassThroughEndpointHelpers:
|
||||
def remove_endpoint_routes(endpoint_id: str):
|
||||
"""Remove all routes for a specific endpoint ID from the registry"""
|
||||
keys_to_remove = [
|
||||
key for key, value in _registered_pass_through_routes.items() if value["endpoint_id"] == endpoint_id
|
||||
key
|
||||
for key, value in _registered_pass_through_routes.items()
|
||||
if value["endpoint_id"] == endpoint_id
|
||||
]
|
||||
for key in keys_to_remove:
|
||||
del _registered_pass_through_routes[key]
|
||||
verbose_proxy_logger.debug("Removed pass-through route from registry: %s", key)
|
||||
verbose_proxy_logger.debug(
|
||||
"Removed pass-through route from registry: %s", key
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def is_registered_pass_through_route(route: str) -> bool:
|
||||
@@ -1625,11 +1683,13 @@ class InitPassThroughEndpointHelpers:
|
||||
if len(parts) == 3:
|
||||
route_type = parts[1]
|
||||
registered_path = parts[2]
|
||||
|
||||
|
||||
if route_type == "exact" and route == registered_path:
|
||||
return True
|
||||
elif route_type == "subpath":
|
||||
if route == registered_path or route.startswith(registered_path + "/"):
|
||||
if route == registered_path or route.startswith(
|
||||
registered_path + "/"
|
||||
):
|
||||
return True
|
||||
|
||||
return False
|
||||
@@ -1669,7 +1729,9 @@ async def initialize_pass_through_endpoints(
|
||||
if _path is None:
|
||||
raise ValueError("Path is required for pass-through endpoint")
|
||||
_custom_headers = endpoint.get("headers", None)
|
||||
_custom_headers = await set_env_variables_in_header(custom_headers=_custom_headers)
|
||||
_custom_headers = await set_env_variables_in_header(
|
||||
custom_headers=_custom_headers
|
||||
)
|
||||
_forward_headers = endpoint.get("forward_headers", None)
|
||||
_merge_query_params = endpoint.get("merge_query_params", None)
|
||||
_auth = endpoint.get("auth", None)
|
||||
@@ -1688,7 +1750,9 @@ async def initialize_pass_through_endpoints(
|
||||
continue
|
||||
|
||||
# Add exact path route
|
||||
verbose_proxy_logger.debug("Initializing pass through endpoint: %s (ID: %s)", _path, endpoint_id)
|
||||
verbose_proxy_logger.debug(
|
||||
"Initializing pass through endpoint: %s (ID: %s)", _path, endpoint_id
|
||||
)
|
||||
InitPassThroughEndpointHelpers.add_exact_path_route(
|
||||
app=app,
|
||||
path=_path,
|
||||
@@ -1715,7 +1779,9 @@ async def initialize_pass_through_endpoints(
|
||||
endpoint_id=endpoint_id,
|
||||
)
|
||||
|
||||
verbose_proxy_logger.debug("Added new pass through endpoint: %s (ID: %s)", _path, endpoint_id)
|
||||
verbose_proxy_logger.debug(
|
||||
"Added new pass through endpoint: %s (ID: %s)", _path, endpoint_id
|
||||
)
|
||||
|
||||
|
||||
async def _get_pass_through_endpoints_from_db(
|
||||
@@ -1819,7 +1885,11 @@ async def update_pass_through_endpoints(
|
||||
# Find the index for updating the list
|
||||
endpoint_index = None
|
||||
for idx, endpoint in enumerate(pass_through_endpoint_data):
|
||||
_endpoint = PassThroughGenericEndpoint(**endpoint) if isinstance(endpoint, dict) else endpoint
|
||||
_endpoint = (
|
||||
PassThroughGenericEndpoint(**endpoint)
|
||||
if isinstance(endpoint, dict)
|
||||
else endpoint
|
||||
)
|
||||
if _endpoint.id == endpoint_id:
|
||||
endpoint_index = idx
|
||||
break
|
||||
@@ -1827,7 +1897,9 @@ async def update_pass_through_endpoints(
|
||||
if endpoint_index is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={"error": f"Could not find index for endpoint with ID '{endpoint_id}'"},
|
||||
detail={
|
||||
"error": f"Could not find index for endpoint with ID '{endpoint_id}'"
|
||||
},
|
||||
)
|
||||
|
||||
# Get the update data as dict, excluding None values for partial updates
|
||||
@@ -1858,9 +1930,13 @@ async def update_pass_through_endpoints(
|
||||
field_value=pass_through_endpoint_data,
|
||||
config_type="general_settings",
|
||||
)
|
||||
await update_config_general_settings(data=updated_data, user_api_key_dict=user_api_key_dict)
|
||||
await update_config_general_settings(
|
||||
data=updated_data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
|
||||
return PassThroughEndpointResponse(endpoints=[updated_endpoint] if updated_endpoint else [])
|
||||
return PassThroughEndpointResponse(
|
||||
endpoints=[updated_endpoint] if updated_endpoint else []
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
@@ -1887,7 +1963,9 @@ async def create_pass_through_endpoints(
|
||||
field_name="pass_through_endpoints", user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
except Exception:
|
||||
response = ConfigFieldInfo(field_name="pass_through_endpoints", field_value=None)
|
||||
response = ConfigFieldInfo(
|
||||
field_name="pass_through_endpoints", field_value=None
|
||||
)
|
||||
|
||||
## Auto-generate ID if not provided
|
||||
data_dict = data.model_dump()
|
||||
@@ -1905,7 +1983,9 @@ async def create_pass_through_endpoints(
|
||||
field_value=response.field_value,
|
||||
config_type="general_settings",
|
||||
)
|
||||
await update_config_general_settings(data=updated_data, user_api_key_dict=user_api_key_dict)
|
||||
await update_config_general_settings(
|
||||
data=updated_data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
|
||||
# Return the created endpoint with the generated ID
|
||||
created_endpoint = PassThroughGenericEndpoint(**data_dict)
|
||||
@@ -1938,7 +2018,9 @@ async def delete_pass_through_endpoints(
|
||||
field_name="pass_through_endpoints", user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
except Exception:
|
||||
response = ConfigFieldInfo(field_name="pass_through_endpoints", field_value=None)
|
||||
response = ConfigFieldInfo(
|
||||
field_name="pass_through_endpoints", field_value=None
|
||||
)
|
||||
|
||||
## Update field by removing endpoint
|
||||
pass_through_endpoint_data: Optional[List] = response.field_value
|
||||
@@ -1954,13 +2036,21 @@ async def delete_pass_through_endpoints(
|
||||
if found_endpoint is None:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={"error": "Endpoint with ID '{}' was not found in pass-through endpoint list.".format(endpoint_id)},
|
||||
detail={
|
||||
"error": "Endpoint with ID '{}' was not found in pass-through endpoint list.".format(
|
||||
endpoint_id
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
# Find the index for deleting from the list
|
||||
endpoint_index = None
|
||||
for idx, endpoint in enumerate(pass_through_endpoint_data):
|
||||
_endpoint = PassThroughGenericEndpoint(**endpoint) if isinstance(endpoint, dict) else endpoint
|
||||
_endpoint = (
|
||||
PassThroughGenericEndpoint(**endpoint)
|
||||
if isinstance(endpoint, dict)
|
||||
else endpoint
|
||||
)
|
||||
if _endpoint.id == endpoint_id:
|
||||
endpoint_index = idx
|
||||
break
|
||||
@@ -1968,7 +2058,9 @@ async def delete_pass_through_endpoints(
|
||||
if endpoint_index is None:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={"error": f"Could not find index for endpoint with ID '{endpoint_id}'"},
|
||||
detail={
|
||||
"error": f"Could not find index for endpoint with ID '{endpoint_id}'"
|
||||
},
|
||||
)
|
||||
|
||||
# Remove the endpoint
|
||||
@@ -1984,7 +2076,9 @@ async def delete_pass_through_endpoints(
|
||||
field_value=pass_through_endpoint_data,
|
||||
config_type="general_settings",
|
||||
)
|
||||
await update_config_general_settings(data=updated_data, user_api_key_dict=user_api_key_dict)
|
||||
await update_config_general_settings(
|
||||
data=updated_data, user_api_key_dict=user_api_key_dict
|
||||
)
|
||||
|
||||
return PassThroughEndpointResponse(endpoints=[response_obj])
|
||||
|
||||
@@ -2022,4 +2116,6 @@ async def initialize_pass_through_endpoints_in_db():
|
||||
Gets all pass-through endpoints from db and initializes them in the proxy server.
|
||||
"""
|
||||
pass_through_endpoints = await _get_pass_through_endpoints_from_db()
|
||||
await initialize_pass_through_endpoints(pass_through_endpoints=pass_through_endpoints)
|
||||
await initialize_pass_through_endpoints(
|
||||
pass_through_endpoints=pass_through_endpoints
|
||||
)
|
||||
|
||||
@@ -253,9 +253,7 @@ from litellm.proxy.management_endpoints.customer_endpoints import (
|
||||
from litellm.proxy.management_endpoints.internal_user_endpoints import (
|
||||
router as internal_user_router,
|
||||
)
|
||||
from litellm.proxy.management_endpoints.internal_user_endpoints import (
|
||||
user_update,
|
||||
)
|
||||
from litellm.proxy.management_endpoints.internal_user_endpoints import user_update
|
||||
from litellm.proxy.management_endpoints.key_management_endpoints import (
|
||||
delete_verification_tokens,
|
||||
duration_in_seconds,
|
||||
@@ -302,9 +300,7 @@ from litellm.proxy.middleware.prometheus_auth_middleware import PrometheusAuthMi
|
||||
from litellm.proxy.openai_files_endpoints.files_endpoints import (
|
||||
router as openai_files_router,
|
||||
)
|
||||
from litellm.proxy.openai_files_endpoints.files_endpoints import (
|
||||
set_files_config,
|
||||
)
|
||||
from litellm.proxy.openai_files_endpoints.files_endpoints import set_files_config
|
||||
from litellm.proxy.pass_through_endpoints.llm_passthrough_endpoints import (
|
||||
passthrough_endpoint_router,
|
||||
)
|
||||
@@ -467,9 +463,9 @@ except ImportError:
|
||||
server_root_path = os.getenv("SERVER_ROOT_PATH", "")
|
||||
_license_check = LicenseCheck()
|
||||
premium_user: bool = _license_check.is_premium()
|
||||
premium_user_data: Optional[
|
||||
"EnterpriseLicenseData"
|
||||
] = _license_check.airgapped_license_data
|
||||
premium_user_data: Optional["EnterpriseLicenseData"] = (
|
||||
_license_check.airgapped_license_data
|
||||
)
|
||||
global_max_parallel_request_retries_env: Optional[str] = os.getenv(
|
||||
"LITELLM_GLOBAL_MAX_PARALLEL_REQUEST_RETRIES"
|
||||
)
|
||||
@@ -966,9 +962,9 @@ model_max_budget_limiter = _PROXY_VirtualKeyModelMaxBudgetLimiter(
|
||||
dual_cache=user_api_key_cache
|
||||
)
|
||||
litellm.logging_callback_manager.add_litellm_callback(model_max_budget_limiter)
|
||||
redis_usage_cache: Optional[
|
||||
RedisCache
|
||||
] = None # redis cache used for tracking spend, tpm/rpm limits
|
||||
redis_usage_cache: Optional[RedisCache] = (
|
||||
None # redis cache used for tracking spend, tpm/rpm limits
|
||||
)
|
||||
user_custom_auth = None
|
||||
user_custom_key_generate = None
|
||||
user_custom_sso = None
|
||||
@@ -1299,9 +1295,9 @@ async def update_cache( # noqa: PLR0915
|
||||
_id = "team_id:{}".format(team_id)
|
||||
try:
|
||||
# Fetch the existing cost for the given user
|
||||
existing_spend_obj: Optional[
|
||||
LiteLLM_TeamTable
|
||||
] = await user_api_key_cache.async_get_cache(key=_id)
|
||||
existing_spend_obj: Optional[LiteLLM_TeamTable] = (
|
||||
await user_api_key_cache.async_get_cache(key=_id)
|
||||
)
|
||||
if existing_spend_obj is None:
|
||||
# do nothing if team not in api key cache
|
||||
return
|
||||
@@ -1878,9 +1874,7 @@ class ProxyConfig:
|
||||
f"{blue_color_code}Set Global BitBucket Config on LiteLLM Proxy{reset_color_code}"
|
||||
)
|
||||
elif key == "global_gitlab_config":
|
||||
from litellm.integrations.gitlab import (
|
||||
set_global_gitlab_config,
|
||||
)
|
||||
from litellm.integrations.gitlab import set_global_gitlab_config
|
||||
|
||||
set_global_gitlab_config(value)
|
||||
verbose_proxy_logger.info(
|
||||
@@ -2541,10 +2535,14 @@ class ProxyConfig:
|
||||
_model_list: list = []
|
||||
for m in new_models:
|
||||
_litellm_params = m.litellm_params
|
||||
if isinstance(_litellm_params, BaseModel):
|
||||
_litellm_params = _litellm_params.model_dump()
|
||||
if isinstance(_litellm_params, dict):
|
||||
# decrypt values
|
||||
for k, v in _litellm_params.items():
|
||||
decrypted_value = decrypt_value_helper(value=v, key=k)
|
||||
decrypted_value = decrypt_value_helper(
|
||||
value=v, key=k, return_original_value=True
|
||||
)
|
||||
_litellm_params[k] = decrypted_value
|
||||
_litellm_params = LiteLLM_Params(**_litellm_params)
|
||||
else:
|
||||
@@ -2628,7 +2626,7 @@ class ProxyConfig:
|
||||
) -> None:
|
||||
"""
|
||||
Helper method to add a single callback to litellm for specified event types.
|
||||
|
||||
|
||||
Args:
|
||||
callback: The callback name to add
|
||||
event_types: List of event types (e.g., ["success"], ["failure"], or ["success", "failure"])
|
||||
@@ -3153,10 +3151,10 @@ class ProxyConfig:
|
||||
)
|
||||
|
||||
try:
|
||||
guardrails_in_db: List[
|
||||
Guardrail
|
||||
] = await GuardrailRegistry.get_all_guardrails_from_db(
|
||||
prisma_client=prisma_client
|
||||
guardrails_in_db: List[Guardrail] = (
|
||||
await GuardrailRegistry.get_all_guardrails_from_db(
|
||||
prisma_client=prisma_client
|
||||
)
|
||||
)
|
||||
verbose_proxy_logger.debug(
|
||||
"guardrails from the DB %s", str(guardrails_in_db)
|
||||
@@ -3386,9 +3384,9 @@ async def initialize( # noqa: PLR0915
|
||||
user_api_base = api_base
|
||||
dynamic_config[user_model]["api_base"] = api_base
|
||||
if api_version:
|
||||
os.environ[
|
||||
"AZURE_API_VERSION"
|
||||
] = api_version # set this for azure - litellm can read this from the env
|
||||
os.environ["AZURE_API_VERSION"] = (
|
||||
api_version # set this for azure - litellm can read this from the env
|
||||
)
|
||||
if max_tokens: # model-specific param
|
||||
dynamic_config[user_model]["max_tokens"] = max_tokens
|
||||
if temperature: # model-specific param
|
||||
@@ -3888,10 +3886,10 @@ class ProxyStartupEvent:
|
||||
LITELLM_KEY_ROTATION_CHECK_INTERVAL_SECONDS,
|
||||
LITELLM_KEY_ROTATION_ENABLED,
|
||||
)
|
||||
|
||||
|
||||
key_rotation_enabled: Optional[bool] = str_to_bool(LITELLM_KEY_ROTATION_ENABLED)
|
||||
verbose_proxy_logger.debug(f"key_rotation_enabled: {key_rotation_enabled}")
|
||||
|
||||
|
||||
if key_rotation_enabled is True:
|
||||
try:
|
||||
from litellm.proxy.common_utils.key_rotation_manager import (
|
||||
@@ -3902,19 +3900,25 @@ class ProxyStartupEvent:
|
||||
global prisma_client
|
||||
if prisma_client is not None:
|
||||
key_rotation_manager = KeyRotationManager(prisma_client)
|
||||
verbose_proxy_logger.debug(f"Key rotation background job scheduled every {LITELLM_KEY_ROTATION_CHECK_INTERVAL_SECONDS} seconds (LITELLM_KEY_ROTATION_ENABLED=true)")
|
||||
verbose_proxy_logger.debug(
|
||||
f"Key rotation background job scheduled every {LITELLM_KEY_ROTATION_CHECK_INTERVAL_SECONDS} seconds (LITELLM_KEY_ROTATION_ENABLED=true)"
|
||||
)
|
||||
scheduler.add_job(
|
||||
key_rotation_manager.process_rotations,
|
||||
"interval",
|
||||
seconds=LITELLM_KEY_ROTATION_CHECK_INTERVAL_SECONDS,
|
||||
id="key_rotation_job"
|
||||
id="key_rotation_job",
|
||||
)
|
||||
else:
|
||||
verbose_proxy_logger.warning("Key rotation enabled but prisma_client not available")
|
||||
verbose_proxy_logger.warning(
|
||||
"Key rotation enabled but prisma_client not available"
|
||||
)
|
||||
except Exception as e:
|
||||
verbose_proxy_logger.warning(f"Failed to setup key rotation job: {e}")
|
||||
else:
|
||||
verbose_proxy_logger.debug("Key rotation disabled (set LITELLM_KEY_ROTATION_ENABLED=true to enable)")
|
||||
verbose_proxy_logger.debug(
|
||||
"Key rotation disabled (set LITELLM_KEY_ROTATION_ENABLED=true to enable)"
|
||||
)
|
||||
|
||||
@classmethod
|
||||
async def _setup_prisma_client(
|
||||
@@ -8745,9 +8749,9 @@ async def get_config_list(
|
||||
hasattr(sub_field_info, "description")
|
||||
and sub_field_info.description is not None
|
||||
):
|
||||
nested_fields[
|
||||
idx
|
||||
].field_description = sub_field_info.description
|
||||
nested_fields[idx].field_description = (
|
||||
sub_field_info.description
|
||||
)
|
||||
idx += 1
|
||||
|
||||
_stored_in_db = None
|
||||
|
||||
@@ -179,6 +179,7 @@ model LiteLLM_MCPServerTable {
|
||||
mcp_info Json? @default("{}")
|
||||
mcp_access_groups String[]
|
||||
allowed_tools String[] @default([])
|
||||
extra_headers String[] @default([])
|
||||
// Health check status
|
||||
status String? @default("unknown")
|
||||
last_health_check DateTime?
|
||||
|
||||
+38
-31
@@ -1395,9 +1395,12 @@ class ProxyLogging:
|
||||
3. /image/generation
|
||||
4. /files
|
||||
"""
|
||||
from litellm.types.guardrails import GuardrailEventHooks
|
||||
|
||||
for callback in litellm.callbacks:
|
||||
try:
|
||||
guardrail_callbacks: List[CustomGuardrail] = []
|
||||
other_callbacks: List[CustomLogger] = []
|
||||
try:
|
||||
for callback in litellm.callbacks:
|
||||
_callback: Optional[CustomLogger] = None
|
||||
if isinstance(callback, str):
|
||||
_callback = litellm.litellm_core_utils.litellm_logging.get_custom_logger_compatible_class(
|
||||
@@ -1407,36 +1410,37 @@ class ProxyLogging:
|
||||
_callback = callback # type: ignore
|
||||
|
||||
if _callback is not None:
|
||||
if isinstance(_callback, CustomGuardrail):
|
||||
guardrail_callbacks.append(_callback)
|
||||
else:
|
||||
other_callbacks.append(_callback)
|
||||
############## Handle Guardrails ########################################
|
||||
#############################################################################
|
||||
if isinstance(callback, CustomGuardrail):
|
||||
# Main - V2 Guardrails implementation
|
||||
from litellm.types.guardrails import GuardrailEventHooks
|
||||
|
||||
if (
|
||||
callback.should_run_guardrail(
|
||||
data=data, event_type=GuardrailEventHooks.post_call
|
||||
)
|
||||
is not True
|
||||
):
|
||||
continue
|
||||
for callback in guardrail_callbacks:
|
||||
# Main - V2 Guardrails implementation
|
||||
if (
|
||||
callback.should_run_guardrail(
|
||||
data=data, event_type=GuardrailEventHooks.post_call
|
||||
)
|
||||
is not True
|
||||
):
|
||||
continue
|
||||
|
||||
await callback.async_post_call_success_hook(
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
data=data,
|
||||
response=response,
|
||||
)
|
||||
await callback.async_post_call_success_hook(
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
data=data,
|
||||
response=response,
|
||||
)
|
||||
|
||||
############ Handle CustomLogger ###############################
|
||||
#################################################################
|
||||
elif isinstance(_callback, CustomLogger):
|
||||
await _callback.async_post_call_success_hook(
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
data=data,
|
||||
response=response,
|
||||
)
|
||||
except Exception as e:
|
||||
raise e
|
||||
############ Handle CustomLogger ###############################
|
||||
#################################################################
|
||||
for callback in other_callbacks:
|
||||
await callback.async_post_call_success_hook(
|
||||
user_api_key_dict=user_api_key_dict, data=data, response=response
|
||||
)
|
||||
except Exception as e:
|
||||
raise e
|
||||
return response
|
||||
|
||||
async def async_post_call_streaming_hook(
|
||||
@@ -3571,18 +3575,21 @@ def handle_exception_on_proxy(e: Exception) -> ProxyException:
|
||||
)
|
||||
|
||||
|
||||
def _premium_user_check():
|
||||
def _premium_user_check(feature: Optional[str] = None):
|
||||
"""
|
||||
Raises an HTTPException if the user is not a premium user
|
||||
"""
|
||||
from litellm.proxy.proxy_server import premium_user
|
||||
|
||||
if feature:
|
||||
detail_msg = f"This feature is only available for LiteLLM Enterprise users: {feature}. {CommonProxyErrors.not_premium_user.value}"
|
||||
else:
|
||||
detail_msg = f"This feature is only available for LiteLLM Enterprise users. {CommonProxyErrors.not_premium_user.value}"
|
||||
|
||||
if not premium_user:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail={
|
||||
"error": f"This feature is only available for LiteLLM Enterprise users. {CommonProxyErrors.not_premium_user.value}"
|
||||
},
|
||||
detail={"error": detail_msg},
|
||||
)
|
||||
|
||||
|
||||
|
||||
+55
-16
@@ -416,6 +416,9 @@ class Router:
|
||||
|
||||
# Initialize model ID to deployment index mapping for O(1) lookups
|
||||
self.model_id_to_deployment_index_map: Dict[str, int] = {}
|
||||
# Initialize model name to deployment indices mapping for O(1) lookups
|
||||
# Maps model_name -> list of indices in model_list
|
||||
self.model_name_to_deployment_indices: Dict[str, List[int]] = {}
|
||||
|
||||
if model_list is not None:
|
||||
# Build model index immediately to enable O(1) lookups from the start
|
||||
@@ -5097,6 +5100,7 @@ class Router:
|
||||
original_model_list = copy.deepcopy(model_list)
|
||||
self.model_list = []
|
||||
self.model_id_to_deployment_index_map = {} # Reset the index
|
||||
self.model_name_to_deployment_indices = {} # Reset the model_name index
|
||||
# we add api_base/api_key each model so load balancing between azure/gpt on api_base1 and api_base2 works
|
||||
|
||||
for model in original_model_list:
|
||||
@@ -5138,6 +5142,9 @@ class Router:
|
||||
f"\nInitialized Model List {self.get_model_names()}"
|
||||
)
|
||||
self.model_names = [m["model_name"] for m in model_list]
|
||||
|
||||
# Build model_name index for O(1) lookups
|
||||
self._build_model_name_index(self.model_list)
|
||||
|
||||
def _add_deployment(self, deployment: Deployment) -> Deployment:
|
||||
import os
|
||||
@@ -5365,20 +5372,27 @@ class Router:
|
||||
self, model: dict, model_id: Optional[str] = None
|
||||
) -> None:
|
||||
"""
|
||||
Helper method to add a model to the model_list and update the model_id_to_deployment_index_map.
|
||||
Helper method to add a model to the model_list and update both indices.
|
||||
|
||||
Parameters:
|
||||
- model: dict - the model to add to the list
|
||||
- model_id: Optional[str] - the model ID to use for indexing. If None, will try to get from model["model_info"]["id"]
|
||||
"""
|
||||
idx = len(self.model_list)
|
||||
self.model_list.append(model)
|
||||
# Update model index for O(1) lookup
|
||||
|
||||
# Update model_id index for O(1) lookup
|
||||
if model_id is not None:
|
||||
self.model_id_to_deployment_index_map[model_id] = len(self.model_list) - 1
|
||||
self.model_id_to_deployment_index_map[model_id] = idx
|
||||
elif model.get("model_info", {}).get("id") is not None:
|
||||
self.model_id_to_deployment_index_map[model["model_info"]["id"]] = (
|
||||
len(self.model_list) - 1
|
||||
)
|
||||
self.model_id_to_deployment_index_map[model["model_info"]["id"]] = idx
|
||||
|
||||
# Update model_name index for O(1) lookup
|
||||
model_name = model.get("model_name")
|
||||
if model_name:
|
||||
if model_name not in self.model_name_to_deployment_indices:
|
||||
self.model_name_to_deployment_indices[model_name] = []
|
||||
self.model_name_to_deployment_indices[model_name].append(idx)
|
||||
|
||||
def upsert_deployment(self, deployment: Deployment) -> Optional[Deployment]:
|
||||
"""
|
||||
@@ -6094,6 +6108,22 @@ class Router:
|
||||
additional_headers[header] = value
|
||||
return response
|
||||
|
||||
def _build_model_name_index(self, model_list: list) -> None:
|
||||
"""
|
||||
Build model_name -> deployment indices mapping for O(1) lookups.
|
||||
|
||||
This index allows us to find all deployments for a given model_name in O(1) time
|
||||
instead of O(n) linear scan through the entire model_list.
|
||||
"""
|
||||
self.model_name_to_deployment_indices.clear()
|
||||
|
||||
for idx, model in enumerate(model_list):
|
||||
model_name = model.get("model_name")
|
||||
if model_name:
|
||||
if model_name not in self.model_name_to_deployment_indices:
|
||||
self.model_name_to_deployment_indices[model_name] = []
|
||||
self.model_name_to_deployment_indices[model_name].append(idx)
|
||||
|
||||
def _build_model_id_to_deployment_index_map(self, model_list: list):
|
||||
"""
|
||||
Build model index from model list to enable O(1) lookups immediately.
|
||||
@@ -6198,18 +6228,27 @@ class Router:
|
||||
Used for accurate 'get_model_list'.
|
||||
|
||||
if team_id specified, only return team-specific models
|
||||
|
||||
Optimized with O(1) index lookup instead of O(n) linear scan.
|
||||
"""
|
||||
returned_models: List[DeploymentTypedDict] = []
|
||||
for model in self.model_list:
|
||||
if self.should_include_deployment(
|
||||
model_name=model_name, model=model, team_id=team_id
|
||||
):
|
||||
if model_alias is not None:
|
||||
alias_model = copy.deepcopy(model)
|
||||
alias_model["model_name"] = model_alias
|
||||
returned_models.append(alias_model)
|
||||
else:
|
||||
returned_models.append(model)
|
||||
|
||||
# O(1) lookup in model_name index
|
||||
if model_name in self.model_name_to_deployment_indices:
|
||||
indices = self.model_name_to_deployment_indices[model_name]
|
||||
|
||||
# O(k) where k = deployments for this model_name (typically 1-10)
|
||||
for idx in indices:
|
||||
model = self.model_list[idx]
|
||||
if self.should_include_deployment(
|
||||
model_name=model_name, model=model, team_id=team_id
|
||||
):
|
||||
if model_alias is not None:
|
||||
alias_model = copy.deepcopy(model)
|
||||
alias_model["model_name"] = model_alias
|
||||
returned_models.append(alias_model)
|
||||
else:
|
||||
returned_models.append(model)
|
||||
|
||||
return returned_models
|
||||
|
||||
|
||||
@@ -426,13 +426,13 @@ class PrometheusMetricLabels:
|
||||
|
||||
# Buffer monitoring metrics - these typically don't need additional labels
|
||||
litellm_pod_lock_manager_size: List[str] = []
|
||||
|
||||
|
||||
litellm_in_memory_daily_spend_update_queue_size: List[str] = []
|
||||
|
||||
|
||||
litellm_redis_daily_spend_update_queue_size: List[str] = []
|
||||
|
||||
|
||||
litellm_in_memory_spend_update_queue_size: List[str] = []
|
||||
|
||||
|
||||
litellm_redis_spend_update_queue_size: List[str] = []
|
||||
|
||||
@staticmethod
|
||||
|
||||
@@ -1867,6 +1867,7 @@ class StandardLoggingUserAPIKeyMetadata(TypedDict):
|
||||
user_api_key_team_alias: Optional[str]
|
||||
user_api_key_end_user_id: Optional[str]
|
||||
user_api_key_request_route: Optional[str]
|
||||
user_api_key_auth_metadata: Optional[Dict[str, str]]
|
||||
|
||||
|
||||
class StandardLoggingMCPToolCall(TypedDict, total=False):
|
||||
@@ -2077,10 +2078,12 @@ class StandardLoggingGuardrailInformation(TypedDict, total=False):
|
||||
|
||||
StandardLoggingPayloadStatus = Literal["success", "failure"]
|
||||
|
||||
|
||||
class CachingDetails(TypedDict):
|
||||
"""
|
||||
Track all caching related metrics, fields for a given request
|
||||
"""
|
||||
|
||||
cache_hit: Optional[bool]
|
||||
"""
|
||||
Whether the request hit the cache
|
||||
@@ -2090,12 +2093,16 @@ class CachingDetails(TypedDict):
|
||||
Duration for reading from cache
|
||||
"""
|
||||
|
||||
|
||||
class CostBreakdown(TypedDict):
|
||||
"""
|
||||
Detailed cost breakdown for a request
|
||||
"""
|
||||
|
||||
input_cost: float # Cost of input/prompt tokens
|
||||
output_cost: float # Cost of output/completion tokens (includes reasoning if applicable)
|
||||
output_cost: (
|
||||
float # Cost of output/completion tokens (includes reasoning if applicable)
|
||||
)
|
||||
total_cost: float # Total cost (input + output + tool usage)
|
||||
tool_usage_cost: float # Cost of usage of built-in tools
|
||||
|
||||
|
||||
@@ -3324,28 +3324,27 @@
|
||||
"supports_web_search": true
|
||||
},
|
||||
"azure_ai/grok-4-fast-non-reasoning": {
|
||||
"input_cost_per_token": 5e-06,
|
||||
"input_cost_per_token": 0.43e-06,
|
||||
"output_cost_per_token": 1.73e-06,
|
||||
"litellm_provider": "azure_ai",
|
||||
"max_input_tokens": 131072,
|
||||
"max_output_tokens": 131072,
|
||||
"max_tokens": 131072,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 2.5e-03,
|
||||
"source": "https://azure.microsoft.com/en-us/blog/grok-4-is-now-available-in-azure-ai-foundry-unlock-frontier-intelligence-and-business-ready-capabilities/",
|
||||
"supports_function_calling": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_web_search": true
|
||||
},
|
||||
"azure_ai/grok-4-fast-reasoning": {
|
||||
"input_cost_per_token": 5.8e-06,
|
||||
"input_cost_per_token": 0.43e-06,
|
||||
"output_cost_per_token": 1.73e-06,
|
||||
"litellm_provider": "azure_ai",
|
||||
"max_input_tokens": 131072,
|
||||
"max_output_tokens": 131072,
|
||||
"max_tokens": 131072,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 2.9e-03,
|
||||
"source": "https://azure.microsoft.com/en-us/blog/grok-4-is-now-available-in-azure-ai-foundry-unlock-frontier-intelligence-and-business-ready-capabilities/",
|
||||
"source": "https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/announcing-the-grok-4-fast-models-from-xai-now-available-in-azure-ai-foundry/4456701",
|
||||
"supports_function_calling": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
@@ -22174,6 +22173,307 @@
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/bigscience/mt0-xxl-13b": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.0005,
|
||||
"output_cost_per_token": 0.002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/core42/jais-13b-chat": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.0005,
|
||||
"output_cost_per_token": 0.002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/google/flan-t5-xl-3b": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.0001,
|
||||
"output_cost_per_token": 0.00025,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-13b-chat-v2": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.0005,
|
||||
"output_cost_per_token": 0.002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-13b-instruct-v2": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.0005,
|
||||
"output_cost_per_token": 0.002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-3-3-8b-instruct": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.00025,
|
||||
"output_cost_per_token": 0.001,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-4-h-small": {
|
||||
"max_tokens": 20480,
|
||||
"max_input_tokens": 20480,
|
||||
"max_output_tokens": 20480,
|
||||
"input_cost_per_token": 0.000625,
|
||||
"output_cost_per_token": 0.0025,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-guardian-3-2-2b": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.00015,
|
||||
"output_cost_per_token": 0.0006,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-guardian-3-3-8b": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.00025,
|
||||
"output_cost_per_token": 0.001,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-ttm-1024-96-r2": {
|
||||
"max_tokens": 512,
|
||||
"max_input_tokens": 512,
|
||||
"max_output_tokens": 512,
|
||||
"input_cost_per_token": 0.000625,
|
||||
"output_cost_per_token": 0.000625,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-ttm-1536-96-r2": {
|
||||
"max_tokens": 512,
|
||||
"max_input_tokens": 512,
|
||||
"max_output_tokens": 512,
|
||||
"input_cost_per_token": 0.000625,
|
||||
"output_cost_per_token": 0.000625,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-ttm-512-96-r2": {
|
||||
"max_tokens": 512,
|
||||
"max_input_tokens": 512,
|
||||
"max_output_tokens": 512,
|
||||
"input_cost_per_token": 0.000625,
|
||||
"output_cost_per_token": 0.000625,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/ibm/granite-vision-3-2-2b": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.00015,
|
||||
"output_cost_per_token": 0.0006,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": true
|
||||
},
|
||||
"watsonx/meta-llama/llama-3-2-11b-vision-instruct": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.00025,
|
||||
"output_cost_per_token": 0.001,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"watsonx/meta-llama/llama-3-2-1b-instruct": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.0001,
|
||||
"output_cost_per_token": 0.0002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/meta-llama/llama-3-2-3b-instruct": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.00015,
|
||||
"output_cost_per_token": 0.0006,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/meta-llama/llama-3-2-90b-vision-instruct": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.002,
|
||||
"output_cost_per_token": 0.008,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"watsonx/meta-llama/llama-3-3-70b-instruct": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.002,
|
||||
"output_cost_per_token": 0.006,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/meta-llama/llama-4-maverick-17b": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.0005,
|
||||
"output_cost_per_token": 0.002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/meta-llama/llama-guard-3-11b-vision": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.00025,
|
||||
"output_cost_per_token": 0.001,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": true
|
||||
},
|
||||
"watsonx/mistralai/mistral-medium-2505": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.00225,
|
||||
"output_cost_per_token": 0.00675,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/mistralai/mistral-small-2503": {
|
||||
"max_tokens": 32000,
|
||||
"max_input_tokens": 32000,
|
||||
"max_output_tokens": 32000,
|
||||
"input_cost_per_token": 0.0002,
|
||||
"output_cost_per_token": 0.0006,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/mistralai/pixtral-12b-2409": {
|
||||
"max_tokens": 128000,
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"input_cost_per_token": 0.00015,
|
||||
"output_cost_per_token": 0.00015,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": true
|
||||
},
|
||||
"watsonx/openai/gpt-oss-120b": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.004,
|
||||
"output_cost_per_token": 0.016,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
"watsonx/sdaia/allam-1-13b-instruct": {
|
||||
"max_tokens": 8192,
|
||||
"max_input_tokens": 8192,
|
||||
"max_output_tokens": 8192,
|
||||
"input_cost_per_token": 0.0005,
|
||||
"output_cost_per_token": 0.002,
|
||||
"litellm_provider": "watsonx",
|
||||
"mode": "chat",
|
||||
"supports_function_calling": false,
|
||||
"supports_parallel_function_calling": false,
|
||||
"supports_vision": false
|
||||
},
|
||||
|
||||
"whisper-1": {
|
||||
"input_cost_per_second": 0.0001,
|
||||
"litellm_provider": "openai",
|
||||
|
||||
Generated
+4
-4
@@ -3804,15 +3804,15 @@ files = [
|
||||
|
||||
[[package]]
|
||||
name = "litellm-proxy-extras"
|
||||
version = "0.2.22"
|
||||
version = "0.2.25"
|
||||
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
|
||||
optional = true
|
||||
python-versions = "!=2.7.*,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,!=3.7.*,>=3.8"
|
||||
groups = ["main"]
|
||||
markers = "extra == \"proxy\""
|
||||
files = [
|
||||
{file = "litellm_proxy_extras-0.2.22-py3-none-any.whl", hash = "sha256:e64b19b48e8d84cad56bb136c7f31d9ae601a10628327c922634d7081803c205"},
|
||||
{file = "litellm_proxy_extras-0.2.22.tar.gz", hash = "sha256:59c395bff3353de57d67b7637e8ce0a8a4e096ce55e2ee2df4d9d4bda94f6ef0"},
|
||||
{file = "litellm_proxy_extras-0.2.25-py3-none-any.whl", hash = "sha256:334ac3c04511258e2cbbd7a1ddb6e30619a6e693b267db92033732f4d981baab"},
|
||||
{file = "litellm_proxy_extras-0.2.25.tar.gz", hash = "sha256:9cf363570a5dc3349bea6ad1fba00ce9aeb90232fc69adc32881e53bec2cbf8f"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -9598,4 +9598,4 @@ utils = ["numpydoc"]
|
||||
[metadata]
|
||||
lock-version = "2.1"
|
||||
python-versions = ">=3.8.1,<4.0, !=3.9.7"
|
||||
content-hash = "dd6b1b42d43c2049fd8fcc95a6627581c5d9c60b3afd5eab60659d8f5d6ae641"
|
||||
content-hash = "ef5f8d965a4d77f6ae7d424306e2c88082708bc7e374896e6b022a13ce7c1962"
|
||||
|
||||
+1
-1
@@ -59,7 +59,7 @@ websockets = {version = "^13.1.0", optional = true}
|
||||
boto3 = {version = "1.36.0", optional = true}
|
||||
redisvl = {version = "^0.4.1", optional = true, markers = "python_version >= '3.9' and python_version < '3.14'"}
|
||||
mcp = {version = "^1.10.0", optional = true, python = ">=3.10"}
|
||||
litellm-proxy-extras = {version = "0.2.22", optional = true}
|
||||
litellm-proxy-extras = {version = "0.2.25", optional = true}
|
||||
rich = {version = "13.7.1", optional = true}
|
||||
litellm-enterprise = {version = "0.1.20", optional = true}
|
||||
diskcache = {version = "^5.6.1", optional = true}
|
||||
|
||||
+1
-1
@@ -43,7 +43,7 @@ sentry_sdk==2.21.0 # for sentry error handling
|
||||
detect-secrets==1.5.0 # Enterprise - secret detection / masking in LLM requests
|
||||
cryptography==44.0.1
|
||||
tzdata==2025.1 # IANA time zone database
|
||||
litellm-proxy-extras==0.2.22 # for proxy extras - e.g. prisma migrations
|
||||
litellm-proxy-extras==0.2.25 # for proxy extras - e.g. prisma migrations
|
||||
### LITELLM PACKAGE DEPENDENCIES
|
||||
python-dotenv==1.0.1 # for env
|
||||
tiktoken==0.8.0 # for calculating usage
|
||||
|
||||
@@ -179,6 +179,7 @@ model LiteLLM_MCPServerTable {
|
||||
mcp_info Json? @default("{}")
|
||||
mcp_access_groups String[]
|
||||
allowed_tools String[] @default([])
|
||||
extra_headers String[] @default([])
|
||||
// Health check status
|
||||
status String? @default("unknown")
|
||||
last_health_check DateTime?
|
||||
|
||||
+132
-99
@@ -6,7 +6,6 @@ sys.path.insert(0, os.path.abspath("../.."))
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from litellm._uuid import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from unittest.mock import MagicMock, call, patch
|
||||
|
||||
@@ -16,6 +15,7 @@ from prometheus_client import REGISTRY, CollectorRegistry
|
||||
import litellm
|
||||
from litellm import completion
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm._uuid import uuid
|
||||
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
|
||||
from litellm.types.utils import (
|
||||
StandardLoggingHiddenParams,
|
||||
@@ -1033,10 +1033,10 @@ def test_deployment_state_management(prometheus_logger):
|
||||
# Test set_deployment_healthy (state=0)
|
||||
prometheus_logger.set_deployment_healthy(**test_params)
|
||||
prometheus_logger.litellm_deployment_state.labels.assert_called_with(
|
||||
test_params["litellm_model_name"],
|
||||
test_params["model_id"],
|
||||
test_params["api_base"],
|
||||
test_params["api_provider"],
|
||||
litellm_model_name=test_params["litellm_model_name"],
|
||||
model_id=test_params["model_id"],
|
||||
api_base=test_params["api_base"],
|
||||
api_provider=test_params["api_provider"],
|
||||
)
|
||||
prometheus_logger.litellm_deployment_state.labels().set.assert_called_with(0)
|
||||
|
||||
@@ -1153,22 +1153,28 @@ def test_get_custom_labels_from_tags_wildcard_patterns(monkeypatch):
|
||||
|
||||
# Configure tags with wildcard patterns
|
||||
monkeypatch.setattr(
|
||||
"litellm.custom_prometheus_tags",
|
||||
["User-Agent: curl/*", "User-Agent: python-requests/*", "Environment: prod*", "Service: api-gateway*", "exact-match"]
|
||||
"litellm.custom_prometheus_tags",
|
||||
[
|
||||
"User-Agent: curl/*",
|
||||
"User-Agent: python-requests/*",
|
||||
"Environment: prod*",
|
||||
"Service: api-gateway*",
|
||||
"exact-match",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
# Test tags that should match the wildcard patterns
|
||||
tags = [
|
||||
"User-Agent: curl/7.68.0",
|
||||
"User-Agent: python-requests/2.28.1",
|
||||
"User-Agent: curl/7.68.0",
|
||||
"User-Agent: python-requests/2.28.1",
|
||||
"Environment: production",
|
||||
"Service: api-gateway-v2",
|
||||
"exact-match",
|
||||
"other-tag"
|
||||
"other-tag",
|
||||
]
|
||||
|
||||
|
||||
result = get_custom_labels_from_tags(tags)
|
||||
|
||||
|
||||
expected = {
|
||||
"tag_User_Agent__curl__": "true", # matches "User-Agent: curl/*"
|
||||
"tag_User_Agent__python_requests__": "true", # matches "User-Agent: python-requests/*"
|
||||
@@ -1176,7 +1182,7 @@ def test_get_custom_labels_from_tags_wildcard_patterns(monkeypatch):
|
||||
"tag_Service__api_gateway_": "true", # matches "Service: api-gateway*"
|
||||
"tag_exact_match": "true", # exact match
|
||||
}
|
||||
|
||||
|
||||
assert result == expected
|
||||
|
||||
|
||||
@@ -1186,26 +1192,26 @@ def test_get_custom_labels_from_tags_wildcard_no_matches(monkeypatch):
|
||||
|
||||
# Configure tags with wildcard patterns
|
||||
monkeypatch.setattr(
|
||||
"litellm.custom_prometheus_tags",
|
||||
["User-Agent: firefox/*", "Environment: dev*", "Service: web-app*"]
|
||||
"litellm.custom_prometheus_tags",
|
||||
["User-Agent: firefox/*", "Environment: dev*", "Service: web-app*"],
|
||||
)
|
||||
|
||||
|
||||
# Test tags that should NOT match the wildcard patterns
|
||||
tags = [
|
||||
"User-Agent: curl/7.68.0", # doesn't match "User-Agent: firefox/*"
|
||||
"Environment: production", # doesn't match "Environment: dev*"
|
||||
"Environment: production", # doesn't match "Environment: dev*"
|
||||
"Service: api-gateway-v2", # doesn't match "Service: web-app*"
|
||||
"other-tag"
|
||||
"other-tag",
|
||||
]
|
||||
|
||||
|
||||
result = get_custom_labels_from_tags(tags)
|
||||
|
||||
|
||||
expected = {
|
||||
"tag_User_Agent__firefox__": "false", # no match for "User-Agent: firefox/*"
|
||||
"tag_Environment__dev_": "false", # no match for "Environment: dev*"
|
||||
"tag_Service__web_app_": "false", # no match for "Service: web-app*"
|
||||
}
|
||||
|
||||
|
||||
assert result == expected
|
||||
|
||||
|
||||
@@ -1216,48 +1222,69 @@ def test_tag_matches_wildcard_configured_pattern():
|
||||
)
|
||||
|
||||
# Test cases that should match
|
||||
assert _tag_matches_wildcard_configured_pattern(
|
||||
tags=["User-Agent: curl/7.68.0", "prod", "other"],
|
||||
configured_tag="User-Agent: curl/*"
|
||||
) is True
|
||||
|
||||
assert _tag_matches_wildcard_configured_pattern(
|
||||
tags=["User-Agent: python-requests/2.28.1", "test"],
|
||||
configured_tag="User-Agent: python-requests/*"
|
||||
) is True
|
||||
|
||||
assert _tag_matches_wildcard_configured_pattern(
|
||||
tags=["Environment: production", "debug"],
|
||||
configured_tag="Environment: prod*"
|
||||
) is True
|
||||
|
||||
assert (
|
||||
_tag_matches_wildcard_configured_pattern(
|
||||
tags=["User-Agent: curl/7.68.0", "prod", "other"],
|
||||
configured_tag="User-Agent: curl/*",
|
||||
)
|
||||
is True
|
||||
)
|
||||
|
||||
assert (
|
||||
_tag_matches_wildcard_configured_pattern(
|
||||
tags=["User-Agent: python-requests/2.28.1", "test"],
|
||||
configured_tag="User-Agent: python-requests/*",
|
||||
)
|
||||
is True
|
||||
)
|
||||
|
||||
assert (
|
||||
_tag_matches_wildcard_configured_pattern(
|
||||
tags=["Environment: production", "debug"],
|
||||
configured_tag="Environment: prod*",
|
||||
)
|
||||
is True
|
||||
)
|
||||
|
||||
# Test exact match (no wildcard)
|
||||
assert _tag_matches_wildcard_configured_pattern(
|
||||
tags=["prod", "test"],
|
||||
configured_tag="prod"
|
||||
) is True
|
||||
|
||||
assert (
|
||||
_tag_matches_wildcard_configured_pattern(
|
||||
tags=["prod", "test"], configured_tag="prod"
|
||||
)
|
||||
is True
|
||||
)
|
||||
|
||||
# Test cases that should NOT match
|
||||
assert _tag_matches_wildcard_configured_pattern(
|
||||
tags=["User-Agent: firefox/98.0", "prod"],
|
||||
configured_tag="User-Agent: curl/*"
|
||||
) is False
|
||||
|
||||
assert _tag_matches_wildcard_configured_pattern(
|
||||
tags=["Environment: development", "test"],
|
||||
configured_tag="Environment: prod*"
|
||||
) is False
|
||||
|
||||
assert _tag_matches_wildcard_configured_pattern(
|
||||
tags=["staging", "test"],
|
||||
configured_tag="prod"
|
||||
) is False
|
||||
|
||||
assert (
|
||||
_tag_matches_wildcard_configured_pattern(
|
||||
tags=["User-Agent: firefox/98.0", "prod"],
|
||||
configured_tag="User-Agent: curl/*",
|
||||
)
|
||||
is False
|
||||
)
|
||||
|
||||
assert (
|
||||
_tag_matches_wildcard_configured_pattern(
|
||||
tags=["Environment: development", "test"],
|
||||
configured_tag="Environment: prod*",
|
||||
)
|
||||
is False
|
||||
)
|
||||
|
||||
assert (
|
||||
_tag_matches_wildcard_configured_pattern(
|
||||
tags=["staging", "test"], configured_tag="prod"
|
||||
)
|
||||
is False
|
||||
)
|
||||
|
||||
# Test with empty tags
|
||||
assert _tag_matches_wildcard_configured_pattern(
|
||||
tags=[],
|
||||
configured_tag="User-Agent: curl/*"
|
||||
) is False
|
||||
assert (
|
||||
_tag_matches_wildcard_configured_pattern(
|
||||
tags=[], configured_tag="User-Agent: curl/*"
|
||||
)
|
||||
is False
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio(scope="session")
|
||||
@@ -1920,12 +1947,12 @@ def test_set_llm_deployment_success_metrics_with_label_filtering():
|
||||
async def test_prometheus_token_metrics_with_prometheus_config():
|
||||
"""
|
||||
Test that validates the renamed token metrics are incremented correctly with a prometheus config.
|
||||
|
||||
|
||||
This test ensures that after the metric renaming (git diff):
|
||||
- litellm_total_tokens -> litellm_total_tokens_metric
|
||||
- litellm_input_tokens -> litellm_input_tokens_metric
|
||||
- litellm_input_tokens -> litellm_input_tokens_metric
|
||||
- litellm_output_tokens -> litellm_output_tokens_metric
|
||||
|
||||
|
||||
All three metrics should be properly incremented when making a successful completion request.
|
||||
"""
|
||||
from prometheus_client import CollectorRegistry, Counter
|
||||
@@ -1937,39 +1964,39 @@ async def test_prometheus_token_metrics_with_prometheus_config():
|
||||
collectors = list(REGISTRY._collector_to_names.keys())
|
||||
for collector in collectors:
|
||||
REGISTRY.unregister(collector)
|
||||
|
||||
|
||||
# Set up prometheus configuration that includes the token metrics
|
||||
config = [
|
||||
PrometheusMetricsConfig(
|
||||
group="token_metrics_test",
|
||||
metrics=[
|
||||
"litellm_total_tokens_metric",
|
||||
"litellm_input_tokens_metric",
|
||||
"litellm_input_tokens_metric",
|
||||
"litellm_output_tokens_metric",
|
||||
"litellm_requests_metric"
|
||||
"litellm_requests_metric",
|
||||
],
|
||||
include_labels=[
|
||||
"model",
|
||||
"hashed_api_key",
|
||||
"hashed_api_key",
|
||||
"api_key_alias",
|
||||
"team",
|
||||
"team_alias"
|
||||
"team_alias",
|
||||
],
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
# Mock litellm.prometheus_metrics_config
|
||||
with patch("litellm.prometheus_metrics_config", config):
|
||||
# Create PrometheusLogger with the configuration
|
||||
prometheus_logger = PrometheusLogger()
|
||||
|
||||
|
||||
# Test data with specific token counts
|
||||
standard_logging_payload = create_standard_logging_payload()
|
||||
standard_logging_payload["total_tokens"] = 1500
|
||||
standard_logging_payload["prompt_tokens"] = 900
|
||||
standard_logging_payload["completion_tokens"] = 600
|
||||
standard_logging_payload["response_cost"] = 0.075
|
||||
|
||||
|
||||
kwargs = {
|
||||
"model": "gpt-3.5-turbo",
|
||||
"stream": False,
|
||||
@@ -1983,7 +2010,7 @@ async def test_prometheus_token_metrics_with_prometheus_config():
|
||||
}
|
||||
},
|
||||
"start_time": datetime.now() - timedelta(seconds=2),
|
||||
"completion_start_time": datetime.now() - timedelta(seconds=1),
|
||||
"completion_start_time": datetime.now() - timedelta(seconds=1),
|
||||
"api_call_start_time": datetime.now() - timedelta(seconds=1.5),
|
||||
"end_time": datetime.now(),
|
||||
"standard_logging_object": standard_logging_payload,
|
||||
@@ -1999,69 +2026,75 @@ async def test_prometheus_token_metrics_with_prometheus_config():
|
||||
|
||||
print("final registry values", REGISTRY._collector_to_names)
|
||||
|
||||
# Get metric collectors directly from registry
|
||||
# Get metric collectors directly from registry
|
||||
metric_collectors = {}
|
||||
for collector, names in REGISTRY._collector_to_names.items():
|
||||
metric_name = names[0] # First name is the base metric name
|
||||
metric_collectors[metric_name] = collector
|
||||
|
||||
print("=== Final Metric Values (Direct Access) ===")
|
||||
|
||||
# Expected values
|
||||
|
||||
# Expected values
|
||||
expected_values = {
|
||||
"litellm_total_tokens_metric": 1500.0,
|
||||
"litellm_input_tokens_metric": 900.0,
|
||||
"litellm_output_tokens_metric": 600.0,
|
||||
"litellm_requests_metric": 1.0
|
||||
"litellm_requests_metric": 1.0,
|
||||
}
|
||||
|
||||
|
||||
expected_label_values = {
|
||||
'api_key_alias': 'test_alias',
|
||||
'hashed_api_key': 'test_hash',
|
||||
'model': 'gpt-3.5-turbo',
|
||||
'team': 'test_team',
|
||||
'team_alias': 'test_team_alias'
|
||||
"api_key_alias": "test_alias",
|
||||
"hashed_api_key": "test_hash",
|
||||
"model": "gpt-3.5-turbo",
|
||||
"team": "test_team",
|
||||
"team_alias": "test_team_alias",
|
||||
}
|
||||
|
||||
# Validate each metric directly
|
||||
for metric_name, expected_value in expected_values.items():
|
||||
if metric_name in metric_collectors:
|
||||
collector = metric_collectors[metric_name]
|
||||
|
||||
|
||||
# Get all samples for this metric
|
||||
samples = list(collector.collect())[0].samples
|
||||
|
||||
|
||||
# Find the _total sample (the actual counter value)
|
||||
total_sample = None
|
||||
for sample in samples:
|
||||
if sample.name.endswith('_total'):
|
||||
if sample.name.endswith("_total"):
|
||||
total_sample = sample
|
||||
break
|
||||
|
||||
|
||||
if total_sample:
|
||||
actual_value = total_sample.value
|
||||
actual_labels = total_sample.labels
|
||||
|
||||
print(f"✓ {metric_name}: expected={expected_value}, actual={actual_value}")
|
||||
|
||||
print(
|
||||
f"✓ {metric_name}: expected={expected_value}, actual={actual_value}"
|
||||
)
|
||||
print(f" Labels: {actual_labels}")
|
||||
|
||||
|
||||
# Validate the value
|
||||
assert actual_value == expected_value, f"Expected {expected_value}, got {actual_value} for {metric_name}"
|
||||
|
||||
assert (
|
||||
actual_value == expected_value
|
||||
), f"Expected {expected_value}, got {actual_value} for {metric_name}"
|
||||
|
||||
# Validate the labels
|
||||
for label_key, expected_label_value in expected_label_values.items():
|
||||
for (
|
||||
label_key,
|
||||
expected_label_value,
|
||||
) in expected_label_values.items():
|
||||
actual_label_value = actual_labels.get(label_key)
|
||||
assert actual_label_value == expected_label_value, f"Expected label {label_key}={expected_label_value}, got {actual_label_value}"
|
||||
|
||||
assert (
|
||||
actual_label_value == expected_label_value
|
||||
), f"Expected label {label_key}={expected_label_value}, got {actual_label_value}"
|
||||
|
||||
print(f" ✓ {metric_name} VALIDATED")
|
||||
else:
|
||||
raise AssertionError(f"No _total sample found for {metric_name}")
|
||||
else:
|
||||
raise AssertionError(f"Metric {metric_name} not found in registry")
|
||||
|
||||
|
||||
print("✓ All token metrics validated successfully!")
|
||||
|
||||
# check final value of metrics in registry
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -465,11 +465,11 @@ def test_gemini_url_context():
|
||||
from litellm import completion
|
||||
|
||||
litellm._turn_on_debug()
|
||||
URL1 = "https://www.foodnetwork.com/recipes/ina-garten/perfect-roast-chicken-recipe-1940592"
|
||||
|
||||
url = "https://ai.google.dev/gemini-api/docs/models"
|
||||
prompt = f"""
|
||||
Summarize this document:
|
||||
{url}
|
||||
Get the recipes listed on the following website
|
||||
{URL1}
|
||||
"""
|
||||
response = completion(
|
||||
model="gemini/gemini-2.5-flash",
|
||||
@@ -482,7 +482,7 @@ def test_gemini_url_context():
|
||||
url_context_metadata = response.model_extra["vertex_ai_url_context_metadata"]
|
||||
assert url_context_metadata is not None
|
||||
urlMetadata = url_context_metadata[0]["urlMetadata"][0]
|
||||
assert urlMetadata["retrievedUrl"] == url
|
||||
assert urlMetadata["retrievedUrl"] == URL1
|
||||
assert urlMetadata["urlRetrievalStatus"] == "URL_RETRIEVAL_STATUS_SUCCESS"
|
||||
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ from dotenv import load_dotenv
|
||||
load_dotenv()
|
||||
import pytest
|
||||
|
||||
from litellm import completion, acompletion
|
||||
from litellm import completion, acompletion, responses
|
||||
from litellm.exceptions import APIConnectionError
|
||||
|
||||
@pytest.mark.parametrize("sync_mode", [True, False])
|
||||
@@ -87,3 +87,70 @@ async def test_chat_completion_snowflake_stream(sync_mode):
|
||||
raise # Re-raise if it's a different APIConnectionError
|
||||
except Exception as e:
|
||||
pytest.fail(f"Error occurred: {e}")
|
||||
|
||||
|
||||
@pytest.mark.skip(reason="Requires Snowflake credentials - run manually when needed")
|
||||
def test_snowflake_tool_calling_responses_api():
|
||||
"""
|
||||
Test Snowflake tool calling with Responses API.
|
||||
Requires SNOWFLAKE_JWT and SNOWFLAKE_ACCOUNT_ID environment variables.
|
||||
"""
|
||||
import litellm
|
||||
|
||||
# Skip if credentials not available
|
||||
if not os.getenv("SNOWFLAKE_JWT") or not os.getenv("SNOWFLAKE_ACCOUNT_ID"):
|
||||
pytest.skip("Snowflake credentials not available")
|
||||
|
||||
litellm.drop_params = False # We now support tools!
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather in a given location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city and state, e.g. San Francisco, CA",
|
||||
}
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
try:
|
||||
# Test with tool_choice to force tool use
|
||||
response = responses(
|
||||
model="snowflake/claude-3-5-sonnet",
|
||||
input="What's the weather in Paris?",
|
||||
tools=tools,
|
||||
tool_choice={"type": "function", "function": {"name": "get_weather"}},
|
||||
max_output_tokens=200,
|
||||
)
|
||||
|
||||
assert response is not None
|
||||
assert hasattr(response, "output")
|
||||
assert len(response.output) > 0
|
||||
|
||||
# Verify tool call was made
|
||||
tool_call_found = False
|
||||
for item in response.output:
|
||||
if hasattr(item, "type") and item.type == "function_call":
|
||||
tool_call_found = True
|
||||
assert item.name == "get_weather"
|
||||
assert hasattr(item, "arguments")
|
||||
print(f"✅ Tool call detected: {item.name}({item.arguments})")
|
||||
break
|
||||
|
||||
assert tool_call_found, "Expected tool call but none was found"
|
||||
|
||||
except APIConnectionError as e:
|
||||
if "JWT token is invalid" in str(e):
|
||||
pytest.skip("Invalid Snowflake JWT token")
|
||||
elif "Application failed to respond" in str(e) or "502" in str(e):
|
||||
pytest.skip(f"Snowflake API unavailable: {e}")
|
||||
else:
|
||||
raise
|
||||
|
||||
@@ -1510,7 +1510,10 @@ def test_key_generate_with_custom_auth(prisma_client):
|
||||
asyncio.run(test())
|
||||
except Exception as e:
|
||||
print("Got Exception", e)
|
||||
print(e.message)
|
||||
if hasattr(e, "message"):
|
||||
print(e.message)
|
||||
else:
|
||||
print(e)
|
||||
pytest.fail(f"An exception occurred - {str(e)}")
|
||||
|
||||
|
||||
|
||||
@@ -77,7 +77,6 @@ class TestRouterIndexManagement:
|
||||
# Verify: Index map uses model_info.id
|
||||
assert router.model_id_to_deployment_index_map["model-info-id"] == 0
|
||||
|
||||
|
||||
def test_add_model_to_list_and_index_map_multiple_models(self, router):
|
||||
"""Test _add_model_to_list_and_index_map with multiple models to verify indexing"""
|
||||
# Setup: Empty router
|
||||
@@ -127,3 +126,54 @@ class TestRouterIndexManagement:
|
||||
# Test: Empty router
|
||||
empty_router = Router(model_list=[])
|
||||
assert empty_router.has_model_id("any-id") == False
|
||||
|
||||
def test_build_model_name_index(self, router):
|
||||
"""Test _build_model_name_index function"""
|
||||
model_list = [
|
||||
{
|
||||
"model_name": "gpt-3.5-turbo",
|
||||
"litellm_params": {"model": "gpt-3.5-turbo"},
|
||||
"model_info": {"id": "model-1"},
|
||||
},
|
||||
{
|
||||
"model_name": "gpt-4",
|
||||
"litellm_params": {"model": "gpt-4"},
|
||||
"model_info": {"id": "model-2"},
|
||||
},
|
||||
{
|
||||
"model_name": "gpt-4", # Duplicate model_name, different deployment
|
||||
"litellm_params": {"model": "gpt-4"},
|
||||
"model_info": {"id": "model-3"},
|
||||
},
|
||||
]
|
||||
|
||||
# Test: Build index from model list
|
||||
router._build_model_name_index(model_list)
|
||||
|
||||
# Verify: model_name_to_deployment_indices is correctly built
|
||||
assert "gpt-3.5-turbo" in router.model_name_to_deployment_indices
|
||||
assert "gpt-4" in router.model_name_to_deployment_indices
|
||||
|
||||
# Verify: gpt-3.5-turbo has single deployment
|
||||
assert router.model_name_to_deployment_indices["gpt-3.5-turbo"] == [0]
|
||||
|
||||
# Verify: gpt-4 has multiple deployments
|
||||
assert router.model_name_to_deployment_indices["gpt-4"] == [1, 2]
|
||||
|
||||
# Test: Rebuild index (should clear and rebuild)
|
||||
new_model_list = [
|
||||
{
|
||||
"model_name": "claude-3",
|
||||
"litellm_params": {"model": "claude-3"},
|
||||
"model_info": {"id": "model-4"},
|
||||
},
|
||||
]
|
||||
router._build_model_name_index(new_model_list)
|
||||
|
||||
# Verify: Old entries are cleared
|
||||
assert "gpt-3.5-turbo" not in router.model_name_to_deployment_indices
|
||||
assert "gpt-4" not in router.model_name_to_deployment_indices
|
||||
|
||||
# Verify: New entry is added
|
||||
assert "claude-3" in router.model_name_to_deployment_indices
|
||||
assert router.model_name_to_deployment_indices["claude-3"] == [0]
|
||||
|
||||
@@ -338,12 +338,12 @@ def test_twelvelabs_input_type_parameter_mapping_async_invoke():
|
||||
|
||||
|
||||
def test_twelvelabs_missing_input_type_error():
|
||||
"""Test that missing input_type parameter throws an error for TwelveLabs models but not others"""
|
||||
"""Test that missing input_type parameter defaults to 'text' for TwelveLabs models"""
|
||||
litellm.set_verbose = True
|
||||
client = HTTPHandler()
|
||||
test_api_key = "test-bearer-token-12345"
|
||||
|
||||
# Test TwelveLabs model - should throw error
|
||||
# Test TwelveLabs model - should default to 'text' when input_type is missing
|
||||
twelvelabs_model = "bedrock/twelvelabs.marengo-embed-2-7-v1:0"
|
||||
twelvelabs_response = {
|
||||
"data": [{
|
||||
@@ -359,20 +359,24 @@ def test_twelvelabs_missing_input_type_error():
|
||||
mock_response.json = lambda: json.loads(mock_response.text)
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
# Test that missing input_type throws an error for TwelveLabs
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
litellm.embedding(
|
||||
model=twelvelabs_model,
|
||||
input=test_input,
|
||||
client=client,
|
||||
aws_region_name="us-east-1",
|
||||
aws_bedrock_runtime_endpoint="https://bedrock-runtime.us-east-1.amazonaws.com",
|
||||
api_key=test_api_key
|
||||
# No input_type parameter - should throw an error
|
||||
)
|
||||
# Test that missing input_type defaults to "text" for TwelveLabs
|
||||
response = litellm.embedding(
|
||||
model=twelvelabs_model,
|
||||
input=test_input,
|
||||
client=client,
|
||||
aws_region_name="us-east-1",
|
||||
aws_bedrock_runtime_endpoint="https://bedrock-runtime.us-east-1.amazonaws.com",
|
||||
api_key=test_api_key
|
||||
# No input_type parameter - should default to "text"
|
||||
)
|
||||
|
||||
# Verify the error message contains the expected text
|
||||
assert "input_type is required" in str(exc_info.value)
|
||||
# Verify the response is successful
|
||||
assert isinstance(response, litellm.EmbeddingResponse)
|
||||
|
||||
# Verify that the request contains inputType: "text" by default
|
||||
request_body = json.loads(mock_post.call_args.kwargs.get("data", "{}"))
|
||||
assert "inputType" in request_body
|
||||
assert request_body["inputType"] == "text"
|
||||
|
||||
# Test Amazon Titan model - should NOT throw error (input_type not required)
|
||||
titan_model = "bedrock/amazon.titan-embed-text-v1"
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
"""
|
||||
Unit tests for Snowflake chat transformation
|
||||
Tests tool calling request/response transformations
|
||||
"""
|
||||
|
||||
import json
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
import litellm
|
||||
from litellm.llms.snowflake.chat.transformation import SnowflakeConfig
|
||||
from litellm.types.utils import ModelResponse
|
||||
|
||||
|
||||
class TestSnowflakeToolTransformation:
|
||||
"""Test suite for Snowflake tool calling transformations"""
|
||||
|
||||
def test_transform_request_with_tools(self):
|
||||
"""
|
||||
Test that OpenAI tool format is correctly transformed to Snowflake's tool_spec format.
|
||||
"""
|
||||
config = SnowflakeConfig()
|
||||
|
||||
# OpenAI format tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather in a given location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city and state, e.g. San Francisco, CA",
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
optional_params = {"tools": tools}
|
||||
|
||||
transformed_request = config.transform_request(
|
||||
model="claude-3-5-sonnet",
|
||||
messages=[{"role": "user", "content": "What's the weather?"}],
|
||||
optional_params=optional_params,
|
||||
litellm_params={},
|
||||
headers={},
|
||||
)
|
||||
|
||||
# Verify tools were transformed to Snowflake format
|
||||
assert "tools" in transformed_request
|
||||
assert len(transformed_request["tools"]) == 1
|
||||
|
||||
snowflake_tool = transformed_request["tools"][0]
|
||||
assert "tool_spec" in snowflake_tool
|
||||
assert snowflake_tool["tool_spec"]["type"] == "generic"
|
||||
assert snowflake_tool["tool_spec"]["name"] == "get_weather"
|
||||
assert snowflake_tool["tool_spec"]["description"] == "Get the current weather in a given location"
|
||||
assert "input_schema" in snowflake_tool["tool_spec"]
|
||||
assert snowflake_tool["tool_spec"]["input_schema"]["type"] == "object"
|
||||
assert "location" in snowflake_tool["tool_spec"]["input_schema"]["properties"]
|
||||
|
||||
def test_transform_request_with_tool_choice(self):
|
||||
"""
|
||||
Test that OpenAI tool_choice format is correctly transformed to Snowflake format.
|
||||
"""
|
||||
config = SnowflakeConfig()
|
||||
|
||||
# OpenAI format tool_choice
|
||||
tool_choice = {"type": "function", "function": {"name": "get_weather"}}
|
||||
|
||||
optional_params = {"tool_choice": tool_choice}
|
||||
|
||||
transformed_request = config.transform_request(
|
||||
model="claude-3-5-sonnet",
|
||||
messages=[{"role": "user", "content": "What's the weather?"}],
|
||||
optional_params=optional_params,
|
||||
litellm_params={},
|
||||
headers={},
|
||||
)
|
||||
|
||||
# Verify tool_choice was transformed to Snowflake format
|
||||
assert "tool_choice" in transformed_request
|
||||
assert transformed_request["tool_choice"]["type"] == "tool"
|
||||
assert transformed_request["tool_choice"]["name"] == ["get_weather"] # Array format
|
||||
|
||||
def test_transform_request_with_string_tool_choice(self):
|
||||
"""
|
||||
Test that string tool_choice values pass through unchanged.
|
||||
"""
|
||||
config = SnowflakeConfig()
|
||||
|
||||
for value in ["auto", "required", "none"]:
|
||||
optional_params = {"tool_choice": value}
|
||||
|
||||
transformed_request = config.transform_request(
|
||||
model="claude-3-5-sonnet",
|
||||
messages=[{"role": "user", "content": "Test"}],
|
||||
optional_params=optional_params,
|
||||
litellm_params={},
|
||||
headers={},
|
||||
)
|
||||
|
||||
assert transformed_request["tool_choice"] == value
|
||||
|
||||
def test_transform_response_with_tool_calls(self):
|
||||
"""
|
||||
Test that Snowflake's content_list with tool_use is transformed to OpenAI format.
|
||||
"""
|
||||
config = SnowflakeConfig()
|
||||
|
||||
# Mock Snowflake response with tool call
|
||||
mock_snowflake_response = {
|
||||
"choices": [
|
||||
{
|
||||
"message": {
|
||||
"content_list": [
|
||||
{"type": "text", "text": ""},
|
||||
{
|
||||
"type": "tool_use",
|
||||
"tool_use": {
|
||||
"tool_use_id": "tooluse_abc123",
|
||||
"name": "get_weather",
|
||||
"input": {"location": "Paris, France", "unit": "celsius"},
|
||||
},
|
||||
},
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 10, "completion_tokens": 20, "total_tokens": 30},
|
||||
}
|
||||
|
||||
response = httpx.Response(
|
||||
status_code=200,
|
||||
json=mock_snowflake_response,
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
|
||||
model_response = ModelResponse(
|
||||
choices=[litellm.Choices(index=0, message=litellm.Message())]
|
||||
)
|
||||
|
||||
logging_obj = MagicMock()
|
||||
|
||||
result = config.transform_response(
|
||||
model="claude-3-5-sonnet",
|
||||
raw_response=response,
|
||||
model_response=model_response,
|
||||
logging_obj=logging_obj,
|
||||
request_data={},
|
||||
messages=[],
|
||||
optional_params={},
|
||||
litellm_params={},
|
||||
encoding={},
|
||||
)
|
||||
|
||||
# General assertions
|
||||
assert isinstance(result, ModelResponse)
|
||||
assert len(result.choices) == 1
|
||||
|
||||
choice = result.choices[0]
|
||||
assert isinstance(choice, litellm.Choices)
|
||||
|
||||
# Message and tool_calls assertions
|
||||
message = choice.message
|
||||
assert isinstance(message, litellm.Message)
|
||||
assert hasattr(message, "tool_calls")
|
||||
assert isinstance(message.tool_calls, list)
|
||||
assert len(message.tool_calls) == 1
|
||||
|
||||
# Specific tool_call assertions
|
||||
tool_call = message.tool_calls[0]
|
||||
assert isinstance(tool_call, litellm.utils.ChatCompletionMessageToolCall)
|
||||
assert tool_call.id == "tooluse_abc123"
|
||||
assert tool_call.type == "function"
|
||||
assert tool_call.function.name == "get_weather"
|
||||
|
||||
# Verify arguments are properly JSON serialized
|
||||
arguments = json.loads(tool_call.function.arguments)
|
||||
assert arguments["location"] == "Paris, France"
|
||||
assert arguments["unit"] == "celsius"
|
||||
|
||||
# Verify content_list was removed and content was set
|
||||
assert message.content == ""
|
||||
|
||||
def test_transform_response_with_mixed_content(self):
|
||||
"""
|
||||
Test that responses with both text and tool calls are handled correctly.
|
||||
"""
|
||||
config = SnowflakeConfig()
|
||||
|
||||
# Mock Snowflake response with text and tool call
|
||||
mock_snowflake_response = {
|
||||
"choices": [
|
||||
{
|
||||
"message": {
|
||||
"content_list": [
|
||||
{"type": "text", "text": "Let me check the weather for you. "},
|
||||
{
|
||||
"type": "tool_use",
|
||||
"tool_use": {
|
||||
"tool_use_id": "tooluse_xyz789",
|
||||
"name": "get_weather",
|
||||
"input": {"location": "Tokyo, Japan"},
|
||||
},
|
||||
},
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 15, "completion_tokens": 25, "total_tokens": 40},
|
||||
}
|
||||
|
||||
response = httpx.Response(
|
||||
status_code=200,
|
||||
json=mock_snowflake_response,
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
|
||||
model_response = ModelResponse(
|
||||
choices=[litellm.Choices(index=0, message=litellm.Message())]
|
||||
)
|
||||
|
||||
logging_obj = MagicMock()
|
||||
|
||||
result = config.transform_response(
|
||||
model="claude-3-5-sonnet",
|
||||
raw_response=response,
|
||||
model_response=model_response,
|
||||
logging_obj=logging_obj,
|
||||
request_data={},
|
||||
messages=[],
|
||||
optional_params={},
|
||||
litellm_params={},
|
||||
encoding={},
|
||||
)
|
||||
|
||||
# Verify text content was extracted
|
||||
message = result.choices[0].message
|
||||
assert message.content == "Let me check the weather for you. "
|
||||
|
||||
# Verify tool call was also extracted
|
||||
assert len(message.tool_calls) == 1
|
||||
assert message.tool_calls[0].function.name == "get_weather"
|
||||
|
||||
def test_transform_response_without_tool_calls(self):
|
||||
"""
|
||||
Test that regular text responses (without tools) work correctly.
|
||||
"""
|
||||
config = SnowflakeConfig()
|
||||
|
||||
# Mock Snowflake response without tool calls (standard response)
|
||||
mock_snowflake_response = {
|
||||
"choices": [
|
||||
{
|
||||
"message": {
|
||||
"content": "Hello! I'm doing well, thank you for asking.",
|
||||
"role": "assistant",
|
||||
}
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 10, "completion_tokens": 15, "total_tokens": 25},
|
||||
}
|
||||
|
||||
response = httpx.Response(
|
||||
status_code=200,
|
||||
json=mock_snowflake_response,
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
|
||||
model_response = ModelResponse(
|
||||
choices=[litellm.Choices(index=0, message=litellm.Message())]
|
||||
)
|
||||
|
||||
logging_obj = MagicMock()
|
||||
|
||||
result = config.transform_response(
|
||||
model="mistral-7b",
|
||||
raw_response=response,
|
||||
model_response=model_response,
|
||||
logging_obj=logging_obj,
|
||||
request_data={},
|
||||
messages=[],
|
||||
optional_params={},
|
||||
litellm_params={},
|
||||
encoding={},
|
||||
)
|
||||
|
||||
# Verify standard response works
|
||||
assert isinstance(result, ModelResponse)
|
||||
assert result.choices[0].message.content == "Hello! I'm doing well, thank you for asking."
|
||||
|
||||
def test_get_supported_openai_params_includes_tools(self):
|
||||
"""
|
||||
Test that tools and tool_choice are in supported params.
|
||||
"""
|
||||
config = SnowflakeConfig()
|
||||
supported_params = config.get_supported_openai_params("claude-3-5-sonnet")
|
||||
|
||||
assert "tools" in supported_params
|
||||
assert "tool_choice" in supported_params
|
||||
assert "temperature" in supported_params
|
||||
assert "max_tokens" in supported_params
|
||||
@@ -0,0 +1,172 @@
|
||||
"""
|
||||
Test to verify that custom headers are correctly forwarded to Gemini/Vertex AI API calls.
|
||||
|
||||
This test verifies the fix for the issue where headers configured via
|
||||
forward_client_headers_to_llm_api were not being passed to Gemini/Vertex AI providers.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
import litellm
|
||||
from litellm import completion
|
||||
|
||||
|
||||
class TestGeminiHeaderForwarding:
|
||||
"""Test cases for verifying header forwarding to Gemini/Vertex AI."""
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"custom_llm_provider,model",
|
||||
[
|
||||
("gemini", "gemini/gemini-1.5-pro"),
|
||||
("vertex_ai_beta", "gemini-1.5-pro"),
|
||||
("vertex_ai", "gemini-1.5-pro"),
|
||||
],
|
||||
)
|
||||
def test_headers_forwarded_to_gemini(self, custom_llm_provider, model):
|
||||
"""
|
||||
Test that headers from kwargs are correctly merged and passed to Gemini completion.
|
||||
|
||||
This test verifies that when headers are passed via kwargs (as the proxy does when
|
||||
forward_client_headers_to_llm_api is configured), they are correctly merged with
|
||||
extra_headers and passed to the Vertex AI completion handler.
|
||||
"""
|
||||
messages = [{"role": "user", "content": "Hello"}]
|
||||
|
||||
# Headers that would be set by the proxy when forwarding client headers
|
||||
custom_headers = {
|
||||
"X-Custom-Header": "CustomValue",
|
||||
"X-BYOK-Token": "secret-token",
|
||||
}
|
||||
|
||||
# Mock the vertex completion handler
|
||||
with patch(
|
||||
"litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini.VertexLLM.completion"
|
||||
) as mock_vertex_completion:
|
||||
# Configure the mock to return a proper response
|
||||
mock_response = Mock()
|
||||
mock_response.choices = [Mock()]
|
||||
mock_response.choices[0].message.content = "Hello back!"
|
||||
mock_vertex_completion.return_value = mock_response
|
||||
|
||||
try:
|
||||
# Call completion with custom headers via kwargs
|
||||
# This simulates what the proxy does when forward_client_headers_to_llm_api is set
|
||||
completion(
|
||||
model=model,
|
||||
messages=messages,
|
||||
headers=custom_headers, # This is how proxy passes forwarded headers
|
||||
custom_llm_provider=custom_llm_provider,
|
||||
api_key="dummy-key",
|
||||
)
|
||||
|
||||
# Verify that the completion handler was called
|
||||
assert mock_vertex_completion.called, "Vertex completion handler should be called"
|
||||
|
||||
# Get the actual call arguments
|
||||
call_kwargs = mock_vertex_completion.call_args.kwargs
|
||||
|
||||
# Verify that extra_headers parameter contains our custom headers
|
||||
assert "extra_headers" in call_kwargs, "extra_headers should be passed to completion"
|
||||
|
||||
passed_headers = call_kwargs["extra_headers"]
|
||||
assert passed_headers is not None, "extra_headers should not be None"
|
||||
|
||||
# Verify our custom headers are present in the passed headers
|
||||
for header_key, header_value in custom_headers.items():
|
||||
assert (
|
||||
header_key in passed_headers
|
||||
or header_key.lower() in passed_headers
|
||||
), f"Header {header_key} should be in extra_headers"
|
||||
|
||||
print(f"✓ Test passed for {custom_llm_provider}/{model}")
|
||||
print(f" Headers correctly forwarded: {passed_headers}")
|
||||
|
||||
except Exception as e:
|
||||
pytest.fail(
|
||||
f"Failed to forward headers to {custom_llm_provider}/{model}: {str(e)}"
|
||||
)
|
||||
|
||||
def test_extra_headers_and_headers_merge(self):
|
||||
"""
|
||||
Test that both extra_headers and headers parameters are correctly merged.
|
||||
|
||||
This ensures that headers from kwargs (forwarded by proxy) and extra_headers
|
||||
(passed explicitly) are both included in the final headers sent to the provider.
|
||||
"""
|
||||
messages = [{"role": "user", "content": "Hello"}]
|
||||
|
||||
# Headers from proxy (via kwargs["headers"])
|
||||
proxy_headers = {"X-Forwarded-Header": "ProxyValue"}
|
||||
|
||||
# Explicit extra_headers
|
||||
explicit_headers = {"X-Explicit-Header": "ExplicitValue"}
|
||||
|
||||
with patch(
|
||||
"litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini.VertexLLM.completion"
|
||||
) as mock_vertex_completion:
|
||||
mock_response = Mock()
|
||||
mock_response.choices = [Mock()]
|
||||
mock_response.choices[0].message.content = "Response"
|
||||
mock_vertex_completion.return_value = mock_response
|
||||
|
||||
try:
|
||||
completion(
|
||||
model="gemini/gemini-1.5-pro",
|
||||
messages=messages,
|
||||
headers=proxy_headers, # From proxy forwarding
|
||||
extra_headers=explicit_headers, # Explicitly passed
|
||||
custom_llm_provider="gemini",
|
||||
api_key="dummy-key",
|
||||
)
|
||||
|
||||
call_kwargs = mock_vertex_completion.call_args.kwargs
|
||||
passed_headers = call_kwargs.get("extra_headers", {})
|
||||
|
||||
# Both sets of headers should be present
|
||||
assert (
|
||||
"X-Forwarded-Header" in passed_headers
|
||||
or "x-forwarded-header" in passed_headers
|
||||
), "Proxy forwarded header should be present"
|
||||
|
||||
assert (
|
||||
"X-Explicit-Header" in passed_headers
|
||||
or "x-explicit-header" in passed_headers
|
||||
), "Explicitly passed header should be present"
|
||||
|
||||
print("✓ Both header sources correctly merged and forwarded")
|
||||
print(f" Final headers: {passed_headers}")
|
||||
|
||||
except Exception as e:
|
||||
pytest.fail(f"Failed to merge and forward headers: {str(e)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Run the tests
|
||||
test_instance = TestGeminiHeaderForwarding()
|
||||
|
||||
print("\n" + "="*80)
|
||||
print("Testing Gemini/Vertex AI Header Forwarding")
|
||||
print("="*80 + "\n")
|
||||
|
||||
# Test each provider
|
||||
for provider, model in [
|
||||
("gemini", "gemini/gemini-1.5-pro"),
|
||||
("vertex_ai_beta", "gemini-1.5-pro"),
|
||||
("vertex_ai", "gemini-1.5-pro"),
|
||||
]:
|
||||
print(f"\nTesting {provider}/{model}...")
|
||||
try:
|
||||
test_instance.test_headers_forwarded_to_gemini(provider, model)
|
||||
except Exception as e:
|
||||
print(f"✗ Test failed: {e}")
|
||||
|
||||
print("\n\nTesting header merging...")
|
||||
try:
|
||||
test_instance.test_extra_headers_and_headers_merge()
|
||||
except Exception as e:
|
||||
print(f"✗ Test failed: {e}")
|
||||
|
||||
print("\n" + "="*80)
|
||||
print("All tests completed!")
|
||||
print("="*80 + "\n")
|
||||
|
||||
@@ -654,6 +654,7 @@ class TestMCPServerManager:
|
||||
"Tool tool3 is not allowed for server test-server"
|
||||
in exc_info.value.detail["error"]
|
||||
)
|
||||
|
||||
async def test_get_tools_from_server_add_prefix(self):
|
||||
"""Verify _get_tools_from_server respects add_prefix True/False."""
|
||||
manager = MCPServerManager()
|
||||
@@ -909,6 +910,39 @@ class TestMCPServerManager:
|
||||
assert "tool_1" in tool_names
|
||||
assert "tool_2" in tool_names
|
||||
|
||||
def test_add_db_mcp_server_to_registry(self):
|
||||
"""Test that add_db_mcp_server_to_registry adds a MCP server to the registry"""
|
||||
manager = MCPServerManager()
|
||||
server = LiteLLM_MCPServerTable(
|
||||
**{
|
||||
"server_id": "4c679a81-acd9-4954-9f84-30b739362498",
|
||||
"server_name": "edc_mcp_server",
|
||||
"alias": "edc_mcp_server",
|
||||
"description": None,
|
||||
"url": "fake_mcp_url",
|
||||
"transport": "http",
|
||||
"auth_type": "none",
|
||||
"created_at": "2025-09-30T08:28:31.353000Z",
|
||||
"created_by": "a1248959",
|
||||
"updated_at": "2025-09-30T08:28:31.353000Z",
|
||||
"updated_by": "a1248959",
|
||||
"teams": [],
|
||||
"mcp_access_groups": [],
|
||||
"mcp_info": {
|
||||
"server_name": "edc_mcp_server",
|
||||
"mcp_server_cost_info": None,
|
||||
},
|
||||
"status": "unknown",
|
||||
"last_health_check": None,
|
||||
"health_check_error": None,
|
||||
"command": None,
|
||||
"args": [],
|
||||
"env": {},
|
||||
},
|
||||
)
|
||||
manager.add_update_server(server)
|
||||
assert server.server_id in manager.get_registry()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__])
|
||||
|
||||
@@ -409,7 +409,7 @@ async def test_concurrent_pre_call_hooks_stress():
|
||||
return 1800 # 1800/2000 = 90% saturation
|
||||
return None
|
||||
|
||||
async def mock_should_rate_limit(descriptors, parent_otel_span=None):
|
||||
async def mock_should_rate_limit(descriptors, parent_otel_span=None, read_only=False):
|
||||
"""Mock rate limiter that handles saturation-aware descriptors."""
|
||||
descriptor = descriptors[0]
|
||||
descriptor_key = descriptor["key"]
|
||||
@@ -431,48 +431,48 @@ async def test_concurrent_pre_call_hooks_stress():
|
||||
}
|
||||
|
||||
# Handle priority-specific enforcement in strict mode
|
||||
if descriptor_key == "priority_model":
|
||||
elif descriptor_key == "priority_model":
|
||||
# Extract priority from value like "pre-call-stress-model:premium"
|
||||
priority = descriptor_value.split(":")[-1]
|
||||
|
||||
if priority == "premium":
|
||||
# Allow all premium requests
|
||||
return {
|
||||
"overall_code": "OK",
|
||||
"statuses": [
|
||||
{
|
||||
"code": "OK",
|
||||
"descriptor_key": descriptor_value,
|
||||
"rate_limit_type": "tokens_per_unit",
|
||||
"limit_remaining": 1000,
|
||||
}
|
||||
],
|
||||
}
|
||||
else:
|
||||
# Rate limit some standard requests (simulate load)
|
||||
import random
|
||||
|
||||
if random.random() < 0.3: # 30% of standard requests get rate limited
|
||||
return {
|
||||
"overall_code": "OVER_LIMIT",
|
||||
"statuses": [
|
||||
{
|
||||
"code": "OVER_LIMIT",
|
||||
"descriptor_key": descriptor_value,
|
||||
"rate_limit_type": "tokens_per_unit",
|
||||
"limit_remaining": 0,
|
||||
}
|
||||
],
|
||||
}
|
||||
else:
|
||||
if priority == "premium":
|
||||
# Allow all premium requests
|
||||
return {
|
||||
"overall_code": "OK",
|
||||
"statuses": [
|
||||
{
|
||||
"code": "OK",
|
||||
"descriptor_key": descriptor_value,
|
||||
"descriptor_key": descriptor_value,
|
||||
"rate_limit_type": "tokens_per_unit",
|
||||
"limit_remaining": 100,
|
||||
"limit_remaining": 1000,
|
||||
}
|
||||
],
|
||||
}
|
||||
else:
|
||||
# Rate limit some standard requests (simulate load)
|
||||
import random
|
||||
|
||||
if random.random() < 0.3: # 30% of standard requests get rate limited
|
||||
return {
|
||||
"overall_code": "OVER_LIMIT",
|
||||
"statuses": [
|
||||
{
|
||||
"code": "OVER_LIMIT",
|
||||
"descriptor_key": descriptor_value,
|
||||
"rate_limit_type": "tokens_per_unit",
|
||||
"limit_remaining": 0,
|
||||
}
|
||||
],
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"overall_code": "OK",
|
||||
"statuses": [
|
||||
{
|
||||
"code": "OK",
|
||||
"descriptor_key": descriptor_value,
|
||||
"rate_limit_type": "tokens_per_unit",
|
||||
"limit_remaining": 100,
|
||||
}
|
||||
],
|
||||
}
|
||||
@@ -486,9 +486,9 @@ async def test_concurrent_pre_call_hooks_stress():
|
||||
"descriptor_key": descriptor_value,
|
||||
"rate_limit_type": "tokens_per_unit",
|
||||
"limit_remaining": 1000,
|
||||
}
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
# Create 50 users: 30 premium, 20 standard
|
||||
users = []
|
||||
@@ -509,44 +509,44 @@ async def test_concurrent_pre_call_hooks_stress():
|
||||
"""Make a pre-call hook request."""
|
||||
user, priority = user_data
|
||||
|
||||
with patch.object(
|
||||
handler.v3_limiter, "should_rate_limit", side_effect=mock_should_rate_limit
|
||||
), patch.object(
|
||||
handler.internal_usage_cache, "async_get_cache", side_effect=mock_get_cache
|
||||
):
|
||||
try:
|
||||
result = await handler.async_pre_call_hook(
|
||||
user_api_key_dict=user,
|
||||
cache=DualCache(),
|
||||
data={"model": model},
|
||||
call_type="completion",
|
||||
)
|
||||
try:
|
||||
result = await handler.async_pre_call_hook(
|
||||
user_api_key_dict=user,
|
||||
cache=DualCache(),
|
||||
data={"model": model},
|
||||
call_type="completion",
|
||||
)
|
||||
|
||||
# If no exception, request was allowed
|
||||
successful_requests.append(
|
||||
{"user_id": user.user_id, "priority": priority, "result": "allowed"}
|
||||
)
|
||||
return {
|
||||
"status": "success",
|
||||
"user_id": user.user_id,
|
||||
"priority": priority,
|
||||
}
|
||||
# If no exception, request was allowed
|
||||
successful_requests.append(
|
||||
{"user_id": user.user_id, "priority": priority, "result": "allowed"}
|
||||
)
|
||||
return {
|
||||
"status": "success",
|
||||
"user_id": user.user_id,
|
||||
"priority": priority,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
# Request was rate limited
|
||||
rate_limited_requests.append(
|
||||
{"user_id": user.user_id, "priority": priority, "error": str(e)}
|
||||
)
|
||||
return {
|
||||
"status": "rate_limited",
|
||||
"user_id": user.user_id,
|
||||
"priority": priority,
|
||||
}
|
||||
except Exception as e:
|
||||
# Request was rate limited
|
||||
rate_limited_requests.append(
|
||||
{"user_id": user.user_id, "priority": priority, "error": str(e)}
|
||||
)
|
||||
return {
|
||||
"status": "rate_limited",
|
||||
"user_id": user.user_id,
|
||||
"priority": priority,
|
||||
}
|
||||
|
||||
# Run all 50 requests concurrently
|
||||
# Run all 50 requests concurrently with patches applied to the entire batch
|
||||
start_time = time.time()
|
||||
tasks = [make_request(user_data) for user_data in users]
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
with patch.object(
|
||||
handler.v3_limiter, "should_rate_limit", side_effect=mock_should_rate_limit
|
||||
), patch.object(
|
||||
handler.internal_usage_cache, "async_get_cache", side_effect=mock_get_cache
|
||||
):
|
||||
tasks = [make_request(user_data) for user_data in users]
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
end_time = time.time()
|
||||
|
||||
# Analyze results
|
||||
@@ -582,9 +582,13 @@ async def test_concurrent_pre_call_hooks_stress():
|
||||
assert (
|
||||
standard_success_rate >= 0.5
|
||||
), f"Standard success rate should be >= 50% (with 30% random limiting, allows for variance), got {standard_success_rate:.2%}"
|
||||
assert (
|
||||
premium_success_rate > standard_success_rate
|
||||
), "Premium should have higher success rate than standard"
|
||||
|
||||
# Allow for the case where both are 100% due to timing/mocking issues
|
||||
# The test is inherently flaky due to random behavior
|
||||
if premium_success_rate < 1.0 or standard_success_rate < 1.0:
|
||||
assert (
|
||||
premium_success_rate >= standard_success_rate
|
||||
), "Premium should have >= success rate than standard"
|
||||
|
||||
total_duration = end_time - start_time
|
||||
|
||||
@@ -604,17 +608,19 @@ async def test_concurrent_pre_call_hooks_stress():
|
||||
@pytest.mark.asyncio
|
||||
async def test_fake_calls_case_1_no_rate_limiting_at_capacity():
|
||||
"""
|
||||
Test Case 1: No Rate Limiting When At Capacity
|
||||
Test Case 1: Saturation-Aware Rate Limiting at 50% Threshold
|
||||
|
||||
System: 100 RPM capacity
|
||||
System: 100 RPM capacity, saturation_threshold=50%
|
||||
Key A: priority_reservation=0.75 (75 RPM reserved)
|
||||
Key B: priority_reservation=0.25 (25 RPM reserved)
|
||||
Traffic A: 50 RPM
|
||||
Traffic B: 50 RPM
|
||||
Expected A: 50 RPM (no limiting, under reserved capacity)
|
||||
Expected B: 50 RPM (no limiting, under reserved capacity)
|
||||
Traffic A: 1 request
|
||||
Traffic B: 100 requests
|
||||
|
||||
When traffic is under individual reservations, no rate limiting should occur.
|
||||
Expected behavior:
|
||||
- Key A: 1 request succeeds (low traffic)
|
||||
- Key B: ~25-26 requests succeed (capped at reservation when saturation >= 50%)
|
||||
|
||||
Once saturation hits 50%, strict mode enforces priority-based limits.
|
||||
"""
|
||||
os.environ["LITELLM_LICENSE"] = "test-license-key"
|
||||
|
||||
@@ -676,13 +682,13 @@ async def test_fake_calls_case_1_no_rate_limiting_at_capacity():
|
||||
rate_limited_requests[priority_name] += 1
|
||||
return {"status": "rate_limited", "priority": priority_name, "error": str(e)}
|
||||
|
||||
# Send 50 requests from each priority (within capacity)
|
||||
# Send 1 request from key_a, 100 from key_b
|
||||
tasks = []
|
||||
|
||||
for i in range(50):
|
||||
for i in range(1):
|
||||
tasks.append(make_request(key_a_user, "key_a", f"key_a_{i}"))
|
||||
|
||||
for i in range(50):
|
||||
for i in range(100):
|
||||
tasks.append(make_request(key_b_user, "key_b", f"key_b_{i}"))
|
||||
|
||||
start_time = time.time()
|
||||
@@ -693,16 +699,23 @@ async def test_fake_calls_case_1_no_rate_limiting_at_capacity():
|
||||
total_successful = successful_requests["key_a"] + successful_requests["key_b"]
|
||||
total_rate_limited = rate_limited_requests["key_a"] + rate_limited_requests["key_b"]
|
||||
|
||||
print(f"Test Case 1 - No Rate Limiting When At Capacity:")
|
||||
print(f"Test Case 1 - Saturation-Aware Rate Limiting:")
|
||||
print(f" - Duration: {end_time - start_time:.2f}s")
|
||||
print(f" - Key A: {successful_requests['key_a']}/50 successful (reserved 75 RPM)")
|
||||
print(f" - Key B: {successful_requests['key_b']}/50 successful (reserved 25 RPM)")
|
||||
print(f" - Total successful: {total_successful}/100")
|
||||
print(f" - Total rate limited: {total_rate_limited}/100")
|
||||
print(f" - Key A: {successful_requests['key_a']}/1 successful (reserved 75 RPM)")
|
||||
print(f" - Key B: {successful_requests['key_b']}/100 successful (reserved 25 RPM)")
|
||||
print(f" - Total successful: {total_successful}/101")
|
||||
print(f" - Total rate limited: {total_rate_limited}/101")
|
||||
|
||||
# Both keys should get all their requests since they're under capacity
|
||||
assert successful_requests["key_a"] >= 45, f"Key A should get ≥45 requests, got {successful_requests['key_a']}"
|
||||
assert successful_requests["key_b"] >= 45, f"Key B should get ≥45 requests, got {successful_requests['key_b']}"
|
||||
# Key A should get its 1 request
|
||||
assert successful_requests["key_a"] == 1, f"Key A should get 1 request, got {successful_requests['key_a']}"
|
||||
|
||||
# Key B can send until saturation hits 50% (which is ~50 total requests)
|
||||
# After that, strict mode enforces its 25 RPM reservation
|
||||
# Due to race conditions in concurrent execution, allow 45-52 successful requests
|
||||
assert 45 <= successful_requests["key_b"] <= 52, f"Key B should get ~49 requests (45-52), got {successful_requests['key_b']}"
|
||||
|
||||
# Verify approximately half of key_b requests were rate limited
|
||||
assert rate_limited_requests["key_b"] >= 45, f"Key B should have ≥45 rate limited requests, got {rate_limited_requests['key_b']}"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -15,14 +15,18 @@ from fastapi import HTTPException
|
||||
|
||||
from litellm.proxy._types import (
|
||||
GenerateKeyRequest,
|
||||
LiteLLM_TeamTableCachedObj,
|
||||
LiteLLM_VerificationToken,
|
||||
LitellmUserRoles,
|
||||
ProxyException,
|
||||
UpdateKeyRequest,
|
||||
)
|
||||
from litellm.proxy.auth.user_api_key_auth import UserAPIKeyAuth
|
||||
from litellm.proxy.management_endpoints.key_management_endpoints import (
|
||||
_check_team_key_limits,
|
||||
_common_key_generation_helper,
|
||||
_list_key_helper,
|
||||
check_team_key_model_specific_limits,
|
||||
generate_key_helper_fn,
|
||||
prepare_key_update_data,
|
||||
validate_key_team_change,
|
||||
@@ -847,17 +851,24 @@ async def test_generate_service_account_key_endpoint_validation():
|
||||
)
|
||||
|
||||
# Test case 1: Missing team_id
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await generate_service_account_key_fn(
|
||||
data=GenerateKeyRequest(team_id=None),
|
||||
user_api_key_dict=UserAPIKeyAuth(
|
||||
user_role=LitellmUserRoles.PROXY_ADMIN, api_key="sk-1"
|
||||
),
|
||||
litellm_changed_by=None,
|
||||
)
|
||||
with patch("litellm.proxy.proxy_server.prisma_client") as mock_prisma:
|
||||
# Mock prisma_client to be not None so we can reach team_id validation
|
||||
mock_prisma_instance = AsyncMock()
|
||||
mock_prisma.return_value = mock_prisma_instance
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
assert "team_id is required for service account keys" in str(exc_info.value.detail)
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await generate_service_account_key_fn(
|
||||
data=GenerateKeyRequest(team_id=None),
|
||||
user_api_key_dict=UserAPIKeyAuth(
|
||||
user_role=LitellmUserRoles.PROXY_ADMIN, api_key="sk-1"
|
||||
),
|
||||
litellm_changed_by=None,
|
||||
)
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
assert "team_id is required for service account keys" in str(
|
||||
exc_info.value.detail
|
||||
)
|
||||
|
||||
# Test case 2: Team doesn't exist in database
|
||||
with patch("litellm.proxy.proxy_server.prisma_client") as mock_prisma:
|
||||
@@ -1040,7 +1051,7 @@ async def test_unblock_key_invalid_key_format(monkeypatch):
|
||||
def test_validate_key_team_change_with_member_permissions():
|
||||
"""
|
||||
Test validate_key_team_change function with team member permissions.
|
||||
|
||||
|
||||
This test covers the new logic that allows team members with specific
|
||||
permissions to update keys, not just team admins.
|
||||
"""
|
||||
@@ -1054,111 +1065,107 @@ def test_validate_key_team_change_with_member_permissions():
|
||||
mock_key.models = ["gpt-4"]
|
||||
mock_key.tpm_limit = None
|
||||
mock_key.rpm_limit = None
|
||||
|
||||
|
||||
mock_team = MagicMock()
|
||||
mock_team.team_id = "test-team-456"
|
||||
mock_team.team_id = "test-team-456"
|
||||
mock_team.members_with_roles = []
|
||||
mock_team.tpm_limit = None
|
||||
mock_team.rpm_limit = None
|
||||
|
||||
|
||||
mock_change_initiator = MagicMock()
|
||||
mock_change_initiator.user_id = "test-user-123"
|
||||
|
||||
|
||||
mock_router = MagicMock()
|
||||
|
||||
|
||||
# Mock the member object returned by _get_user_in_team
|
||||
mock_member_object = MagicMock()
|
||||
|
||||
with patch('litellm.proxy.management_endpoints.key_management_endpoints.can_team_access_model'):
|
||||
with patch('litellm.proxy.management_endpoints.key_management_endpoints._get_user_in_team') as mock_get_user:
|
||||
with patch('litellm.proxy.management_endpoints.key_management_endpoints._is_user_team_admin') as mock_is_admin:
|
||||
with patch('litellm.proxy.management_endpoints.key_management_endpoints.TeamMemberPermissionChecks.does_team_member_have_permissions_for_endpoint') as mock_has_perms:
|
||||
|
||||
|
||||
with patch(
|
||||
"litellm.proxy.management_endpoints.key_management_endpoints.can_team_access_model"
|
||||
):
|
||||
with patch(
|
||||
"litellm.proxy.management_endpoints.key_management_endpoints._get_user_in_team"
|
||||
) as mock_get_user:
|
||||
with patch(
|
||||
"litellm.proxy.management_endpoints.key_management_endpoints._is_user_team_admin"
|
||||
) as mock_is_admin:
|
||||
with patch(
|
||||
"litellm.proxy.management_endpoints.key_management_endpoints.TeamMemberPermissionChecks.does_team_member_have_permissions_for_endpoint"
|
||||
) as mock_has_perms:
|
||||
|
||||
mock_get_user.return_value = mock_member_object
|
||||
mock_is_admin.return_value = False
|
||||
mock_has_perms.return_value = True
|
||||
|
||||
|
||||
# This should not raise an exception due to member permissions
|
||||
validate_key_team_change(
|
||||
key=mock_key,
|
||||
team=mock_team,
|
||||
change_initiated_by=mock_change_initiator,
|
||||
llm_router=mock_router
|
||||
llm_router=mock_router,
|
||||
)
|
||||
|
||||
|
||||
# Verify the permission check was called with correct parameters
|
||||
mock_has_perms.assert_called_once_with(
|
||||
team_member_object=mock_member_object,
|
||||
team_table=mock_team,
|
||||
route=KeyManagementRoutes.KEY_UPDATE.value
|
||||
route=KeyManagementRoutes.KEY_UPDATE.value,
|
||||
)
|
||||
|
||||
|
||||
def test_key_rotation_fields_helper():
|
||||
"""
|
||||
Test the key data update logic for rotation fields.
|
||||
|
||||
|
||||
This test focuses on the core logic that adds rotation fields to key_data
|
||||
when auto_rotate is enabled, without the complexity of full key generation.
|
||||
"""
|
||||
# Test Case 1: With rotation enabled
|
||||
key_data = {
|
||||
"models": ["gpt-3.5-turbo"],
|
||||
"user_id": "test-user"
|
||||
}
|
||||
|
||||
key_data = {"models": ["gpt-3.5-turbo"], "user_id": "test-user"}
|
||||
|
||||
auto_rotate = True
|
||||
rotation_interval = "30d"
|
||||
|
||||
|
||||
# Simulate the rotation logic from generate_key_helper_fn
|
||||
if auto_rotate and rotation_interval:
|
||||
key_data.update({
|
||||
"auto_rotate": auto_rotate,
|
||||
"rotation_interval": rotation_interval
|
||||
})
|
||||
|
||||
key_data.update(
|
||||
{"auto_rotate": auto_rotate, "rotation_interval": rotation_interval}
|
||||
)
|
||||
|
||||
# Verify rotation fields are added
|
||||
assert key_data["auto_rotate"] == True
|
||||
assert key_data["rotation_interval"] == "30d"
|
||||
assert key_data["models"] == ["gpt-3.5-turbo"] # Original fields preserved
|
||||
|
||||
|
||||
# Test Case 2: Without rotation enabled
|
||||
key_data2 = {
|
||||
"models": ["gpt-4"],
|
||||
"user_id": "test-user"
|
||||
}
|
||||
|
||||
key_data2 = {"models": ["gpt-4"], "user_id": "test-user"}
|
||||
|
||||
auto_rotate2 = False
|
||||
rotation_interval2 = None
|
||||
|
||||
|
||||
# Simulate the rotation logic
|
||||
if auto_rotate2 and rotation_interval2:
|
||||
key_data2.update({
|
||||
"auto_rotate": auto_rotate2,
|
||||
"rotation_interval": rotation_interval2
|
||||
})
|
||||
|
||||
key_data2.update(
|
||||
{"auto_rotate": auto_rotate2, "rotation_interval": rotation_interval2}
|
||||
)
|
||||
|
||||
# Verify rotation fields are NOT added
|
||||
assert "auto_rotate" not in key_data2
|
||||
assert "rotation_interval" not in key_data2
|
||||
assert key_data2["models"] == ["gpt-4"] # Original fields preserved
|
||||
|
||||
|
||||
# Test Case 3: auto_rotate=True but no interval
|
||||
key_data3 = {
|
||||
"models": ["claude-3"],
|
||||
"user_id": "test-user"
|
||||
}
|
||||
|
||||
key_data3 = {"models": ["claude-3"], "user_id": "test-user"}
|
||||
|
||||
auto_rotate3 = True
|
||||
rotation_interval3 = None
|
||||
|
||||
|
||||
# Simulate the rotation logic
|
||||
if auto_rotate3 and rotation_interval3:
|
||||
key_data3.update({
|
||||
"auto_rotate": auto_rotate3,
|
||||
"rotation_interval": rotation_interval3
|
||||
})
|
||||
|
||||
key_data3.update(
|
||||
{"auto_rotate": auto_rotate3, "rotation_interval": rotation_interval3}
|
||||
)
|
||||
|
||||
# Verify rotation fields are NOT added (missing interval)
|
||||
assert "auto_rotate" not in key_data3
|
||||
assert "rotation_interval" not in key_data3
|
||||
@@ -1181,27 +1188,24 @@ async def test_update_key_fn_auto_rotate_enable():
|
||||
team_id=None,
|
||||
auto_rotate=False,
|
||||
rotation_interval=None,
|
||||
metadata={}
|
||||
metadata={},
|
||||
)
|
||||
|
||||
|
||||
# Test enabling auto rotation
|
||||
update_request = UpdateKeyRequest(
|
||||
key="test-token",
|
||||
auto_rotate=True,
|
||||
rotation_interval="30d"
|
||||
key="test-token", auto_rotate=True, rotation_interval="30d"
|
||||
)
|
||||
|
||||
|
||||
result = await prepare_key_update_data(
|
||||
data=update_request,
|
||||
existing_key_row=existing_key
|
||||
data=update_request, existing_key_row=existing_key
|
||||
)
|
||||
|
||||
|
||||
# Verify rotation fields are included
|
||||
assert result["auto_rotate"] is True
|
||||
assert result["rotation_interval"] == "30d"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_key_fn_auto_rotate_disable():
|
||||
"""Test that update_key_fn properly handles disabling auto rotation."""
|
||||
from litellm.proxy._types import LiteLLM_VerificationToken, UpdateKeyRequest
|
||||
@@ -1218,19 +1222,520 @@ async def test_update_key_fn_auto_rotate_disable():
|
||||
team_id=None,
|
||||
auto_rotate=True,
|
||||
rotation_interval="30d",
|
||||
metadata={}
|
||||
metadata={},
|
||||
)
|
||||
|
||||
|
||||
# Test disabling auto rotation
|
||||
update_request = UpdateKeyRequest(
|
||||
key="test-token",
|
||||
auto_rotate=False
|
||||
)
|
||||
|
||||
update_request = UpdateKeyRequest(key="test-token", auto_rotate=False)
|
||||
|
||||
result = await prepare_key_update_data(
|
||||
data=update_request,
|
||||
existing_key_row=existing_key
|
||||
data=update_request, existing_key_row=existing_key
|
||||
)
|
||||
|
||||
|
||||
# Verify auto_rotate is set to False
|
||||
assert result["auto_rotate"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_check_team_key_limits_no_existing_keys():
|
||||
"""
|
||||
Test _check_team_key_limits when team has no existing keys.
|
||||
Should allow any TPM/RPM limits within team bounds.
|
||||
"""
|
||||
# Mock prisma client
|
||||
mock_prisma_client = AsyncMock()
|
||||
mock_prisma_client.db.litellm_verificationtoken.find_many = AsyncMock(
|
||||
return_value=[]
|
||||
)
|
||||
|
||||
# Create team table with limits
|
||||
team_table = LiteLLM_TeamTableCachedObj(
|
||||
team_id="test-team-123",
|
||||
team_alias="test-team",
|
||||
tpm_limit=10000,
|
||||
rpm_limit=1000,
|
||||
max_budget=100.0,
|
||||
spend=0.0,
|
||||
models=[],
|
||||
blocked=False,
|
||||
members_with_roles=[],
|
||||
)
|
||||
|
||||
# Create request with limits within team bounds
|
||||
data = GenerateKeyRequest(
|
||||
tpm_limit=5000,
|
||||
rpm_limit=500,
|
||||
tpm_limit_type="guaranteed_throughput",
|
||||
rpm_limit_type="guaranteed_throughput",
|
||||
)
|
||||
|
||||
# Should not raise any exception
|
||||
await _check_team_key_limits(
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
prisma_client=mock_prisma_client,
|
||||
)
|
||||
|
||||
# Verify database was queried
|
||||
mock_prisma_client.db.litellm_verificationtoken.find_many.assert_called_once_with(
|
||||
where={"team_id": "test-team-123"}
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_check_team_key_limits_with_existing_keys_within_bounds():
|
||||
"""
|
||||
Test _check_team_key_limits when team has existing keys but total allocation
|
||||
is still within team limits.
|
||||
"""
|
||||
# Create mock existing keys
|
||||
existing_key1 = MagicMock()
|
||||
existing_key1.tpm_limit = 3000
|
||||
existing_key1.rpm_limit = 200
|
||||
|
||||
existing_key2 = MagicMock()
|
||||
existing_key2.tpm_limit = 2000
|
||||
existing_key2.rpm_limit = 300
|
||||
|
||||
existing_key3 = MagicMock()
|
||||
existing_key3.tpm_limit = None # Should be ignored in calculation
|
||||
existing_key3.rpm_limit = None # Should be ignored in calculation
|
||||
|
||||
mock_prisma_client = AsyncMock()
|
||||
mock_prisma_client.db.litellm_verificationtoken.find_many = AsyncMock(
|
||||
return_value=[existing_key1, existing_key2, existing_key3]
|
||||
)
|
||||
|
||||
# Create team table with limits
|
||||
team_table = LiteLLM_TeamTableCachedObj(
|
||||
team_id="test-team-456",
|
||||
team_alias="test-team",
|
||||
tpm_limit=10000, # Total: 3000 + 2000 + 4000 (new) = 9000 < 10000 ✓
|
||||
rpm_limit=1000, # Total: 200 + 300 + 400 (new) = 900 < 1000 ✓
|
||||
max_budget=100.0,
|
||||
spend=0.0,
|
||||
models=[],
|
||||
blocked=False,
|
||||
members_with_roles=[],
|
||||
)
|
||||
|
||||
# Create request that would still be within bounds
|
||||
data = GenerateKeyRequest(
|
||||
tpm_limit=4000,
|
||||
rpm_limit=400,
|
||||
)
|
||||
|
||||
# Should not raise any exception
|
||||
await _check_team_key_limits(
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
prisma_client=mock_prisma_client,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_check_team_key_limits_tpm_overallocation():
|
||||
"""
|
||||
Test _check_team_key_limits when new key would cause TPM overallocation.
|
||||
Should raise HTTPException with appropriate error message.
|
||||
"""
|
||||
# Create mock existing keys with high TPM usage
|
||||
existing_key1 = MagicMock()
|
||||
existing_key1.tpm_limit = 6000
|
||||
existing_key1.rpm_limit = 100
|
||||
existing_key1.metadata = {}
|
||||
|
||||
existing_key2 = MagicMock()
|
||||
existing_key2.tpm_limit = 3000
|
||||
existing_key2.rpm_limit = 200
|
||||
existing_key2.metadata = {}
|
||||
|
||||
mock_prisma_client = AsyncMock()
|
||||
mock_prisma_client.db.litellm_verificationtoken.find_many = AsyncMock(
|
||||
return_value=[existing_key1, existing_key2]
|
||||
)
|
||||
|
||||
# Create team table with limits
|
||||
team_table = LiteLLM_TeamTableCachedObj(
|
||||
team_id="test-team-789",
|
||||
team_alias="test-team",
|
||||
tpm_limit=10000, # Allocated: 6000 + 3000 = 9000, New: 2000, Total: 11000 > 10000 ✗
|
||||
rpm_limit=1000,
|
||||
max_budget=100.0,
|
||||
spend=0.0,
|
||||
models=[],
|
||||
blocked=False,
|
||||
members_with_roles=[],
|
||||
)
|
||||
|
||||
# Create request that would exceed TPM limits
|
||||
data = GenerateKeyRequest(
|
||||
tpm_limit=2000,
|
||||
rpm_limit=100,
|
||||
tpm_limit_type="guaranteed_throughput",
|
||||
)
|
||||
|
||||
# Should raise HTTPException for TPM overallocation
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await _check_team_key_limits(
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
prisma_client=mock_prisma_client,
|
||||
)
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
assert (
|
||||
"Allocated TPM limit=9000 + Key TPM limit=2000 is greater than team TPM limit=10000"
|
||||
in str(exc_info.value.detail)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_check_team_key_limits_rpm_overallocation():
|
||||
"""
|
||||
Test _check_team_key_limits when new key would cause RPM overallocation.
|
||||
Should raise HTTPException with appropriate error message.
|
||||
"""
|
||||
# Create mock existing keys with high RPM usage
|
||||
existing_key1 = MagicMock()
|
||||
existing_key1.tpm_limit = 1000
|
||||
existing_key1.rpm_limit = 600
|
||||
existing_key1.metadata = {}
|
||||
|
||||
existing_key2 = MagicMock()
|
||||
existing_key2.tpm_limit = 2000
|
||||
existing_key2.rpm_limit = 300
|
||||
existing_key2.metadata = {}
|
||||
|
||||
mock_prisma_client = AsyncMock()
|
||||
mock_prisma_client.db.litellm_verificationtoken.find_many = AsyncMock(
|
||||
return_value=[existing_key1, existing_key2]
|
||||
)
|
||||
|
||||
# Create team table with limits
|
||||
team_table = LiteLLM_TeamTableCachedObj(
|
||||
team_id="test-team-101",
|
||||
team_alias="test-team",
|
||||
tpm_limit=10000,
|
||||
rpm_limit=1000, # Allocated: 600 + 300 = 900, New: 200, Total: 1100 > 1000 ✗
|
||||
max_budget=100.0,
|
||||
spend=0.0,
|
||||
models=[],
|
||||
blocked=False,
|
||||
members_with_roles=[],
|
||||
)
|
||||
|
||||
# Create request that would exceed RPM limits
|
||||
data = GenerateKeyRequest(
|
||||
tpm_limit=1000,
|
||||
rpm_limit=200,
|
||||
rpm_limit_type="guaranteed_throughput",
|
||||
)
|
||||
|
||||
# Should raise HTTPException for RPM overallocation
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await _check_team_key_limits(
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
prisma_client=mock_prisma_client,
|
||||
)
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
assert (
|
||||
"Allocated RPM limit=900 + Key RPM limit=200 is greater than team RPM limit=1000"
|
||||
in str(exc_info.value.detail)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_check_team_key_limits_no_team_limits():
|
||||
"""
|
||||
Test _check_team_key_limits when team has no TPM/RPM limits set.
|
||||
Should allow any key limits since there are no team constraints.
|
||||
"""
|
||||
# Create mock existing keys
|
||||
existing_key = MagicMock()
|
||||
existing_key.tpm_limit = 5000
|
||||
existing_key.rpm_limit = 500
|
||||
|
||||
mock_prisma_client = AsyncMock()
|
||||
mock_prisma_client.db.litellm_verificationtoken.find_many = AsyncMock(
|
||||
return_value=[existing_key]
|
||||
)
|
||||
|
||||
# Create team table with no limits
|
||||
team_table = LiteLLM_TeamTableCachedObj(
|
||||
team_id="test-team-202",
|
||||
team_alias="test-team",
|
||||
tpm_limit=None, # No team limit
|
||||
rpm_limit=None, # No team limit
|
||||
max_budget=100.0,
|
||||
spend=0.0,
|
||||
models=[],
|
||||
blocked=False,
|
||||
members_with_roles=[],
|
||||
)
|
||||
|
||||
# Create request with any limits
|
||||
data = GenerateKeyRequest(
|
||||
tpm_limit=10000, # High limit should be allowed
|
||||
rpm_limit=2000, # High limit should be allowed
|
||||
)
|
||||
|
||||
# Should not raise any exception
|
||||
await _check_team_key_limits(
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
prisma_client=mock_prisma_client,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_check_team_key_limits_no_key_limits():
|
||||
"""
|
||||
Test _check_team_key_limits when new key has no TPM/RPM limits.
|
||||
Should not raise any exceptions since no limits are being allocated.
|
||||
"""
|
||||
# Create mock existing keys
|
||||
existing_key = MagicMock()
|
||||
existing_key.tpm_limit = 8000
|
||||
existing_key.rpm_limit = 800
|
||||
|
||||
mock_prisma_client = AsyncMock()
|
||||
mock_prisma_client.db.litellm_verificationtoken.find_many = AsyncMock(
|
||||
return_value=[existing_key]
|
||||
)
|
||||
|
||||
# Create team table with limits
|
||||
team_table = LiteLLM_TeamTableCachedObj(
|
||||
team_id="test-team-303",
|
||||
team_alias="test-team",
|
||||
tpm_limit=10000,
|
||||
rpm_limit=1000,
|
||||
max_budget=100.0,
|
||||
spend=0.0,
|
||||
models=[],
|
||||
blocked=False,
|
||||
members_with_roles=[],
|
||||
)
|
||||
|
||||
# Create request with no limits
|
||||
data = GenerateKeyRequest(
|
||||
tpm_limit=None, # No limit being set
|
||||
rpm_limit=None, # No limit being set
|
||||
)
|
||||
|
||||
# Should not raise any exception
|
||||
await _check_team_key_limits(
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
prisma_client=mock_prisma_client,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_check_team_key_limits_mixed_scenarios():
|
||||
"""
|
||||
Test _check_team_key_limits with mixed scenarios:
|
||||
- Some existing keys have limits, others don't
|
||||
- New key has only one type of limit
|
||||
- Team has only one type of limit
|
||||
"""
|
||||
# Create mock existing keys with mixed limits
|
||||
existing_key1 = MagicMock()
|
||||
existing_key1.tpm_limit = 3000
|
||||
existing_key1.rpm_limit = None # No RPM limit
|
||||
|
||||
existing_key2 = MagicMock()
|
||||
existing_key2.tpm_limit = None # No TPM limit
|
||||
existing_key2.rpm_limit = 400
|
||||
|
||||
existing_key3 = MagicMock()
|
||||
existing_key3.tpm_limit = 2000
|
||||
existing_key3.rpm_limit = 300
|
||||
|
||||
mock_prisma_client = AsyncMock()
|
||||
mock_prisma_client.db.litellm_verificationtoken.find_many = AsyncMock(
|
||||
return_value=[existing_key1, existing_key2, existing_key3]
|
||||
)
|
||||
|
||||
# Create team table with only TPM limit
|
||||
team_table = LiteLLM_TeamTableCachedObj(
|
||||
team_id="test-team-404",
|
||||
team_alias="test-team",
|
||||
tpm_limit=10000, # Allocated: 3000 + 0 + 2000 = 5000, New: 4000, Total: 9000 < 10000 ✓
|
||||
rpm_limit=None, # No team RPM limit
|
||||
max_budget=100.0,
|
||||
spend=0.0,
|
||||
models=[],
|
||||
blocked=False,
|
||||
members_with_roles=[],
|
||||
)
|
||||
|
||||
# Create request with only TPM limit
|
||||
data = GenerateKeyRequest(
|
||||
tpm_limit=4000,
|
||||
rpm_limit=None, # No RPM limit being set
|
||||
)
|
||||
|
||||
# Should not raise any exception
|
||||
await _check_team_key_limits(
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
prisma_client=mock_prisma_client,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_check_team_key_limits_exact_boundary():
|
||||
"""
|
||||
Test _check_team_key_limits when allocation exactly matches team limits.
|
||||
Should allow the allocation (boundary case).
|
||||
"""
|
||||
# Create mock existing keys
|
||||
existing_key = MagicMock()
|
||||
existing_key.tpm_limit = 7000
|
||||
existing_key.rpm_limit = 700
|
||||
|
||||
mock_prisma_client = AsyncMock()
|
||||
mock_prisma_client.db.litellm_verificationtoken.find_many = AsyncMock(
|
||||
return_value=[existing_key]
|
||||
)
|
||||
|
||||
# Create team table with limits
|
||||
team_table = LiteLLM_TeamTableCachedObj(
|
||||
team_id="test-team-505",
|
||||
team_alias="test-team",
|
||||
tpm_limit=10000, # Allocated: 7000, New: 3000, Total: 10000 = 10000 ✓
|
||||
rpm_limit=1000, # Allocated: 700, New: 300, Total: 1000 = 1000 ✓
|
||||
max_budget=100.0,
|
||||
spend=0.0,
|
||||
models=[],
|
||||
blocked=False,
|
||||
members_with_roles=[],
|
||||
)
|
||||
|
||||
# Create request that exactly matches remaining capacity
|
||||
data = GenerateKeyRequest(
|
||||
tpm_limit=3000,
|
||||
rpm_limit=300,
|
||||
)
|
||||
|
||||
# Should not raise any exception (exact boundary should be allowed)
|
||||
await _check_team_key_limits(
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
prisma_client=mock_prisma_client,
|
||||
)
|
||||
|
||||
|
||||
def test_check_team_key_model_specific_limits_no_limits():
|
||||
"""
|
||||
Test check_team_key_model_specific_limits when no model-specific limits are set.
|
||||
Should return without raising any exceptions.
|
||||
"""
|
||||
# Create existing key with no model-specific limits
|
||||
existing_key = LiteLLM_VerificationToken(
|
||||
token="test-token-1",
|
||||
user_id="test-user",
|
||||
team_id="test-team-123",
|
||||
metadata={},
|
||||
)
|
||||
|
||||
keys = [existing_key]
|
||||
|
||||
# Create team table
|
||||
team_table = LiteLLM_TeamTableCachedObj(
|
||||
team_id="test-team-123",
|
||||
team_alias="test-team",
|
||||
tpm_limit=10000,
|
||||
rpm_limit=1000,
|
||||
max_budget=100.0,
|
||||
spend=0.0,
|
||||
models=[],
|
||||
blocked=False,
|
||||
members_with_roles=[],
|
||||
metadata={},
|
||||
)
|
||||
|
||||
# Create request with no model-specific limits
|
||||
data = GenerateKeyRequest(
|
||||
model_rpm_limit=None,
|
||||
model_tpm_limit=None,
|
||||
)
|
||||
|
||||
# Should not raise any exception
|
||||
check_team_key_model_specific_limits(
|
||||
keys=keys,
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
)
|
||||
|
||||
|
||||
def test_check_team_key_model_specific_limits_rpm_overallocation():
|
||||
"""
|
||||
Test check_team_key_model_specific_limits when model-specific RPM would cause overallocation.
|
||||
Should raise HTTPException with appropriate error message.
|
||||
"""
|
||||
# Create existing keys with model-specific RPM limits
|
||||
existing_key1 = LiteLLM_VerificationToken(
|
||||
token="test-token-1",
|
||||
user_id="test-user-1",
|
||||
team_id="test-team-456",
|
||||
metadata={
|
||||
"model_rpm_limit": {
|
||||
"gpt-4": 500,
|
||||
"gpt-3.5-turbo": 300,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
existing_key2 = LiteLLM_VerificationToken(
|
||||
token="test-token-2",
|
||||
user_id="test-user-2",
|
||||
team_id="test-team-456",
|
||||
metadata={
|
||||
"model_rpm_limit": {
|
||||
"gpt-4": 300,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
keys = [existing_key1, existing_key2]
|
||||
|
||||
# Create team table with RPM limit
|
||||
team_table = LiteLLM_TeamTableCachedObj(
|
||||
team_id="test-team-456",
|
||||
team_alias="test-team",
|
||||
tpm_limit=10000,
|
||||
rpm_limit=1000, # Total team RPM limit
|
||||
max_budget=100.0,
|
||||
spend=0.0,
|
||||
models=[],
|
||||
blocked=False,
|
||||
members_with_roles=[],
|
||||
metadata={},
|
||||
)
|
||||
|
||||
# Create request that would exceed model-specific RPM limits
|
||||
# Existing gpt-4: 500 + 300 = 800, New: 300, Total: 1100 > 1000 (team limit)
|
||||
data = GenerateKeyRequest(
|
||||
model_rpm_limit={
|
||||
"gpt-4": 300, # This would cause overallocation
|
||||
},
|
||||
model_tpm_limit=None,
|
||||
)
|
||||
|
||||
# Should raise HTTPException for model-specific RPM overallocation
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
check_team_key_model_specific_limits(
|
||||
keys=keys,
|
||||
team_table=team_table,
|
||||
data=data,
|
||||
)
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
assert (
|
||||
"Allocated RPM limit=800 + Key RPM limit=300 is greater than team RPM limit=1000"
|
||||
in str(exc_info.value.detail)
|
||||
)
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
node_modules
|
||||
.next
|
||||
.out
|
||||
dist
|
||||
build
|
||||
.coverage
|
||||
.vercel
|
||||
.turbo
|
||||
.next-static
|
||||
*.min.js
|
||||
coverage/
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"semi": true,
|
||||
"singleQuote": false,
|
||||
"tabWidth": 2,
|
||||
"printWidth": 120,
|
||||
"trailingComma": "all"
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"semi": false,
|
||||
"tabWidth": 2,
|
||||
"printWidth": 120,
|
||||
"trailingComma": "all",
|
||||
"jsxBracketSameLine": false
|
||||
}
|
||||
@@ -1,12 +1,12 @@
|
||||
/** @type {import('next').NextConfig} */
|
||||
const nextConfig = {
|
||||
output: 'export',
|
||||
basePath: '',
|
||||
assetPrefix: '/litellm-asset-prefix', // If a server_root_path is set, this will be overridden by runtime injection
|
||||
output: "export",
|
||||
basePath: "",
|
||||
assetPrefix: "/litellm-asset-prefix", // If a server_root_path is set, this will be overridden by runtime injection
|
||||
};
|
||||
|
||||
nextConfig.experimental = {
|
||||
missingSuspenseWithCSRBailout: false
|
||||
}
|
||||
missingSuspenseWithCSRBailout: false,
|
||||
};
|
||||
|
||||
export default nextConfig;
|
||||
|
||||
Generated
+1
@@ -17973,6 +17973,7 @@
|
||||
"resolved": "https://registry.npmjs.org/prettier/-/prettier-3.2.5.tgz",
|
||||
"integrity": "sha512-3/GWa9aOC0YeD7LUfvOG2NiDyhOWRvt1k+rcKhOuYnMY24iiCphgneUfJDyFXd6rZCAnuLBv6UeAULtrhT/F4A==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"bin": {
|
||||
"prettier": "bin/prettier.cjs"
|
||||
},
|
||||
|
||||
@@ -8,7 +8,9 @@
|
||||
"start": "next start",
|
||||
"lint": "next lint",
|
||||
"test": "vitest",
|
||||
"test:watch": "vitest -w"
|
||||
"test:watch": "vitest -w",
|
||||
"format": "prettier --write .",
|
||||
"format:check": "prettier --check ."
|
||||
},
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sdk": "^0.54.0",
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user