tiennm99/litellm
1
0
Fork 0
mirror of https://github.com/tiennm99/litellm.git synced 2026-08-11 00:25:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'BerriAI:main' into dev

Browse Source
This commit is contained in:
NULL
2025-08-12 16:51:25 +08:00
committed by GitHub
parent c75e7230d2 008ea864a7
commit 4d2b5076a0
144 changed files with 6105 additions and 1170 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.circleci/config.yml
+1
View File
@@ -957,6 +957,7 @@ jobs:
pip install "responses==0.25.7"
pip install "pytest-xdist==3.6.1"
pip install "semantic_router==0.1.10"
pip install "fastapi-offline==1.7.3"
- setup_litellm_enterprise_pip
# Run pytest and generate JUnit XML report
- run:
.github/workflows/test-litellm.yml
+1
View File
@@ -31,6 +31,7 @@ jobs:
poetry run pip install "pytest-retry==1.6.3"
poetry run pip install pytest-xdist
poetry run pip install "google-genai==1.22.0"
poetry run pip install "fastapi-offline==1.7.3"
- name: Setup litellm-enterprise as local package
run: |
cd enterprise
README.md
+7 -6
View File
@@ -47,7 +47,7 @@ Support for more providers. Missing a provider or LLM Platform, raise a [feature
# Usage ([**Docs**](https://docs.litellm.ai/docs/))
> [!IMPORTANT]
> LiteLLM v1.0.0 now requires `openai>=1.0.0`. Migration guide [here](https://docs.litellm.ai/docs/migration)
> LiteLLM v1.0.0 now requires `openai>=1.0.0`. Migration guide [here](https://docs.litellm.ai/docs/migration)
> LiteLLM v1.40.14+ now requires `pydantic>=2.0.0`. No changes required.
<a target="_blank" href="https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/liteLLM_Getting_Started.ipynb">
@@ -132,7 +132,7 @@ print(response)
## Streaming ([Docs](https://docs.litellm.ai/docs/completion/stream))
liteLLM supports streaming the model response back, pass `stream=True` to get a streaming iterator in response.
liteLLM supports streaming the model response back, pass `stream=True` to get a streaming iterator in response.
Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)
```python
@@ -234,7 +234,7 @@ $ litellm --model huggingface/bigcode/starcoder
> [!IMPORTANT]
> 💡 [Use LiteLLM Proxy with Langchain (Python, JS), OpenAI SDK (Python, JS) Anthropic SDK, Mistral SDK, LlamaIndex, Instructor, Curl](https://docs.litellm.ai/docs/proxy/user_keys)
> 💡 [Use LiteLLM Proxy with Langchain (Python, JS), OpenAI SDK (Python, JS) Anthropic SDK, Mistral SDK, LlamaIndex, Instructor, Curl](https://docs.litellm.ai/docs/proxy/user_keys)
```python
import openai # openai v1.0.0+
@@ -266,7 +266,7 @@ echo 'LITELLM_MASTER_KEY="sk-1234"' > .env
# Add the litellm salt key - you cannot change this after adding a model
# It is used to encrypt / decrypt your LLM API Key credentials
# We recommend - https://1password.com/password-generator/
# We recommend - https://1password.com/password-generator/
# password generator to get a random hash for litellm salt key
echo 'LITELLM_SALT_KEY="sk-1234"' >> .env
@@ -340,6 +340,7 @@ curl 'http://0.0.0.0:4000/key/generate' \
| [xinference [Xorbits Inference]](https://docs.litellm.ai/docs/providers/xinference) | | | | | ✅ | |
| [FriendliAI](https://docs.litellm.ai/docs/providers/friendliai) | ✅ | ✅ | ✅ | ✅ | | |
| [Galadriel](https://docs.litellm.ai/docs/providers/galadriel) | ✅ | ✅ | ✅ | ✅ | | |
| [GradientAI](https://docs.litellm.ai/docs/providers/gradient_ai) | ✅ | ✅ | | | | |
| [Novita AI](https://novita.ai/models/llm?utm_source=github_litellm&utm_medium=github_readme&utm_campaign=github_link) | ✅ | ✅ | ✅ | ✅ | | |
| [Featherless AI](https://docs.litellm.ai/docs/providers/featherless_ai) | ✅ | ✅ | ✅ | ✅ | | |
| [Nebius AI Studio](https://docs.litellm.ai/docs/providers/nebius) | ✅ | ✅ | ✅ | ✅ | ✅ | |
@@ -348,7 +349,7 @@ curl 'http://0.0.0.0:4000/key/generate' \
## Contributing
Interested in contributing? Contributions to LiteLLM Python SDK, Proxy Server, and LLM integrations are both accepted and highly encouraged!
Interested in contributing? Contributions to LiteLLM Python SDK, Proxy Server, and LLM integrations are both accepted and highly encouraged!
**Quick start:** `git clone` → `make install-dev` → `make format` → `make lint` → `make test-unit`
@@ -359,7 +360,7 @@ For companies that need better security, user management and professional suppor
[Talk to founders](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
This covers:
This covers:
- ✅ **Features under the [LiteLLM Commercial License](https://docs.litellm.ai/docs/proxy/enterprise):**
- ✅ **Feature Prioritization**
- ✅ **Custom Integrations**
docs/my-website/docs/providers/azure/azure.md
+79 -2
View File
@@ -9,8 +9,8 @@ import TabItem from '@theme/TabItem';
| Property | Details |
|-------|-------|
| Description | Azure OpenAI Service provides REST API access to OpenAI's powerful language models including o1, o1-mini, GPT-4o, GPT-4o mini, GPT-4 Turbo with Vision, GPT-4, GPT-3.5-Turbo, and Embeddings model series |
| Provider Route on LiteLLM | `azure/`, [`azure/o_series/`](#azure-o-series-models) |
| Description | Azure OpenAI Service provides REST API access to OpenAI's powerful language models including o1, o1-mini, GPT-5, GPT-4o, GPT-4o mini, GPT-4 Turbo with Vision, GPT-4, GPT-3.5-Turbo, and Embeddings model series |
| Provider Route on LiteLLM | `azure/`, [`azure/o_series/`](#o-series-models), [`azure/gpt5_series/`](#gpt-5-models) |
| Supported Operations | [`/chat/completions`](#azure-openai-chat-completion-models), [`/responses`](./azure_responses), [`/completions`](#azure-instruct-models), [`/embeddings`](./azure_embedding), [`/audio/speech`](#azure-text-to-speech-tts), [`/audio/transcriptions`](../audio_transcription), `/fine_tuning`, [`/batches`](#azure-batches-api), `/files`, [`/images`](../image_generation#azure-openai-image-generation-models) |
| Link to Provider Doc | [Azure OpenAI ↗](https://learn.microsoft.com/en-us/azure/ai-services/openai/overview)
@@ -207,6 +207,7 @@ model_list:
|------------------|----------------------------------------|
| o1-mini | `response = completion(model="azure/<your deployment name>", messages=messages)` |
| o1-preview | `response = completion(model="azure/<your deployment name>", messages=messages)` |
| gpt-5 | `response = completion(model="azure/<your deployment name>", messages=messages)` |
| gpt-4o-mini | `completion('azure/<your deployment name>', messages)` |
| gpt-4o | `completion('azure/<your deployment name>', messages)` |
| gpt-4 | `completion('azure/<your deployment name>', messages)` |
@@ -368,6 +369,82 @@ model_list:
</Tabs>
## GPT-5 Models
| Property | Details |
|-------|-------|
| Description | Azure OpenAI GPT-5 models |
| Provider Route on LiteLLM | `azure/gpt5_series/<custom-name>` or `azure/gpt-5-deployment-name` |
LiteLLM supports using Azure GPT-5 models in one of the two ways:
1. Explicit Routing: `model = azure/gpt5_series/<deployment-name>`. In this scenario the model onboarded to litellm follows the format `model=azure/gpt5_series/<deployment-name>`.
2. Inferred Routing (If the azure deployment name contains `gpt-5` in the name): `model = azure/gpt-5-mini`. In this scenario the model onboarded to litellm follows the format `model=azure/gpt-5-mini`.
#### Explicit Routing
Use `azure/gpt5_series/<deployment-name>` for explicit GPT-5 model routing.
<Tabs>
<TabItem value="sdk" label="SDK">
```python
import litellm
response = litellm.completion(
model="azure/gpt5_series/my-gpt-5-deployment",
messages=[{"role": "user", "content": "Hello, world!"}]
)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
```yaml
model_list:
- model_name: gpt-5
litellm_params:
model: azure/gpt5_series/my-gpt-5-deployment
api_base: os.environ/AZURE_API_BASE
api_key: os.environ/AZURE_API_KEY
```
</TabItem>
</Tabs>
#### Inferred Routing (gpt-5 in the deployment name)
If your Azure deployment name contains `gpt-5`, LiteLLM automatically recognizes it as a GPT-5 model.
<Tabs>
<TabItem value="sdk" label="SDK">
```python
import litellm
# Deployment name contains 'gpt-5' - automatically inferred
response = litellm.completion(
model="azure/my-gpt-5-deployment",
messages=[{"role": "user", "content": "Hello, world!"}]
)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
```yaml
model_list:
- model_name: gpt-5-mini
litellm_params:
model: azure/my-gpt-5-deployment # deployment name contains 'gpt-5'
api_base: os.environ/AZURE_API_BASE
api_key: os.environ/AZURE_API_KEY
```
</TabItem>
</Tabs>
## Azure Audio Model
<Tabs>
docs/my-website/docs/providers/gradient_ai.md
+79
View File
@@ -0,0 +1,79 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# GradientAI
https://digitalocean.com/products/gradientai
LiteLLM provides native support for GradientAI models.
To use a GradientAI model, specify it as `gradient_ai/<model-name>` in your LiteLLM requests.
## API Key & Endpoint
Set your credentials and endpoint as environment variables:
```python
import os
os.environ['GRADIENT_AI_API_KEY'] = "your-api-key"
os.environ['GRADIENT_AI_AGENT_ENDPOINT'] = "https://api.gradient_ai.com/api/v1/chat" # default endpoint
```
## Sample Usage
```python
from litellm import completion
import os
os.environ['GRADIENT_AI_API_KEY'] = "your-api-key"
response = completion(
model="gradient_ai/model-name",
messages=[
{"role": "user", "content": "Hello, how are you?"}
],
)
print(response.choices[0].message.content)
```
## Streaming Example
```python
from litellm import completion
import os
os.environ['GRADIENT_AI_API_KEY'] = "your-api-key"
response = completion(
model="gradient_ai/model-name",
messages=[
{"role": "user", "content": "Write a story about a robot learning to love"}
],
stream=True,
)
for chunk in response:
print(chunk.choices[0].delta.content or "", end="")
```
## Supported Parameters
| Parameter | Type | Description |
|-----------------------------------|--------------|--------------------------------------------------------------------|
| `temperature` | float | Controls randomness (0.0-2.0) |
| `top_p` | float | Nucleus sampling parameter (0.0-1.0) |
| `max_tokens` | int | Maximum tokens to generate |
| `max_completion_tokens` | int | Alternative to max_tokens |
| `stream` | bool | Whether to stream the response |
| `k` | int | Top results to return from knowledge bases |
| `retrieval_method` | string | Retrieval strategy (rewrite/step_back/sub_queries/none) |
| `frequency_penalty` | float | Penalizes repeated tokens (-2.0 to 2.0) |
| `presence_penalty` | float | Penalizes tokens based on presence (-2.0 to 2.0) |
| `stop` | string/list | Sequences to stop generation |
| `kb_filters` | List[Dict] | Filters for knowledge base retrieval |
| `instruction_override` | string | Override agent's default instruction |
| `include_retrieval_info` | bool | Include document retrieval metadata |
| `include_guardrails_info` | bool | Include guardrail trigger metadata |
| `provide_citations` | bool | Include citations in response |
---
For more details, see [DigitalOcean GradientAI documentation](https://digitalocean.com/products/gradientai).
docs/my-website/docs/providers/litellm_proxy.md
+35
View File
@@ -211,3 +211,38 @@ response = litellm.completion(
use_litellm_proxy=True
)
```
## Sending `tags` to LiteLLM Proxy
Tags allow you to categorize and track your API requests for monitoring, debugging, and analytics purposes. You can send tags as a list of strings to the LiteLLM Proxy using the `extra_body` parameter.
### Usage
Send tags by including them in the `extra_body` parameter of your completion request:
```python showLineNumbers title="Usage"
import litellm
response = litellm.completion(
model="gpt-4",
messages=[{"role": "user", "content": "What is the capital of France?"}],
api_base="http://localhost:4000",
api_key="sk-1234",
extra_body={"tags": ["user:ishaan", "department:engineering", "priority:high"]}
)
```
### Async Usage
```python showLineNumbers title="Async Usage"
import litellm
response = await litellm.acompletion(
model="gpt-4",
messages=[{"role": "user", "content": "What is the capital of France?"}],
api_base="http://localhost:4000",
api_key="sk-1234",
extra_body={"tags": ["user:ishaan", "department:engineering"]}
)
```
docs/my-website/docs/providers/openai/responses_api.md
+352
View File
@@ -492,3 +492,355 @@ print(response_with_mcp_call)
</Tabs>
## Verbosity Parameter
The `verbosity` parameter is supported for the `responses` API.
<Tabs>
<TabItem value="sdk" label="LiteLLM Python SDK">
```python showLineNumbers title="Verbosity Parameter"
from litellm import responses
question = "Write a poem about a boy and his first pet dog."
for verbosity in ["low", "medium", "high"]:
response = responses(
model="gpt-5-mini",
input=question,
text={"verbosity": verbosity}
)
print(response)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
```python
from openai import OpenAI
import pandas as pd
from IPython.display import display
client = OpenAI(
base_url="http://localhost:4000", # Your proxy URL
api_key="your-api-key" # Your proxy API key
)
question = "Write a poem about a boy and his first pet dog."
data = []
for verbosity in ["low", "medium", "high"]:
response = client.responses.create(
model="gpt-5-mini",
input=question,
text={"verbosity": verbosity}
)
# Extract text
output_text = ""
for item in response.output:
if hasattr(item, "content"):
for content in item.content:
if hasattr(content, "text"):
output_text += content.text
usage = response.usage
data.append({
"Verbosity": verbosity,
"Sample Output": output_text,
"Output Tokens": usage.output_tokens
})
# Create DataFrame
df = pd.DataFrame(data)
# Display nicely with centered headers
pd.set_option('display.max_colwidth', None)
styled_df = df.style.set_table_styles(
[
{'selector': 'th', 'props': [('text-align', 'center')]}, # Center column headers
{'selector': 'td', 'props': [('text-align', 'left')]} # Left-align table cells
]
)
display(styled_df)
```
</TabItem>
</Tabs>
## Free-form Function Calling
<Tabs>
<TabItem value="sdk" label="LiteLLM Python SDK">
```python showLineNumbers title="Free-form Function Calling"
import litellm
response = litellm.responses(
response = client.responses.create(
model="gpt-5-mini",
input="Please use the code_exec tool to calculate the area of a circle with radius equal to the number of 'r's in strawberry",
text={"format": {"type": "text"}},
tools=[
{
"type": "custom",
"name": "code_exec",
"description": "Executes arbitrary python code",
}
]
)
print(response.output)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
```python showLineNumbers title="Free-form Function Calling"
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:4000", # Your proxy URL
api_key="your-api-key" # Your proxy API key
)
response = client.responses.create(
model="gpt-5-mini",
input="Please use the code_exec tool to calculate the area of a circle with radius equal to the number of 'r's in strawberry",
text={"format": {"type": "text"}},
tools=[
{
"type": "custom",
"name": "code_exec",
"description": "Executes arbitrary python code",
}
]
)
print(response.output)
```
</TabItem>
</Tabs>
## Context-Free Grammar
<Tabs>
<TabItem value="sdk" label="LiteLLM Python SDK">
```python showLineNumbers title="Context-Free Grammar"
import litellm
import textwrap
# ----------------- grammars for MS SQL dialect -----------------
mssql_grammar = textwrap.dedent(r"""
// ---------- Punctuation & operators ----------
SP: " "
COMMA: ","
GT: ">"
EQ: "="
SEMI: ";"
// ---------- Start ----------
start: "SELECT" SP "TOP" SP NUMBER SP select_list SP "FROM" SP table SP "WHERE" SP amount_filter SP "AND" SP date_filter SP "ORDER" SP "BY" SP sort_cols SEMI
// ---------- Projections ----------
select_list: column (COMMA SP column)*
column: IDENTIFIER
// ---------- Tables ----------
table: IDENTIFIER
// ---------- Filters ----------
amount_filter: "total_amount" SP GT SP NUMBER
date_filter: "order_date" SP GT SP DATE
// ---------- Sorting ----------
sort_cols: "order_date" SP "DESC"
// ---------- Terminals ----------
IDENTIFIER: /[A-Za-z_][A-Za-z0-9_]*/
NUMBER: /[0-9]+/
DATE: /'[0-9]{4}-[0-9]{2}-[0-9]{2}'/
""")
sql_prompt_mssql = (
"Call the mssql_grammar to generate a query for Microsoft SQL Server that retrieve the "
"five most recent orders per customer, showing customer_id, order_id, order_date, and total_amount, "
"where total_amount > 500 and order_date is after '2025-01-01'. "
)
response = litellm.responses(
model="gpt-5",
input=sql_prompt_mssql,
text={"format": {"type": "text"}},
tools=[
{
"type": "custom",
"name": "mssql_grammar",
"description": "Executes read-only Microsoft SQL Server queries limited to SELECT statements with TOP and basic WHERE/ORDER BY. YOU MUST REASON HEAVILY ABOUT THE QUERY AND MAKE SURE IT OBEYS THE GRAMMAR.",
"format": {
"type": "grammar",
"syntax": "lark",
"definition": mssql_grammar
}
},
],
parallel_tool_calls=False
)
print("--- MS SQL Query ---")
print(response_mssql.output[1].input)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
```python showLineNumbers title="Context-Free Grammar"
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:4000", # Your proxy URL
api_key="your-api-key" # Your proxy API key
)
import textwrap
# ----------------- grammars for MS SQL dialect -----------------
mssql_grammar = textwrap.dedent(r"""
// ---------- Punctuation & operators ----------
SP: " "
COMMA: ","
GT: ">"
EQ: "="
SEMI: ";"
// ---------- Start ----------
start: "SELECT" SP "TOP" SP NUMBER SP select_list SP "FROM" SP table SP "WHERE" SP amount_filter SP "AND" SP date_filter SP "ORDER" SP "BY" SP sort_cols SEMI
// ---------- Projections ----------
select_list: column (COMMA SP column)*
column: IDENTIFIER
// ---------- Tables ----------
table: IDENTIFIER
// ---------- Filters ----------
amount_filter: "total_amount" SP GT SP NUMBER
date_filter: "order_date" SP GT SP DATE
// ---------- Sorting ----------
sort_cols: "order_date" SP "DESC"
// ---------- Terminals ----------
IDENTIFIER: /[A-Za-z_][A-Za-z0-9_]*/
NUMBER: /[0-9]+/
DATE: /'[0-9]{4}-[0-9]{2}-[0-9]{2}'/
""")
sql_prompt_mssql = (
"Call the mssql_grammar to generate a query for Microsoft SQL Server that retrieve the "
"five most recent orders per customer, showing customer_id, order_id, order_date, and total_amount, "
"where total_amount > 500 and order_date is after '2025-01-01'. "
)
response = client.responses.create(
model="gpt-5",
input=sql_prompt_mssql,
text={"format": {"type": "text"}},
tools=[
{
"type": "custom",
"name": "mssql_grammar",
"description": "Executes read-only Microsoft SQL Server queries limited to SELECT statements with TOP and basic WHERE/ORDER BY. YOU MUST REASON HEAVILY ABOUT THE QUERY AND MAKE SURE IT OBEYS THE GRAMMAR.",
"format": {
"type": "grammar",
"syntax": "lark",
"definition": mssql_grammar
}
},
],
parallel_tool_calls=False
)
print("--- MS SQL Query ---")
print(response_mssql.output[1].input)
```
</TabItem>
</Tabs>
## Minimal Reasoning
<Tabs>
<TabItem value="sdk" label="LiteLLM Python SDK">
```python showLineNumbers title="Minimal Reasoning"
import litellm
response = litellm.responses(
model="gpt-5",
input= [{ 'role': 'developer', 'content': prompt },
{ 'role': 'user', 'content': 'The food that the restaurant was great! I recommend it to everyone.' }],
reasoning = {
"effort": "minimal"
},
)
print(response)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
```python showLineNumbers title="Minimal Reasoning"
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:4000", # Your proxy URL
api_key="your-api-key" # Your proxy API key
)
prompt = "Classify sentiment of the review as positive|neutral|negative. Return one word only."
response = client.responses.create(
model="gpt-5",
input= [{ 'role': 'developer', 'content': prompt },
{ 'role': 'user', 'content': 'The food that the restaurant was great! I recommend it to everyone.' }],
reasoning = {
"effort": "minimal"
},
)
# Extract model's text output
output_text = ""
for item in response.output:
if hasattr(item, "content"):
for content in item.content:
if hasattr(content, "text"):
output_text += content.text
# Token usage details
usage = response.usage
print("--------------------------------")
print("Output:")
print(output_text)
```
</TabItem>
</Tabs>
docs/my-website/docs/proxy/native_litellm_prompt.md
+162
View File
@@ -0,0 +1,162 @@
import Image from '@theme/IdealImage';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# LiteLLM Prompt Management (GitOps)
Store prompts as `.prompt` files in your repository and use them directly with LiteLLM. No external services required.
## Quick Start
<Tabs>
<TabItem value="sdk" label="SDK">
**1. Create a .prompt file**
Create `prompts/hello.prompt`:
```yaml
---
model: gpt-4
temperature: 0.7
---
System: You are a helpful assistant.
User: {{user_message}}
```
**2. Use with LiteLLM**
```python
import litellm
# Set the global prompt directory
litellm.global_prompt_directory = "prompts/"
response = litellm.completion(
model="dotprompt/gpt-4",
prompt_id="hello",
prompt_variables={"user_message": "What is the capital of France?"}
)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
**1. Create a .prompt file**
Create `prompts/hello.prompt`:
```yaml
---
model: gpt-4
temperature: 0.7
---
System: You are a helpful assistant.
User: {{user_message}}
```
**2. Setup config.yaml**
```yaml
model_list:
- model_name: my-dotprompt-model
litellm_params:
model: dotprompt/gpt-4
prompt_id: "hello"
api_key: os.environ/OPENAI_API_KEY
litellm_settings:
global_prompt_directory: "./prompts"
```
**3. Start the proxy**
```bash
litellm --config config.yaml --detailed_debug
```
**4. Test it!**
```bash
curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
"model": "my-dotprompt-model",
"messages": [{"role": "user", "content": "IGNORED"}],
"prompt_variables": {
"user_message": "What is the capital of France?"
}
}'
```
</TabItem>
</Tabs>
### .prompt File Format
`.prompt` files use YAML frontmatter for metadata and support Jinja2 templating:
```yaml
---
model: gpt-4 # Model to use
temperature: 0.7 # Optional parameters
max_tokens: 1000
input:
schema:
user_message: string # Input validation (optional)
---
System: You are a helpful {{role}} assistant.
User: {{user_message}}
```
### Advanced Features
**Multi-role conversations:**
```yaml
---
model: gpt-4
temperature: 0.3
---
System: You are a helpful coding assistant.
User: {{user_question}}
```
**Dynamic model selection:**
```yaml
---
model: "{{preferred_model}}" # Model can be a variable
temperature: 0.7
---
System: You are a helpful assistant specialized in {{domain}}.
User: {{user_message}}
```
### API Reference
For dotprompt integration, use these parameters:
```
model: dotprompt/<base_model> # required (e.g., dotprompt/gpt-4)
prompt_id: str # required - the .prompt filename without extension
prompt_variables: Optional[dict] # optional - variables for template rendering
```
**Example API call:**
```python
response = litellm.completion(
model="dotprompt/gpt-4",
prompt_id="hello",
prompt_variables={"user_message": "Hello world"},
messages=[{"role": "user", "content": "This will be ignored"}]
)
```
docs/my-website/docs/proxy/prompt_management.md
+1
View File
@@ -8,6 +8,7 @@ Run experiments or change the specific model (e.g. from gpt-4o to gpt4o-mini fin
| Supported Integrations | Link |
|------------------------|------|
| Native LiteLLM GitOps (.prompt files) | [Get Started](native_litellm_prompt) |
| Langfuse | [Get Started](https://langfuse.com/docs/prompts/get-started) |
| Humanloop | [Get Started](../observability/humanloop) |
docs/my-website/release_notes/v1.74.15-stable/index.md
+3 -3
View File
@@ -1,5 +1,5 @@
---
title: "[Pre-Release] v1.74.15-stable"
title: "v1.74.15-stable"
slug: "v1-74-15"
date: 2025-08-02T10:00:00
authors:
@@ -28,14 +28,14 @@ import TabItem from '@theme/TabItem';
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:1.74.15.rc.1
ghcr.io/berriai/litellm:v1.74.15-stable
```
</TabItem>
<TabItem value="pip" label="Pip">
``` showLineNumbers title="pip install litellm"
pip install litellm==1.74.15.post1
pip install litellm==1.74.15.post2
```
</TabItem>
docs/my-website/release_notes/v1.75.5-stable/index.md
+258
View File
@@ -0,0 +1,258 @@
---
title: "[PRE-RELEASE]v1.75.5-stable"
slug: "v1-75-5"
date: 2025-08-10T10:00:00
authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaffer
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
hide_table_of_contents: false
---
import Image from '@theme/IdealImage';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
## Deploy this version
<Tabs>
<TabItem value="docker" label="Docker">
``` showLineNumbers title="docker run litellm"
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.75.5.rc.1
```
</TabItem>
<TabItem value="pip" label="Pip">
``` showLineNumbers title="pip install litellm"
pip install litellm==1.75.5.post1
```
</TabItem>
</Tabs>
---
---
## New Models / Updated Models
#### New Model Support
| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) |
| ----------- | -------------------------------------- | -------------- | ------------------- | -------------------- |
| Bedrock | `bedrock/us.anthropic.claude-opus-4-1-20250805-v1:0` | 200k | $15 | $75 |
| Bedrock | `bedrock/openai.gpt-oss-20b-1:0` | 200k | 0.07 | 0.3 |
| Bedrock | `bedrock/openai.gpt-oss-120b-1:0` | 200k | 0.15 | 0.6 |
| Fireworks AI | `fireworks_ai/accounts/fireworks/models/glm-4p5` | 128k | 0.55 | 2.19 |
| Fireworks AI | `fireworks_ai/accounts/fireworks/models/glm-4p5-air` | 128k | 0.22 | 0.88 |
| Fireworks AI | `fireworks_ai/accounts/fireworks/models/gpt-oss-120b` | 131072 | 0.15 | 0.6 |
| Fireworks AI | `fireworks_ai/accounts/fireworks/models/gpt-oss-20b` | 131072 | 0.05 | 0.2 |
| Groq | `groq/openai/gpt-oss-20b` | 131072 | 0.1 | 0.5 |
| Groq | `groq/openai/gpt-oss-120b` | 131072 | 0.15 | 0.75 |
| OpenAI | `openai/gpt-5` | 400k | 1.25 | 10 |
| OpenAI | `openai/gpt-5-2025-08-07` | 400k | 1.25 | 10 |
| OpenAI | `openai/gpt-5-mini` | 400k | 0.25 | 2 |
| OpenAI | `openai/gpt-5-mini-2025-08-07` | 400k | 0.25 | 2 |
| OpenAI | `openai/gpt-5-nano` | 400k | 0.05 | 0.4 |
| OpenAI | `openai/gpt-5-nano-2025-08-07` | 400k | 0.05 | 0.4 |
| OpenAI | `openai/gpt-5-chat` | 400k | 1.25 | 10 |
| OpenAI | `openai/gpt-5-chat-latest` | 400k | 1.25 | 10 |
| Azure | `azure/gpt-5` | 400k | 1.25 | 10 |
| Azure | `azure/gpt-5-2025-08-07` | 400k | 1.25 | 10 |
| Azure | `azure/gpt-5-mini` | 400k | 0.25 | 2 |
| Azure | `azure/gpt-5-mini-2025-08-07` | 400k | 0.25 | 2 |
| Azure | `azure/gpt-5-nano-2025-08-07` | 400k | 0.05 | 0.4 |
| Azure | `azure/gpt-5-nano` | 400k | 0.05 | 0.4 |
| Azure | `azure/gpt-5-chat` | 400k | 1.25 | 10 |
| Azure | `azure/gpt-5-chat-latest` | 400k | 1.25 | 10 |
#### Features
- **[OCI](../../docs/providers/oci)**
- New LLM provider - [PR #13206](https://github.com/BerriAI/litellm/pull/13206)
- **[JinaAI](../../docs/providers/jina_ai)**
- support multimodal embedding models - [PR #13181](https://github.com/BerriAI/litellm/pull/13181)
- **GPT-5 ([OpenAI](../../docs/providers/openai)/[Azure](../../docs/providers/azure))**
- Support drop_params for temperature - [PR #13390](https://github.com/BerriAI/litellm/pull/13390)
- Map max_tokens to max_completion_tokens - [PR #13390](https://github.com/BerriAI/litellm/pull/13390)
- **[Anthropic](../../docs/providers/anthropic)**
- Add claude-opus-4-1 on model cost map - [PR #13384](https://github.com/BerriAI/litellm/pull/13384)
- **[OpenRouter](../../docs/providers/openrouter)**
- Add gpt-oss to model cost map - [PR #13442](https://github.com/BerriAI/litellm/pull/13442)
- **[Cerebras](../../docs/providers/cerebras)**
- Add gpt-oss to model cost map - [PR #13442](https://github.com/BerriAI/litellm/pull/13442)
- **[Azure](../../docs/providers/azure)**
- Support drop params for ‘temperature’ on o-series models - [PR #13353](https://github.com/BerriAI/litellm/pull/13353)
- **[GradientAI](../../docs/providers/gradient_ai)**
- New LLM Provider - [PR #12169](https://github.com/BerriAI/litellm/pull/12169)
#### Bugs
- **[OpenAI](../../docs/providers/openai)**
- Add ‘service_tier’ and ‘safety_identifier’ as supported responses api params - [PR #13258](https://github.com/BerriAI/litellm/pull/13258)
- Correct pricing for web search on 4o-mini - [PR #13269](https://github.com/BerriAI/litellm/pull/13269)
- **[Mistral](../../docs/providers/mistral)**
- Handle $id and $schema fields when calling mistral - [PR #13389](https://github.com/BerriAI/litellm/pull/13389)
---
## LLM API Endpoints
#### Features
- `/responses`
- Responses API Session Handling w/ support for images - [PR #13347](https://github.com/BerriAI/litellm/pull/13347)
- failed if input containing ResponseReasoningItem - [PR #13465](https://github.com/BerriAI/litellm/pull/13465)
- Support custom tools - [PR #13418](https://github.com/BerriAI/litellm/pull/13418)
#### Bugs
- `/chat/completions`
- Fix completion_token_details usage object missing ‘text’ tokens - [PR #13234](https://github.com/BerriAI/litellm/pull/13234)
- (SDK) handle tool being a pydantic object - [PR #13274](https://github.com/BerriAI/litellm/pull/13274)
- include cost in streaming usage object - [PR #13418](https://github.com/BerriAI/litellm/pull/13418)
- Exclude none fields on /chat/completion - allows usage with n8n - [PR #13320](https://github.com/BerriAI/litellm/pull/13320)
- `/responses`
- Transform function call in response for non-openai models (gemini/anthropic) - [PR #13260](https://github.com/BerriAI/litellm/pull/13260)
- Fix unsupported operand error with model groups - [PR #13293](https://github.com/BerriAI/litellm/pull/13293)
- Responses api session management for streaming responses - [PR #13396](https://github.com/BerriAI/litellm/pull/13396)
- `/v1/messages`
- Added litellm claude code count tokens - [PR #13261](https://github.com/BerriAI/litellm/pull/13261)
- `/vector_stores`
- Fix create/search vector store errors - [PR #13285](https://github.com/BerriAI/litellm/pull/13285)
---
## [MCP Gateway](../../docs/mcp)
#### Features
- Add route check for internal users - [PR #13350](https://github.com/BerriAI/litellm/pull/13350)
- MCP Guardrails - docs - [PR #13392](https://github.com/BerriAI/litellm/pull/13392)
#### Bugs
- Fix auth on UI for bearer token servers - [PR #13312](https://github.com/BerriAI/litellm/pull/13312)
- allow access group on mcp tool retrieval - [PR #13425](https://github.com/BerriAI/litellm/pull/13425)
---
## Management Endpoints / UI
#### Features
- **Teams**
- Add team deletion check for teams with keys - [PR #12953](https://github.com/BerriAI/litellm/pull/12953)
- **Models**
- Add ability to set model alias per key/team - [PR #13276](https://github.com/BerriAI/litellm/pull/13276)
- New button to reload model pricing from model cost map - [PR #13464](https://github.com/BerriAI/litellm/pull/13464), [PR #13470](https://github.com/BerriAI/litellm/pull/13470)
- **Keys**
- Make ‘team’ field required when creating service account keys - [PR #13302](https://github.com/BerriAI/litellm/pull/13302)
- Gray out key-based logging settings for non-enterprise users - prevents confusion on if ‘logging’ all up is supported - [PR #13431](https://github.com/BerriAI/litellm/pull/13431)
- **Navbar**
- Add logo customization for LiteLLM admin UI - [PR #12958](https://github.com/BerriAI/litellm/pull/12958)
- **Logs**
- Add token breakdowns on logs + session page - [PR #13357](https://github.com/BerriAI/litellm/pull/13357)
- **Usage**
- Ensure Usage Page loads after the DB has large entries - [PR #13400](https://github.com/BerriAI/litellm/pull/13400)
- **Test Key Page**
- allow uploading images for /chat/completions and /responses - [PR #13445](https://github.com/BerriAI/litellm/pull/13445)
- **MCP**
- Add auth tokens to local storage auth - [PR #13473](https://github.com/BerriAI/litellm/pull/13473)
#### Bugs
- **Custom Root Path**
- Fix login route when SSO is enabled - [PR #13267](https://github.com/BerriAI/litellm/pull/13267)
- **Customers/End-users**
- Allow calling /v1/models when end user over budget - allows model listing to work on OpenWebUI when customer over budget - [PR #13320](https://github.com/BerriAI/litellm/pull/13320)
- **Teams**
- Remove user - team membership, when user removed from team - [PR #13433](https://github.com/BerriAI/litellm/pull/13433)
- **Errors**
- Bubble up network errors to user for Logging and Alerts page - [PR #13427](https://github.com/BerriAI/litellm/pull/13427)
- **Model Hub**
- Show pricing for azure models, when base model is set - [PR #13418](https://github.com/BerriAI/litellm/pull/13418)
---
## Logging / Guardrail Integrations
#### Features
- **Bedrock Guardrails**
- Redacted sensitive information in bedrock guardrails error message - [PR #13356](https://github.com/BerriAI/litellm/pull/13356)
- **Standard Logging Payload**
- Fix ‘can’t register atextexit’ bug - [PR #13436](https://github.com/BerriAI/litellm/pull/13436)
#### Bugs
- **Braintrust**
- Allow setting of braintrust callback base url - [PR #13368](https://github.com/BerriAI/litellm/pull/13368)
- **OTEL**
- Track pre_call hook latency - [PR #13362](https://github.com/BerriAI/litellm/pull/13362)
---
## Performance / Loadbalancing / Reliability improvements
#### Features
- **Team-BYOK models**
- Add wildcard model support - [PR #13278](https://github.com/BerriAI/litellm/pull/13278)
- **Caching**
- GCP IAM auth support for caching - [PR #13275](https://github.com/BerriAI/litellm/pull/13275)
- **Latency**
- reduce p99 latency w/ redis enabled by 50% - only updates model usage if tpm/rpm limits set - [PR #13362](https://github.com/BerriAI/litellm/pull/13362)
---
## General Proxy Improvements
#### Features
- **Models**
- Support /v1/models/\{model_id\} retrieval - [PR #13268](https://github.com/BerriAI/litellm/pull/13268)
- **Multi-instance**
- Ensure disable_llm_api_endpoints works - [PR #13278](https://github.com/BerriAI/litellm/pull/13278)
- **Logs**
- Add apscheduler log suppress - [PR #13299](https://github.com/BerriAI/litellm/pull/13299)
- **Helm**
- Add labels to migrations job template - [PR #13343](https://github.com/BerriAI/litellm/pull/13343) s/o [@unique-jakub](https://github.com/unique-jakub)
#### Bugs
- **Non-root image**
- Fix non-root image for migration - [PR #13379](https://github.com/BerriAI/litellm/pull/13379)
- **Get Routes**
- Load get routes when using fastapi-offline - [PR #13466](https://github.com/BerriAI/litellm/pull/13466)
- **Health checks**
- Generate unique trace IDs for Langfuse health checks - [PR #13468](https://github.com/BerriAI/litellm/pull/13468)
- **Swagger**
- Allow using Swagger for /chat/completions - [PR #13469](https://github.com/BerriAI/litellm/pull/13469)
- **Auth**
- Fix JWTs access not working with model access groups - [PR #13474](https://github.com/BerriAI/litellm/pull/13474)
---
## New Contributors
* @bbartels made their first contribution in https://github.com/BerriAI/litellm/pull/13244
* @breno-aumo made their first contribution in https://github.com/BerriAI/litellm/pull/13206
* @pascalwhoop made their first contribution in https://github.com/BerriAI/litellm/pull/13122
* @ZPerling made their first contribution in https://github.com/BerriAI/litellm/pull/13045
* @zjx20 made their first contribution in https://github.com/BerriAI/litellm/pull/13181
* @edwarddamato made their first contribution in https://github.com/BerriAI/litellm/pull/13368
* @msannan2 made their first contribution in https://github.com/BerriAI/litellm/pull/12169
## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.74.15-stable...v1.75.5-stable.rc-draft)**
docs/my-website/sidebars.js
+7 -5
View File
@@ -65,6 +65,7 @@ const sidebars = {
label: "[Beta] Prompt Management",
items: [
"proxy/prompt_management",
"proxy/native_litellm_prompt",
"proxy/custom_prompt_management"
].sort()
},
@@ -81,12 +82,12 @@ const sidebars = {
"tutorials/cost_tracking_coding",
]
},
],
// But you can create a sidebar manually
tutorialSidebar: [
{ type: "doc", id: "index" }, // NEW
{
type: "category",
label: "LiteLLM Proxy Server",
@@ -213,7 +214,7 @@ const sidebars = {
"proxy/dynamic_logging"
],
},
{
type: "category",
label: "Secret Managers",
@@ -466,6 +467,7 @@ const sidebars = {
"providers/custom_llm_server",
"providers/petals",
"providers/snowflake",
"providers/gradient_ai",
"providers/featherless_ai",
"providers/nebius",
"providers/dashscope",
@@ -504,7 +506,7 @@ const sidebars = {
]
},
{
type: "category",
label: "Routing, Loadbalancing & Fallbacks",
@@ -535,7 +537,7 @@ const sidebars = {
},
],
},
{
type: "category",
label: "Load Testing",
litellm/__init__.py
+9 -1
View File
@@ -231,6 +231,7 @@ aleph_alpha_key: Optional[str] = None
nlp_cloud_key: Optional[str] = None
novita_api_key: Optional[str] = None
snowflake_key: Optional[str] = None
gradient_ai_api_key: Optional[str] = None
nebius_key: Optional[str] = None
cometapi_key: Optional[str] = None
common_cloud_provider_auth_params: dict = {
@@ -521,6 +522,7 @@ sambanova_models: List = []
novita_models: List = []
assemblyai_models: List = []
snowflake_models: List = []
gradient_ai_models: List = []
llama_models: List = []
nscale_models: List = []
nebius_models: List = []
@@ -705,6 +707,8 @@ def add_known_models():
jina_ai_models.append(key)
elif value.get("litellm_provider") == "snowflake":
snowflake_models.append(key)
elif value.get("litellm_provider") == "gradient_ai":
gradient_ai_models.append(key)
elif value.get("litellm_provider") == "featherless_ai":
featherless_ai_models.append(key)
elif value.get("litellm_provider") == "deepgram":
@@ -806,6 +810,7 @@ model_list = (
+ assemblyai_models
+ jina_ai_models
+ snowflake_models
+ gradient_ai_models
+ llama_models
+ featherless_ai_models
+ nscale_models
@@ -880,6 +885,7 @@ models_by_provider: dict = {
"assemblyai": assemblyai_models,
"jina_ai": jina_ai_models,
"snowflake": snowflake_models,
"gradient_ai": gradient_ai_models,
"meta_llama": llama_models,
"nscale": nscale_models,
"featherless_ai": featherless_ai_models,
@@ -1140,13 +1146,14 @@ from .llms.azure_ai.chat.transformation import AzureAIStudioConfig
from .llms.mistral.chat.transformation import MistralConfig
from .llms.openai.responses.transformation import OpenAIResponsesAPIConfig
from .llms.azure.responses.transformation import AzureOpenAIResponsesAPIConfig
from .llms.azure.responses.o_series_transformation import AzureOpenAIOSeriesResponsesAPIConfig
from .llms.openai.chat.o_series_transformation import (
OpenAIOSeriesConfig as OpenAIO1Config, # maintain backwards compatibility
OpenAIOSeriesConfig,
)
from .llms.snowflake.chat.transformation import SnowflakeConfig
from .llms.gradient_ai.chat.transformation import GradientAIConfig
openaiOSeriesConfig = OpenAIOSeriesConfig()
from .llms.openai.chat.gpt_transformation import (
OpenAIGPTConfig,
@@ -1199,6 +1206,7 @@ from .llms.azure.azure import (
)
from .llms.cometapi.chat.transformation import CometAPIConfig
from .llms.azure.chat.gpt_transformation import AzureOpenAIConfig
from .llms.azure.chat.gpt_5_transformation import AzureOpenAIGPT5Config
from .llms.azure.completion.transformation import AzureOpenAITextConfig
from .llms.hosted_vllm.chat.transformation import HostedVLLMChatConfig
from .llms.llamafile.chat.transformation import LlamafileChatConfig
litellm/caching/caching.py
+49 -25
View File
@@ -28,13 +28,13 @@ from .azure_blob_cache import AzureBlobCache
from .base_cache import BaseCache
from .disk_cache import DiskCache
from .dual_cache import DualCache # noqa
from .gcs_cache import GCSCache
from .in_memory_cache import InMemoryCache
from .qdrant_semantic_cache import QdrantSemanticCache
from .redis_cache import RedisCache
from .redis_cluster_cache import RedisClusterCache
from .redis_semantic_cache import RedisSemanticCache
from .s3_cache import S3Cache
from .gcs_cache import GCSCache
def print_verbose(print_statement):
@@ -177,7 +177,7 @@ class Cache:
cluster_kwargs["gcp_service_account"] = gcp_service_account
if gcp_ssl_ca_certs is not None:
cluster_kwargs["gcp_ssl_ca_certs"] = gcp_ssl_ca_certs
self.cache: BaseCache = RedisClusterCache(**cluster_kwargs)
else:
self.cache = RedisCache(
@@ -481,7 +481,7 @@ class Cache:
return cached_response
return cached_result
def get_cache(self, **kwargs):
def get_cache(self, dynamic_cache_object: Optional[BaseCache] = None, **kwargs):
"""
Retrieves the cached result for the given arguments.
@@ -507,8 +507,12 @@ class Cache:
or cache_control_args.get("s-max-age")
or float("inf")
)
cached_result = self.cache.get_cache(cache_key, messages=messages)
cached_result = self.cache.get_cache(cache_key, messages=messages)
if dynamic_cache_object is not None:
cached_result = dynamic_cache_object.get_cache(
cache_key, messages=messages
)
else:
cached_result = self.cache.get_cache(cache_key, messages=messages)
return self._get_cache_logic(
cached_result=cached_result, max_age=max_age
)
@@ -516,7 +520,9 @@ class Cache:
print_verbose(f"An exception occurred: {traceback.format_exc()}")
return None
async def async_get_cache(self, **kwargs):
async def async_get_cache(
self, dynamic_cache_object: Optional[BaseCache] = None, **kwargs
):
"""
Async get cache implementation.
@@ -537,7 +543,14 @@ class Cache:
max_age = cache_control_args.get(
"s-max-age", cache_control_args.get("s-maxage", float("inf"))
)
cached_result = await self.cache.async_get_cache(cache_key, **kwargs)
if dynamic_cache_object is not None:
cached_result = await dynamic_cache_object.async_get_cache(
cache_key, **kwargs
)
else:
cached_result = await self.cache.async_get_cache(
cache_key, **kwargs
)
return self._get_cache_logic(
cached_result=cached_result, max_age=max_age
)
@@ -596,7 +609,9 @@ class Cache:
except Exception as e:
verbose_logger.exception(f"LiteLLM Cache: Excepton add_cache: {str(e)}")
async def async_add_cache(self, result, **kwargs):
async def async_add_cache(
self, result, dynamic_cache_object: Optional[BaseCache] = None, **kwargs
):
"""
Async implementation of add_cache
"""
@@ -610,12 +625,18 @@ class Cache:
cache_key, cached_data, kwargs = self._add_cache_logic(
result=result, **kwargs
)
await self.cache.async_set_cache(cache_key, cached_data, **kwargs)
if dynamic_cache_object is not None:
await dynamic_cache_object.async_set_cache(
cache_key, cached_data, **kwargs
)
else:
await self.cache.async_set_cache(cache_key, cached_data, **kwargs)
except Exception as e:
verbose_logger.exception(f"LiteLLM Cache: Excepton add_cache: {str(e)}")
def _convert_to_cached_embedding(self, embedding_response: Any, model: Optional[str]) -> CachedEmbedding:
def _convert_to_cached_embedding(
self, embedding_response: Any, model: Optional[str]
) -> CachedEmbedding:
"""
Convert any embedding response into the standardized CachedEmbedding TypedDict format.
"""
@@ -627,7 +648,7 @@ class Cache:
"object": embedding_response.get("object"),
"model": model,
}
elif hasattr(embedding_response, 'model_dump'):
elif hasattr(embedding_response, "model_dump"):
data = embedding_response.model_dump()
return {
"embedding": data.get("embedding"),
@@ -646,7 +667,6 @@ class Cache:
except KeyError as e:
raise ValueError(f"Missing expected key in embedding response: {e}")
def add_embedding_response_to_cache(
self,
result: EmbeddingResponse,
@@ -657,18 +677,22 @@ class Cache:
preset_cache_key = self.get_cache_key(**{**kwargs, "input": input})
kwargs["cache_key"] = preset_cache_key
embedding_response = result.data[idx_in_result_data]
# Always convert to properly typed CachedEmbedding
model_name = result.model
embedding_dict: CachedEmbedding = self._convert_to_cached_embedding(embedding_response, model_name)
embedding_dict: CachedEmbedding = self._convert_to_cached_embedding(
embedding_response, model_name
)
cache_key, cached_data, kwargs = self._add_cache_logic(
result=embedding_dict,
**kwargs,
)
return cache_key, cached_data, kwargs
async def async_add_cache_pipeline(self, result, **kwargs):
async def async_add_cache_pipeline(
self, result, dynamic_cache_object: Optional[BaseCache] = None, **kwargs
):
"""
Async implementation of add_cache for Embedding calls
@@ -697,14 +721,14 @@ class Cache:
)
cache_list.append((cache_key, cached_data))
await self.cache.async_set_cache_pipeline(cache_list=cache_list, **kwargs)
# if async_set_cache_pipeline:
# await async_set_cache_pipeline(cache_list=cache_list, **kwargs)
# else:
# tasks = []
# for val in cache_list:
# tasks.append(self.cache.async_set_cache(val[0], val[1], **kwargs))
# await asyncio.gather(*tasks)
if dynamic_cache_object is not None:
await dynamic_cache_object.async_set_cache_pipeline(
cache_list=cache_list, **kwargs
)
else:
await self.cache.async_set_cache_pipeline(
cache_list=cache_list, **kwargs
)
except Exception as e:
verbose_logger.exception(f"LiteLLM Cache: Excepton add_cache: {str(e)}")
litellm/caching/caching_handler.py
+56 -14
View File
@@ -1,5 +1,5 @@
"""
This contains LLMCachingHandler
This contains LLMCachingHandler
This exposes two methods:
- async_get_cache
@@ -35,11 +35,12 @@ from pydantic import BaseModel
import litellm
from litellm._logging import print_verbose, verbose_logger
from litellm.caching import InMemoryCache
from litellm.caching.caching import S3Cache
from litellm.types.caching import CachedEmbedding
from litellm.litellm_core_utils.logging_utils import (
_assemble_complete_response_from_streaming_chunks,
)
from litellm.types.caching import CachedEmbedding
from litellm.types.rerank import RerankResponse
from litellm.types.utils import (
CallTypes,
@@ -68,7 +69,12 @@ class CachingHandlerResponse(BaseModel):
cached_result: Optional[Any] = None
final_embedding_cached_response: Optional[EmbeddingResponse] = None
embedding_all_elements_cache_hit: bool = False # this is set to True when all elements in the list have a cache hit in the embedding cache, if true return the final_embedding_cached_response no need to make an API call
embedding_all_elements_cache_hit: bool = (
False # this is set to True when all elements in the list have a cache hit in the embedding cache, if true return the final_embedding_cached_response no need to make an API call
)
in_memory_cache_obj = InMemoryCache()
class LLMCachingHandler:
@@ -78,11 +84,20 @@ class LLMCachingHandler:
request_kwargs: Dict[str, Any],
start_time: datetime.datetime,
):
from litellm.caching import DualCache, RedisCache
self.async_streaming_chunks: List[ModelResponse] = []
self.sync_streaming_chunks: List[ModelResponse] = []
self.request_kwargs = request_kwargs
self.original_function = original_function
self.start_time = start_time
if litellm.cache is not None and isinstance(litellm.cache.cache, RedisCache):
self.dual_cache: Optional[DualCache] = DualCache(
redis_cache=litellm.cache.cache,
in_memory_cache=in_memory_cache_obj,
)
else:
self.dual_cache = None
pass
async def _async_get_cache(
@@ -115,10 +130,16 @@ class LLMCachingHandler:
Raises:
None
"""
from litellm.litellm_core_utils.core_helpers import (
_get_parent_otel_span_from_kwargs,
)
from litellm.utils import CustomStreamWrapper
kwargs = kwargs.copy()
args = args or ()
parent_otel_span = _get_parent_otel_span_from_kwargs(kwargs)
kwargs["parent_otel_span"] = parent_otel_span
final_embedding_cached_response: Optional[EmbeddingResponse] = None
embedding_all_elements_cache_hit: bool = False
cached_result: Optional[Any] = None
@@ -306,13 +327,15 @@ class LLMCachingHandler:
else:
raise ValueError("input must be a string or a list")
def _extract_model_from_cached_results(self, non_null_list: List[Tuple[int, CachedEmbedding]]) -> Optional[str]:
def _extract_model_from_cached_results(
self, non_null_list: List[Tuple[int, CachedEmbedding]]
) -> Optional[str]:
"""
Helper method to extract the model name from cached results.
Args:
non_null_list: List of (idx, cr) tuples where cr is the cached result dict
Returns:
Optional[str]: The model name if found, None otherwise
"""
@@ -558,7 +581,12 @@ class LLMCachingHandler:
preset_cache_key = litellm.cache.get_cache_key(
**{**new_kwargs, "input": i}
)
tasks.append(litellm.cache.async_get_cache(cache_key=preset_cache_key))
tasks.append(
litellm.cache.async_get_cache(
cache_key=preset_cache_key,
dynamic_cache_object=self.dual_cache,
)
)
cached_result = await asyncio.gather(*tasks)
## check if cached result is None ##
if cached_result is not None and isinstance(cached_result, list):
@@ -567,9 +595,14 @@ class LLMCachingHandler:
cached_result = None
else:
if litellm.cache._supports_async() is True:
cached_result = await litellm.cache.async_get_cache(**new_kwargs)
## check if dual cache is supported ##
cached_result = await litellm.cache.async_get_cache(
dynamic_cache_object=self.dual_cache, **new_kwargs
)
else: # for s3 caching. [NOT RECOMMENDED IN PROD - this will slow down responses since boto3 is sync]
cached_result = litellm.cache.get_cache(**new_kwargs)
cached_result = litellm.cache.get_cache(
dynamic_cache_object=self.dual_cache, **new_kwargs
)
return cached_result
def _convert_cached_result_to_model_response(
@@ -735,6 +768,9 @@ class LLMCachingHandler:
Raises:
None
"""
from litellm.litellm_core_utils.core_helpers import (
_get_parent_otel_span_from_kwargs,
)
if litellm.cache is None:
return
@@ -746,6 +782,8 @@ class LLMCachingHandler:
args,
)
)
parent_otel_span = _get_parent_otel_span_from_kwargs(new_kwargs)
new_kwargs["parent_otel_span"] = parent_otel_span
# [OPTIONAL] ADD TO CACHE
if self._should_store_result_in_cache(
original_function=original_function, kwargs=new_kwargs
@@ -764,7 +802,9 @@ class LLMCachingHandler:
) # s3 doesn't support bulk writing. Exclude.
):
asyncio.create_task(
litellm.cache.async_add_cache_pipeline(result, **new_kwargs)
litellm.cache.async_add_cache_pipeline(
result, dynamic_cache_object=self.dual_cache, **new_kwargs
)
)
elif isinstance(litellm.cache.cache, S3Cache):
threading.Thread(
@@ -775,7 +815,9 @@ class LLMCachingHandler:
else:
asyncio.create_task(
litellm.cache.async_add_cache(
result.model_dump_json(), **new_kwargs
result.model_dump_json(),
dynamic_cache_object=self.dual_cache,
**new_kwargs,
)
)
else:
@@ -933,9 +975,9 @@ class LLMCachingHandler:
}
if litellm.cache is not None:
litellm_params[
"preset_cache_key"
] = litellm.cache._get_preset_cache_key_from_kwargs(**kwargs)
litellm_params["preset_cache_key"] = (
litellm.cache._get_preset_cache_key_from_kwargs(**kwargs)
)
else:
litellm_params["preset_cache_key"] = None
litellm/caching/redis_cache.py
+71 -32
View File
@@ -43,6 +43,45 @@ else:
Span = Any
def _get_call_stack_info(num_frames: int = 2) -> str:
"""
Get the function names from the previous 1-2 functions in the call stack.
Args:
num_frames: Number of previous frames to include (default: 2)
Returns:
A string with format "current_function <- caller_function [<- grandparent_function]"
"""
try:
current_frame = inspect.currentframe()
if current_frame is None:
return "unknown"
# Skip this function and the immediate caller (which sets call_type)
f_back = current_frame.f_back
if f_back is None:
return "unknown"
frame = f_back.f_back
if frame is None:
return "unknown"
function_names = []
for _ in range(num_frames):
if frame is None:
break
func_name = frame.f_code.co_name
function_names.append(func_name)
frame = frame.f_back
if not function_names:
return "unknown"
return " <- ".join(function_names)
except Exception:
return "unknown"
class RedisCache(BaseCache):
# if users don't provider one, use the default litellm cache
@@ -181,7 +220,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="set_cache",
call_type=f"set_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -205,7 +244,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="increment_cache",
call_type=f"increment_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -219,7 +258,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="increment_cache_ttl",
call_type=f"increment_cache_ttl <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -232,7 +271,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="increment_cache_expire",
call_type=f"increment_cache_expire <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -271,7 +310,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="async_scan_iter",
call_type=f"async_scan_iter <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -287,7 +326,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
call_type="async_scan_iter",
call_type=f"async_scan_iter <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -341,7 +380,7 @@ class RedisCache(BaseCache):
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
call_type="async_set_cache",
call_type=f"async_set_cache <- {_get_call_stack_info()}",
)
)
verbose_logger.error(
@@ -374,7 +413,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="async_set_cache",
call_type=f"async_set_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -390,7 +429,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
call_type="async_set_cache",
call_type=f"async_set_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -463,7 +502,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="async_set_cache_pipeline",
call_type=f"async_set_cache_pipeline <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -479,7 +518,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
call_type="async_set_cache_pipeline",
call_type=f"async_set_cache_pipeline <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -528,7 +567,7 @@ class RedisCache(BaseCache):
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
call_type="async_set_cache_sadd",
call_type=f"async_set_cache_sadd <- {_get_call_stack_info()}",
)
)
# NON blocking - notify users Redis is throwing an exception
@@ -554,7 +593,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="async_set_cache_sadd",
call_type=f"async_set_cache_sadd <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -568,7 +607,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
call_type="async_set_cache_sadd",
call_type=f"async_set_cache_sadd <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -620,7 +659,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="async_increment",
call_type=f"async_increment <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -636,7 +675,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
call_type="async_increment",
call_type=f"async_increment <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -683,7 +722,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="get_cache",
call_type=f"get_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -745,7 +784,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="batch_get_cache",
call_type=f"batch_get_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -790,7 +829,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="async_get_cache",
call_type=f"async_get_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -806,7 +845,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
call_type="async_get_cache",
call_type=f"async_get_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -851,7 +890,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="async_batch_get_cache",
call_type=f"async_batch_get_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -879,7 +918,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
call_type="async_batch_get_cache",
call_type=f"async_batch_get_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -903,7 +942,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="sync_ping",
call_type=f"sync_ping <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -917,7 +956,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
call_type="sync_ping",
call_type=f"sync_ping <- {_get_call_stack_info()}",
)
verbose_logger.error(
f"LiteLLM Redis Cache PING: - Got exception from REDIS : {str(e)}"
@@ -938,7 +977,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="async_ping",
call_type=f"async_ping <- {_get_call_stack_info()}",
)
)
return response
@@ -952,7 +991,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
call_type="async_ping",
call_type=f"async_ping <- {_get_call_stack_info()}",
)
)
verbose_logger.error(
@@ -1051,7 +1090,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="async_increment_pipeline",
call_type=f"async_increment_pipeline <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -1067,7 +1106,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
call_type="async_increment_pipeline",
call_type=f"async_increment_pipeline <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -1131,7 +1170,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="async_rpush",
call_type=f"async_rpush <- {_get_call_stack_info()}",
)
)
return response
@@ -1145,7 +1184,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
call_type="async_rpush",
call_type=f"async_rpush <- {_get_call_stack_info()}",
)
)
verbose_logger.error(
@@ -1202,7 +1241,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
call_type="async_lpop",
call_type=f"async_lpop <- {_get_call_stack_info()}",
)
)
@@ -1230,7 +1269,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
call_type="async_lpop",
call_type=f"async_lpop <- {_get_call_stack_info()}",
)
)
verbose_logger.error(
litellm/constants.py
+1
View File
@@ -271,6 +271,7 @@ LITELLM_CHAT_PROVIDERS = [
"llamafile",
"lm_studio",
"galadriel",
"gradient_ai",
"github_copilot", # GitHub Copilot Chat API
"novita",
"meta_llama",
litellm/integrations/langfuse/langfuse_otel.py
+48 -9
View File
@@ -1,15 +1,16 @@
import base64
import os
import json # <--- NEW
from typing import TYPE_CHECKING, Any, Union
from urllib.parse import quote
import os
from typing import TYPE_CHECKING, Any, Optional, Union
from litellm._logging import verbose_logger
from litellm.integrations.arize import _utils
from litellm.integrations.opentelemetry import OpenTelemetry
from litellm.types.integrations.langfuse_otel import (
LangfuseOtelConfig,
LangfuseSpanAttributes,
)
from litellm.types.utils import StandardCallbackDynamicParams
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
@@ -33,7 +34,11 @@ LANGFUSE_CLOUD_US_ENDPOINT = "https://us.cloud.langfuse.com/api/public/otel"
class LangfuseOtelLogger:
class LangfuseOtelLogger(OpenTelemetry):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@staticmethod
def set_langfuse_otel_attributes(span: Span, kwargs, response_obj):
"""
@@ -174,11 +179,11 @@ class LangfuseOtelLogger:
endpoint = LANGFUSE_CLOUD_US_ENDPOINT
verbose_logger.debug(f"Using Langfuse US cloud endpoint: {endpoint}")
# Create Basic Auth header
auth_string = f"{public_key}:{secret_key}"
auth_header = base64.b64encode(auth_string.encode()).decode()
# URL encode the entire header value as required by OpenTelemetry specification
otlp_auth_headers = f"Authorization={quote(f'Basic {auth_header}')}"
auth_header = LangfuseOtelLogger._get_langfuse_authorization_header(
public_key=public_key,
secret_key=secret_key
)
otlp_auth_headers = f"Authorization={auth_header}"
# Set standard OTEL environment variables
os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = endpoint
@@ -187,3 +192,37 @@ class LangfuseOtelLogger:
return LangfuseOtelConfig(
otlp_auth_headers=otlp_auth_headers, protocol="otlp_http"
)
@staticmethod
def _get_langfuse_authorization_header(public_key: str, secret_key: str) -> str:
"""
Get the authorization header for Langfuse OpenTelemetry.
"""
auth_string = f"{public_key}:{secret_key}"
auth_header = base64.b64encode(auth_string.encode()).decode()
return f'Basic {auth_header}'
def construct_dynamic_otel_headers(
self,
standard_callback_dynamic_params: StandardCallbackDynamicParams
) -> Optional[dict]:
"""
Construct dynamic Langfuse headers from standard callback dynamic params
This is used for team/key based logging.
Returns:
dict: A dictionary of dynamic Langfuse headers
"""
dynamic_headers = {}
dynamic_langfuse_public_key = standard_callback_dynamic_params.get("langfuse_public_key")
dynamic_langfuse_secret_key = standard_callback_dynamic_params.get("langfuse_secret_key")
if dynamic_langfuse_public_key and dynamic_langfuse_secret_key:
auth_header = LangfuseOtelLogger._get_langfuse_authorization_header(
public_key=dynamic_langfuse_public_key,
secret_key=dynamic_langfuse_secret_key
)
dynamic_headers["Authorization"] = auth_header
return dynamic_headers
litellm/litellm_core_utils/get_llm_provider_logic.py
+9
View File
@@ -351,6 +351,8 @@ def get_llm_provider( # noqa: PLR0915
custom_llm_provider = "openai"
elif model in litellm.empower_models:
custom_llm_provider = "empower"
elif model in litellm.gradient_ai_models:
custom_llm_provider = "gradient_ai"
elif model == "*":
custom_llm_provider = "openai"
# bytez models
@@ -667,6 +669,13 @@ def _get_openai_compatible_provider_info( # noqa: PLR0915
or f"https://{get_secret('SNOWFLAKE_ACCOUNT_ID')}.snowflakecomputing.com/api/v2/cortex/inference:complete"
) # type: ignore
dynamic_api_key = api_key or get_secret_str("SNOWFLAKE_JWT")
elif custom_llm_provider == "gradient_ai":
(
api_base,
dynamic_api_key,
) = litellm.GradientAIConfig()._get_openai_compatible_provider_info(
api_base, api_key
)
elif custom_llm_provider == "featherless_ai":
(
api_base,
litellm/litellm_core_utils/get_supported_openai_params.py
+4
View File
@@ -121,6 +121,10 @@ def get_supported_openai_params( # noqa: PLR0915
return litellm.AzureOpenAIO1Config().get_supported_openai_params(
model=model
)
elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
return litellm.AzureOpenAIGPT5Config().get_supported_openai_params(
model=model
)
else:
return litellm.AzureOpenAIConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "openrouter":
litellm/litellm_core_utils/health_check_helpers.py
+7 -5
View File
@@ -1,12 +1,13 @@
"""
Helper functions for health check calls.
"""
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging
class HealthCheckHelpers:
@staticmethod
@@ -38,10 +39,9 @@ class HealthCheckHelpers:
model_params["model"] = cheapest_models[0]
model_params["litellm_logging_obj"] = litellm_logging_obj
model_params["fallbacks"] = fallback_models
model_params["max_tokens"] = 1
model_params["max_tokens"] = 10 # gpt-5-nano throws errors for max_tokens=1
await acompletion(**model_params)
return {}
@staticmethod
def _update_model_params_with_health_check_tracking_information(
@@ -57,6 +57,7 @@ class HealthCheckHelpers:
"""
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.litellm_pre_call_utils import LiteLLMProxyRequestSetup
_metadata_variable_name = "litellm_metadata"
litellm_metadata = HealthCheckHelpers._get_metadata_for_health_check_call()
model_params[_metadata_variable_name] = litellm_metadata
@@ -66,13 +67,14 @@ class HealthCheckHelpers:
_metadata_variable_name=_metadata_variable_name,
)
return model_params
@staticmethod
def _get_metadata_for_health_check_call():
"""
Returns the metadata for the health check call.
"""
from litellm.constants import LITTELM_INTERNAL_HEALTH_SERVICE_ACCOUNT_NAME
return {
"tags": [LITTELM_INTERNAL_HEALTH_SERVICE_ACCOUNT_NAME],
}
}
litellm/litellm_core_utils/litellm_logging.py
+4 -3
View File
@@ -131,7 +131,6 @@ from ..integrations.humanloop import HumanloopLogger
from ..integrations.lago import LagoLogger
from ..integrations.langfuse.langfuse import LangFuseLogger
from ..integrations.langfuse.langfuse_handler import LangFuseHandler
from ..integrations.langfuse.langfuse_otel import LangfuseOtelLogger
from ..integrations.langfuse.langfuse_prompt_management import LangfusePromptManagement
from ..integrations.langsmith import LangsmithLogger
from ..integrations.literal_ai import LiteralAILogger
@@ -3457,6 +3456,7 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
_in_memory_loggers.append(langfuse_logger)
return langfuse_logger # type: ignore
elif logging_integration == "langfuse_otel":
from litellm.integrations.langfuse.langfuse_otel import LangfuseOtelLogger
from litellm.integrations.opentelemetry import (
OpenTelemetry,
OpenTelemetryConfig,
@@ -3467,15 +3467,16 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
# The endpoint and headers are now set as environment variables by get_langfuse_otel_config()
otel_config = OpenTelemetryConfig(
exporter=langfuse_otel_config.protocol,
headers=langfuse_otel_config.otlp_auth_headers,
)
for callback in _in_memory_loggers:
if (
isinstance(callback, OpenTelemetry)
isinstance(callback, LangfuseOtelLogger)
and callback.callback_name == "langfuse_otel"
):
return callback # type: ignore
_otel_logger = OpenTelemetry(
_otel_logger = LangfuseOtelLogger(
config=otel_config, callback_name="langfuse_otel"
)
_in_memory_loggers.append(_otel_logger)
litellm/llms/azure/azure.py
+8
View File
@@ -230,6 +230,14 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
)
data = {"model": None, "messages": messages, **optional_params}
elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
data = litellm.AzureOpenAIGPT5Config().transform_request(
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
headers=headers or {},
)
else:
data = litellm.AzureOpenAIConfig().transform_request(
model=model,
litellm/llms/azure/chat/gpt_5_transformation.py
+58
View File
@@ -0,0 +1,58 @@
"""Support for Azure OpenAI gpt-5 model family."""
from typing import List
from litellm.llms.openai.chat.gpt_5_transformation import OpenAIGPT5Config
from litellm.types.llms.openai import AllMessageValues
from .gpt_transformation import AzureOpenAIConfig
class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
"""Azure specific handling for gpt-5 models."""
GPT5_SERIES_ROUTE = "gpt5_series/"
@classmethod
def is_model_gpt_5_model(cls, model: str) -> bool:
"""Check if the Azure model string refers to a gpt-5 variant.
Accepts both explicit gpt-5 model names and the ``gpt5_series/`` prefix
used for manual routing.
"""
return "gpt-5" in model or "gpt5_series" in model
def get_supported_openai_params(self, model: str) -> List[str]:
return OpenAIGPT5Config.get_supported_openai_params(self, model=model)
def map_openai_params(
self,
non_default_params: dict,
optional_params: dict,
model: str,
drop_params: bool,
api_version: str = "",
) -> dict:
return OpenAIGPT5Config.map_openai_params(
self,
non_default_params=non_default_params,
optional_params=optional_params,
model=model,
drop_params=drop_params,
)
def transform_request(
self,
model: str,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
model = model.replace(self.GPT5_SERIES_ROUTE, "")
return super().transform_request(
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
headers=headers,
)
litellm/llms/azure/common_utils.py
+5
View File
@@ -662,6 +662,11 @@ class BaseAzureLLM(BaseOpenAILLM):
headers: dict, litellm_params: Optional[GenericLiteLLMParams]
) -> dict:
litellm_params = litellm_params or GenericLiteLLMParams()
# If api-key is already in headers, preserve it
if "api-key" in headers:
return headers
api_key = (
litellm_params.api_key
or litellm.api_key
litellm/llms/azure/responses/o_series_transformation.py
+93
View File
@@ -0,0 +1,93 @@
"""
Support for Azure OpenAI O-series models (o1, o3, etc.) in Responses API
https://platform.openai.com/docs/guides/reasoning
Translations handled by LiteLLM:
- temperature => drop param (if user opts in to dropping param)
- Other parameters follow base Azure OpenAI Responses API behavior
"""
from typing import TYPE_CHECKING, Any, Dict
from litellm._logging import verbose_logger
from litellm.types.llms.openai import ResponsesAPIOptionalRequestParams
from litellm.utils import supports_reasoning
from .transformation import AzureOpenAIResponsesAPIConfig
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
LiteLLMLoggingObj = _LiteLLMLoggingObj
else:
LiteLLMLoggingObj = Any
class AzureOpenAIOSeriesResponsesAPIConfig(AzureOpenAIResponsesAPIConfig):
"""
Configuration for Azure OpenAI O-series models in Responses API.
O-series models (o1, o3, etc.) do not support the temperature parameter
in the responses API, so we need to drop it when drop_params is enabled.
"""
def get_supported_openai_params(self, model: str) -> list:
"""
Get supported parameters for Azure OpenAI O-series Responses API.
O-series models don't support temperature parameter in responses API.
"""
# Get the base Azure supported params
base_supported_params = super().get_supported_openai_params(model)
# O-series models don't support temperature parameter in responses API
o_series_unsupported_params = ["temperature"]
# Filter out unsupported parameters for O-series models
o_series_supported_params = [
param for param in base_supported_params
if param not in o_series_unsupported_params
]
return o_series_supported_params
def map_openai_params(
self,
response_api_optional_params: ResponsesAPIOptionalRequestParams,
model: str,
drop_params: bool,
) -> Dict:
"""
Map OpenAI parameters for Azure OpenAI O-series Responses API.
Drops temperature parameter if drop_params is True since O-series models
don't support temperature in the responses API.
"""
mapped_params = dict(response_api_optional_params)
# If drop_params is enabled, remove temperature parameter for O-series models
if drop_params and "temperature" in mapped_params:
verbose_logger.debug(
f"Dropping unsupported parameter 'temperature' for Azure OpenAI O-series responses API model {model}"
)
mapped_params.pop("temperature", None)
return mapped_params
def is_o_series_model(self, model: str) -> bool:
"""
Check if the model is an O-series model.
Args:
model: The model name to check
Returns:
True if it's an O-series model, False otherwise
"""
# Check if model name contains o_series or if it's a known O-series model
if "o_series" in model.lower():
return True
# Check if the model supports reasoning (which is O-series specific)
return supports_reasoning(model)
litellm/llms/gradient_ai/chat/transformation.py
+147
View File
@@ -0,0 +1,147 @@
from typing import List, Optional, Tuple, Union, Dict, Literal
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import (
AllMessageValues,
)
from ...openai_like.chat.transformation import OpenAILikeChatConfig
# Default GradientAI endpoint
GRADIENT_AI_SERVERLESS_ENDPOINT = "https://inference.do-ai.run"
class GradientAIConfig(OpenAILikeChatConfig):
k: Optional[int] = None
kb_filters: Optional[List[Dict]] = None
filter_kb_content_by_query_metadata: Optional[bool] = None
instruction_override: Optional[str] = None
include_functions_info: Optional[bool] = None
include_retrieval_info: Optional[bool] = None
include_guardrails_info: Optional[bool] = None
provide_citations: Optional[bool] = None
retrieval_method: Optional[Literal["rewrite", "step_back", "sub_queries", "none"]] = None
def __init__(
self,
frequency_penalty: Optional[float] = None,
max_tokens: Optional[int] = None,
max_completion_tokens: Optional[int] = None,
presence_penalty: Optional[float] = None,
retrieval_method: Optional[str] = None,
stop: Optional[Union[str, List[str]]] = None,
stream: Optional[bool] = None,
temperature: Optional[float] = None,
top_p: Optional[float] = None,
k: Optional[int] = None,
kb_filters: Optional[List[Dict]] = None,
filter_kb_content_by_query_metadata: Optional[bool] = None,
instruction_override: Optional[str] = None,
include_functions_info: Optional[bool] = None,
include_retrieval_info: Optional[bool] = None,
include_guardrails_info: Optional[bool] = None,
provide_citations: Optional[bool] = None,
) -> None:
locals_ = locals().copy()
for key, value in locals_.items():
if key != "self" and value is not None:
setattr(self.__class__, key, value)
@classmethod
def get_config(cls):
return super().get_config()
def get_supported_openai_params(self, model: str) -> list:
supported_params = [
"frequency_penalty",
"max_tokens",
"max_completion_tokens",
"presence_penalty",
"stop",
"stream",
"stream_options",
"temperature",
"top_p",
# GradientAI specific parameters
"k",
"kb_filters",
"filter_kb_content_by_query_metadata",
"instruction_override",
"include_functions_info",
"include_retrieval_info",
"include_guardrails_info",
"provide_citations",
"retrieval_method",
]
return supported_params
def validate_environment(self,
headers: dict,
model: str,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None):
api_key = api_key or get_secret_str("GRADIENT_AI_API_KEY")
if api_key is None:
raise ValueError("GradientAI API key not found")
if headers is None:
headers = {}
headers["Authorization"] = f"Bearer {api_key}"
headers["Content-Type"] = "application/json"
return headers
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
gradient_ai_endpoint = get_secret_str("GRADIENT_AI_AGENT_ENDPOINT")
complete_url = f"{GRADIENT_AI_SERVERLESS_ENDPOINT}/v1/chat/completions"
if api_base and api_base != GRADIENT_AI_SERVERLESS_ENDPOINT:
complete_url = f"{api_base}/api/v1/chat/completions"
elif gradient_ai_endpoint and gradient_ai_endpoint != GRADIENT_AI_SERVERLESS_ENDPOINT:
complete_url = f"{gradient_ai_endpoint}/api/v1/chat/completions"
return complete_url
def _get_openai_compatible_provider_info(
self, api_base: Optional[str], api_key: Optional[str]
) -> Tuple[Optional[str], Optional[str]]:
gradient_ai_endpoint = get_secret_str("GRADIENT_AI_AGENT_ENDPOINT")
if not api_base and not gradient_ai_endpoint:
api_base = GRADIENT_AI_SERVERLESS_ENDPOINT
else:
api_base = api_base or gradient_ai_endpoint
dynamic_api_key = api_key or get_secret_str("GRADIENT_AI_API_KEY")
return api_base, dynamic_api_key
def map_openai_params(
self,
non_default_params: dict,
optional_params: dict,
model: str,
drop_params: bool = False,
replace_max_completion_tokens_with_max_tokens: bool = False,
) -> dict:
supported_openai_params = self.get_supported_openai_params(model=model)
for param, value in non_default_params.items():
if param in supported_openai_params:
optional_params[param] = value
elif not drop_params:
from litellm.utils import UnsupportedParamsError
raise UnsupportedParamsError(
status_code=400,
message=f"GradientAI does not support parameter '{param}'. To drop unsupported params, set `drop_params=True`."
)
return optional_params
litellm/llms/oci/chat/transformation.py
+9 -2
View File
@@ -2,7 +2,7 @@ import base64
import datetime
import hashlib
import json
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
from typing import TYPE_CHECKING, Any, AsyncIterator, Dict, List, Optional, Tuple, Union
from urllib.parse import urlparse
import httpx
@@ -586,8 +586,15 @@ class OCIChatConfig(BaseConfig):
completion_stream = response.aiter_text()
async def split_chunks(completion_stream: AsyncIterator[str]):
async for item in completion_stream:
for chunk in item.split("\n\n"):
if not chunk:
continue
yield chunk.strip()
streaming_response = OCIStreamWrapper(
completion_stream=completion_stream,
completion_stream=split_chunks(completion_stream),
model=model,
custom_llm_provider=custom_llm_provider,
logging_obj=logging_obj,
litellm/llms/openai/responses/transformation.py
+25 -1
View File
@@ -1,6 +1,7 @@
from typing import TYPE_CHECKING, Any, Dict, Optional, Union, cast
import httpx
from pydantic import BaseModel
import litellm
from litellm._logging import verbose_logger
@@ -75,12 +76,35 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
headers: dict,
) -> Dict:
"""No transform applied since inputs are in OpenAI spec already"""
return dict(
input = self._validate_input_param(input)
final_request_params = dict(
ResponsesAPIRequestParams(
model=model, input=input, **response_api_optional_request_params
)
)
return final_request_params
def _validate_input_param(self, input: Union[str, ResponseInputParam]) -> Union[str, ResponseInputParam]:
"""
Ensure all input fields if pydantic are converted to dict
OpenAI API Fails when we try to JSON dumps specific input pydantic fields.
This function ensures all input fields are converted to dict.
"""
if isinstance(input, list):
validated_input = []
for item in input:
# if it's pydantic, convert to dict
if isinstance(item, BaseModel):
validated_input.append(item.model_dump(exclude_none=True))
else:
validated_input.append(item)
return validated_input
# Input is expected to be either str or List, no single BaseModel expected
return input
def transform_response_api_response(
self,
model: str,
litellm/main.py
+21 -2
View File
@@ -3342,6 +3342,25 @@ def completion( # type: ignore # noqa: PLR0915
additional_args={"headers": headers},
)
raise e
elif custom_llm_provider == "gradient_ai":
api_base = litellm.api_base or api_base
response = base_llm_http_handler.completion(
model=model,
stream=stream,
messages=messages,
acompletion=acompletion,
api_base=api_base,
model_response=model_response,
optional_params=optional_params,
litellm_params=litellm_params,
custom_llm_provider="gradient_ai",
timeout=timeout,
headers=headers,
encoding=encoding,
api_key=api_key,
logging_obj=logging,
)
elif custom_llm_provider == "bytez":
api_key = (
@@ -5553,9 +5572,9 @@ async def ahealth_check(
messages=[],
stream=False,
call_type="acompletion",
litellm_call_id="1234",
litellm_call_id=str(uuid.uuid4()),
start_time=datetime.datetime.now(),
function_id="1234",
function_id=str(uuid.uuid4()),
log_raw_request_response=True,
)
model_params["litellm_logging_obj"] = litellm_logging_obj
litellm/model_prices_and_context_window_backup.json
+132 -8
View File
@@ -2457,7 +2457,7 @@
},
"azure/gpt-5-chat": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_input_tokens": 400000,
"max_output_tokens": 128000,
"input_cost_per_token": 1.25e-06,
"output_cost_per_token": 1e-05,
@@ -2490,7 +2490,7 @@
},
"azure/gpt-5-chat-latest": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_input_tokens": 400000,
"max_output_tokens": 128000,
"input_cost_per_token": 1.25e-06,
"output_cost_per_token": 1e-05,
@@ -12582,8 +12582,8 @@
},
"openai.gpt-oss-20b-1:0": {
"max_tokens": 128000,
"max_input_tokens": 200000,
"max_output_tokens": 32000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 7e-08,
"output_cost_per_token": 3e-07,
"litellm_provider": "bedrock_converse",
@@ -12596,8 +12596,8 @@
},
"openai.gpt-oss-120b-1:0": {
"max_tokens": 128000,
"max_input_tokens": 200000,
"max_output_tokens": 32000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 1.5e-07,
"output_cost_per_token": 6e-07,
"litellm_provider": "bedrock_converse",
@@ -15605,7 +15605,7 @@
"supports_tool_choice": false
},
"fireworks_ai/accounts/fireworks/models/glm-4p5": {
"max_tokens": 128000,
"max_tokens": 96000,
"max_input_tokens": 128000,
"max_output_tokens": 96000,
"input_cost_per_token": 5.5e-07,
@@ -15618,7 +15618,7 @@
"source": "https://fireworks.ai/models/fireworks/glm-4p5"
},
"fireworks_ai/accounts/fireworks/models/glm-4p5-air": {
"max_tokens": 128000,
"max_tokens": 96000,
"max_input_tokens": 128000,
"max_output_tokens": 96000,
"input_cost_per_token": 2.2e-07,
@@ -17098,6 +17098,130 @@
"litellm_provider": "snowflake",
"mode": "chat"
},
"gradient_ai/anthropic-claude-3.7-sonnet": {
"input_cost_per_token": 3e-06,
"output_cost_per_token": 15e-06,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 1024,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/anthropic-claude-3.5-sonnet": {
"input_cost_per_token": 3e-06,
"output_cost_per_token": 15e-06,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 1024,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/anthropic-claude-3.5-haiku": {
"input_cost_per_token": 8e-07,
"output_cost_per_token": 4e-06,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 1024,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/anthropic-claude-3-opus": {
"input_cost_per_token": 15e-06,
"output_cost_per_token": 75e-06,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 1024,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/deepseek-r1-distill-llama-70b": {
"input_cost_per_token": 99e-08,
"output_cost_per_token": 99e-08,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 8000,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/llama3.3-70b-instruct": {
"input_cost_per_token": 65e-08,
"output_cost_per_token": 65e-08,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 2048,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/llama3-8b-instruct": {
"input_cost_per_token": 2e-07,
"output_cost_per_token": 2e-07,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 512,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/mistral-nemo-instruct-2407": {
"input_cost_per_token": 3e-07,
"output_cost_per_token": 3e-07,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 512,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/openai-o3": {
"input_cost_per_token": 2e-06,
"output_cost_per_token": 8e-06,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 100000,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/openai-o3-mini": {
"input_cost_per_token": 11e-07,
"output_cost_per_token": 44e-07,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 100000,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/openai-gpt-4o": {
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 16384,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/openai-gpt-4o-mini": {
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 16384,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/alibaba-qwen3-32b": {
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 2048,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"nscale/meta-llama/Llama-4-Scout-17B-16E-Instruct": {
"input_cost_per_token": 9e-08,
"output_cost_per_token": 2.9e-07,
litellm/proxy/_experimental/mcp_server/db.py
+15 -7
View File
@@ -1,6 +1,7 @@
import uuid
from typing import Any, Dict, Iterable, List, Optional, Set, Union
from litellm._logging import verbose_proxy_logger
from litellm.proxy._types import (
LiteLLM_MCPServerTable,
LiteLLM_ObjectPermissionTable,
@@ -53,11 +54,20 @@ async def get_all_mcp_servers(
"""
Returns all of the mcp servers from the db
"""
mcp_servers = await prisma_client.db.litellm_mcpservertable.find_many()
try:
mcp_servers = await prisma_client.db.litellm_mcpservertable.find_many()
return [
LiteLLM_MCPServerTable(**mcp_server.model_dump()) for mcp_server in mcp_servers
]
return [
LiteLLM_MCPServerTable(**mcp_server.model_dump())
for mcp_server in mcp_servers
]
except Exception as e:
verbose_proxy_logger.debug(
"litellm.proxy._experimental.mcp_server.db.py::get_all_mcp_servers - {}".format(
str(e)
)
)
return []
async def get_mcp_server(
@@ -91,9 +101,7 @@ async def get_mcp_servers(
)
final_mcp_servers: List[LiteLLM_MCPServerTable] = []
for _mcp_server in _mcp_servers:
final_mcp_servers.append(
LiteLLM_MCPServerTable(**_mcp_server.model_dump())
)
final_mcp_servers.append(LiteLLM_MCPServerTable(**_mcp_server.model_dump()))
return final_mcp_servers
litellm/proxy/_experimental/out/_next/static/zvEphJL9GILaCMSn4AXPU/_buildManifest.js → litellm/proxy/_experimental/out/_next/static/ILJ2l6ZzNB2f7RRsIZVJI/_buildManifest.js
View File
litellm/proxy/_experimental/out/_next/static/zvEphJL9GILaCMSn4AXPU/_ssgManifest.js → litellm/proxy/_experimental/out/_next/static/ILJ2l6ZzNB2f7RRsIZVJI/_ssgManifest.js
View File
litellm/proxy/_experimental/out/_next/static/chunks/154-7bf3bbb913e68f71.js
+1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/154-ff9562264ad409e1.js
-1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/162-741f64e7b75eb970.js
-1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/162-f2925685093720a4.js
+1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/172-2755b782e3848de3.js → litellm/proxy/_experimental/out/_next/static/chunks/172-25e8f67ccf021150.js
+1 -1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/247-7557228b7131016b.js
+12
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/487-79ed94231812dae7.js
+4
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/760-4dbc0194adfbea2a.js
-12
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/85-52b9060394399707.js
+1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/85-aa9694ed291cdedf.js
-1
View File
File diff suppressed because one or more lines are too long
ui/litellm-dashboard/out/_next/static/chunks/866-3523e0e07cf314f6.js → litellm/proxy/_experimental/out/_next/static/chunks/866-9e1803a09e9ae8da.js
+1 -1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/90-d2b5ed6f7f6e342e.js
-4
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/app/layout-25a743106e1c9456.js
-1
View File
@@ -1 +0,0 @@
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[185],{6580:function(n,e,t){Promise.resolve().then(t.t.bind(t,39974,23)),Promise.resolve().then(t.t.bind(t,2778,23))},2778:function(){},39974:function(n){n.exports={style:{fontFamily:"'__Inter_b0dd8a', '__Inter_Fallback_b0dd8a'",fontStyle:"normal"},className:"__className_b0dd8a"}}},function(n){n.O(0,[919,986,971,117,744],function(){return n(n.s=6580)}),_N_E=n.O()}]);
litellm/proxy/_experimental/out/_next/static/chunks/app/layout-f4acf18888f0aa20.js
+1
View File
@@ -0,0 +1 @@
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[185],{96443:function(n,e,t){Promise.resolve().then(t.t.bind(t,39974,23)),Promise.resolve().then(t.t.bind(t,2778,23))},2778:function(){},39974:function(n){n.exports={style:{fontFamily:"'__Inter_b0dd8a', '__Inter_Fallback_b0dd8a'",fontStyle:"normal"},className:"__className_b0dd8a"}}},function(n){n.O(0,[919,986,971,117,744],function(){return n(n.s=96443)}),_N_E=n.O()}]);
litellm/proxy/_experimental/out/_next/static/chunks/app/model_hub/page-8fa4c9ea2a6c311e.js → litellm/proxy/_experimental/out/_next/static/chunks/app/model_hub/page-72c8a8dbd0d3984f.js
+1 -1
View File
@@ -1 +1 @@
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[418],{11790:function(e,n,t){Promise.resolve().then(t.bind(t,52829))},52829:function(e,n,t){"use strict";t.r(n),t.d(n,{default:function(){return f}});var u=t(57437),s=t(2265),c=t(99376),r=t(72162);function f(){let e=(0,c.useSearchParams)().get("key"),[n,t]=(0,s.useState)(null);return(0,s.useEffect)(()=>{e&&t(e)},[e]),(0,u.jsx)(r.Z,{accessToken:n})}}},function(e){e.O(0,[85,90,154,162,971,117,744],function(){return e(e.s=11790)}),_N_E=e.O()}]);
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[418],{21024:function(e,n,t){Promise.resolve().then(t.bind(t,52829))},52829:function(e,n,t){"use strict";t.r(n),t.d(n,{default:function(){return f}});var u=t(57437),s=t(2265),c=t(99376),r=t(72162);function f(){let e=(0,c.useSearchParams)().get("key"),[n,t]=(0,s.useState)(null);return(0,s.useEffect)(()=>{e&&t(e)},[e]),(0,u.jsx)(r.Z,{accessToken:n})}}},function(e){e.O(0,[85,487,154,162,971,117,744],function(){return e(e.s=21024)}),_N_E=e.O()}]);
litellm/proxy/_experimental/out/_next/static/chunks/app/model_hub_table/page-41ea0aa42ba475f3.js → litellm/proxy/_experimental/out/_next/static/chunks/app/model_hub_table/page-10c235fc18df224f.js
+1 -1
View File
@@ -1 +1 @@
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[25],{58538:function(e,n,u){Promise.resolve().then(u.bind(u,22775))},22775:function(e,n,u){"use strict";u.r(n),u.d(n,{default:function(){return f}});var t=u(57437),s=u(2265),r=u(99376),c=u(36172);function f(){let e=(0,r.useSearchParams)().get("key"),[n,u]=(0,s.useState)(null);return(0,s.useEffect)(()=>{e&&u(e)},[e]),(0,t.jsx)(c.Z,{accessToken:n,publicPage:!0,premiumUser:!1,userRole:null})}}},function(e){e.O(0,[85,90,866,154,162,172,971,117,744],function(){return e(e.s=58538)}),_N_E=e.O()}]);
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[25],{64563:function(e,n,u){Promise.resolve().then(u.bind(u,22775))},22775:function(e,n,u){"use strict";u.r(n),u.d(n,{default:function(){return f}});var t=u(57437),s=u(2265),r=u(99376),c=u(36172);function f(){let e=(0,r.useSearchParams)().get("key"),[n,u]=(0,s.useState)(null);return(0,s.useEffect)(()=>{e&&u(e)},[e]),(0,t.jsx)(c.Z,{accessToken:n,publicPage:!0,premiumUser:!1,userRole:null})}}},function(e){e.O(0,[85,487,866,154,162,172,971,117,744],function(){return e(e.s=64563)}),_N_E=e.O()}]);
litellm/proxy/_experimental/out/_next/static/chunks/app/onboarding/page-6b5d568180da3ccd.js
+1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/app/onboarding/page-94ef2a34b1440aa2.js
-1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/app/page-c64580821d04b2c5.js
+1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/app/page-e71feaa99a0e3050.js
-1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/chunks/main-app-475d6efe4080647d.js → litellm/proxy/_experimental/out/_next/static/chunks/main-app-4f7318ae681a6d94.js
+1 -1
View File
@@ -1 +1 @@
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[744],{20169:function(e,n,t){Promise.resolve().then(t.t.bind(t,12846,23)),Promise.resolve().then(t.t.bind(t,19107,23)),Promise.resolve().then(t.t.bind(t,61060,23)),Promise.resolve().then(t.t.bind(t,4707,23)),Promise.resolve().then(t.t.bind(t,80,23)),Promise.resolve().then(t.t.bind(t,36423,23))}},function(e){var n=function(n){return e(e.s=n)};e.O(0,[971,117],function(){return n(54278),n(20169)}),_N_E=e.O()}]);
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[744],{10264:function(e,n,t){Promise.resolve().then(t.t.bind(t,12846,23)),Promise.resolve().then(t.t.bind(t,19107,23)),Promise.resolve().then(t.t.bind(t,61060,23)),Promise.resolve().then(t.t.bind(t,4707,23)),Promise.resolve().then(t.t.bind(t,80,23)),Promise.resolve().then(t.t.bind(t,36423,23))}},function(e){var n=function(n){return e(e.s=n)};e.O(0,[971,117],function(){return n(54278),n(10264)}),_N_E=e.O()}]);
litellm/proxy/_experimental/out/_next/static/css/3c0e0d4261b19d44.css
-3
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/_next/static/css/c8d591a6ccd18f71.css
+3
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/index.html
+1 -1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/index.txt
+2 -2
View File
@@ -1,7 +1,7 @@
2:I[19107,[],"ClientPageRoot"]
3:I[76377,["665","static/chunks/3014691f-b7b79b78e27792f3.js","990","static/chunks/13b76428-ebdf3012af0e4489.js","85","static/chunks/85-aa9694ed291cdedf.js","90","static/chunks/90-d2b5ed6f7f6e342e.js","866","static/chunks/866-3523e0e07cf314f6.js","760","static/chunks/760-4dbc0194adfbea2a.js","154","static/chunks/154-ff9562264ad409e1.js","162","static/chunks/162-741f64e7b75eb970.js","172","static/chunks/172-2755b782e3848de3.js","931","static/chunks/app/page-e71feaa99a0e3050.js"],"default",1]
3:I[39105,["665","static/chunks/3014691f-b7b79b78e27792f3.js","990","static/chunks/13b76428-ebdf3012af0e4489.js","85","static/chunks/85-52b9060394399707.js","487","static/chunks/487-79ed94231812dae7.js","866","static/chunks/866-9e1803a09e9ae8da.js","247","static/chunks/247-7557228b7131016b.js","154","static/chunks/154-7bf3bbb913e68f71.js","162","static/chunks/162-f2925685093720a4.js","172","static/chunks/172-25e8f67ccf021150.js","931","static/chunks/app/page-c64580821d04b2c5.js"],"default",1]
4:I[4707,[],""]
5:I[36423,[],""]
0:["zvEphJL9GILaCMSn4AXPU",[[["",{"children":["__PAGE__",{}]},"$undefined","$undefined",true],["",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/31b7f215e119031e.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/3c0e0d4261b19d44.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_b0dd8a","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
0:["ILJ2l6ZzNB2f7RRsIZVJI",[[["",{"children":["__PAGE__",{}]},"$undefined","$undefined",true],["",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/31b7f215e119031e.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/c8d591a6ccd18f71.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_b0dd8a","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
6:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"LiteLLM Dashboard"}],["$","meta","3",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","4",{"rel":"icon","href":"/favicon.ico","type":"image/x-icon","sizes":"16x16"}],["$","link","5",{"rel":"icon","href":"./favicon.ico"}],["$","meta","6",{"name":"next-size-adjust"}]]
1:null
litellm/proxy/_experimental/out/model_hub.txt
+2 -2
View File
@@ -1,7 +1,7 @@
2:I[19107,[],"ClientPageRoot"]
3:I[52829,["85","static/chunks/85-aa9694ed291cdedf.js","90","static/chunks/90-d2b5ed6f7f6e342e.js","154","static/chunks/154-ff9562264ad409e1.js","162","static/chunks/162-741f64e7b75eb970.js","418","static/chunks/app/model_hub/page-8fa4c9ea2a6c311e.js"],"default",1]
3:I[52829,["85","static/chunks/85-52b9060394399707.js","487","static/chunks/487-79ed94231812dae7.js","154","static/chunks/154-7bf3bbb913e68f71.js","162","static/chunks/162-f2925685093720a4.js","418","static/chunks/app/model_hub/page-72c8a8dbd0d3984f.js"],"default",1]
4:I[4707,[],""]
5:I[36423,[],""]
0:["zvEphJL9GILaCMSn4AXPU",[[["",{"children":["model_hub",{"children":["__PAGE__",{}]}]},"$undefined","$undefined",true],["",{"children":["model_hub",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[null,["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children","model_hub","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/31b7f215e119031e.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/3c0e0d4261b19d44.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_b0dd8a","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
0:["ILJ2l6ZzNB2f7RRsIZVJI",[[["",{"children":["model_hub",{"children":["__PAGE__",{}]}]},"$undefined","$undefined",true],["",{"children":["model_hub",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[null,["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children","model_hub","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/31b7f215e119031e.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/c8d591a6ccd18f71.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_b0dd8a","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
6:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"LiteLLM Dashboard"}],["$","meta","3",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","4",{"rel":"icon","href":"/favicon.ico","type":"image/x-icon","sizes":"16x16"}],["$","link","5",{"rel":"icon","href":"./favicon.ico"}],["$","meta","6",{"name":"next-size-adjust"}]]
1:null
litellm/proxy/_experimental/out/model_hub_table.html
-1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/model_hub_table.txt
+2 -2
View File
@@ -1,7 +1,7 @@
2:I[19107,[],"ClientPageRoot"]
3:I[22775,["85","static/chunks/85-aa9694ed291cdedf.js","90","static/chunks/90-d2b5ed6f7f6e342e.js","866","static/chunks/866-3523e0e07cf314f6.js","154","static/chunks/154-ff9562264ad409e1.js","162","static/chunks/162-741f64e7b75eb970.js","172","static/chunks/172-2755b782e3848de3.js","25","static/chunks/app/model_hub_table/page-41ea0aa42ba475f3.js"],"default",1]
3:I[22775,["85","static/chunks/85-52b9060394399707.js","487","static/chunks/487-79ed94231812dae7.js","866","static/chunks/866-9e1803a09e9ae8da.js","154","static/chunks/154-7bf3bbb913e68f71.js","162","static/chunks/162-f2925685093720a4.js","172","static/chunks/172-25e8f67ccf021150.js","25","static/chunks/app/model_hub_table/page-10c235fc18df224f.js"],"default",1]
4:I[4707,[],""]
5:I[36423,[],""]
0:["zvEphJL9GILaCMSn4AXPU",[[["",{"children":["model_hub_table",{"children":["__PAGE__",{}]}]},"$undefined","$undefined",true],["",{"children":["model_hub_table",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[null,["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children","model_hub_table","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/31b7f215e119031e.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/3c0e0d4261b19d44.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_b0dd8a","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
0:["ILJ2l6ZzNB2f7RRsIZVJI",[[["",{"children":["model_hub_table",{"children":["__PAGE__",{}]}]},"$undefined","$undefined",true],["",{"children":["model_hub_table",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[null,["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children","model_hub_table","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/31b7f215e119031e.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/c8d591a6ccd18f71.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_b0dd8a","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
6:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"LiteLLM Dashboard"}],["$","meta","3",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","4",{"rel":"icon","href":"/favicon.ico","type":"image/x-icon","sizes":"16x16"}],["$","link","5",{"rel":"icon","href":"./favicon.ico"}],["$","meta","6",{"name":"next-size-adjust"}]]
1:null
litellm/proxy/_experimental/out/model_hub_table/index.html
+1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/onboarding.html
-1
View File
File diff suppressed because one or more lines are too long
litellm/proxy/_experimental/out/onboarding.txt
+2 -2
View File
@@ -1,7 +1,7 @@
2:I[19107,[],"ClientPageRoot"]
3:I[12011,["665","static/chunks/3014691f-b7b79b78e27792f3.js","85","static/chunks/85-aa9694ed291cdedf.js","154","static/chunks/154-ff9562264ad409e1.js","461","static/chunks/app/onboarding/page-94ef2a34b1440aa2.js"],"default",1]
3:I[12011,["665","static/chunks/3014691f-b7b79b78e27792f3.js","85","static/chunks/85-52b9060394399707.js","154","static/chunks/154-7bf3bbb913e68f71.js","461","static/chunks/app/onboarding/page-6b5d568180da3ccd.js"],"default",1]
4:I[4707,[],""]
5:I[36423,[],""]
0:["zvEphJL9GILaCMSn4AXPU",[[["",{"children":["onboarding",{"children":["__PAGE__",{}]}]},"$undefined","$undefined",true],["",{"children":["onboarding",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[null,["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children","onboarding","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/31b7f215e119031e.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/3c0e0d4261b19d44.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_b0dd8a","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
0:["ILJ2l6ZzNB2f7RRsIZVJI",[[["",{"children":["onboarding",{"children":["__PAGE__",{}]}]},"$undefined","$undefined",true],["",{"children":["onboarding",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[null,["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children","onboarding","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/31b7f215e119031e.css","precedence":"next","crossOrigin":"$undefined"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/css/c8d591a6ccd18f71.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"__className_b0dd8a","children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
6:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"LiteLLM Dashboard"}],["$","meta","3",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","4",{"rel":"icon","href":"/favicon.ico","type":"image/x-icon","sizes":"16x16"}],["$","link","5",{"rel":"icon","href":"./favicon.ico"}],["$","meta","6",{"name":"next-size-adjust"}]]
1:null
litellm/proxy/_new_secret_config.yaml
+12 -4
View File
@@ -1,5 +1,13 @@
model_list:
- model_name: openai-test
litellm_params:
model: gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY
- model_name: fake-openai-endpoint
litellm_params:
model: openai/fake
api_key: fake-key
api_base: https://exampleopenaiendpoint-production.up.railway.app/
litellm_settings:
cache: true
cache_params:
type: redis
ttl: 600
supported_call_types: ["acompletion", "completion"]
litellm/proxy/_types.py
+41 -11
View File
@@ -16,11 +16,7 @@ from pydantic import (
from typing_extensions import Required, TypedDict
from litellm.types.integrations.slack_alerting import AlertType
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionRequest,
OpenAIFileObject,
)
from litellm.types.llms.openai import AllMessageValues, OpenAIFileObject
from litellm.types.mcp import (
MCPAuthType,
MCPSpecVersion,
@@ -576,13 +572,47 @@ class LiteLLMPromptInjectionParams(LiteLLMPydanticObjectBase):
######### Request Class Definition ######
class ProxyChatCompletionRequest(ChatCompletionRequest):
class ProxyChatCompletionRequest(LiteLLMPydanticObjectBase):
"""
Pydantic model for chat completion requests that includes both OpenAI standard fields
and LiteLLM-specific parameters. This replaces the previous TypedDict version.
"""
# Required fields (from ChatCompletionRequest)
model: str
messages: List[AllMessageValues]
# Standard OpenAI completion parameters (all optional)
frequency_penalty: Optional[float] = None
logit_bias: Optional[Dict[str, float]] = None
logprobs: Optional[bool] = None
top_logprobs: Optional[int] = None
max_tokens: Optional[int] = None
n: Optional[int] = None
presence_penalty: Optional[float] = None
response_format: Optional[Dict[str, Any]] = None
seed: Optional[int] = None
service_tier: Optional[str] = None
stop: Optional[Union[str, List[str]]] = None
stream_options: Optional[Dict[str, Any]] = None
temperature: Optional[float] = None
top_p: Optional[float] = None
tools: Optional[List[Dict[str, Any]]] = None
tool_choice: Optional[Union[str, Dict[str, Any]]] = None
parallel_tool_calls: Optional[bool] = None
function_call: Optional[Union[str, Dict[str, Any]]] = None
functions: Optional[List[Dict[str, Any]]] = None
user: Optional[str] = None
stream: Optional[bool] = None
# LiteLLM-specific metadata param (from original ChatCompletionRequest)
metadata: Optional[Dict[str, Any]] = None
# Optional LiteLLM params
guardrails: Optional[List[str]]
caching: Optional[bool]
num_retries: Optional[int]
context_window_fallback_dict: Optional[Dict[str, str]]
fallbacks: Optional[List[str]]
guardrails: Optional[List[str]] = None
caching: Optional[bool] = None
num_retries: Optional[int] = None
context_window_fallback_dict: Optional[Dict[str, str]] = None
fallbacks: Optional[List[str]] = None
class ModelInfoDelete(LiteLLMPydanticObjectBase):
litellm/proxy/auth/handle_jwt.py
+2 -1
View File
@@ -763,6 +763,7 @@ class JWTAuthManager:
proxy_logging_obj: ProxyLogging,
) -> Tuple[Optional[str], Optional[LiteLLM_TeamTable]]:
"""Find first team with access to the requested model"""
from litellm.proxy.proxy_server import llm_router
if not team_ids:
if jwt_handler.litellm_jwtauth.enforce_team_based_model_access:
@@ -789,7 +790,7 @@ class JWTAuthManager:
or can_team_access_model(
model=requested_model,
team_object=team_object,
llm_router=None,
llm_router=llm_router,
team_model_aliases=None,
)
):
litellm/proxy/common_utils/custom_openapi_spec.py
+88 -4
View File
@@ -72,7 +72,8 @@ class CustomOpenAPISpec:
@staticmethod
def add_request_body_to_paths(openapi_schema: Dict[str, Any], paths: List[str], schema_ref: str) -> None:
"""
Add request body schema reference to specified paths.
Add request body with expanded form fields for better Swagger UI display.
This keeps the request body but expands it to show individual fields in the UI.
Args:
openapi_schema: The OpenAPI schema dict to modify
@@ -81,16 +82,99 @@ class CustomOpenAPISpec:
"""
for path in paths:
if path in openapi_schema.get("paths", {}) and "post" in openapi_schema["paths"][path]:
# Get the actual schema to extract ALL field definitions
schema_name = schema_ref.split("/")[-1] # Extract "ProxyChatCompletionRequest" from the ref
actual_schema = openapi_schema.get("components", {}).get("schemas", {}).get(schema_name, {})
schema_properties = actual_schema.get("properties", {})
required_fields = actual_schema.get("required", [])
# Create an expanded inline schema instead of just a $ref
# This makes Swagger UI show all individual fields in the request body editor
expanded_schema = {
"type": "object",
"required": required_fields,
"properties": {}
}
# Add all properties with their full definitions
for field_name, field_def in schema_properties.items():
expanded_field = CustomOpenAPISpec._expand_field_definition(field_def)
# Add a simple example for the messages field
if field_name == "messages":
expanded_field["example"] = [
{"role": "user", "content": "Hello, how are you?"}
]
expanded_schema["properties"][field_name] = expanded_field
# Include $defs from the original schema to support complex types like AllMessageValues
# This ensures that message types and other complex union types work properly
if "$defs" in actual_schema:
expanded_schema["$defs"] = actual_schema["$defs"]
# Set the request body with the expanded schema
openapi_schema["paths"][path]["post"]["requestBody"] = {
"required": True,
"content": {
"application/json": {
"schema": {
"$ref": schema_ref
}
"schema": expanded_schema
}
}
}
# Keep any existing parameters (like path parameters) but remove conflicting query params
if "parameters" in openapi_schema["paths"][path]["post"]:
existing_params = openapi_schema["paths"][path]["post"]["parameters"]
# Only keep path parameters, remove query params that conflict with request body
filtered_params = [
param for param in existing_params
if param.get("in") == "path"
]
openapi_schema["paths"][path]["post"]["parameters"] = filtered_params
@staticmethod
def _extract_field_schema(field_def: Dict[str, Any]) -> Dict[str, Any]:
"""
Extract a simple schema from a Pydantic field definition for parameter display.
Args:
field_def: Pydantic field definition
Returns:
Simplified schema for OpenAPI parameter
"""
# Handle simple types
if "type" in field_def:
return {"type": field_def["type"]}
# Handle anyOf (Optional fields in Pydantic v2)
if "anyOf" in field_def:
any_of = field_def["anyOf"]
# Find the non-null type
for option in any_of:
if option.get("type") != "null":
return option
# Fallback to string if all else fails
return {"type": "string"}
# Default fallback
return {"type": "string"}
@staticmethod
def _expand_field_definition(field_def: Dict[str, Any]) -> Dict[str, Any]:
"""
Expand a Pydantic field definition for inline use in OpenAPI schema.
This creates a full field definition that Swagger UI can render as individual form fields.
Args:
field_def: Pydantic field definition
Returns:
Expanded field definition for OpenAPI schema
"""
# Return the field definition as-is since Pydantic already provides proper schemas
return field_def.copy()
@staticmethod
def add_request_schema(
litellm/proxy/common_utils/get_routes.py
+22 -3
View File
@@ -2,10 +2,12 @@
Utility class for getting routes from a FastAPI app.
"""
from typing import Any, Dict, List
from typing import Any, Dict, List, Optional
from starlette.routing import BaseRoute
from litellm._logging import verbose_logger
class GetRoutes:
@staticmethod
@@ -53,8 +55,25 @@ class GetRoutes:
"path": full_path,
"methods": getattr(sub_route, "methods", ["GET", "POST"]),
"name": getattr(sub_route, "name", None),
"endpoint": endpoint_func.__name__ if callable(endpoint_func) else None,
"endpoint": GetRoutes._safe_get_endpoint_name(endpoint_func),
"mounted_app": True,
}
routes.append(route_info)
return routes
return routes
@staticmethod
def _safe_get_endpoint_name(endpoint_function: Any) -> Optional[str]:
"""
Safely get the name of the endpoint function.
"""
try:
if hasattr(endpoint_function, '__name__'):
return getattr(endpoint_function, '__name__')
elif hasattr(endpoint_function, '__class__') and hasattr(endpoint_function.__class__, '__name__'):
return getattr(endpoint_function.__class__, '__name__')
else:
return None
except Exception:
verbose_logger.exception(f"Error getting endpoint name for route: {endpoint_function}")
return None
litellm/proxy/health_endpoints/_health_endpoints.py
+57 -48
View File
@@ -324,7 +324,7 @@ def _convert_health_check_to_dict(check) -> dict:
def _check_prisma_client():
"""Helper to check if prisma_client is available and raise appropriate error"""
from litellm.proxy.proxy_server import prisma_client
if prisma_client is None:
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
@@ -334,23 +334,23 @@ def _check_prisma_client():
async def _save_health_check_to_db(
prisma_client,
model_name: str,
healthy_endpoints: list,
unhealthy_endpoints: list,
start_time: float,
prisma_client,
model_name: str,
healthy_endpoints: list,
unhealthy_endpoints: list,
start_time: float,
user_id: Optional[str],
model_id: Optional[str] = None
model_id: Optional[str] = None,
):
"""Helper function to save health check results to database"""
try:
# Extract error message from first unhealthy endpoint if available
error_message = (
str(unhealthy_endpoints[0]["error"])[:500]
if unhealthy_endpoints and unhealthy_endpoints[0].get("error")
str(unhealthy_endpoints[0]["error"])[:500]
if unhealthy_endpoints and unhealthy_endpoints[0].get("error")
else None
)
await prisma_client.save_health_check_result(
model_name=model_name,
model_id=model_id,
@@ -363,7 +363,9 @@ async def _save_health_check_to_db(
checked_by=user_id,
)
except Exception as db_error:
verbose_proxy_logger.warning(f"Failed to save health check to database for model {model_name}: {db_error}")
verbose_proxy_logger.warning(
f"Failed to save health check to database for model {model_name}: {db_error}"
)
# Continue execution - don't let database save failure break health checks
@@ -375,31 +377,30 @@ async def _perform_health_check_and_save(
prisma_client,
start_time,
user_id,
model_id=None
model_id=None,
):
"""Helper function to perform health check and save results to database"""
healthy_endpoints, unhealthy_endpoints = await perform_health_check(
model_list=model_list,
cli_model=cli_model,
model=target_model,
details=details
model_list=model_list, cli_model=cli_model, model=target_model, details=details
)
# Optionally save health check result to database (non-blocking)
if prisma_client is not None:
# For CLI model, use cli_model name; for router models, use target_model
model_name_for_db = cli_model if cli_model is not None else target_model
if model_name_for_db is not None:
asyncio.create_task(_save_health_check_to_db(
prisma_client,
model_name_for_db,
healthy_endpoints,
unhealthy_endpoints,
start_time,
user_id,
model_id=model_id
))
asyncio.create_task(
_save_health_check_to_db(
prisma_client,
model_name_for_db,
healthy_endpoints,
unhealthy_endpoints,
start_time,
user_id,
model_id=model_id,
)
)
return {
"healthy_endpoints": healthy_endpoints,
"unhealthy_endpoints": unhealthy_endpoints,
@@ -434,19 +435,20 @@ async def health_endpoint(
```
else, the health checks will be run on models when /health is called.
"""
import time
from litellm.proxy.proxy_server import (
health_check_details,
health_check_results,
llm_model_list,
llm_router,
prisma_client,
use_background_health_checks,
user_model,
prisma_client,
)
import time
start_time = time.time()
# Handle model_id parameter - convert to model name for health check
target_model = model
if model_id and not model:
@@ -462,7 +464,9 @@ async def health_endpoint(
detail={"error": f"Model with ID {model_id} not found"},
)
except Exception as e:
verbose_proxy_logger.error(f"Error getting deployment for model_id {model_id}: {e}")
verbose_proxy_logger.error(
f"Error getting deployment for model_id {model_id}: {e}"
)
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail={"error": f"Model with ID {model_id} not found"},
@@ -472,7 +476,7 @@ async def health_endpoint(
status_code=status.HTTP_404_NOT_FOUND,
detail={"error": f"Model with ID {model_id} not found"},
)
try:
if llm_model_list is None:
# if no router set, check if user set a model using litellm --model ollama/llama2
@@ -485,7 +489,7 @@ async def health_endpoint(
prisma_client=prisma_client,
start_time=start_time,
user_id=user_api_key_dict.user_id,
model_id=None # CLI model doesn't have model_id
model_id=None, # CLI model doesn't have model_id
)
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
@@ -508,7 +512,7 @@ async def health_endpoint(
prisma_client=prisma_client,
start_time=start_time,
user_id=user_api_key_dict.user_id,
model_id=model_id
model_id=model_id,
)
except Exception as e:
verbose_proxy_logger.error(
@@ -520,7 +524,9 @@ async def health_endpoint(
raise e
@router.get("/health/history", tags=["health"], dependencies=[Depends(user_api_key_auth)])
@router.get(
"/health/history", tags=["health"], dependencies=[Depends(user_api_key_auth)]
)
async def health_check_history_endpoint(
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
model: Optional[str] = fastapi.Query(
@@ -532,17 +538,15 @@ async def health_check_history_endpoint(
limit: int = fastapi.Query(
100, description="Number of records to return", ge=1, le=1000
),
offset: int = fastapi.Query(
0, description="Number of records to skip", ge=0
),
offset: int = fastapi.Query(0, description="Number of records to skip", ge=0),
):
"""
Get health check history for models
Returns historical health check data with optional filtering.
"""
prisma_client = _check_prisma_client()
try:
history = await prisma_client.get_health_check_history(
model_name=model,
@@ -550,10 +554,10 @@ async def health_check_history_endpoint(
offset=offset,
status_filter=status_filter,
)
# Convert to dict format for JSON response using helper function
history_data = [_convert_health_check_to_dict(check) for check in history]
return {
"health_checks": history_data,
"total_records": len(history_data),
@@ -568,26 +572,30 @@ async def health_check_history_endpoint(
)
@router.get("/health/latest", tags=["health"], dependencies=[Depends(user_api_key_auth)])
@router.get(
"/health/latest", tags=["health"], dependencies=[Depends(user_api_key_auth)]
)
async def latest_health_checks_endpoint(
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""
Get the latest health check status for all models
Returns the most recent health check result for each model.
"""
prisma_client = _check_prisma_client()
try:
latest_checks = await prisma_client.get_all_latest_health_checks()
# Convert to dict format for JSON response using helper function
checks_data = {
(check.model_id if check.model_id else check.model_name): _convert_health_check_to_dict(check)
(
check.model_id if check.model_id else check.model_name
): _convert_health_check_to_dict(check)
for check in latest_checks
}
return {
"latest_health_checks": checks_data,
"total_models": len(checks_data),
@@ -896,6 +904,7 @@ async def test_model_connection(
litellm_params=litellm_params,
)
mode = mode or litellm_params.pop("mode", None)
result = await run_with_timeout(
litellm.ahealth_check(
model_params=litellm_params,
litellm/proxy/hooks/parallel_request_limiter_v3.py
+51 -34
View File
@@ -1,8 +1,9 @@
"""
This is a rate limiter implementation based on a similar one by Envoy proxy.
This is a rate limiter implementation based on a similar one by Envoy proxy.
This is currently in development and not yet ready for production.
"""
import os
from datetime import datetime
from typing import (
@@ -309,13 +310,15 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
continue
key_metadata[window_key] = {
"requests_limit": int(requests_limit)
if requests_limit is not None
else None,
"requests_limit": (
int(requests_limit) if requests_limit is not None else None
),
"tokens_limit": int(tokens_limit) if tokens_limit is not None else None,
"max_parallel_requests_limit": int(max_parallel_requests_limit)
if max_parallel_requests_limit is not None
else None,
"max_parallel_requests_limit": (
int(max_parallel_requests_limit)
if max_parallel_requests_limit is not None
else None
),
"window_size": int(window_size),
"descriptor_key": descriptor_key,
}
@@ -394,7 +397,11 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
descriptors = []
# API Key rate limits
if user_api_key_dict.api_key:
if user_api_key_dict.api_key and (
user_api_key_dict.rpm_limit is not None
or user_api_key_dict.tpm_limit is not None
or user_api_key_dict.max_parallel_requests is not None
):
descriptors.append(
RateLimitDescriptor(
key="api_key",
@@ -409,7 +416,10 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
)
# User rate limits
if user_api_key_dict.user_id:
if user_api_key_dict.user_id and (
user_api_key_dict.user_rpm_limit is not None
or user_api_key_dict.user_tpm_limit is not None
):
descriptors.append(
RateLimitDescriptor(
key="user",
@@ -423,7 +433,10 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
)
# Team rate limits
if user_api_key_dict.team_id:
if user_api_key_dict.team_id and (
user_api_key_dict.team_rpm_limit is not None
or user_api_key_dict.team_tpm_limit is not None
):
descriptors.append(
RateLimitDescriptor(
key="team",
@@ -437,7 +450,10 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
)
# End user rate limits
if user_api_key_dict.end_user_id:
if user_api_key_dict.end_user_id and (
user_api_key_dict.end_user_rpm_limit is not None
or user_api_key_dict.end_user_tpm_limit is not None
):
descriptors.append(
RateLimitDescriptor(
key="end_user",
@@ -483,28 +499,29 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
)
)
# Check rate limits
response = await self.should_rate_limit(
descriptors=descriptors,
parent_otel_span=user_api_key_dict.parent_otel_span,
)
# Only check rate limits if we have descriptors with actual limits
if descriptors:
response = await self.should_rate_limit(
descriptors=descriptors,
parent_otel_span=user_api_key_dict.parent_otel_span,
)
if response["overall_code"] == "OVER_LIMIT":
# Find which descriptor hit the limit
for i, status in enumerate(response["statuses"]):
if status["code"] == "OVER_LIMIT":
descriptor = descriptors[i]
raise HTTPException(
status_code=429,
detail=f"Rate limit exceeded for {descriptor['key']}: {descriptor['value']}. Remaining: {status['limit_remaining']}",
headers={
"retry-after": str(self.window_size)
}, # Retry after 1 minute
)
if response["overall_code"] == "OVER_LIMIT":
# Find which descriptor hit the limit
for i, status in enumerate(response["statuses"]):
if status["code"] == "OVER_LIMIT":
descriptor = descriptors[i]
raise HTTPException(
status_code=429,
detail=f"Rate limit exceeded for {descriptor['key']}: {descriptor['value']}. Remaining: {status['limit_remaining']}",
headers={
"retry-after": str(self.window_size)
}, # Retry after 1 minute
)
else:
# add descriptors to request headers
data["litellm_proxy_rate_limit_response"] = response
else:
# add descriptors to request headers
data["litellm_proxy_rate_limit_response"] = response
def _create_pipeline_operations(
self,
@@ -690,9 +707,9 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
from litellm.types.caching import RedisPipelineIncrementOperation
try:
litellm_parent_otel_span: Union[
Span, None
] = _get_parent_otel_span_from_kwargs(kwargs)
litellm_parent_otel_span: Union[Span, None] = (
_get_parent_otel_span_from_kwargs(kwargs)
)
user_api_key = kwargs["litellm_params"]["metadata"].get("user_api_key")
pipeline_operations: List[RedisPipelineIncrementOperation] = []
litellm/proxy/proxy_config.yaml
+7 -11
View File
@@ -1,15 +1,11 @@
model_list:
- model_name: gemini/*
- model_name: openai/*
litellm_params:
model: gemini/*
model: openai/*
- model_name: anthropic/*
litellm_params:
model: anthropic/*
litellm_settings:
callbacks: ["s3_v2"]
s3_callback_params:
s3_bucket_name: litellm-logs # AWS Bucket Name for S3
s3_region_name: us-west-2
general_settings:
cold_storage_custom_logger: s3_v2
store_prompts_in_cold_storage: true
store_prompts_in_spend_logs: true
callbacks:
- langfuse_otel
litellm/proxy/proxy_server.py
+428 -7
View File
@@ -11,6 +11,7 @@ import time
import traceback
import uuid
import warnings
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from datetime import datetime, timedelta
from typing import (
TYPE_CHECKING,
@@ -731,6 +732,11 @@ def get_openapi_schema():
}
}
# Add LLM API request schema bodies for documentation
from litellm.proxy.common_utils.custom_openapi_spec import CustomOpenAPISpec
openapi_schema = CustomOpenAPISpec.add_llm_api_request_schema_body(openapi_schema)
app.openapi_schema = openapi_schema
return app.openapi_schema
@@ -759,6 +765,9 @@ def custom_openapi():
if os.getenv("DOCS_FILTERED", "False") == "True" and premium_user:
app.openapi = custom_openapi # type: ignore
else:
# For regular users, use get_openapi_schema to include LLM API schemas
app.openapi = get_openapi_schema # type: ignore
class UserAPIKeyCacheTTLEnum(enum.Enum):
@@ -973,6 +982,11 @@ proxy_logging_obj = ProxyLogging(
async_result = None
celery_app_conn = None
celery_fn = None # Redis Queue for handling requests
# Global variables for model cost map reload scheduling
scheduler = None
last_model_cost_map_reload = None
### DB WRITER ###
db_writer_client: Optional[AsyncHTTPHandler] = None
### logger ###
@@ -2712,7 +2726,10 @@ class ProxyConfig:
alert_types=general_settings["alert_types"], llm_router=llm_router
)
if _general_settings is not None and "alert_to_webhook_url" in _general_settings:
if (
_general_settings is not None
and "alert_to_webhook_url" in _general_settings
):
general_settings["alert_to_webhook_url"] = _general_settings[
"alert_to_webhook_url"
]
@@ -2928,13 +2945,105 @@ class ProxyConfig:
await self._init_mcp_servers_in_db()
await self._init_pass_through_endpoints_in_db()
await self._init_prompts_in_db(prisma_client=prisma_client)
await self._check_and_reload_model_cost_map(prisma_client=prisma_client)
async def _check_and_reload_model_cost_map(self, prisma_client: PrismaClient):
"""
Check if model cost map needs to be reloaded based on database configuration.
This function runs every 10 seconds as part of _init_non_llm_objects_in_db.
"""
try:
# Get model cost map reload configuration from database
config_record = await prisma_client.db.litellm_config.find_unique(
where={"param_name": "model_cost_map_reload_config"}
)
if config_record is None or config_record.param_value is None:
return # No configuration found, skip reload
config = config_record.param_value
interval_hours = config.get("interval_hours")
force_reload = config.get("force_reload", False)
if interval_hours is None and force_reload is False:
return # No interval configured, skip reload
current_time = datetime.utcnow()
# Check if we need to reload based on interval or force reload
should_reload = False
if force_reload:
should_reload = True
verbose_proxy_logger.info("Model cost map reload triggered by force reload flag")
elif interval_hours is not None:
# Use pod's in-memory last reload time
global last_model_cost_map_reload
if last_model_cost_map_reload is not None:
try:
last_reload_time = datetime.fromisoformat(last_model_cost_map_reload)
time_since_last_reload = current_time - last_reload_time
hours_since_last_reload = time_since_last_reload.total_seconds() / 3600
if hours_since_last_reload >= interval_hours:
should_reload = True
verbose_proxy_logger.info(f"Model cost map reload triggered by interval. Hours since last reload: {hours_since_last_reload:.2f}, Interval: {interval_hours}")
except Exception as e:
verbose_proxy_logger.warning(f"Error parsing last reload time: {e}")
# If we can't parse the last reload time, reload anyway
should_reload = True
else:
# No last reload time recorded, reload now
should_reload = True
verbose_proxy_logger.info("Model cost map reload triggered - no previous reload time recorded")
if should_reload:
# Perform the reload
from litellm.litellm_core_utils.get_model_cost_map import get_model_cost_map
model_cost_map_url = litellm.model_cost_map_url
new_model_cost_map = get_model_cost_map(url=model_cost_map_url)
litellm.model_cost = new_model_cost_map
# Update pod's in-memory last reload time
last_model_cost_map_reload = current_time.isoformat()
# Clear force reload flag in database
await prisma_client.db.litellm_config.upsert(
where={"param_name": "model_cost_map_reload_config"},
data={
"create": {
"param_name": "model_cost_map_reload_config",
"param_value": safe_dumps({
"interval_hours": interval_hours,
"force_reload": False
})
},
"update": {
"param_value": safe_dumps({
"force_reload": False
})
}
}
)
verbose_proxy_logger.info(f"Model cost map reloaded successfully. Models count: {len(new_model_cost_map) if new_model_cost_map else 0}")
except Exception as e:
verbose_proxy_logger.exception(f"Error in _check_and_reload_model_cost_map: {str(e)}")
async def _init_prompts_in_db(self, prisma_client: PrismaClient):
from litellm.proxy.prompts.prompt_registry import IN_MEMORY_PROMPT_REGISTRY
prompts_in_db = await prisma_client.db.litellm_prompttable.find_many()
for prompt in prompts_in_db:
IN_MEMORY_PROMPT_REGISTRY.initialize_prompt(prompt=prompt)
try:
prompts_in_db = await prisma_client.db.litellm_prompttable.find_many()
for prompt in prompts_in_db:
IN_MEMORY_PROMPT_REGISTRY.initialize_prompt(prompt=prompt)
except Exception as e:
verbose_proxy_logger.debug(
"litellm.proxy.proxy_server.py::ProxyConfig:_init_prompts_in_db - {}".format(
str(e)
)
)
async def _init_guardrails_in_db(self, prisma_client: PrismaClient):
from litellm.proxy.guardrails.guardrail_registry import (
@@ -3519,6 +3628,7 @@ class ProxyStartupEvent:
args=[prisma_client, db_writer_client, proxy_logging_obj],
)
### ADD NEW MODELS ###
store_model_in_db = (
get_secret_bool("STORE_MODEL_IN_DB", store_model_in_db) or store_model_in_db
@@ -3857,8 +3967,22 @@ async def model_info(
# Validate that the requested model is accessible
validate_model_access(model_id=model_id, available_models=all_models)
# Get provider information
_, provider, _, _ = litellm.get_llm_provider(model=model_id)
# Get provider information from the router deployment
if llm_router is None:
raise HTTPException(
status_code=500,
detail="Router not initialized"
)
deployment = llm_router.get_deployment_by_model_group_name(model_id)
if deployment is None:
raise HTTPException(
status_code=404,
detail=f"Model '{model_id}' not found in router configuration"
)
# Use the actual litellm model from the deployment to get provider info
_, provider, _, _ = litellm.get_llm_provider(model=deployment.litellm_params.model)
# Return the model information in the same format as the list endpoint
return create_model_info_response(
@@ -8858,7 +8982,16 @@ async def config_yaml_endpoint(config_info: ConfigYAML):
include_in_schema=False,
dependencies=[Depends(user_api_key_auth)],
)
async def get_litellm_model_cost_map():
async def get_litellm_model_cost_map(
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
# Check if user is admin
if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
raise HTTPException(
status_code=403,
detail=f"Access denied. Admin role required. Current role: {user_api_key_dict.user_role}",
)
try:
_model_cost_map = litellm.model_cost
return _model_cost_map
@@ -8869,6 +9002,294 @@ async def get_litellm_model_cost_map():
)
@router.post(
"/reload/model_cost_map",
tags=["model management"],
dependencies=[Depends(user_api_key_auth)],
include_in_schema=False,
)
async def reload_model_cost_map(
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""
ADMIN ONLY / MASTER KEY Only Endpoint
Manually reload the model cost map from the remote source.
This will fetch fresh pricing data from the model_prices_and_context_window.json file.
"""
# Check if user is admin
if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
raise HTTPException(
status_code=403,
detail=f"Access denied. Admin role required. Current role: {user_api_key_dict.user_role}",
)
try:
global prisma_client
if prisma_client is None:
raise HTTPException(
status_code=500,
detail="Database connection not available"
)
# Immediately reload the model cost map in the current pod
from litellm.litellm_core_utils.get_model_cost_map import get_model_cost_map
model_cost_map_url = litellm.model_cost_map_url
new_model_cost_map = get_model_cost_map(url=model_cost_map_url)
litellm.model_cost = new_model_cost_map
# Update pod's in-memory last reload time
global last_model_cost_map_reload
current_time = datetime.utcnow()
last_model_cost_map_reload = current_time.isoformat()
# Set force reload flag in database for other pods
await prisma_client.db.litellm_config.upsert(
where={"param_name": "model_cost_map_reload_config"},
data={
"create": {
"param_name": "model_cost_map_reload_config",
"param_value": safe_dumps({
"interval_hours": None,
"force_reload": True
})
},
"update": {
"param_value": safe_dumps({
"force_reload": True
})
}
}
)
models_count = len(new_model_cost_map) if new_model_cost_map else 0
verbose_proxy_logger.info(f"Model cost map reloaded successfully in current pod. Models count: {models_count}")
return {
"message": f"Price data reloaded successfully! {models_count} models updated.",
"status": "success",
"models_count": models_count,
"timestamp": current_time.isoformat()
}
except Exception as e:
verbose_proxy_logger.exception(f"Failed to reload model cost map: {str(e)}")
raise HTTPException(
status_code=500,
detail=f"Failed to reload model cost map: {str(e)}"
)
@router.post(
"/schedule/model_cost_map_reload",
tags=["model management"],
dependencies=[Depends(user_api_key_auth)],
include_in_schema=False,
)
async def schedule_model_cost_map_reload(
hours: int,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""
ADMIN ONLY / MASTER KEY Only Endpoint
Schedule periodic reload of the model cost map.
This will create a background job that reloads the model cost map every specified hours.
"""
# Check if user is admin
if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
raise HTTPException(
status_code=403,
detail=f"Access denied. Admin role required. Current role: {user_api_key_dict.user_role}",
)
if hours <= 0:
raise HTTPException(
status_code=400,
detail="Hours must be greater than 0"
)
try:
global prisma_client
if prisma_client is None:
raise HTTPException(
status_code=500,
detail="Database connection not available"
)
# Update database with new reload configuration
await prisma_client.db.litellm_config.upsert(
where={"param_name": "model_cost_map_reload_config"},
data={
"create": {
"param_name": "model_cost_map_reload_config",
"param_value": safe_dumps({
"interval_hours": hours,
"force_reload": False
})
},
"update": {
"param_value": safe_dumps({
"interval_hours": hours,
"force_reload": False
})
}
}
)
verbose_proxy_logger.info(f"Model cost map reload scheduled for every {hours} hours")
return {
"message": f"Model cost map reload scheduled for every {hours} hours",
"status": "success",
"interval_hours": hours,
"timestamp": datetime.utcnow().isoformat()
}
except Exception as e:
verbose_proxy_logger.exception(f"Failed to schedule model cost map reload: {str(e)}")
raise HTTPException(
status_code=500,
detail=f"Failed to schedule model cost map reload: {str(e)}"
)
@router.delete(
"/schedule/model_cost_map_reload",
tags=["model management"],
dependencies=[Depends(user_api_key_auth)],
include_in_schema=False,
)
async def cancel_model_cost_map_reload(
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""
ADMIN ONLY / MASTER KEY Only Endpoint
Cancel the scheduled periodic reload of the model cost map.
"""
# Check if user is admin
if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
raise HTTPException(
status_code=403,
detail=f"Access denied. Admin role required. Current role: {user_api_key_dict.user_role}",
)
try:
global prisma_client
if prisma_client is None:
raise HTTPException(
status_code=500,
detail="Database connection not available"
)
# Remove reload configuration from database
await prisma_client.db.litellm_config.delete(
where={"param_name": "model_cost_map_reload_config"}
)
verbose_proxy_logger.info("Model cost map reload schedule cancelled")
return {
"message": "Model cost map reload schedule cancelled",
"status": "success",
"timestamp": datetime.utcnow().isoformat()
}
except Exception as e:
verbose_proxy_logger.exception(f"Failed to cancel model cost map reload: {str(e)}")
raise HTTPException(
status_code=500,
detail=f"Failed to cancel model cost map reload: {str(e)}"
)
@router.get(
"/schedule/model_cost_map_reload/status",
tags=["model management"],
dependencies=[Depends(user_api_key_auth)],
include_in_schema=False,
)
async def get_model_cost_map_reload_status(
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""
ADMIN ONLY / MASTER KEY Only Endpoint
Get the status of the scheduled model cost map reload job.
"""
# Check if user is admin
if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
raise HTTPException(
status_code=403,
detail=f"Access denied. Admin role required. Current role: {user_api_key_dict.user_role}",
)
try:
global prisma_client, last_model_cost_map_reload
verbose_proxy_logger.info(f"Checking model cost map reload status. Last reload: {last_model_cost_map_reload}")
if prisma_client is None:
verbose_proxy_logger.info("No database connection, returning not scheduled")
return {
"scheduled": False,
"interval_hours": None,
"last_run": None,
"next_run": None
}
# Get reload configuration from database
config_record = await prisma_client.db.litellm_config.find_unique(
where={"param_name": "model_cost_map_reload_config"}
)
if config_record is None or config_record.param_value is None:
verbose_proxy_logger.info("No model cost map reload configuration found")
return {
"scheduled": False,
"interval_hours": None,
"last_run": None,
"next_run": None
}
config = config_record.param_value
interval_hours = config.get("interval_hours")
if interval_hours is None:
verbose_proxy_logger.info("No interval configured, returning not scheduled")
return {
"scheduled": False,
"interval_hours": None,
"last_run": None,
"next_run": None
}
current_time = datetime.utcnow()
next_run = None
# Use pod's in-memory last reload time
if last_model_cost_map_reload is not None:
try:
last_reload_time = datetime.fromisoformat(last_model_cost_map_reload)
time_since_last_reload = current_time - last_reload_time
hours_since_last_reload = time_since_last_reload.total_seconds() / 3600
if hours_since_last_reload < interval_hours:
next_run = (last_reload_time + timedelta(hours=interval_hours)).isoformat()
except Exception as e:
verbose_proxy_logger.warning(f"Error parsing last reload time: {e}")
return {
"scheduled": True,
"interval_hours": interval_hours,
"last_run": last_model_cost_map_reload,
"next_run": next_run
}
except Exception as e:
verbose_proxy_logger.exception(f"Failed to get model cost map reload status: {str(e)}")
raise HTTPException(
status_code=500,
detail=f"Failed to get model cost map reload status: {str(e)}"
)
@router.get("/", dependencies=[Depends(user_api_key_auth)])
async def home(request: Request):
return "LiteLLM: RUNNING"
litellm/proxy/utils.py
+60 -33
View File
@@ -22,7 +22,7 @@ from typing import (
overload,
)
from litellm.constants import MAX_TEAM_LIST_LIMIT, DEFAULT_MODEL_CREATED_AT_TIME
from litellm.constants import DEFAULT_MODEL_CREATED_AT_TIME, MAX_TEAM_LIST_LIMIT
from litellm.proxy._types import (
DB_CONNECTION_ERROR_TYPES,
CommonProxyErrors,
@@ -448,7 +448,6 @@ class ProxyLogging:
litellm_parent_otel_span=None,
)
def _convert_user_api_key_auth_to_dict(self, user_api_key_auth_obj):
"""
Helper function to convert UserAPIKeyAuth object to dictionary.
@@ -728,15 +727,19 @@ class ProxyLogging:
}
return result
def _create_mcp_request_object_from_kwargs(self, kwargs: dict) -> "MCPPreCallRequestObject":
def _create_mcp_request_object_from_kwargs(
self, kwargs: dict
) -> "MCPPreCallRequestObject":
"""
Helper function to create MCPPreCallRequestObject from kwargs for standard pre_call_hook.
"""
from litellm.types.llms.base import HiddenParams
from litellm.types.mcp import MCPPreCallRequestObject
user_api_key_auth_dict = self._convert_user_api_key_auth_to_dict(kwargs.get("user_api_key_auth"))
user_api_key_auth_dict = self._convert_user_api_key_auth_to_dict(
kwargs.get("user_api_key_auth")
)
return MCPPreCallRequestObject(
tool_name=kwargs.get("name", ""),
arguments=kwargs.get("arguments", {}),
@@ -745,22 +748,23 @@ class ProxyLogging:
hidden_params=HiddenParams(),
)
def _convert_mcp_hook_response_to_kwargs(self, response_data: Optional[dict], original_kwargs: dict) -> dict:
def _convert_mcp_hook_response_to_kwargs(
self, response_data: Optional[dict], original_kwargs: dict
) -> dict:
"""
Helper function to convert pre_call_hook response back to kwargs for MCP usage.
"""
if not response_data:
return original_kwargs
# Apply any argument modifications from the hook response
modified_kwargs = original_kwargs.copy()
# If the response contains modified arguments, apply them
if response_data.get("modified_arguments"):
modified_kwargs["arguments"] = response_data["modified_arguments"]
return modified_kwargs
return modified_kwargs
async def process_pre_call_hook_response(self, response, data, call_type):
if isinstance(response, Exception):
@@ -894,6 +898,7 @@ class ProxyLogging:
try:
for callback in litellm.callbacks:
start_time = time.time()
_callback = None
if isinstance(callback, str):
_callback = litellm.litellm_core_utils.litellm_logging.get_custom_logger_compatible_class(
@@ -903,15 +908,13 @@ class ProxyLogging:
_callback = callback # type: ignore
if _callback is not None and isinstance(_callback, CustomGuardrail):
from litellm.types.guardrails import GuardrailEventHooks
event_type = GuardrailEventHooks.pre_call
if call_type == "mcp_call":
event_type = GuardrailEventHooks.pre_mcp_call
if (
_callback.should_run_guardrail(
data=data, event_type=event_type
)
_callback.should_run_guardrail(data=data, event_type=event_type)
is not True
):
continue
@@ -936,7 +939,7 @@ class ProxyLogging:
):
if call_type == "mcp_call" and user_api_key_dict is None:
continue
response = await _callback.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=self.call_details["user_api_key_cache"],
@@ -948,6 +951,19 @@ class ProxyLogging:
response=response, data=data, call_type=call_type
)
end_time = time.time()
duration = end_time - start_time
if (
hasattr(self, "service_logging_obj") and duration > 0.01
): # only if duration is non-negligible - don't spam the logs
await self.service_logging_obj.async_service_success_hook(
service=ServiceTypes.PROXY_PRE_CALL,
duration=duration,
call_type=f"{_callback.__class__.__name__}",
parent_otel_span=user_api_key_dict.parent_otel_span,
start_time=start_time,
end_time=end_time,
)
return data
except Exception as e:
raise e
@@ -999,7 +1015,9 @@ class ProxyLogging:
continue
# Convert user_api_key_dict to proper format for async_moderation_hook
if call_type == "mcp_call":
user_api_key_auth_dict = self._convert_user_api_key_auth_to_dict(user_api_key_dict)
user_api_key_auth_dict = (
self._convert_user_api_key_auth_to_dict(user_api_key_dict)
)
else:
user_api_key_auth_dict = user_api_key_dict
@@ -3668,9 +3686,10 @@ def construct_database_url_from_env_vars() -> Optional[str]:
database_url = f"postgresql://{database_username_enc}@{database_host}/{database_name_enc}"
return database_url
return None
async def count_tokens_with_anthropic_api(
model_to_use: str,
messages: Optional[List[Dict[str, Any]]],
@@ -3691,9 +3710,10 @@ async def count_tokens_with_anthropic_api(
return None
try:
import anthropic
import os
import anthropic
# Get Anthropic API key from deployment config
anthropic_api_key = None
if deployment is not None:
@@ -3712,7 +3732,7 @@ async def count_tokens_with_anthropic_api(
response = client.beta.messages.count_tokens(
model=model_to_use,
messages=messages, # type: ignore
betas=["token-counting-2024-11-01"]
betas=["token-counting-2024-11-01"],
)
total_tokens = response.input_tokens
tokenizer_used = "anthropic_api"
@@ -3723,11 +3743,16 @@ async def count_tokens_with_anthropic_api(
}
except ImportError:
verbose_proxy_logger.warning("Anthropic library not available, falling back to LiteLLM tokenizer")
verbose_proxy_logger.warning(
"Anthropic library not available, falling back to LiteLLM tokenizer"
)
except Exception as e:
verbose_proxy_logger.warning(f"Error calling Anthropic API: {e}, falling back to LiteLLM tokenizer")
verbose_proxy_logger.warning(
f"Error calling Anthropic API: {e}, falling back to LiteLLM tokenizer"
)
return None
async def get_available_models_for_user(
user_api_key_dict: "UserAPIKeyAuth",
llm_router: Optional["Router"],
@@ -3743,7 +3768,7 @@ async def get_available_models_for_user(
) -> List[str]:
"""
Get the list of models available to a user based on their API key and team permissions.
Args:
user_api_key_dict: User API key authentication object
llm_router: LiteLLM router instance
@@ -3755,18 +3780,18 @@ async def get_available_models_for_user(
include_model_access_groups: Whether to include model access groups
only_model_access_groups: Whether to only return model access groups
return_wildcard_routes: Whether to return wildcard routes
Returns:
List of model names available to the user
"""
from litellm.proxy.auth.auth_checks import get_team_object
from litellm.proxy.auth.model_checks import (
get_complete_model_list,
get_key_models,
get_team_models,
get_complete_model_list,
)
from litellm.proxy.auth.auth_checks import get_team_object
from litellm.proxy.management_endpoints.team_endpoints import validate_membership
# Get proxy model list and access groups
if llm_router is None:
proxy_model_list = []
@@ -3831,19 +3856,19 @@ def create_model_info_response(
) -> dict:
"""
Create a standardized model info response.
Args:
model_id: The model ID
provider: The model provider
include_metadata: Whether to include metadata
fallback_type: Type of fallbacks to include
llm_router: LiteLLM router instance
Returns:
Dictionary containing model information
"""
from litellm.proxy.auth.model_checks import get_all_fallbacks
model_info = {
"id": model_id,
"object": "model",
@@ -3886,16 +3911,18 @@ def validate_model_access(
) -> None:
"""
Validate that a model is accessible to the user.
Args:
model_id: The model ID to validate
available_models: List of models available to the user
Raises:
HTTPException: If the model is not accessible
"""
if model_id not in available_models:
raise HTTPException(
status_code=404,
detail="The model `{}` does not exist or is not accessible".format(model_id)
detail="The model `{}` does not exist or is not accessible".format(
model_id
),
)
litellm/router.py
+70 -9
View File
@@ -4308,6 +4308,8 @@ class Router:
"""
Track remaining tpm/rpm quota for model in model_list
"""
from litellm.types.caching import RedisPipelineIncrementOperation
try:
standard_logging_object: Optional[StandardLoggingPayload] = kwargs.get(
"standard_logging_object", None
@@ -4327,6 +4329,39 @@ class Router:
elif isinstance(id, int):
id = str(id)
## get deployment info
deployment_info = self.get_deployment(model_id=id)
if deployment_info is None:
return
else:
deployment_model_info = self.get_router_model_info(
deployment=deployment_info.model_dump(),
received_model_name=model_group,
)
# get tpm/rpm from deployment info
tpm = deployment_info.get("tpm", None)
rpm = deployment_info.get("rpm", None)
## check tpm/rpm in litellm_params
tpm_litellm_params = deployment_info.litellm_params.tpm
rpm_litellm_params = deployment_info.litellm_params.rpm
## check tpm/rpm in model_info
tpm_model_info = deployment_model_info.get("tpm", None)
rpm_model_info = deployment_model_info.get("rpm", None)
## if all are none, return - no need to track current tpm/rpm usage for models with no tpm/rpm set
if (
tpm is None
and rpm is None
and tpm_litellm_params is None
and rpm_litellm_params is None
and tpm_model_info is None
and rpm_model_info is None
):
return
parent_otel_span = _get_parent_otel_span_from_kwargs(kwargs)
total_tokens: float = standard_logging_object.get("total_tokens", 0)
@@ -4346,23 +4381,32 @@ class Router:
# ------------
# update cache
pipeline_operations: List[RedisPipelineIncrementOperation] = []
## TPM
await self.cache.async_increment_cache(
key=tpm_key,
value=total_tokens,
parent_otel_span=parent_otel_span,
ttl=RoutingArgs.ttl.value,
pipeline_operations.append(
RedisPipelineIncrementOperation(
key=tpm_key,
increment_value=total_tokens,
ttl=RoutingArgs.ttl.value,
)
)
## RPM
rpm_key = RouterCacheEnum.RPM.value.format(
id=id, current_minute=current_minute, model=deployment_name
)
await self.cache.async_increment_cache(
key=rpm_key,
value=1,
pipeline_operations.append(
RedisPipelineIncrementOperation(
key=rpm_key,
increment_value=1,
ttl=RoutingArgs.ttl.value,
)
)
await self.cache.async_increment_cache_pipeline(
increment_list=pipeline_operations,
parent_otel_span=parent_otel_span,
ttl=RoutingArgs.ttl.value,
)
increment_deployment_successes_for_current_minute(
@@ -5496,6 +5540,23 @@ class Router:
except Exception:
pass
## check for base model
try:
if custom_model_info is not None:
base_model = custom_model_info.get("base_model", None)
if base_model is not None:
## update litellm model info with base model info
base_model_info = litellm.get_model_info(model=base_model)
if base_model_info is not None:
custom_model_info = custom_model_info or {}
# Base model provides defaults, custom model info overrides
custom_model_info = _update_dictionary(
cast(dict, base_model_info),
custom_model_info,
)
except Exception:
pass
if custom_model_info is not None and litellm_model_name_model_info is not None:
model_info = cast(
ModelInfo,
litellm/types/utils.py
+2 -1
View File
@@ -1618,7 +1618,7 @@ class ImageResponse(OpenAIImageResponse, BaseLiteLLMOpenAIResponseObject):
usage: Optional[ImageUsage] = None # type: ignore
"""
Users might use litellm with older python versions, we don't want this to break for them.
Users might use litellm with older python versions, we don't want this to break for them.
Happens when their OpenAIImageResponse has the old OpenAI usage class.
"""
@@ -2324,6 +2324,7 @@ class LlmProviders(str, Enum):
ASSEMBLYAI = "assemblyai"
GITHUB_COPILOT = "github_copilot"
SNOWFLAKE = "snowflake"
GRADIENT_AI = "gradient_ai"
LLAMA = "meta_llama"
NSCALE = "nscale"
PG_VECTOR = "pg_vector"
litellm/utils.py
+25 -1
View File
@@ -3927,6 +3927,17 @@ def get_optional_params( # noqa: PLR0915
else False
),
)
elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
optional_params = litellm.AzureOpenAIGPT5Config().map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
model=model,
drop_params=(
drop_params
if drop_params is not None and isinstance(drop_params, bool)
else False
),
)
else:
verbose_logger.debug(
"Azure optional params - api_version: api_version={}, litellm.api_version={}, os.environ['AZURE_API_VERSION']={}".format(
@@ -6755,6 +6766,11 @@ class ProviderConfigManager:
and litellm.openaiOSeriesConfig.is_model_o_series_model(model=model)
):
return litellm.openaiOSeriesConfig
elif (
provider == LlmProviders.OPENAI
and litellm.OpenAIGPT5Config.is_model_gpt_5_model(model=model)
):
return litellm.OpenAIGPT5Config()
elif litellm.LlmProviders.DEEPSEEK == provider:
return litellm.DeepSeekChatConfig()
elif litellm.LlmProviders.GROQ == provider:
@@ -6857,6 +6873,8 @@ class ProviderConfigManager:
elif litellm.LlmProviders.AZURE == provider:
if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
return litellm.AzureOpenAIO1Config()
if litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
return litellm.AzureOpenAIGPT5Config()
return litellm.AzureOpenAIConfig()
elif litellm.LlmProviders.AZURE_AI == provider:
return litellm.AzureAIStudioConfig()
@@ -6965,6 +6983,8 @@ class ProviderConfigManager:
return litellm.LiteLLMProxyChatConfig()
elif litellm.LlmProviders.OPENAI == provider:
return litellm.OpenAIGPTConfig()
elif litellm.LlmProviders.GRADIENT_AI == provider:
return litellm.GradientAIConfig()
elif litellm.LlmProviders.NSCALE == provider:
return litellm.NscaleConfig()
elif litellm.LlmProviders.OCI == provider:
@@ -7076,7 +7096,11 @@ class ProviderConfigManager:
if litellm.LlmProviders.OPENAI == provider:
return litellm.OpenAIResponsesAPIConfig()
elif litellm.LlmProviders.AZURE == provider:
return litellm.AzureOpenAIResponsesAPIConfig()
# Check if it's an O-series model
if model and ("o_series" in model.lower() or supports_reasoning(model)):
return litellm.AzureOpenAIOSeriesResponsesAPIConfig()
else:
return litellm.AzureOpenAIResponsesAPIConfig()
return None
@staticmethod
model_prices_and_context_window.json
+132 -8
View File
@@ -2457,7 +2457,7 @@
},
"azure/gpt-5-chat": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_input_tokens": 400000,
"max_output_tokens": 128000,
"input_cost_per_token": 1.25e-06,
"output_cost_per_token": 1e-05,
@@ -2490,7 +2490,7 @@
},
"azure/gpt-5-chat-latest": {
"max_tokens": 128000,
"max_input_tokens": 128000,
"max_input_tokens": 400000,
"max_output_tokens": 128000,
"input_cost_per_token": 1.25e-06,
"output_cost_per_token": 1e-05,
@@ -12582,8 +12582,8 @@
},
"openai.gpt-oss-20b-1:0": {
"max_tokens": 128000,
"max_input_tokens": 200000,
"max_output_tokens": 32000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 7e-08,
"output_cost_per_token": 3e-07,
"litellm_provider": "bedrock_converse",
@@ -12596,8 +12596,8 @@
},
"openai.gpt-oss-120b-1:0": {
"max_tokens": 128000,
"max_input_tokens": 200000,
"max_output_tokens": 32000,
"max_input_tokens": 128000,
"max_output_tokens": 128000,
"input_cost_per_token": 1.5e-07,
"output_cost_per_token": 6e-07,
"litellm_provider": "bedrock_converse",
@@ -15605,7 +15605,7 @@
"supports_tool_choice": false
},
"fireworks_ai/accounts/fireworks/models/glm-4p5": {
"max_tokens": 128000,
"max_tokens": 96000,
"max_input_tokens": 128000,
"max_output_tokens": 96000,
"input_cost_per_token": 5.5e-07,
@@ -15618,7 +15618,7 @@
"source": "https://fireworks.ai/models/fireworks/glm-4p5"
},
"fireworks_ai/accounts/fireworks/models/glm-4p5-air": {
"max_tokens": 128000,
"max_tokens": 96000,
"max_input_tokens": 128000,
"max_output_tokens": 96000,
"input_cost_per_token": 2.2e-07,
@@ -17098,6 +17098,130 @@
"litellm_provider": "snowflake",
"mode": "chat"
},
"gradient_ai/anthropic-claude-3.7-sonnet": {
"input_cost_per_token": 3e-06,
"output_cost_per_token": 15e-06,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 1024,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/anthropic-claude-3.5-sonnet": {
"input_cost_per_token": 3e-06,
"output_cost_per_token": 15e-06,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 1024,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/anthropic-claude-3.5-haiku": {
"input_cost_per_token": 8e-07,
"output_cost_per_token": 4e-06,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 1024,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/anthropic-claude-3-opus": {
"input_cost_per_token": 15e-06,
"output_cost_per_token": 75e-06,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 1024,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/deepseek-r1-distill-llama-70b": {
"input_cost_per_token": 99e-08,
"output_cost_per_token": 99e-08,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 8000,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/llama3.3-70b-instruct": {
"input_cost_per_token": 65e-08,
"output_cost_per_token": 65e-08,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 2048,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/llama3-8b-instruct": {
"input_cost_per_token": 2e-07,
"output_cost_per_token": 2e-07,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 512,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/mistral-nemo-instruct-2407": {
"input_cost_per_token": 3e-07,
"output_cost_per_token": 3e-07,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 512,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/openai-o3": {
"input_cost_per_token": 2e-06,
"output_cost_per_token": 8e-06,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 100000,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/openai-o3-mini": {
"input_cost_per_token": 11e-07,
"output_cost_per_token": 44e-07,
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 100000,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/openai-gpt-4o": {
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 16384,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/openai-gpt-4o-mini": {
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 16384,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"gradient_ai/alibaba-qwen3-32b": {
"litellm_provider": "gradient_ai",
"mode": "chat",
"max_tokens": 2048,
"supported_endpoints": ["/v1/chat/completions"],
"supported_modalities": ["text"],
"supports_tool_choice": false
},
"nscale/meta-llama/Llama-4-Scout-17B-16E-Instruct": {
"input_cost_per_token": 9e-08,
"output_cost_per_token": 2.9e-07,
poetry.lock
Generated
+151 -517
View File
File diff suppressed because it is too large Load Diff
pyproject.toml
+2 -2
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "litellm"
version = "1.75.4"
version = "1.75.5"
description = "Library to easily interface with LLM API providers"
authors = ["BerriAI"]
license = "MIT"
@@ -154,7 +154,7 @@ requires = ["poetry-core", "wheel"]
build-backend = "poetry.core.masonry.api"
[tool.commitizen]
version = "1.75.4"
version = "1.75.5"
version_files = [
"pyproject.toml:^version"
]
tests/litellm/llms/gradient_ai/chat/test_gradient_ai_chat_transformation.py
+91
View File
@@ -0,0 +1,91 @@
import os
import sys
import pytest
sys.path.insert(
0, os.path.abspath("../../../../..")
) # Adds the parent directory to the system path
from litellm.llms.gradient_ai.chat.transformation import GradientAIConfig, GRADIENT_AI_SERVERLESS_ENDPOINT
DO_ENDPOINT_PATH = "/api/v1/chat/completions"
DO_BASE_URL = "https://api.gradient_ai.com"
@pytest.fixture
def config():
return GradientAIConfig()
def test_validate_environment_sets_headers(monkeypatch, config):
monkeypatch.setenv("GRADIENT_AI_API_KEY", "test-key")
headers = {}
result = config.validate_environment(
headers=headers,
model="gradient_ai/test-model",
messages=[],
optional_params={},
litellm_params={},
api_key=None,
api_base=None,
)
assert result["Authorization"] == "Bearer test-key"
assert result["Content-Type"] == "application/json"
def test_get_complete_url_custom_base(config):
url = config.get_complete_url(
api_base=DO_BASE_URL,
api_key="test-key",
model="gradient_ai/test-model",
optional_params={},
litellm_params={},
stream=False,
)
assert url == f"{DO_BASE_URL}{DO_ENDPOINT_PATH}"
def test_get_complete_url_default_serverless(monkeypatch, config):
monkeypatch.delenv("GRADIENT_AI_AGENT_ENDPOINT", raising=False)
url = config.get_complete_url(
api_base=None,
api_key="test-key",
model="gradient_ai/test-model",
optional_params={},
litellm_params={},
stream=False,
)
assert url == f"{GRADIENT_AI_SERVERLESS_ENDPOINT}/v1/chat/completions"
def test_get_complete_url_with_env_endpoint(monkeypatch, config):
monkeypatch.setenv("GRADIENT_AI_AGENT_ENDPOINT", DO_BASE_URL)
url = config.get_complete_url(
api_base=None,
api_key="test-key",
model="gradient_ai/test-model",
optional_params={},
litellm_params={},
stream=False,
)
assert url == f"{DO_BASE_URL}{DO_ENDPOINT_PATH}"
def test_transform_messages_handles_dicts_only(config):
messages = [
{"role": "assistant", "content": "Hello!"},
{"role": "user", "content": "Hi!"},
]
out = config._transform_messages(messages, model="gradient_ai/test-model")
assert out[0]["role"] == "assistant"
assert out[0]["content"] == "Hello!"
assert out[1]["role"] == "user"
assert out[1]["content"] == "Hi!"
def test_get_openai_compatible_provider_info_env(monkeypatch, config):
monkeypatch.setenv("GRADIENT_AI_AGENT_ENDPOINT", DO_BASE_URL)
monkeypatch.setenv("GRADIENT_AI_API_KEY", "env-key")
api_base, api_key = config._get_openai_compatible_provider_info(None, None)
assert api_base == DO_BASE_URL
assert api_key == "env-key"
def test_get_openai_compatible_provider_info_default(monkeypatch, config):
monkeypatch.delenv("GRADIENT_AI_AGENT_ENDPOINT", raising=False)
monkeypatch.setenv("GRADIENT_AI_API_KEY", "env-key")
api_base, api_key = config._get_openai_compatible_provider_info(None, None)
assert api_base == GRADIENT_AI_SERVERLESS_ENDPOINT
assert api_key == "env-key"
tests/llm_responses_api_testing/base_responses_api.py
+94
View File
@@ -112,6 +112,10 @@ class BaseResponsesAPITest(ABC):
"""Must return the base completion call args"""
pass
def get_base_completion_reasoning_call_args(self) -> dict:
"""Must return the base completion reasoning call args"""
return None
@pytest.mark.parametrize("sync_mode", [True, False])
@pytest.mark.asyncio
@@ -440,3 +444,93 @@ class BaseResponsesAPITest(ABC):
assert response is not None
assert "output" in response
assert len(response["output"]) > 0
@pytest.mark.asyncio
async def test_responses_api_multi_turn_with_reasoning_and_structured_output(self):
"""
Test multi-turn conversation with reasoning, structured output, and tool calls.
This test validates:
- First call: Model uses reasoning to process a question and makes a tool call
- Tool call handling: Function call output is properly processed
- Second call: Model produces structured output incorporating tool results
- Structured output: Response conforms to defined Pydantic model schema
"""
from pydantic import BaseModel
litellm._turn_on_debug()
litellm.set_verbose = True
base_completion_call_args = self.get_base_completion_reasoning_call_args()
if base_completion_call_args is None:
pytest.skip("Skipping test due to no base completion reasoning call args")
# Define tools for the conversation
tools = [{"type": "function", "name": "get_today"}]
# Define structured output schema
class Output(BaseModel):
today: str
number_of_r: str
# Initial conversation input
input_messages = [
{
"role": "user",
"content": "How many r in strrawberrry? While you're thinking, you should call tool get_today. Then you output the today and number of r",
}
]
# First call - should trigger reasoning and tool call
response = await litellm.aresponses(
input=input_messages,
tools=tools,
reasoning={"effort": "low", "summary": "detailed"},
text_format=Output,
**base_completion_call_args
)
print("First call output:")
print(json.dumps(response.output, indent=4, default=str))
# Validate first response structure
validate_responses_api_response(response, final_chunk=True)
assert response.output is not None
assert len(response.output) > 0
# Extend input with first response output
input_messages.extend(response.output)
# Process any tool calls and add function outputs
function_outputs = []
for item in response.output:
if hasattr(item, 'type') and item.type in ["function_call", "custom_tool_call"]:
if hasattr(item, 'name') and item.name == "get_today":
function_outputs.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": "2025-01-15"
})
# Add function outputs to conversation
input_messages.extend(function_outputs)
print("Second call input:")
print(json.dumps(input_messages, indent=4, default=str))
# Second call - should produce structured output
final_response = await litellm.aresponses(
input=input_messages,
tools=tools,
reasoning={"effort": "low", "summary": "detailed"},
text_format=Output,
**base_completion_call_args
)
print("Second call output:")
print(json.dumps(final_response.output, indent=4, default=str))
# Validate final response structure
validate_responses_api_response(final_response, final_chunk=True)
assert final_response.output is not None
assert len(final_response.output) > 0
tests/llm_responses_api_testing/test_openai_responses_api.py
+89
View File
@@ -30,6 +30,10 @@ class TestOpenAIResponsesAPITest(BaseResponsesAPITest):
return {
"model": "openai/gpt-4o",
}
def get_base_completion_reasoning_call_args(self):
return {
"model": "openai/gpt-5-mini",
}
class TestCustomLogger(CustomLogger):
@@ -1394,4 +1398,89 @@ async def test_aresponses_service_tier_and_safety_identifier():
print("Response:", json.dumps(response, indent=4, default=str))
@pytest.mark.asyncio
async def test_openai_gpt5_reasoning_effort_parameter():
"""Test that reasoning_effort parameter is properly sent in the HTTP request for GPT-5 models."""
# Mock response for GPT-5 responses API (correct format)
mock_response = {
"id": "resp_01ABC123",
"object": "response",
"created_at": 1729621667,
"status": "completed",
"model": "gpt-5-mini",
"output": [
{
"type": "message",
"id": "msg_123",
"status": "completed",
"role": "assistant",
"content": [
{"type": "output_text", "text": "The capital of France is Paris.", "annotations": []}
],
}
],
"parallel_tool_calls": True,
"usage": {
"input_tokens": 15,
"input_tokens_details": {"cached_tokens": 0},
"output_tokens": 8,
"output_tokens_details": {"reasoning_tokens": 0},
"total_tokens": 23,
},
"text": {"format": {"type": "text"}},
"error": None,
"incomplete_details": None,
"instructions": None,
"metadata": {},
"temperature": 1.0,
"tool_choice": "auto",
"tools": [],
"top_p": 1.0,
"max_output_tokens": None,
"previous_response_id": None,
"reasoning": {"effort": "low", "summary": None},
"truncation": "disabled",
"user": None,
}
class MockResponse:
def __init__(self, json_data, status_code):
self._json_data = json_data
self.status_code = status_code
self.text = json.dumps(json_data)
def json(self):
return self._json_data
with patch(
"litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
new_callable=AsyncMock,
) as mock_post:
# Configure the mock to return our response
mock_post.return_value = MockResponse(mock_response, 200)
litellm._turn_on_debug()
litellm.set_verbose = True
# Call aresponses with reasoning_effort parameter
response = await litellm.aresponses(
model="openai/gpt-5-mini",
input="What is the capital of France?",
reasoning={"effort": "minimal"},
)
# Verify the request was made correctly
mock_post.assert_called_once()
request_body = mock_post.call_args.kwargs["json"]
print("request_body=", json.dumps(request_body, indent=4, default=str))
print("reasoning=", request_body["reasoning"])
# Validate that reasoning_effort is present in the request body
assert "reasoning" in request_body, "reasoning should be present in request body"
assert request_body["reasoning"]["effort"] == "minimal", "reasoning_effort should be 'minimal' in request body"
assert request_body["model"] == "gpt-5-mini"
assert request_body["input"] == "What is the capital of France?"
# Validate the response
print("Response:", json.dumps(response, indent=4, default=str))
tests/llm_translation/test_azure_ai.py
+16
View File
@@ -297,3 +297,19 @@ async def test_azure_ai_request_format():
model=model,
messages=messages,
)
@pytest.mark.asyncio
@pytest.mark.parametrize("model", ["azure/gpt5_series/gpt-5", "azure/gpt-5"])
async def test_azure_gpt5_reasoning(model):
litellm._turn_on_debug()
response = await litellm.acompletion(
model="azure/gpt5_series/gpt-5",
messages=[{"role": "user", "content": "What is the capital of France?"}],
reasoning_effort="minimal",
max_tokens=10,
api_base=os.getenv("AZURE_GPT5_API_BASE"),
api_key=os.getenv("AZURE_GPT5_API_KEY"),
)
print("response: ", response)
assert response.choices[0].message.content is not None
tests/llm_translation/test_openai.py
+10
View File
@@ -654,3 +654,13 @@ def test_openai_tool_calling():
}
response = litellm.completion(**completion_params)
@pytest.mark.asyncio
async def test_openai_gpt5_reasoning():
response = await litellm.acompletion(
model="openai/gpt-5-mini",
messages=[{"role": "user", "content": "What is the capital of France?"}],
reasoning_effort="minimal",
)
print("response: ", response)
assert response.choices[0].message.content is not None
tests/local_testing/test_amazing_vertex_completion.py
+1 -5
View File
@@ -167,7 +167,6 @@ async def test_get_response():
pytest.fail(f"An error occurred - {str(e)}")
@pytest.mark.skip(
reason="Local test. Vertex AI Quota is low. Leads to rate limit errors on ci/cd."
)
@@ -547,7 +546,6 @@ def test_completion_function_plus_pdf(load_pdf):
except Exception as e:
pytest.fail("Got={}".format(str(e)))
def encode_image(image_path):
import base64
@@ -765,9 +763,7 @@ def test_gemini_pro_grounding(value_in_dict):
# @pytest.mark.skip(reason="exhausted vertex quota. need to refactor to mock the call")
@pytest.mark.parametrize(
"model", ["vertex_ai_beta/gemini-1.5-pro"]
) # "vertex_ai",
@pytest.mark.parametrize("model", ["vertex_ai_beta/gemini-1.5-pro"]) # "vertex_ai",
@pytest.mark.parametrize("sync_mode", [True]) # "vertex_ai",
@pytest.mark.asyncio
@pytest.mark.flaky(retries=3, delay=1)
tests/local_testing/test_router_utils.py
+26 -14
View File
@@ -240,7 +240,7 @@ async def test_call_router_callbacks_on_success():
)
with patch.object(
router.cache, "async_increment_cache", new=AsyncMock()
router.cache, "async_increment_cache_pipeline", new=AsyncMock()
) as mock_callback:
await router.acompletion(
model="gemini/gemini-1.5-flash",
@@ -248,18 +248,22 @@ async def test_call_router_callbacks_on_success():
mock_response="Hello, I'm good.",
)
await asyncio.sleep(1)
assert mock_callback.call_count == 2
assert mock_callback.call_count == 1
assert (
mock_callback.call_args_list[0]
.kwargs["key"]
.startswith("global_router:1:gemini/gemini-1.5-flash:tpm")
)
assert (
mock_callback.call_args_list[1]
.kwargs["key"]
.startswith("global_router:1:gemini/gemini-1.5-flash:rpm")
)
increment_list = mock_callback.call_args_list[0].kwargs["increment_list"]
assert len(increment_list) == 2
for increment in increment_list:
if "tpm" in increment["key"]:
assert increment["key"].startswith(
"global_router:1:gemini/gemini-1.5-flash:tpm"
)
assert increment["increment_value"] == 30
elif "rpm" in increment["key"]:
assert increment["key"].startswith(
"global_router:1:gemini/gemini-1.5-flash:rpm"
)
assert increment["increment_value"] == 1
@pytest.mark.asyncio
@@ -456,7 +460,15 @@ def test_router_get_deployment_credentials():
def test_router_get_deployment_model_info():
router = Router(
model_list=[{"model_name": "gemini/*", "litellm_params": {"model": "gemini/*"}, "model_info": {"id": "1"}}]
model_list=[
{
"model_name": "gemini/*",
"litellm_params": {"model": "gemini/*"},
"model_info": {"id": "1"},
}
]
)
model_info = router.get_deployment_model_info(
model_id="1", model_name="gemini/gemini-1.5-flash"
)
model_info = router.get_deployment_model_info(model_id="1", model_name="gemini/gemini-1.5-flash")
assert model_info is not None
tests/router_unit_tests/test_router_helper_utils.py
+175 -160
View File
@@ -374,13 +374,25 @@ def test_get_fallback_model_group_from_fallbacks(model_list):
@pytest.mark.parametrize("sync_mode", [True, False])
@pytest.mark.asyncio
async def test_deployment_callback_on_success(model_list, sync_mode):
async def test_deployment_callback_on_success(sync_mode):
"""Test if the '_deployment_callback_on_success' function is working correctly"""
import time
model_list = [
{
"model_name": "gpt-3.5-turbo",
"litellm_params": {
"model": "gpt-3.5-turbo",
"api_key": os.getenv("OPENAI_API_KEY"),
"rpm": 100,
},
"model_info": {"id": "100"},
}
]
router = Router(model_list=model_list)
standard_logging_payload = create_standard_logging_payload()
standard_logging_payload["total_tokens"] = 100
standard_logging_payload["model_id"] = "100"
kwargs = {
"litellm_params": {
"metadata": {
@@ -1181,6 +1193,7 @@ def test_cached_get_model_group_info(model_list):
def test_init_responses_api_endpoints(model_list):
"""Test if the '_init_responses_api_endpoints' function is working correctly"""
from typing import Callable
router = Router(model_list=model_list)
assert router.aget_responses is not None
@@ -1215,29 +1228,29 @@ def test_mock_router_testing_params_str_to_bool_conversion(
):
"""Test if MockRouterTestingParams.from_kwargs correctly converts string values to booleans using str_to_bool"""
from litellm.types.router import MockRouterTestingParams
kwargs = {
"mock_testing_fallbacks": mock_testing_fallbacks,
"mock_testing_context_fallbacks": mock_testing_context_fallbacks,
"mock_testing_content_policy_fallbacks": mock_testing_content_policy_fallbacks,
"other_param": "should_remain", # This should not be affected
}
# Make a copy to verify kwargs are properly popped
original_kwargs = kwargs.copy()
mock_params = MockRouterTestingParams.from_kwargs(kwargs)
# Verify the converted values
assert mock_params.mock_testing_fallbacks == expected_fallbacks
assert mock_params.mock_testing_context_fallbacks == expected_context
assert mock_params.mock_testing_content_policy_fallbacks == expected_content_policy
# Verify that the mock testing params were popped from kwargs
assert "mock_testing_fallbacks" not in kwargs
assert "mock_testing_context_fallbacks" not in kwargs
assert "mock_testing_content_policy_fallbacks" not in kwargs
# Verify other params remain unchanged
assert kwargs["other_param"] == "should_remain"
@@ -1245,50 +1258,49 @@ def test_mock_router_testing_params_str_to_bool_conversion(
def test_is_auto_router_deployment(model_list):
"""Test if the '_is_auto_router_deployment' function correctly identifies auto-router deployments"""
router = Router(model_list=model_list)
# Test case 1: Model starts with "auto_router/" - should return True
litellm_params_auto = LiteLLM_Params(model="auto_router/my-auto-router")
assert router._is_auto_router_deployment(litellm_params_auto) is True
# Test case 2: Model doesn't start with "auto_router/" - should return False
litellm_params_regular = LiteLLM_Params(model="gpt-3.5-turbo")
assert router._is_auto_router_deployment(litellm_params_regular) is False
# Test case 3: Model is empty string - should return False
litellm_params_empty = LiteLLM_Params(model="")
assert router._is_auto_router_deployment(litellm_params_empty) is False
# Test case 4: Model contains "auto_router/" but doesn't start with it - should return False
litellm_params_contains = LiteLLM_Params(model="prefix_auto_router/something")
assert router._is_auto_router_deployment(litellm_params_contains) is False
@patch('litellm.router_strategy.auto_router.auto_router.AutoRouter')
@patch("litellm.router_strategy.auto_router.auto_router.AutoRouter")
def test_init_auto_router_deployment_success(mock_auto_router, model_list):
"""Test if the 'init_auto_router_deployment' function successfully initializes auto-router when all params provided"""
router = Router(model_list=model_list)
# Create a mock AutoRouter instance
mock_auto_router_instance = MagicMock()
mock_auto_router.return_value = mock_auto_router_instance
# Test case: All required parameters provided
litellm_params = LiteLLM_Params(
model="auto_router/test",
auto_router_config_path="/path/to/config",
auto_router_default_model="gpt-3.5-turbo",
auto_router_embedding_model="text-embedding-ada-002"
auto_router_embedding_model="text-embedding-ada-002",
)
deployment = Deployment(
model_name="test-auto-router",
model_name="test-auto-router",
litellm_params=litellm_params,
model_info={"id": "test-id"}
model_info={"id": "test-id"},
)
# Should not raise any exception
router.init_auto_router_deployment(deployment)
# Verify AutoRouter was called with correct parameters
mock_auto_router.assert_called_once_with(
model_name="test-auto-router",
@@ -1298,86 +1310,96 @@ def test_init_auto_router_deployment_success(mock_auto_router, model_list):
embedding_model="text-embedding-ada-002",
litellm_router_instance=router,
)
# Verify the auto-router was added to the router's auto_routers dict
assert "test-auto-router" in router.auto_routers
assert router.auto_routers["test-auto-router"] == mock_auto_router_instance
@patch('litellm.router_strategy.auto_router.auto_router.AutoRouter')
@patch("litellm.router_strategy.auto_router.auto_router.AutoRouter")
def test_init_auto_router_deployment_duplicate_model_name(mock_auto_router, model_list):
"""Test if the 'init_auto_router_deployment' function raises ValueError when model_name already exists"""
router = Router(model_list=model_list)
# Create a mock AutoRouter instance
mock_auto_router_instance = MagicMock()
mock_auto_router.return_value = mock_auto_router_instance
# Add an existing auto-router
router.auto_routers["test-auto-router"] = mock_auto_router_instance
# Try to add another auto-router with the same name
litellm_params = LiteLLM_Params(
model="auto_router/test",
auto_router_config_path="/path/to/config",
auto_router_default_model="gpt-3.5-turbo",
auto_router_embedding_model="text-embedding-ada-002"
auto_router_embedding_model="text-embedding-ada-002",
)
deployment = Deployment(
model_name="test-auto-router",
model_name="test-auto-router",
litellm_params=litellm_params,
model_info={"id": "test-id"}
model_info={"id": "test-id"},
)
with pytest.raises(ValueError, match="Auto-router deployment test-auto-router already exists"):
with pytest.raises(
ValueError, match="Auto-router deployment test-auto-router already exists"
):
router.init_auto_router_deployment(deployment)
def test_generate_model_id_with_deployment_model_name(model_list):
"""Test that _generate_model_id works correctly with deployment model_name and handles None values properly"""
router = Router(model_list=model_list)
# Test case 1: Normal case with valid model_group and litellm_params
model_group = "gpt-4.1"
litellm_params = {
"model": "gpt-4.1",
"api_key": "test_key",
"api_base": "https://api.openai.com/v1"
"api_base": "https://api.openai.com/v1",
}
try:
result = router._generate_model_id(model_group=model_group, litellm_params=litellm_params)
result = router._generate_model_id(
model_group=model_group, litellm_params=litellm_params
)
assert isinstance(result, str)
assert len(result) > 0
print(f"✓ Success with valid model_group: {result}")
except Exception as e:
pytest.fail(f"Failed with valid model_group: {e}")
# Test case 2: Edge case with None model_group (this should fail as expected - our fix prevents this from happening)
try:
result = router._generate_model_id(model_group=None, litellm_params=litellm_params)
pytest.fail("Expected TypeError when model_group is None - this confirms our fix is needed")
result = router._generate_model_id(
model_group=None, litellm_params=litellm_params
)
pytest.fail(
"Expected TypeError when model_group is None - this confirms our fix is needed"
)
except TypeError as e:
assert "unsupported operand type(s) for +=" in str(e)
print(f"✓ Correctly failed with None model_group (as expected): {e}")
except Exception as e:
pytest.fail(f"Unexpected error with None model_group: {e}")
# Test case 3: Edge case with None key in litellm_params
litellm_params_with_none_key = {
"model": "gpt-4.1",
"api_key": "test_key",
None: "should_be_skipped" # This should be handled gracefully
None: "should_be_skipped", # This should be handled gracefully
}
try:
result = router._generate_model_id(model_group=model_group, litellm_params=litellm_params_with_none_key)
result = router._generate_model_id(
model_group=model_group, litellm_params=litellm_params_with_none_key
)
assert isinstance(result, str)
assert len(result) > 0
print(f"✓ Success with None key in litellm_params: {result}")
except Exception as e:
pytest.fail(f"Failed with None key in litellm_params: {e}")
# Test case 4: Edge case with empty litellm_params
try:
result = router._generate_model_id(model_group=model_group, litellm_params={})
@@ -1386,216 +1408,213 @@ def test_generate_model_id_with_deployment_model_name(model_list):
print(f"✓ Success with empty litellm_params: {result}")
except Exception as e:
pytest.fail(f"Failed with empty litellm_params: {e}")
# Test case 5: Verify that the same inputs produce the same result (deterministic)
result1 = router._generate_model_id(model_group=model_group, litellm_params=litellm_params)
result2 = router._generate_model_id(model_group=model_group, litellm_params=litellm_params)
result1 = router._generate_model_id(
model_group=model_group, litellm_params=litellm_params
)
result2 = router._generate_model_id(
model_group=model_group, litellm_params=litellm_params
)
assert result1 == result2, "Model ID generation should be deterministic"
print("✓ All _generate_model_id tests passed!")
def test_handle_clientside_credential_with_deployment_model_name(model_list):
"""Test that _handle_clientside_credential uses deployment model_name correctly"""
router = Router(model_list=model_list)
# Mock deployment with model_name
deployment = {
"model_name": "gpt-4.1",
"litellm_params": {
"model": "gpt-4.1",
"api_key": "test_key"
}
"litellm_params": {"model": "gpt-4.1", "api_key": "test_key"},
}
# Mock kwargs with empty metadata (simulating the original issue)
kwargs = {
"metadata": {}, # Empty metadata, no model_group
"litellm_params": {
"api_key": "client_side_key",
"api_base": "https://api.openai.com/v1"
}
"api_base": "https://api.openai.com/v1",
},
}
# Mock dynamic_litellm_params that would be returned by get_dynamic_litellm_params
dynamic_litellm_params = {
"api_key": "client_side_key",
"api_base": "https://api.openai.com/v1"
"api_base": "https://api.openai.com/v1",
}
# Test that the method doesn't fail when metadata is empty
try:
# This would normally call _generate_model_id internally
# We're testing that the fix prevents the TypeError
model_group = deployment["model_name"] # This is what our fix does
assert model_group == "gpt-4.1"
# Verify that _generate_model_id works with this model_group
result = router._generate_model_id(model_group=model_group, litellm_params=dynamic_litellm_params)
result = router._generate_model_id(
model_group=model_group, litellm_params=dynamic_litellm_params
)
assert isinstance(result, str)
assert len(result) > 0
print(f"✓ Success with deployment model_name: {result}")
except Exception as e:
pytest.fail(f"Failed with deployment model_name: {e}")
print("✓ _handle_clientside_credential test passed!")
@pytest.mark.parametrize("function_name, expected_metadata_key", [
("acompletion", "metadata"),
("_ageneric_api_call_with_fallbacks", "litellm_metadata"),
("batch", "litellm_metadata"),
("completion", "metadata"),
("acreate_file", "litellm_metadata"),
("aget_file", "litellm_metadata"),
])
def test_handle_clientside_credential_metadata_loading(model_list, function_name, expected_metadata_key):
@pytest.mark.parametrize(
"function_name, expected_metadata_key",
[
("acompletion", "metadata"),
("_ageneric_api_call_with_fallbacks", "litellm_metadata"),
("batch", "litellm_metadata"),
("completion", "metadata"),
("acreate_file", "litellm_metadata"),
("aget_file", "litellm_metadata"),
],
)
def test_handle_clientside_credential_metadata_loading(
model_list, function_name, expected_metadata_key
):
"""Test that _handle_clientside_credential correctly loads metadata based on function name"""
router = Router(model_list=model_list)
# Mock deployment
deployment = {
"model_name": "gpt-4.1",
"litellm_params": {
"model": "gpt-4.1",
"api_key": "test_key"
},
"model_info": {
"id": "original-id-123"
}
"litellm_params": {"model": "gpt-4.1", "api_key": "test_key"},
"model_info": {"id": "original-id-123"},
}
# Mock kwargs with clientside credentials and metadata
kwargs = {
"api_key": "client_side_key",
"api_base": "https://api.openai.com/v1",
expected_metadata_key: {
"model_group": "gpt-4.1",
"custom_field": "test_value"
}
expected_metadata_key: {"model_group": "gpt-4.1", "custom_field": "test_value"},
}
# Call the function
result_deployment = router._handle_clientside_credential(
deployment=deployment,
kwargs=kwargs,
function_name=function_name
deployment=deployment, kwargs=kwargs, function_name=function_name
)
# Verify the result is a Deployment object
assert isinstance(result_deployment, Deployment)
# Verify the deployment has the correct model_name (should be the model_group from metadata)
assert result_deployment.model_name == "gpt-4.1"
# Verify the litellm_params contain the clientside credentials
assert result_deployment.litellm_params.api_key == "client_side_key"
assert result_deployment.litellm_params.api_base == "https://api.openai.com/v1"
# Verify the model_info has been updated with a new ID
assert result_deployment.model_info.id != "original-id-123"
assert result_deployment.model_info.original_model_id == "original-id-123"
# Verify the deployment was added to the router
assert len(router.model_list) == len(model_list) + 1
# Test that the function correctly uses the right metadata key
# For acompletion, it should use "metadata"
# For _ageneric_api_call_with_fallbacks/batch, it should use "litellm_metadata"
if function_name == "acompletion":
assert "metadata" in kwargs
assert "litellm_metadata" not in kwargs
elif function_name in ["_ageneric_api_call_with_fallbacks", "batch", "acreate_file", "aget_file"]:
elif function_name in [
"_ageneric_api_call_with_fallbacks",
"batch",
"acreate_file",
"aget_file",
]:
assert "litellm_metadata" in kwargs
# Note: acompletion would not have litellm_metadata, but other functions might have both
print(f"✓ Success with function_name '{function_name}' using '{expected_metadata_key}' metadata key")
print(
f"✓ Success with function_name '{function_name}' using '{expected_metadata_key}' metadata key"
)
@pytest.mark.parametrize("function_name, metadata_key", [
("acompletion", "metadata"),
("_ageneric_api_call_with_fallbacks", "litellm_metadata"),
])
def test_handle_clientside_credential_metadata_variable_name(model_list, function_name, metadata_key):
@pytest.mark.parametrize(
"function_name, metadata_key",
[
("acompletion", "metadata"),
("_ageneric_api_call_with_fallbacks", "litellm_metadata"),
],
)
def test_handle_clientside_credential_metadata_variable_name(
model_list, function_name, metadata_key
):
"""Test that _handle_clientside_credential uses the correct metadata variable name based on function name"""
from litellm.router_utils.batch_utils import _get_router_metadata_variable_name
router = Router(model_list=model_list)
# Verify the metadata variable name is correct for each function
expected_metadata_key = _get_router_metadata_variable_name(function_name=function_name)
expected_metadata_key = _get_router_metadata_variable_name(
function_name=function_name
)
assert expected_metadata_key == metadata_key
# Mock deployment
deployment = {
"model_name": "gpt-4.1",
"litellm_params": {
"model": "gpt-4.1",
"api_key": "test_key"
},
"model_info": {
"id": "original-id-456"
}
"litellm_params": {"model": "gpt-4.1", "api_key": "test_key"},
"model_info": {"id": "original-id-456"},
}
# Mock kwargs with clientside credentials and the correct metadata key
kwargs = {
"api_key": "client_side_key",
"api_base": "https://api.openai.com/v1",
metadata_key: {
"model_group": "gpt-4.1",
"test_field": "test_value"
}
metadata_key: {"model_group": "gpt-4.1", "test_field": "test_value"},
}
# Call the function
result_deployment = router._handle_clientside_credential(
deployment=deployment,
kwargs=kwargs,
function_name=function_name
deployment=deployment, kwargs=kwargs, function_name=function_name
)
# Verify the function correctly extracted model_group from the right metadata key
assert result_deployment.model_name == "gpt-4.1"
# Verify the deployment was created with the correct metadata
assert result_deployment.litellm_params.api_key == "client_side_key"
assert result_deployment.litellm_params.api_base == "https://api.openai.com/v1"
print(f"✓ Success with function_name '{function_name}' correctly using '{metadata_key}' for metadata")
print(
f"✓ Success with function_name '{function_name}' correctly using '{metadata_key}' for metadata"
)
def test_handle_clientside_credential_no_metadata(model_list):
"""Test that _handle_clientside_credential handles cases where no metadata is provided"""
router = Router(model_list=model_list)
# Mock deployment
deployment = {
"model_name": "gpt-4.1",
"litellm_params": {
"model": "gpt-4.1",
"api_key": "test_key"
},
"model_info": {
"id": "original-id-789"
}
"litellm_params": {"model": "gpt-4.1", "api_key": "test_key"},
"model_info": {"id": "original-id-789"},
}
# Mock kwargs with clientside credentials but NO metadata
kwargs = {
"api_key": "client_side_key",
"api_base": "https://api.openai.com/v1"
"api_base": "https://api.openai.com/v1",
# No metadata key at all
}
# This should fail because there's no model_group in metadata
# The function expects to find model_group in the metadata
try:
result_deployment = router._handle_clientside_credential(
deployment=deployment,
kwargs=kwargs,
function_name="acompletion"
deployment=deployment, kwargs=kwargs, function_name="acompletion"
)
# If we get here, the function should have used deployment.model_name as fallback
assert result_deployment.model_name == "gpt-4.1"
@@ -1603,19 +1622,19 @@ def test_handle_clientside_credential_no_metadata(model_list):
except Exception as e:
# This is expected behavior - the function needs model_group to generate model_id
print(f"✓ Correctly handled no metadata case: {e}")
# Test with empty metadata
kwargs_with_empty_metadata = {
"api_key": "client_side_key",
"api_base": "https://api.openai.com/v1",
"metadata": {} # Empty metadata
"metadata": {}, # Empty metadata
}
try:
result_deployment = router._handle_clientside_credential(
deployment=deployment,
kwargs=kwargs_with_empty_metadata,
function_name="acompletion"
function_name="acompletion",
)
# Should fail because empty metadata has no model_group
pytest.fail("Expected failure with empty metadata")
@@ -1626,36 +1645,31 @@ def test_handle_clientside_credential_no_metadata(model_list):
def test_handle_clientside_credential_with_responses_function(model_list):
"""Test that _handle_clientside_credential works correctly with responses function name"""
router = Router(model_list=model_list)
# Mock deployment
deployment = {
"model_name": "gpt-4.1",
"litellm_params": {
"model": "gpt-4.1",
"api_key": "test_key"
},
"model_info": {
"id": "original-id-responses"
}
"litellm_params": {"model": "gpt-4.1", "api_key": "test_key"},
"model_info": {"id": "original-id-responses"},
}
# Mock kwargs with clientside credentials and litellm_metadata (for responses function)
kwargs = {
"api_key": "client_side_key",
"api_base": "https://api.openai.com/v1",
"litellm_metadata": {
"model_group": "gpt-4.1",
"responses_field": "responses_value"
}
"responses_field": "responses_value",
},
}
# Call the function with _ageneric_api_call_with_fallbacks function name (which handles responses)
result_deployment = router._handle_clientside_credential(
deployment=deployment,
kwargs=kwargs,
function_name="_ageneric_api_call_with_fallbacks"
function_name="_ageneric_api_call_with_fallbacks",
)
# Verify the result
assert isinstance(result_deployment, Deployment)
assert result_deployment.model_name == "gpt-4.1"
@@ -1663,9 +1677,10 @@ def test_handle_clientside_credential_with_responses_function(model_list):
assert result_deployment.litellm_params.api_base == "https://api.openai.com/v1"
assert result_deployment.model_info.id != "original-id-responses"
assert result_deployment.model_info.original_model_id == "original-id-responses"
# Verify the deployment was added to the router
assert len(router.model_list) == len(model_list) + 1
print("✓ Success with _ageneric_api_call_with_fallbacks function name and litellm_metadata")
print(
"✓ Success with _ageneric_api_call_with_fallbacks function name and litellm_metadata"
)
tests/test_litellm/integrations/test_langfuse_otel.py
+46 -2
View File
@@ -1,6 +1,6 @@
import json
import os
from unittest.mock import MagicMock, patch
import json
import pytest
@@ -108,7 +108,8 @@ class TestLangfuseOtelIntegration:
def test_extract_langfuse_metadata_with_header_enrichment(self, monkeypatch):
"""_extract_langfuse_metadata should call LangFuseLogger.add_metadata_from_header when available."""
import sys, types
import sys
import types
# Build a stub module + class on-the-fly
stub_module = types.ModuleType("litellm.integrations.langfuse.langfuse")
@@ -186,6 +187,49 @@ class TestLangfuseOtelIntegration:
assert actual == expected, "Mismatch between expected and actual OTEL attribute mapping."
def test_construct_dynamic_otel_headers_with_langfuse_keys(self):
"""Test that construct_dynamic_otel_headers creates proper auth headers when langfuse keys are provided."""
from litellm.types.utils import StandardCallbackDynamicParams
# Create dynamic params with langfuse keys
dynamic_params = StandardCallbackDynamicParams(
langfuse_public_key="test_public_key",
langfuse_secret_key="test_secret_key"
)
logger = LangfuseOtelLogger()
result = logger.construct_dynamic_otel_headers(dynamic_params)
# Should return a dict with otlp_auth_headers
assert result is not None
assert "Authorization" in result
# The auth header should contain the basic auth format
auth_header = result["Authorization"]
assert auth_header.startswith("Basic ")
# Verify the header format by decoding
import base64
# Extract the base64 part from "Authorization=Basic <base64>"
base64_part = auth_header.replace("Basic ", "")
decoded = base64.b64decode(base64_part).decode()
assert decoded == "test_public_key:test_secret_key"
def test_construct_dynamic_otel_headers_empty_params(self):
"""Test that construct_dynamic_otel_headers returns empty dict when no langfuse keys are provided."""
from litellm.types.utils import StandardCallbackDynamicParams
# Create dynamic params without langfuse keys
dynamic_params = StandardCallbackDynamicParams()
logger = LangfuseOtelLogger()
result = logger.construct_dynamic_otel_headers(dynamic_params)
# Should return an empty dict
assert result == {}
if __name__ == "__main__":
pytest.main([__file__])
tests/test_litellm/llms/azure/chat/test_azure_gpt5_transformation.py
+48
View File
@@ -0,0 +1,48 @@
import pytest
import litellm
from litellm.llms.azure.chat.gpt_5_transformation import AzureOpenAIGPT5Config
@pytest.fixture()
def config() -> AzureOpenAIGPT5Config:
return AzureOpenAIGPT5Config()
def test_azure_gpt5_supports_reasoning_effort(config: AzureOpenAIGPT5Config):
assert "reasoning_effort" in config.get_supported_openai_params(model="gpt-5")
assert "reasoning_effort" in config.get_supported_openai_params(model="gpt5_series/my-deployment")
def test_azure_gpt5_maps_max_tokens(config: AzureOpenAIGPT5Config):
params = config.map_openai_params(
non_default_params={"max_tokens": 5},
optional_params={},
model="gpt5_series/gpt-5",
drop_params=False,
api_version="2024-05-01-preview",
)
assert params["max_completion_tokens"] == 5
assert "max_tokens" not in params
def test_azure_gpt5_temperature_error(config: AzureOpenAIGPT5Config):
with pytest.raises(litellm.utils.UnsupportedParamsError):
config.map_openai_params(
non_default_params={"temperature": 0.2},
optional_params={},
model="gpt-5",
drop_params=False,
api_version="2024-05-01-preview",
)
def test_azure_gpt5_series_transform_request(config: AzureOpenAIGPT5Config):
request = config.transform_request(
model="gpt5_series/gpt-5",
messages=[],
optional_params={},
litellm_params={},
headers={},
)
assert request["model"] == "gpt-5"
tests/test_litellm/llms/azure/response/test_azure_transformation.py
+161 -34
View File
@@ -9,7 +9,9 @@ sys.path.insert(
) # Adds the parent directory to the system path
from litellm.llms.azure.responses.transformation import AzureOpenAIResponsesAPIConfig
from litellm.llms.azure.responses.o_series_transformation import AzureOpenAIOSeriesResponsesAPIConfig
from litellm.types.router import GenericLiteLLMParams
from litellm.types.llms.openai import ResponsesAPIOptionalRequestParams
@pytest.mark.serial
@@ -54,47 +56,172 @@ def test_validate_environment_azure_key_within_litellm():
assert result == expected
@pytest.mark.serial
def test_validate_environment_azure_openai_api_key_within_secret_str():
def test_validate_environment_azure_key_within_headers():
azure_openai_responses_apiconfig = AzureOpenAIResponsesAPIConfig()
headers = {"api-key": "test-api-key-from-headers"}
litellm_params = GenericLiteLLMParams()
with patch("litellm.api_key", None), \
patch("litellm.azure_key", None), \
patch("litellm.llms.azure.common_utils.get_secret_str") as mock_get_secret_str:
# Configure the mock to return "test-api-key" when called with "AZURE_OPENAI_API_KEY"
mock_get_secret_str.side_effect = (
lambda key: "test-api-key" if key == "AZURE_OPENAI_API_KEY" else None
)
result = azure_openai_responses_apiconfig.validate_environment(
headers=headers, model="", litellm_params=litellm_params
)
litellm_params = GenericLiteLLMParams()
result = azure_openai_responses_apiconfig.validate_environment(
headers={}, model="", litellm_params=litellm_params
)
expected = {"api-key": "test-api-key"}
expected = {"api-key": "test-api-key-from-headers"}
assert result == expected
assert result == expected
@pytest.mark.serial
def test_validate_environment_azure_api_key_within_secret_str():
def test_get_complete_url():
"""
Test the get_complete_url function
"""
azure_openai_responses_apiconfig = AzureOpenAIResponsesAPIConfig()
api_base = "https://litellm8397336933.openai.azure.com"
litellm_params = {"api_version": "2024-05-01-preview"}
with patch("litellm.api_key", None), \
patch("litellm.azure_key", None), \
patch("litellm.llms.azure.common_utils.get_secret_str") as mock_get_secret_str:
# Configure the mock to return None for "AZURE_OPENAI_API_KEY" and "test-api-key" for "AZURE_API_KEY"
def mock_side_effect(key):
if key == "AZURE_OPENAI_API_KEY":
return None
elif key == "AZURE_API_KEY":
return "test-api-key"
else:
return None
mock_get_secret_str.side_effect = mock_side_effect
result = azure_openai_responses_apiconfig.get_complete_url(
api_base=api_base, litellm_params=litellm_params
)
litellm_params = GenericLiteLLMParams()
result = azure_openai_responses_apiconfig.validate_environment(
headers={}, model="", litellm_params=litellm_params
)
expected = {"api-key": "test-api-key"}
expected = "https://litellm8397336933.openai.azure.com/openai/responses?api-version=2024-05-01-preview"
assert result == expected
assert result == expected
@pytest.mark.serial
def test_azure_o_series_responses_api_supported_params():
"""Test that Azure OpenAI O-series responses API excludes temperature from supported parameters."""
config = AzureOpenAIOSeriesResponsesAPIConfig()
supported_params = config.get_supported_openai_params("o_series/gpt-o1")
# Temperature should not be in supported params for O-series models
assert "temperature" not in supported_params
# Other parameters should still be supported
assert "input" in supported_params
assert "max_output_tokens" in supported_params
assert "stream" in supported_params
assert "top_p" in supported_params
@pytest.mark.serial
def test_azure_o_series_responses_api_drop_temperature_param():
"""Test that temperature parameter is dropped when drop_params is True for O-series models."""
config = AzureOpenAIOSeriesResponsesAPIConfig()
# Create request params with temperature
request_params = ResponsesAPIOptionalRequestParams(
temperature=0.7,
max_output_tokens=1000,
stream=False,
top_p=0.9
)
# Test with drop_params=True
mapped_params_with_drop = config.map_openai_params(
response_api_optional_params=request_params,
model="o_series/gpt-o1",
drop_params=True
)
# Temperature should be dropped
assert "temperature" not in mapped_params_with_drop
# Other params should remain
assert mapped_params_with_drop["max_output_tokens"] == 1000
assert mapped_params_with_drop["top_p"] == 0.9
# Test with drop_params=False
mapped_params_without_drop = config.map_openai_params(
response_api_optional_params=request_params,
model="o_series/gpt-o1",
drop_params=False
)
# Temperature should still be present when drop_params=False
assert mapped_params_without_drop["temperature"] == 0.7
assert mapped_params_without_drop["max_output_tokens"] == 1000
assert mapped_params_without_drop["top_p"] == 0.9
@pytest.mark.serial
def test_azure_o_series_responses_api_drop_params_no_temperature():
"""Test that map_openai_params works correctly when temperature is not present for O-series models."""
config = AzureOpenAIOSeriesResponsesAPIConfig()
# Create request params without temperature
request_params = ResponsesAPIOptionalRequestParams(
max_output_tokens=1000,
stream=False,
top_p=0.9
)
# Should work fine even with drop_params=True
mapped_params = config.map_openai_params(
response_api_optional_params=request_params,
model="o_series/gpt-o1",
drop_params=True
)
assert "temperature" not in mapped_params
assert mapped_params["max_output_tokens"] == 1000
assert mapped_params["top_p"] == 0.9
@pytest.mark.serial
def test_azure_regular_responses_api_supports_temperature():
"""Test that regular Azure OpenAI responses API (non-O-series) supports temperature parameter."""
config = AzureOpenAIResponsesAPIConfig()
supported_params = config.get_supported_openai_params("gpt-4o")
# Regular Azure models should support temperature
assert "temperature" in supported_params
# Other parameters should still be supported
assert "input" in supported_params
assert "max_output_tokens" in supported_params
assert "stream" in supported_params
assert "top_p" in supported_params
@pytest.mark.serial
def test_o_series_model_detection():
"""Test that the O-series configuration correctly identifies O-series models."""
config = AzureOpenAIOSeriesResponsesAPIConfig()
# Test explicit o_series naming
assert config.is_o_series_model("o_series/gpt-o1") == True
assert config.is_o_series_model("azure/o_series/gpt-o3") == True
# Test regular models
assert config.is_o_series_model("gpt-4o") == False
assert config.is_o_series_model("gpt-3.5-turbo") == False
@pytest.mark.serial
def test_provider_config_manager_o_series_selection():
"""Test that ProviderConfigManager returns the correct config for O-series vs regular models."""
from litellm.utils import ProviderConfigManager
import litellm
# Test O-series model selection
o_series_config = ProviderConfigManager.get_provider_responses_api_config(
provider=litellm.LlmProviders.AZURE,
model="o_series/gpt-o1"
)
assert isinstance(o_series_config, AzureOpenAIOSeriesResponsesAPIConfig)
# Test regular model selection
regular_config = ProviderConfigManager.get_provider_responses_api_config(
provider=litellm.LlmProviders.AZURE,
model="gpt-4o"
)
assert isinstance(regular_config, AzureOpenAIResponsesAPIConfig)
assert not isinstance(regular_config, AzureOpenAIOSeriesResponsesAPIConfig)
# Test with no model specified (should default to regular)
default_config = ProviderConfigManager.get_provider_responses_api_config(
provider=litellm.LlmProviders.AZURE,
model=None
)
assert isinstance(default_config, AzureOpenAIResponsesAPIConfig)
assert not isinstance(default_config, AzureOpenAIOSeriesResponsesAPIConfig)
tests/test_litellm/proxy/auth/test_handle_jwt.py
+60 -8
View File
@@ -459,9 +459,9 @@ async def test_nested_jwt_field_access():
2. Backward compatibility is maintained for flat field names
3. Missing nested paths return appropriate defaults
"""
from litellm.proxy.auth.handle_jwt import JWTHandler
from litellm.proxy._types import LiteLLM_JWTAuth
from litellm.proxy.auth.handle_jwt import JWTHandler
# Create JWT handler
jwt_handler = JWTHandler()
@@ -536,7 +536,7 @@ async def test_nested_jwt_field_access():
assert jwt_handler.get_org_id(flat_token, None) == "org456"
# Test 5: object_id_jwt_field with nested access (requires role_mappings)
from litellm.proxy._types import RoleMapping, LitellmUserRoles
from litellm.proxy._types import LitellmUserRoles, RoleMapping
jwt_handler.litellm_jwtauth = LiteLLM_JWTAuth(
object_id_jwt_field="profile.object_id",
role_mappings=[RoleMapping(role="admin", internal_role=LitellmUserRoles.INTERNAL_USER)]
@@ -588,9 +588,9 @@ async def test_nested_jwt_field_missing_paths():
2. Partial paths that exist but don't have the final key return defaults
3. team_id_default fallback works with nested fields
"""
from litellm.proxy.auth.handle_jwt import JWTHandler
from litellm.proxy._types import LiteLLM_JWTAuth
from litellm.proxy.auth.handle_jwt import JWTHandler
# Create JWT handler
jwt_handler = JWTHandler()
@@ -626,7 +626,7 @@ async def test_nested_jwt_field_missing_paths():
assert jwt_handler.get_org_id(incomplete_token, "default_org") == "default_org"
# Test 5: Missing profile.object_id should return default (requires role_mappings)
from litellm.proxy._types import RoleMapping, LitellmUserRoles
from litellm.proxy._types import LitellmUserRoles, RoleMapping
jwt_handler.litellm_jwtauth = LiteLLM_JWTAuth(
object_id_jwt_field="profile.object_id",
role_mappings=[RoleMapping(role="admin", internal_role=LitellmUserRoles.INTERNAL_USER)]
@@ -663,9 +663,9 @@ async def test_metadata_prefix_handling_in_nested_fields():
The get_nested_value function should remove metadata. prefix before traversing
"""
from litellm.proxy.auth.handle_jwt import JWTHandler
from litellm.proxy._types import LiteLLM_JWTAuth
from litellm.proxy.auth.handle_jwt import JWTHandler
# Create JWT handler
jwt_handler = JWTHandler()
@@ -685,3 +685,55 @@ async def test_metadata_prefix_handling_in_nested_fields():
# Test 2: user.sub should work normally without metadata prefix
jwt_handler.litellm_jwtauth = LiteLLM_JWTAuth(user_id_jwt_field="sub")
assert jwt_handler.get_user_id(token, None) == "u123"
@pytest.mark.asyncio
async def test_find_team_with_model_access_model_group(monkeypatch):
from litellm.caching import DualCache
from litellm.proxy.utils import ProxyLogging
from litellm.router import Router
router = Router(
model_list=[
{
"model_name": "gpt-4o-mini",
"litellm_params": {"model": "gpt-4o-mini"},
"model_info": {"access_groups": ["test-group"]},
}
]
)
import sys
import types
proxy_server_module = types.ModuleType("proxy_server")
proxy_server_module.llm_router = router
monkeypatch.setitem(sys.modules, "litellm.proxy.proxy_server", proxy_server_module)
team = LiteLLM_TeamTable(team_id="team-1", models=["test-group"])
async def mock_get_team_object(*args, **kwargs): # type: ignore
return team
monkeypatch.setattr(
"litellm.proxy.auth.handle_jwt.get_team_object", mock_get_team_object
)
jwt_handler = JWTHandler()
jwt_handler.litellm_jwtauth = LiteLLM_JWTAuth()
user_api_key_cache = DualCache()
proxy_logging_obj = ProxyLogging(user_api_key_cache=user_api_key_cache)
team_id, team_obj = await JWTAuthManager.find_team_with_model_access(
team_ids={"team-1"},
requested_model="gpt-4o-mini",
route="/chat/completions",
jwt_handler=jwt_handler,
prisma_client=None,
user_api_key_cache=user_api_key_cache,
parent_otel_span=None,
proxy_logging_obj=proxy_logging_obj,
)
assert team_id == "team-1"
assert team_obj.team_id == "team-1"
tests/test_litellm/proxy/common_utils/test_get_routes.py
+54
View File
@@ -166,3 +166,57 @@ class TestGetRoutes:
assert mount_route["endpoint"] == "handle_streamable_http_mcp"
assert mount_route["mounted_app"] is True
def test_get_routes_for_mounted_app_with_static_files(self):
"""
Test getting routes for mounted app with StaticFiles object (reproduces AttributeError bug).
This test reproduces the exact stacktrace scenario:
AttributeError: 'StaticFiles' object has no attribute '__name__'. Did you mean: '__ne__'?
The original bug occurred when the code tried to access endpoint_func.__name__
directly on a StaticFiles object. The fix uses _safe_get_endpoint_name() which
gracefully handles objects without __name__ by falling back to class name.
"""
# Mock the main mount route (e.g., /ui)
mock_mount_route = Mock()
mock_mount_route.path = "/ui"
# Mock sub-app with routes
mock_sub_app = Mock()
mock_sub_app.routes = []
# Create a mock StaticFiles route (this is the problematic case)
mock_static_route = Mock(spec=['path', 'name', 'endpoint', 'app'])
mock_static_route.path = ""
mock_static_route.name = "ui"
mock_static_route.endpoint = None
# Mock StaticFiles object - this is the key part that caused the AttributeError
# Real StaticFiles objects don't have __name__ attribute
# Create a mock that simulates StaticFiles behavior (no __name__ attribute)
class StaticFiles:
"""Mock class that simulates real StaticFiles without __name__ attribute"""
pass
mock_static_files = StaticFiles()
# Verify no __name__ attribute exists on the instance (reproduces bug condition)
assert not hasattr(mock_static_files, '__name__')
mock_static_route.app = mock_static_files
mock_sub_app.routes.append(mock_static_route)
mock_mount_route.app = mock_sub_app
# This should NOT raise AttributeError thanks to _safe_get_endpoint_name
# In the old code, this would fail with: 'StaticFiles' object has no attribute '__name__'
result = GetRoutes.get_routes_for_mounted_app(mock_mount_route)
# Should handle StaticFiles gracefully without throwing AttributeError
assert len(result) == 1
assert result[0]["path"] == "/ui"
assert result[0]["methods"] == ["GET", "POST"] # Default methods
assert result[0]["name"] == "ui"
# Should fall back to class name since instance doesn't have __name__ attribute
assert result[0]["endpoint"] == "StaticFiles" # Falls back to class name
assert result[0]["mounted_app"] is True
tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.py
+207
View File
@@ -1,6 +1,7 @@
"""
Unit Tests for the max parallel request limiter v3 for the proxy
"""
import asyncio
import os
import sys
@@ -515,3 +516,209 @@ async def test_async_log_failure_event_v3():
assert op["key"] == f"{{api_key:{_api_key}}}:max_parallel_requests"
assert op["increment_value"] == -1
assert op["ttl"] == 60 # default window size
@pytest.mark.asyncio
async def test_should_rate_limit_only_called_when_limits_exist_v3():
"""
Test that should_rate_limit is only called when actual rate limits are configured.
This verifies the optimization that avoids unnecessary rate limit checks.
"""
_api_key = "sk-12345"
_api_key = hash_token(_api_key)
local_cache = DualCache()
parallel_request_handler = _PROXY_MaxParallelRequestsHandler(
internal_usage_cache=InternalUsageCache(local_cache)
)
# Mock should_rate_limit to track if it's called
should_rate_limit_called = False
async def mock_should_rate_limit(*args, **kwargs):
nonlocal should_rate_limit_called
should_rate_limit_called = True
return {"overall_code": "OK", "statuses": []}
parallel_request_handler.should_rate_limit = mock_should_rate_limit
# Test 1: No rate limits configured - should_rate_limit should NOT be called
should_rate_limit_called = False
user_api_key_dict_no_limits = UserAPIKeyAuth(
api_key=_api_key,
user_id="test_user",
team_id="test_team",
end_user_id="test_end_user",
# No rpm_limit, tpm_limit, max_parallel_requests, etc.
)
await parallel_request_handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict_no_limits,
cache=local_cache,
data={"model": "gpt-3.5-turbo"},
call_type="",
)
assert (
not should_rate_limit_called
), "should_rate_limit should not be called when no rate limits are configured"
# Test 2: API key rate limits configured - should_rate_limit SHOULD be called
should_rate_limit_called = False
user_api_key_dict_with_api_limits = UserAPIKeyAuth(
api_key=_api_key,
rpm_limit=100, # Rate limit configured
)
await parallel_request_handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict_with_api_limits,
cache=local_cache,
data={"model": "gpt-3.5-turbo"},
call_type="",
)
assert (
should_rate_limit_called
), "should_rate_limit should be called when API key rate limits are configured"
# Test 3: User rate limits configured - should_rate_limit SHOULD be called
should_rate_limit_called = False
user_api_key_dict_with_user_limits = UserAPIKeyAuth(
api_key=_api_key,
user_id="test_user",
user_tpm_limit=1000, # User rate limit configured
)
await parallel_request_handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict_with_user_limits,
cache=local_cache,
data={"model": "gpt-3.5-turbo"},
call_type="",
)
assert (
should_rate_limit_called
), "should_rate_limit should be called when user rate limits are configured"
# Test 4: Team rate limits configured - should_rate_limit SHOULD be called
should_rate_limit_called = False
user_api_key_dict_with_team_limits = UserAPIKeyAuth(
api_key=_api_key,
team_id="test_team",
team_rpm_limit=500, # Team rate limit configured
)
await parallel_request_handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict_with_team_limits,
cache=local_cache,
data={"model": "gpt-3.5-turbo"},
call_type="",
)
assert (
should_rate_limit_called
), "should_rate_limit should be called when team rate limits are configured"
# Test 5: End user rate limits configured - should_rate_limit SHOULD be called
should_rate_limit_called = False
user_api_key_dict_with_end_user_limits = UserAPIKeyAuth(
api_key=_api_key,
end_user_id="test_end_user",
end_user_rpm_limit=200, # End user rate limit configured
)
await parallel_request_handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict_with_end_user_limits,
cache=local_cache,
data={"model": "gpt-3.5-turbo"},
call_type="",
)
assert (
should_rate_limit_called
), "should_rate_limit should be called when end user rate limits are configured"
# Test 6: Max parallel requests configured - should_rate_limit SHOULD be called
should_rate_limit_called = False
user_api_key_dict_with_parallel_limits = UserAPIKeyAuth(
api_key=_api_key,
max_parallel_requests=5, # Max parallel requests configured
)
await parallel_request_handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict_with_parallel_limits,
cache=local_cache,
data={"model": "gpt-3.5-turbo"},
call_type="",
)
assert (
should_rate_limit_called
), "should_rate_limit should be called when max parallel requests are configured"
@pytest.mark.asyncio
async def test_model_specific_rate_limits_only_called_when_configured_v3():
"""
Test that model-specific rate limits only trigger should_rate_limit when actually configured for the requested model.
"""
from litellm.proxy.auth.auth_utils import (
get_key_model_rpm_limit,
get_key_model_tpm_limit,
)
_api_key = "sk-12345"
_api_key = hash_token(_api_key)
local_cache = DualCache()
parallel_request_handler = _PROXY_MaxParallelRequestsHandler(
internal_usage_cache=InternalUsageCache(local_cache)
)
# Mock should_rate_limit to track if it's called
should_rate_limit_called = False
async def mock_should_rate_limit(*args, **kwargs):
nonlocal should_rate_limit_called
should_rate_limit_called = True
return {"overall_code": "OK", "statuses": []}
parallel_request_handler.should_rate_limit = mock_should_rate_limit
# Test 1: Model-specific rate limits configured but for different model - should NOT be called
should_rate_limit_called = False
user_api_key_dict_with_model_limits = UserAPIKeyAuth(
api_key=_api_key,
metadata={
"model_tpm_limit": {"gpt-4": 1000}
}, # Rate limit for gpt-4, not gpt-3.5-turbo
)
await parallel_request_handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict_with_model_limits,
cache=local_cache,
data={"model": "gpt-3.5-turbo"}, # Requesting different model
call_type="",
)
assert (
not should_rate_limit_called
), "should_rate_limit should not be called when model-specific limits don't match requested model"
# Test 2: Model-specific rate limits configured for requested model - SHOULD be called
should_rate_limit_called = False
user_api_key_dict_with_matching_model_limits = UserAPIKeyAuth(
api_key=_api_key,
metadata={
"model_tpm_limit": {"gpt-3.5-turbo": 1000}
}, # Rate limit for requested model
)
await parallel_request_handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict_with_matching_model_limits,
cache=local_cache,
data={"model": "gpt-3.5-turbo"}, # Requesting same model
call_type="",
)
assert (
should_rate_limit_called
), "should_rate_limit should be called when model-specific limits match requested model"
tests/test_litellm/proxy/test_fastapi_offline_routes.py
+125
View File
@@ -0,0 +1,125 @@
"""
Unit test for testing /routes endpoint with FastAPIOffline app initialization.
This test verifies that the /routes endpoint works correctly when the proxy
server is initialized using FastAPIOffline instead of regular FastAPI.
"""
import os
import sys
sys.path.insert(
0, os.path.abspath("../../..")
) # Adds the parent directory to the system path
import pytest
from fastapi.testclient import TestClient
from fastapi_offline import FastAPIOffline
class TestFastAPIOfflineRoutes:
"""Test that /routes endpoint works with FastAPIOffline app initialization."""
def test_routes_endpoint_with_fastapi_offline(self):
"""
Test that /routes endpoint responds correctly when using FastAPIOffline.
This test verifies that when the proxy server app is initialized using
FastAPIOffline instead of regular FastAPI, the /routes endpoint still
functions properly without throwing the StaticFiles AttributeError.
"""
from litellm.proxy.proxy_server import router
# Initialize app using FastAPIOffline instead of regular FastAPI
app = FastAPIOffline()
# Add a simple root endpoint to verify app is working
@app.get("/")
async def root():
return {"message": "Hello World"}
# Include the litellm proxy router which contains the /routes endpoint
app.include_router(router)
# Create test client
client = TestClient(app)
# Test the root endpoint first to ensure app is working
response = client.get("/")
assert response.status_code == 200
assert response.json() == {"message": "Hello World"}
# Test the /routes endpoint - this should not fail even with FastAPIOffline
# The important part is that it doesn't fail with the StaticFiles AttributeError
response = client.get("/routes")
# Print response for debugging
print(f"Response status: {response.status_code}")
print(f"Response content: {response.text}")
# The key test: we should NOT get a 500 (Internal Server Error)
# which would indicate the StaticFiles AttributeError bug
assert response.status_code != 500, f"Got 500 error: {response.text}"
# We accept either 200 (success) or 401 (auth required) - both are valid
assert response.status_code in [200, 401], f"Unexpected status: {response.status_code}"
if response.status_code == 200:
# If successful, verify it has the expected structure
response_json = response.json()
assert "routes" in response_json
assert isinstance(response_json["routes"], list)
print("✓ /routes endpoint returns valid routes data with FastAPIOffline")
else:
# If auth fails, ensure it's a proper JSON error response
response_json = response.json()
assert "detail" in response_json
print("✓ /routes endpoint handles auth properly with FastAPIOffline")
# If we get here without any AttributeError exceptions, the fix is working
print("✓ /routes endpoint handles FastAPIOffline initialization correctly")
def test_routes_endpoint_with_auth_token_fastapi_offline(self):
"""
Test /routes endpoint with auth token using FastAPIOffline.
This test provides a mock auth token to actually test the routes response.
"""
from unittest.mock import patch
from litellm.proxy.proxy_server import router
# Initialize app using FastAPIOffline
app = FastAPIOffline()
@app.get("/")
async def root():
return {"message": "Hello World"}
app.include_router(router)
client = TestClient(app)
# Mock the authentication to bypass the auth requirement
with patch('litellm.proxy.auth.user_api_key_auth.user_api_key_auth') as mock_auth:
# Configure mock to return a successful auth response
mock_auth.return_value = {"user_id": "test_user", "api_key": "test_key"}
# Test with Authorization header
headers = {"Authorization": "Bearer sk-test-token"}
response = client.get("/routes", headers=headers)
# If authentication is properly mocked, we should get a 200 response
# If not, we might get 401, but we should NOT get 500 (AttributeError)
assert response.status_code in [200, 401], f"Unexpected status code: {response.status_code}"
if response.status_code == 200:
# If we get a successful response, verify it has the expected structure
response_json = response.json()
assert "routes" in response_json
assert isinstance(response_json["routes"], list)
print("✓ /routes endpoint returns valid response with FastAPIOffline")
else:
# Even if auth fails, ensure it's a proper JSON error response
response_json = response.json()
assert "detail" in response_json
print("✓ /routes endpoint handles auth properly with FastAPIOffline")
tests/test_litellm/proxy/test_proxy_server.py
+440 -2
View File
@@ -5,6 +5,7 @@ import os
import socket
import subprocess
import sys
from datetime import datetime
from unittest import mock
from unittest.mock import AsyncMock, MagicMock, mock_open, patch
@@ -21,6 +22,7 @@ sys.path.insert(
import litellm
from litellm.proxy.proxy_server import app, initialize
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
example_embedding_result = {
"object": "list",
@@ -1117,9 +1119,9 @@ async def test_chat_completion_result_no_nested_none_values():
)
mock_model_response.choices = [mock_choice]
mock_model_response.usage = litellm.Usage(
setattr(mock_model_response, "usage", litellm.Usage(
prompt_tokens=10, completion_tokens=5, total_tokens=15
)
))
# Verify the mock has None values before serialization
raw_dict = mock_model_response.model_dump()
@@ -1190,3 +1192,439 @@ async def test_chat_completion_result_no_nested_none_values():
assert (
field not in message
), f"Field '{field}' should be excluded when it's None"
# ============================================================================
# Price Data Reload Tests
# ============================================================================
class TestPriceDataReloadAPI:
"""Test cases for price data reload API endpoints"""
@pytest.fixture
def client_with_auth(self):
"""Create a test client with authentication"""
from litellm.proxy.proxy_server import cleanup_router_config_variables
from litellm.proxy._types import LitellmUserRoles
cleanup_router_config_variables()
filepath = os.path.dirname(os.path.abspath(__file__))
config_fp = f"{filepath}/test_configs/test_config_no_auth.yaml"
asyncio.run(initialize(config=config_fp, debug=True))
# Mock admin user authentication
mock_auth = MagicMock()
mock_auth.user_role = LitellmUserRoles.PROXY_ADMIN
app.dependency_overrides[user_api_key_auth] = lambda: mock_auth
return TestClient(app)
def test_reload_model_cost_map_admin_access(self, client_with_auth):
"""Test that admin users can access the reload endpoint"""
with patch('litellm.litellm_core_utils.get_model_cost_map.get_model_cost_map') as mock_get_map:
mock_get_map.return_value = {"gpt-3.5-turbo": {"input_cost_per_token": 0.001}}
# Mock the database connection
with patch('litellm.proxy.proxy_server.prisma_client') as mock_prisma:
mock_prisma.db.litellm_config.upsert = AsyncMock(return_value=None)
response = client_with_auth.post("/reload/model_cost_map")
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert "message" in data
assert "timestamp" in data
assert "models_count" in data
# The new implementation immediately reloads and returns the count
assert "Price data reloaded successfully! 1 models updated." in data["message"]
assert data["models_count"] == 1
def test_reload_model_cost_map_non_admin_access(self, client_with_auth):
"""Test that non-admin users cannot access the reload endpoint"""
# Mock non-admin user
mock_auth = MagicMock()
mock_auth.user_role = "user" # Non-admin role
app.dependency_overrides[user_api_key_auth] = lambda: mock_auth
response = client_with_auth.post("/reload/model_cost_map")
assert response.status_code == 403
data = response.json()
assert "Access denied" in data["detail"]
assert "Admin role required" in data["detail"]
def test_get_model_cost_map_admin_access(self, client_with_auth):
"""Test that admin users can access the get model cost map endpoint"""
with patch('litellm.model_cost', {"gpt-3.5-turbo": {"input_cost_per_token": 0.001}}):
response = client_with_auth.get("/get/litellm_model_cost_map")
assert response.status_code == 200
data = response.json()
assert "gpt-3.5-turbo" in data
def test_get_model_cost_map_non_admin_access(self, client_with_auth):
"""Test that non-admin users cannot access the get model cost map endpoint"""
# Mock non-admin user
mock_auth = MagicMock()
mock_auth.user_role = "user" # Non-admin role
app.dependency_overrides[user_api_key_auth] = lambda: mock_auth
response = client_with_auth.get("/get/litellm_model_cost_map")
assert response.status_code == 403
data = response.json()
assert "Access denied" in data["detail"]
assert "Admin role required" in data["detail"]
def test_reload_model_cost_map_error_handling(self, client_with_auth):
"""Test error handling in the reload endpoint"""
with patch('litellm.litellm_core_utils.get_model_cost_map.get_model_cost_map') as mock_get_map:
mock_get_map.side_effect = Exception("Network error")
# Mock the database connection
with patch('litellm.proxy.proxy_server.prisma_client') as mock_prisma:
mock_prisma.db.litellm_config.upsert = AsyncMock(return_value=None)
response = client_with_auth.post("/reload/model_cost_map")
assert response.status_code == 500 # The new implementation immediately reloads and fails on error
data = response.json()
assert "Failed to reload model cost map" in data["detail"]
def test_schedule_model_cost_map_reload_admin_access(self, client_with_auth):
"""Test that admin users can schedule periodic reload"""
with patch('litellm.proxy.proxy_server.prisma_client') as mock_prisma:
# Mock database upsert
mock_prisma.db.litellm_config.upsert = AsyncMock(return_value=None)
response = client_with_auth.post("/schedule/model_cost_map_reload?hours=6")
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert data["interval_hours"] == 6
assert "message" in data
assert "timestamp" in data
def test_schedule_model_cost_map_reload_non_admin_access(self, client_with_auth):
"""Test that non-admin users cannot schedule periodic reload"""
# Mock non-admin user
mock_auth = MagicMock()
mock_auth.user_role = "user" # Non-admin role
app.dependency_overrides[user_api_key_auth] = lambda: mock_auth
response = client_with_auth.post("/schedule/model_cost_map_reload?hours=6")
assert response.status_code == 403
data = response.json()
assert "Access denied" in data["detail"]
assert "Admin role required" in data["detail"]
def test_schedule_model_cost_map_reload_invalid_hours(self, client_with_auth):
"""Test that invalid hours parameter is rejected"""
response = client_with_auth.post("/schedule/model_cost_map_reload?hours=0")
assert response.status_code == 400
data = response.json()
assert "Hours must be greater than 0" in data["detail"]
def test_cancel_model_cost_map_reload_admin_access(self, client_with_auth):
"""Test that admin users can cancel periodic reload"""
with patch('litellm.proxy.proxy_server.prisma_client') as mock_prisma:
# Mock database delete
mock_prisma.db.litellm_config.delete = AsyncMock(return_value=None)
response = client_with_auth.delete("/schedule/model_cost_map_reload")
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert "message" in data
assert "timestamp" in data
def test_cancel_model_cost_map_reload_non_admin_access(self, client_with_auth):
"""Test that non-admin users cannot cancel periodic reload"""
# Mock non-admin user
mock_auth = MagicMock()
mock_auth.user_role = "user" # Non-admin role
app.dependency_overrides[user_api_key_auth] = lambda: mock_auth
response = client_with_auth.delete("/schedule/model_cost_map_reload")
assert response.status_code == 403
data = response.json()
assert "Access denied" in data["detail"]
assert "Admin role required" in data["detail"]
def test_get_model_cost_map_reload_status_admin_access(self, client_with_auth):
"""Test that admin users can get reload status"""
with patch('litellm.proxy.proxy_server.prisma_client') as mock_prisma:
# Mock database config record
mock_config = MagicMock()
mock_config.param_value = {
"interval_hours": 6,
"force_reload": False
}
mock_prisma.db.litellm_config.find_unique = AsyncMock(return_value=mock_config)
# Mock the last reload time and current time
with patch('litellm.proxy.proxy_server.last_model_cost_map_reload', "2024-01-01T06:00:00"):
with patch('litellm.proxy.proxy_server.datetime') as mock_datetime:
# Mock current time to be 1 hour after last reload
mock_datetime.utcnow.return_value = datetime(2024, 1, 1, 7, 0, 0)
mock_datetime.fromisoformat = datetime.fromisoformat
response = client_with_auth.get("/schedule/model_cost_map_reload/status")
assert response.status_code == 200
data = response.json()
assert data["scheduled"] == True
assert data["interval_hours"] == 6
assert data["last_run"] == "2024-01-01T06:00:00"
assert data["next_run"] == "2024-01-01T12:00:00"
def test_get_model_cost_map_reload_status_non_admin_access(self, client_with_auth):
"""Test that non-admin users cannot get reload status"""
# Mock non-admin user
mock_auth = MagicMock()
mock_auth.user_role = "user" # Non-admin role
app.dependency_overrides[user_api_key_auth] = lambda: mock_auth
response = client_with_auth.get("/schedule/model_cost_map_reload/status")
assert response.status_code == 403
data = response.json()
assert "Access denied" in data["detail"]
assert "Admin role required" in data["detail"]
def test_get_model_cost_map_reload_status_no_config(self, client_with_auth):
"""Test that status returns not scheduled when no config exists"""
with patch('litellm.proxy.proxy_server.prisma_client') as mock_prisma:
mock_prisma.db.litellm_config.find_unique = AsyncMock(return_value=None)
response = client_with_auth.get("/schedule/model_cost_map_reload/status")
assert response.status_code == 200
data = response.json()
assert data["scheduled"] == False
assert data["interval_hours"] == None
assert data["last_run"] == None
assert data["next_run"] == None
def test_get_model_cost_map_reload_status_no_interval(self, client_with_auth):
"""Test that status returns not scheduled when no interval is configured"""
with patch('litellm.proxy.proxy_server.prisma_client') as mock_prisma:
# Mock config with no interval
mock_config = MagicMock()
mock_config.param_value = {
"interval_hours": None,
"force_reload": False
}
mock_prisma.db.litellm_config.find_unique = AsyncMock(return_value=mock_config)
response = client_with_auth.get("/schedule/model_cost_map_reload/status")
assert response.status_code == 200
data = response.json()
assert data["scheduled"] == False
assert data["interval_hours"] == None
assert data["last_run"] == None
assert data["next_run"] == None
class TestPriceDataReloadIntegration:
"""Integration tests for the complete price data reload feature"""
@pytest.fixture
def client_with_auth(self):
"""Create a test client with authentication"""
from litellm.proxy.proxy_server import cleanup_router_config_variables
from litellm.proxy._types import LitellmUserRoles
cleanup_router_config_variables()
filepath = os.path.dirname(os.path.abspath(__file__))
config_fp = f"{filepath}/test_configs/test_config_no_auth.yaml"
asyncio.run(initialize(config=config_fp, debug=True))
# Mock admin user authentication
mock_auth = MagicMock()
mock_auth.user_role = LitellmUserRoles.PROXY_ADMIN
app.dependency_overrides[user_api_key_auth] = lambda: mock_auth
return TestClient(app)
def test_complete_reload_flow(self, client_with_auth):
"""Test the complete reload flow from API to model cost update"""
# Mock the model cost map
mock_cost_map = {
"gpt-3.5-turbo": {
"input_cost_per_token": 0.001,
"output_cost_per_token": 0.002
},
"gpt-4": {
"input_cost_per_token": 0.03,
"output_cost_per_token": 0.06
}
}
with patch('litellm.litellm_core_utils.get_model_cost_map.get_model_cost_map') as mock_get_map:
mock_get_map.return_value = mock_cost_map
# Mock the database connection
with patch('litellm.proxy.proxy_server.prisma_client') as mock_prisma:
mock_prisma.db.litellm_config.upsert = AsyncMock(return_value=None)
# Test reload endpoint
response = client_with_auth.post("/reload/model_cost_map")
assert response.status_code == 200
# Test get endpoint
response = client_with_auth.get("/get/litellm_model_cost_map")
assert response.status_code == 200
def test_distributed_reload_check_function(self):
"""Test the _check_and_reload_model_cost_map function"""
from litellm.proxy.proxy_server import ProxyConfig
proxy_config = ProxyConfig()
# Mock prisma client
mock_prisma = MagicMock()
# Test case 1: No config in database
mock_prisma.db.litellm_config.find_unique = AsyncMock(return_value=None)
# Should return early without reloading
asyncio.run(proxy_config._check_and_reload_model_cost_map(mock_prisma))
# Test case 2: Config with interval but not time to reload
mock_config = MagicMock()
mock_config.param_value = {
"interval_hours": 6,
"force_reload": False
}
mock_prisma.db.litellm_config.find_unique = AsyncMock(return_value=mock_config)
# Mock current time and last reload time
with patch('litellm.proxy.proxy_server.last_model_cost_map_reload', "2024-01-01T06:00:00"):
with patch('litellm.proxy.proxy_server.datetime') as mock_datetime:
mock_datetime.utcnow.return_value = datetime(2024, 1, 1, 7, 0, 0) # 1 hour later
# Should not reload (only 1 hour passed, need 6)
asyncio.run(proxy_config._check_and_reload_model_cost_map(mock_prisma))
# Test case 3: Config with force reload
mock_config.param_value = {
"interval_hours": 6,
"force_reload": True
}
mock_prisma.db.litellm_config.find_unique = AsyncMock(return_value=mock_config)
mock_prisma.db.litellm_config.upsert = AsyncMock(return_value=None)
with patch('litellm.litellm_core_utils.get_model_cost_map.get_model_cost_map') as mock_get_map:
mock_get_map.return_value = {"gpt-3.5-turbo": {"input_cost_per_token": 0.001}}
# Should reload due to force flag
asyncio.run(proxy_config._check_and_reload_model_cost_map(mock_prisma))
# Verify force_reload was reset to False
mock_prisma.db.litellm_config.upsert.assert_called()
call_args = mock_prisma.db.litellm_config.upsert.call_args
# The param_value is now a JSON string, so we need to parse it
param_value_json = call_args[1]['data']['update']['param_value']
param_value_dict = json.loads(param_value_json)
assert param_value_dict['force_reload'] == False
def test_config_file_parsing(self):
"""Test parsing of config file with reload settings"""
config_content = """
general_settings:
master_key: sk-1234
model_cost_map_reload_interval: 21600
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: gpt-3.5-turbo
- model_name: gpt-4
litellm_params:
model: gpt-4
"""
# Parse the config
config = yaml.safe_load(config_content)
# Verify the reload setting is present
assert "general_settings" in config
assert "model_cost_map_reload_interval" in config["general_settings"]
assert config["general_settings"]["model_cost_map_reload_interval"] == 21600
# Verify models are present
assert "model_list" in config
assert len(config["model_list"]) == 2
def test_database_config_storage(self):
"""Test that configuration is properly stored in database"""
# Mock prisma client
mock_prisma = MagicMock()
# Test the database upsert call that would be made by the schedule endpoint
mock_prisma.db.litellm_config.upsert = AsyncMock(return_value=None)
# Simulate the database call that the schedule endpoint would make
asyncio.run(mock_prisma.db.litellm_config.upsert(
where={"param_name": "model_cost_map_reload_config"},
data={
"create": {
"param_name": "model_cost_map_reload_config",
"param_value": {
"interval_hours": 6,
"force_reload": False
}
},
"update": {
"param_value": {
"interval_hours": 6,
"force_reload": False
}
}
}
))
# Verify database upsert was called with correct data
mock_prisma.db.litellm_config.upsert.assert_called_once()
call_args = mock_prisma.db.litellm_config.upsert.call_args
assert call_args[1]['where']['param_name'] == "model_cost_map_reload_config"
assert call_args[1]['data']['create']['param_value']['interval_hours'] == 6
assert call_args[1]['data']['create']['param_value']['force_reload'] == False
def test_manual_reload_force_flag(self):
"""Test that manual reload sets force flag correctly"""
# Mock prisma client
mock_prisma = MagicMock()
# Test the database upsert call that would be made by the manual reload endpoint
mock_prisma.db.litellm_config.upsert = AsyncMock(return_value=None)
# Simulate the database call that the manual reload endpoint would make
asyncio.run(mock_prisma.db.litellm_config.upsert(
where={"param_name": "model_cost_map_reload_config"},
data={
"create": {
"param_name": "model_cost_map_reload_config",
"param_value": {
"interval_hours": None,
"force_reload": True
}
},
"update": {
"param_value": {
"force_reload": True
}
}
}
))
# Verify force_reload flag was set
mock_prisma.db.litellm_config.upsert.assert_called_once()
call_args = mock_prisma.db.litellm_config.upsert.call_args
assert call_args[1]['data']['update']['param_value']['force_reload'] == True
tests/test_litellm/proxy/test_swagger_chat_completions.py
+310
View File
@@ -0,0 +1,310 @@
"""
Unit test to validate that /chat/completions has the expected schema in Swagger after add_llm_api_request_schema_body runs.
This test ensures that the ProxyChatCompletionRequest Pydantic model is properly added to the OpenAPI schema
for the /chat/completions endpoint, showing all expected fields in the Swagger documentation.
"""
from unittest.mock import Mock, patch
import pytest
from fastapi.testclient import TestClient
from litellm.proxy.common_utils.custom_openapi_spec import CustomOpenAPISpec
from litellm.proxy.proxy_server import app
class TestSwaggerChatCompletions:
"""Test suite for validating /chat/completions schema in Swagger documentation."""
@pytest.fixture
def client(self):
"""FastAPI test client for the proxy server."""
return TestClient(app)
def test_openapi_schema_includes_chat_completions_request_body(self, client):
"""
Test that the OpenAPI schema includes ProxyChatCompletionRequest schema
for /chat/completions endpoints after add_llm_api_request_schema_body runs.
"""
# Clear any cached schema to ensure we get the latest version
from litellm.proxy.proxy_server import app
app.openapi_schema = None
# Get the OpenAPI schema from the running app
response = client.get("/openapi.json")
assert response.status_code == 200
openapi_schema = response.json()
# Verify the schema has the expected structure
assert "openapi" in openapi_schema
assert "paths" in openapi_schema
assert "components" in openapi_schema
assert "schemas" in openapi_schema["components"]
# Check that ProxyChatCompletionRequest schema is in components
assert "ProxyChatCompletionRequest" in openapi_schema["components"]["schemas"]
# Get the ProxyChatCompletionRequest schema
chat_completion_schema = openapi_schema["components"]["schemas"]["ProxyChatCompletionRequest"]
# Verify it has the expected properties structure
assert "properties" in chat_completion_schema
properties = chat_completion_schema["properties"]
# Check for core OpenAI chat completion fields
expected_core_fields = [
"model",
"messages",
"temperature",
"top_p",
"max_tokens",
"stream",
"stop",
"presence_penalty",
"frequency_penalty",
"logit_bias",
"user",
"response_format",
"seed",
"tools",
"tool_choice",
"logprobs",
"top_logprobs"
]
for field in expected_core_fields:
assert field in properties, f"Expected field '{field}' not found in ProxyChatCompletionRequest schema"
# Check for LiteLLM-specific fields added by ProxyChatCompletionRequest
expected_litellm_fields = [
"guardrails",
"caching",
"num_retries",
"context_window_fallback_dict",
"fallbacks"
]
for field in expected_litellm_fields:
assert field in properties, f"Expected LiteLLM field '{field}' not found in ProxyChatCompletionRequest schema"
# Verify model and messages are required fields
if "required" in chat_completion_schema:
required_fields = chat_completion_schema["required"]
assert "model" in required_fields, "Field 'model' should be required"
assert "messages" in required_fields, "Field 'messages' should be required"
def test_chat_completions_endpoints_have_expanded_request_body(self, client):
"""
Test that /chat/completions endpoint has an expanded request body schema
with all individual fields visible (not just a $ref).
"""
# Clear any cached schema to ensure we get the latest version
from litellm.proxy.proxy_server import app
app.openapi_schema = None
# Get the OpenAPI schema
response = client.get("/openapi.json")
assert response.status_code == 200
openapi_schema = response.json()
paths = openapi_schema["paths"]
# Check main chat completion path
path_to_check = "/chat/completions"
assert path_to_check in paths, f"Path {path_to_check} not found in OpenAPI schema"
assert "post" in paths[path_to_check], f"POST method not found for path {path_to_check}"
post_spec = paths[path_to_check]["post"]
# Should have request body with expanded schema (not just $ref)
assert "requestBody" in post_spec, f"Path {path_to_check} should have requestBody"
request_body = post_spec["requestBody"]
# Check request body structure
assert "content" in request_body
assert "application/json" in request_body["content"]
json_content = request_body["content"]["application/json"]
assert "schema" in json_content
schema_def = json_content["schema"]
# Should be an expanded object schema, not a $ref
assert schema_def.get("type") == "object", "Schema should be an expanded object type"
assert "properties" in schema_def, "Schema should have expanded properties"
assert "$ref" not in schema_def, "Schema should not be a reference (should be expanded inline)"
# Should have all Pydantic fields as individual properties
properties = schema_def["properties"]
assert len(properties) >= 25, f"Expected at least 25 properties, got {len(properties)}"
# Should have core OpenAI fields
core_fields = ["model", "messages", "temperature", "max_tokens", "stream"]
for field in core_fields:
assert field in properties, f"Core field '{field}' should be in expanded properties"
# Should have LiteLLM-specific fields
litellm_fields = ["guardrails", "caching", "fallbacks", "num_retries"]
for field in litellm_fields:
assert field in properties, f"LiteLLM field '{field}' should be in expanded properties"
# Check required fields
required_fields = schema_def.get("required", [])
assert "model" in required_fields, "Model should be marked as required"
assert "messages" in required_fields, "Messages should be marked as required"
# Should have minimal parameters (only path parameters)
parameters = post_spec.get("parameters", [])
# All parameters should be path parameters, no query parameters
for param in parameters:
assert param.get("in") == "path", f"Only path parameters expected, found {param.get('in')} parameter: {param.get('name')}"
@patch('litellm.proxy.common_utils.custom_openapi_spec.CustomOpenAPISpec.add_chat_completion_request_schema')
def test_add_llm_api_request_schema_body_calls_chat_completion_method(self, mock_add_chat):
"""
Test that add_llm_api_request_schema_body calls add_chat_completion_request_schema.
"""
# Create a mock schema
mock_schema = {
"openapi": "3.0.0",
"info": {"title": "Test API", "version": "1.0.0"},
"paths": {}
}
# Configure the mock to return the schema
mock_add_chat.return_value = mock_schema
# Call the main method
result = CustomOpenAPISpec.add_llm_api_request_schema_body(mock_schema)
# Verify the chat completion method was called
mock_add_chat.assert_called_once_with(mock_schema)
assert result == mock_schema
def test_custom_openapi_spec_chat_completion_paths_constant(self):
"""
Test that the CHAT_COMPLETION_PATHS constant includes all expected endpoints.
"""
expected_paths = [
"/v1/chat/completions",
"/chat/completions",
"/engines/{model}/chat/completions",
"/openai/deployments/{model}/chat/completions"
]
assert hasattr(CustomOpenAPISpec, 'CHAT_COMPLETION_PATHS')
actual_paths = CustomOpenAPISpec.CHAT_COMPLETION_PATHS
for expected_path in expected_paths:
assert expected_path in actual_paths, f"Expected path '{expected_path}' not found in CHAT_COMPLETION_PATHS"
def test_proxy_chat_completion_request_pydantic_model_works(self):
"""
Test that ProxyChatCompletionRequest properly generates schemas
and includes the expected LiteLLM-specific fields.
"""
from litellm.proxy._types import ProxyChatCompletionRequest
# Check that we can get the schema
try:
# Try Pydantic v2 method first
schema = ProxyChatCompletionRequest.model_json_schema()
except AttributeError:
try:
# Fallback to Pydantic v1 method
schema = ProxyChatCompletionRequest.schema()
except AttributeError:
pytest.fail("Could not get schema from ProxyChatCompletionRequest using either Pydantic v1 or v2 methods")
# Verify schema has properties
assert "properties" in schema
properties = schema["properties"]
# Check for core required fields
assert "model" in properties, "Field 'model' should be in schema"
assert "messages" in properties, "Field 'messages' should be in schema"
# Check for LiteLLM-specific fields
litellm_fields = ["guardrails", "caching", "num_retries", "context_window_fallback_dict", "fallbacks"]
for field in litellm_fields:
assert field in properties, f"LiteLLM field '{field}' should be in ProxyChatCompletionRequest schema"
def test_messages_field_has_example(self, client):
"""
Test that the messages field in the expanded request body includes a helpful example.
"""
# Clear any cached schema to ensure we get the latest version
from litellm.proxy.proxy_server import app
app.openapi_schema = None
# Get the OpenAPI schema
response = client.get("/openapi.json")
assert response.status_code == 200
openapi_schema = response.json()
# Navigate to the chat completions request body schema
chat_completions_post = openapi_schema["paths"]["/chat/completions"]["post"]
request_body = chat_completions_post["requestBody"]
schema_def = request_body["content"]["application/json"]["schema"]
# Check that messages field has an example
messages_field = schema_def["properties"]["messages"]
assert "example" in messages_field, "Messages field should have an example"
# Verify the example structure
example = messages_field["example"]
assert isinstance(example, list), "Messages example should be a list"
assert len(example) >= 1, "Messages example should have at least 1 message"
# Check that example messages have proper structure
for message in example:
assert "role" in message, "Each example message should have a role"
assert "content" in message, "Each example message should have content"
assert message["role"] in ["user", "assistant", "system"], f"Invalid role: {message['role']}"
assert isinstance(message["content"], str), "Message content should be a string"
def test_request_body_accepts_actual_chat_request(self, client):
"""
Test that the expanded request body schema accepts a real chat completion request.
This ensures our schema modifications don't break actual API functionality.
"""
# Test data that should be valid according to our expanded schema
test_request = {
"model": "gpt-4o",
"messages": [
{"role": "user", "content": "Hello, how are you?"},
{"role": "assistant", "content": "I'm doing well, thank you!"}
],
"temperature": 0.7,
"max_tokens": 100,
"guardrails": ["no-harmful-content"],
"caching": True
}
# This should validate against our schema without errors
# Note: We're not actually calling the endpoint (which would require API keys)
# but testing that the request structure is accepted by the schema
# Get the OpenAPI schema to verify our test data matches
response = client.get("/openapi.json")
assert response.status_code == 200
openapi_schema = response.json()
chat_completions_post = openapi_schema["paths"]["/chat/completions"]["post"]
# Should have expanded request body (not just $ref)
assert "requestBody" in chat_completions_post
request_body = chat_completions_post["requestBody"]
schema_def = request_body["content"]["application/json"]["schema"]
# Verify our test request has fields that exist in the schema
properties = schema_def["properties"]
for field_name in test_request.keys():
assert field_name in properties, f"Field '{field_name}' should be in expanded schema properties"
# Verify required fields are present in test request
required_fields = schema_def.get("required", [])
for required_field in required_fields:
assert required_field in test_request, f"Required field '{required_field}' should be in test request"
tests/test_litellm/test_router.py
+289
View File
@@ -1401,3 +1401,292 @@ def test_should_include_deployment():
model_name=model_name,
team_id=team_id,
)
def test_get_deployment_model_info_base_model_flow():
"""Test that get_deployment_model_info correctly handles the base model flow"""
from unittest.mock import patch
router = litellm.Router(
model_list=[
{
"model_name": "test-model",
"litellm_params": {"model": "gpt-3.5-turbo"},
}
],
)
# Mock data for the test
mock_custom_model_info = {
"base_model": "gpt-3.5-turbo",
"input_cost_per_token": 0.001,
"output_cost_per_token": 0.002,
"custom_field": "custom_value",
}
mock_base_model_info = {
"key": "gpt-3.5-turbo",
"max_tokens": 4096,
"max_input_tokens": 4096,
"max_output_tokens": 4096,
"input_cost_per_token": 0.0015, # This should be overridden by custom model info
"output_cost_per_token": 0.002,
"litellm_provider": "openai",
"mode": "chat",
"supported_openai_params": ["temperature", "max_tokens"],
}
mock_litellm_model_name_info = {
"key": "test-model",
"max_tokens": 2048,
"max_input_tokens": 2048,
"max_output_tokens": 2048,
"input_cost_per_token": 0.0005,
"output_cost_per_token": 0.001,
"litellm_provider": "test_provider",
"mode": "completion",
"supported_openai_params": ["temperature"],
}
# Test Case 1: Base model flow with custom model info that has base_model
with patch.object(
litellm, "model_cost", {"test-custom-model": mock_custom_model_info}
):
with patch.object(litellm, "get_model_info") as mock_get_model_info:
# Configure mock returns
mock_get_model_info.side_effect = lambda model: {
"gpt-3.5-turbo": mock_base_model_info,
"test-model": mock_litellm_model_name_info,
}.get(model)
result = router.get_deployment_model_info(
model_id="test-custom-model", model_name="test-model"
)
# Verify that get_model_info was called for both base model and model name
assert mock_get_model_info.call_count == 2
mock_get_model_info.assert_any_call(
model="gpt-3.5-turbo"
) # base model call
mock_get_model_info.assert_any_call(model="test-model") # model name call
# Verify the result contains merged information
assert result is not None
# Test the correct merging behavior after fix:
# 1. base_model_info provides defaults, custom_model_info overrides (correct priority)
# 2. The result of step 1 gets merged into litellm_model_name_info (custom+base override litellm)
# Fields from custom model (should override base model values)
assert (
result["input_cost_per_token"] == 0.001
) # From custom model (overrides base 0.0015)
assert (
result["output_cost_per_token"] == 0.002
) # From custom model (same as base)
assert result["custom_field"] == "custom_value" # From custom model
# Fields from base model that weren't overridden by custom
assert result["max_tokens"] == 4096 # From base model
assert result["litellm_provider"] == "openai" # From base model
assert (
result["mode"] == "chat"
) # From base model (overrides litellm "completion")
# The key field comes from base model since both base and litellm have it
# and base model info overrides litellm model name info in final merge
assert (
result["key"] == "gpt-3.5-turbo"
) # From base model (overrides litellm key)
# Test Case 2: Custom model info without base_model
mock_custom_model_info_no_base = {
"input_cost_per_token": 0.001,
"output_cost_per_token": 0.002,
"custom_field": "custom_value",
}
with patch.object(
litellm,
"model_cost",
{"test-custom-model-no-base": mock_custom_model_info_no_base},
):
with patch.object(litellm, "get_model_info") as mock_get_model_info:
mock_get_model_info.side_effect = lambda model: {
"test-model": mock_litellm_model_name_info,
}.get(model)
result = router.get_deployment_model_info(
model_id="test-custom-model-no-base", model_name="test-model"
)
# Should only call get_model_info once for model name (no base model)
assert mock_get_model_info.call_count == 1
mock_get_model_info.assert_called_with(model="test-model")
# Verify the result contains merged information
assert result is not None
assert result["input_cost_per_token"] == 0.001 # From custom model
assert result["max_tokens"] == 2048 # From litellm model name info
assert result["custom_field"] == "custom_value" # From custom model
assert result["mode"] == "completion" # From litellm model name info
# Test Case 3: No custom model info, only litellm model name info
with patch.object(litellm, "model_cost", {}): # Empty model cost
with patch.object(litellm, "get_model_info") as mock_get_model_info:
mock_get_model_info.side_effect = lambda model: {
"test-model": mock_litellm_model_name_info,
}.get(model)
result = router.get_deployment_model_info(
model_id="non-existent-model", model_name="test-model"
)
# Should only call get_model_info once for model name
assert mock_get_model_info.call_count == 1
mock_get_model_info.assert_called_with(model="test-model")
# Result should be just the litellm model name info
assert result is not None
assert result == mock_litellm_model_name_info
# Test Case 4: Base model info retrieval fails (exception handling)
mock_custom_model_info_invalid_base = {
"base_model": "invalid-base-model",
"input_cost_per_token": 0.001,
"output_cost_per_token": 0.002,
}
with patch.object(
litellm,
"model_cost",
{"test-custom-model-invalid": mock_custom_model_info_invalid_base},
):
with patch.object(litellm, "get_model_info") as mock_get_model_info:
# Mock get_model_info to raise exception for invalid base model
def mock_get_model_info_side_effect(model):
if model == "invalid-base-model":
raise Exception("Model not found")
elif model == "test-model":
return mock_litellm_model_name_info
return None
mock_get_model_info.side_effect = mock_get_model_info_side_effect
result = router.get_deployment_model_info(
model_id="test-custom-model-invalid", model_name="test-model"
)
# Should handle exception gracefully and still return merged result
assert result is not None
assert result["input_cost_per_token"] == 0.001 # From custom model
assert result["mode"] == "completion" # From litellm model name info
# Test Case 5: Both model_cost.get() and get_model_info() return None
with patch.object(litellm, "model_cost", {}):
with patch.object(
litellm, "get_model_info", side_effect=Exception("Not found")
):
result = router.get_deployment_model_info(
model_id="non-existent", model_name="non-existent"
)
# Should return None when no model info is found
assert result is None
print("✓ All base model flow test cases passed!")
@patch("litellm.model_cost", {})
def test_get_deployment_model_info_base_model_merge_priority():
"""Test that base model info merging respects the correct priority order"""
from unittest.mock import patch
router = litellm.Router(
model_list=[
{
"model_name": "test-model",
"litellm_params": {"model": "gpt-3.5-turbo"},
}
],
)
# Test data with overlapping fields to test merge priority
mock_custom_model_info = {
"base_model": "gpt-4",
"input_cost_per_token": 0.01, # Should override base model value
"max_tokens": 8000, # Should override base model value
"custom_only_field": "custom_value",
}
mock_base_model_info = {
"key": "gpt-4",
"max_tokens": 4096, # Should be overridden by custom model
"input_cost_per_token": 0.03, # Should be overridden by custom model
"output_cost_per_token": 0.06, # Should be preserved (not in custom)
"litellm_provider": "openai",
"base_only_field": "base_value",
}
mock_litellm_model_name_info = {
"key": "test-model",
"max_tokens": 2048, # Should be overridden by final custom model info
"input_cost_per_token": 0.005, # Should be overridden by final custom model info
"output_cost_per_token": 0.01, # Should be overridden by final custom model info
"mode": "completion",
"litellm_only_field": "litellm_value",
}
with patch.object(
litellm, "model_cost", {"custom-model-id": mock_custom_model_info}
):
with patch.object(litellm, "get_model_info") as mock_get_model_info:
mock_get_model_info.side_effect = lambda model: {
"gpt-4": mock_base_model_info,
"test-model": mock_litellm_model_name_info,
}.get(model)
result = router.get_deployment_model_info(
model_id="custom-model-id", model_name="test-model"
)
assert result is not None
# Test correct merge priority after fix:
# 1. base_model_info provides defaults
# 2. custom_model_info overrides base_model_info
# 3. Result from steps 1-2 overrides litellm_model_name_info
# Fields that should come from custom model info (highest priority)
assert (
result["input_cost_per_token"] == 0.01
) # From custom model (overrides base 0.03)
assert (
result["max_tokens"] == 8000
) # From custom model (overrides base 4096)
assert result["custom_only_field"] == "custom_value" # From custom model
# Fields that should come from base model (not overridden by custom)
assert (
result["output_cost_per_token"] == 0.06
) # From base model (not in custom)
assert (
result["litellm_provider"] == "openai"
) # From base model (not in custom)
assert (
result["base_only_field"] == "base_value"
) # From base model (not in custom)
# Fields that should come from litellm model name info (not overridden by custom+base)
assert (
result["mode"] == "completion"
) # From litellm model name info (not in custom or base)
assert (
result["litellm_only_field"] == "litellm_value"
) # From litellm model name info (not in custom or base)
# Key comes from base model since both base and litellm have key fields
# and the merged custom+base overrides litellm in the final merge
assert result["key"] == "gpt-4"
print("✓ Base model merge priority test passed!")
ui/litellm-dashboard/out/404.html
+1 -1
View File
File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show More