mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-17 08:25:03 +00:00
Litellm sameer nov 3 stable branch (#16963)
* Add openai metadata filed in the request * Add docs related to openai metadata * Add utils * test_completion_openai_metadata[True] * Added support for though signature for gemini 3 in responses api (#16872) * Added support for though signature for gemini 3 * Update docs with all supported endpoints and cost tracking * Added config based routing support for batches and files * fix lint errors * Litellm anthropic image url support (#16868) * Add image as url support to anthropic * fix mypy errors * fix tests * Fix: Populate spend_logs_metadata in batch and files endpoints (#16921) * Add spend-logs-metadata to the metadata * Add tests for spend logs metadata in batches * use better names * Remove support for penalty param for gemini 3 (#16907) * Remove support for penalty param * remove halucinated model names * fix mypy/test errors * fix tests * fix too many lines error * fix too many lines error * Add config for cicd test case * Fix final tests * fix batch tests * fix batch tests
This commit is contained in:
Sameer Kankute
@@ -4,8 +4,8 @@ title: "DAY 0 Support: Gemini 3 on LiteLLM"
|
||||
date: 2025-11-19T10:00:00
|
||||
authors:
|
||||
- name: Sameer Kankute
|
||||
title: "SWE @ LiteLLM (LLM Translation)"
|
||||
url: https://in.linkedin.com/in/sameer-kankute
|
||||
title: SWE @ LiteLLM (LLM Translation)
|
||||
url: https://www.linkedin.com/in/sameer-kankute/
|
||||
image_url: https://media.licdn.com/dms/image/v2/D4D03AQHB_loQYd5gjg/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1719137160975?e=1765411200&v=beta&t=c8396f--_lH6Fb_pVvx_jGholPfcl0bvwmNynbNdnII
|
||||
- name: Krrish Dholakia
|
||||
title: "CEO, LiteLLM"
|
||||
@@ -88,9 +88,11 @@ curl http://0.0.0.0:4000/v1/chat/completions \
|
||||
LiteLLM provides **full end-to-end support** for Gemini 3 Pro Preview on:
|
||||
|
||||
- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
|
||||
- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
|
||||
- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
|
||||
- ✅ `/v1/generateContent` – [Google Gemini API](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini#rest) compatible endpoint (for code, see: `client.models.generate_content(...)`)
|
||||
|
||||
Both endpoints support:
|
||||
All endpoints support:
|
||||
- Streaming and non-streaming responses
|
||||
- Function calling with thought signatures
|
||||
- Multi-turn conversations
|
||||
@@ -548,6 +550,129 @@ curl http://localhost:4000/v1/chat/completions \
|
||||
|
||||
3. **Automatic Defaults**: If you don't specify `reasoning_effort`, LiteLLM automatically sets `thinking_level="low"` for optimal performance.
|
||||
|
||||
## Cost Tracking: Prompt Caching & Context Window
|
||||
|
||||
LiteLLM provides comprehensive cost tracking for Gemini 3 Pro Preview, including support for prompt caching and tiered pricing based on context window size.
|
||||
|
||||
### Prompt Caching Cost Tracking
|
||||
|
||||
Gemini 3 supports prompt caching, which allows you to cache frequently used prompt prefixes to reduce costs. LiteLLM automatically tracks and calculates costs for:
|
||||
|
||||
- **Cache Hit Tokens**: Tokens that are read from cache (charged at a lower rate)
|
||||
- **Cache Creation Tokens**: Tokens that are written to cache (one-time cost)
|
||||
- **Text Tokens**: Regular prompt tokens that are processed normally
|
||||
|
||||
#### How It Works
|
||||
|
||||
LiteLLM extracts caching information from the `prompt_tokens_details` field in the usage object:
|
||||
|
||||
```python
|
||||
{
|
||||
"usage": {
|
||||
"prompt_tokens": 50000,
|
||||
"completion_tokens": 1000,
|
||||
"total_tokens": 51000,
|
||||
"prompt_tokens_details": {
|
||||
"cached_tokens": 30000, # Cache hit tokens
|
||||
"cache_creation_tokens": 5000, # Tokens written to cache
|
||||
"text_tokens": 15000 # Regular processed tokens
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Context Window Tiered Pricing
|
||||
|
||||
Gemini 3 Pro Preview supports up to 1M tokens of context, with tiered pricing that automatically applies when your prompt exceeds 200k tokens.
|
||||
|
||||
#### Automatic Tier Detection
|
||||
|
||||
LiteLLM automatically detects when your prompt exceeds the 200k token threshold and applies the appropriate tiered pricing:
|
||||
|
||||
```python
|
||||
from litellm import completion_cost
|
||||
|
||||
# Example: Small prompt (< 200k tokens)
|
||||
response_small = completion(
|
||||
model="gemini/gemini-3-pro-preview",
|
||||
messages=[{"role": "user", "content": "Hello!"}]
|
||||
)
|
||||
# Uses base pricing: $0.000002/input token, $0.000012/output token
|
||||
|
||||
# Example: Large prompt (> 200k tokens)
|
||||
response_large = completion(
|
||||
model="gemini/gemini-3-pro-preview",
|
||||
messages=[{"role": "user", "content": "..." * 250000}] # 250k tokens
|
||||
)
|
||||
# Automatically uses tiered pricing: $0.000004/input token, $0.000018/output token
|
||||
```
|
||||
|
||||
#### Cost Breakdown
|
||||
|
||||
The cost calculation includes:
|
||||
|
||||
1. **Text Processing Cost**: Regular tokens processed at base or tiered rate
|
||||
2. **Cache Read Cost**: Cached tokens read at discounted rate
|
||||
3. **Cache Creation Cost**: One-time cost for writing tokens to cache (applies tiered rate if above 200k)
|
||||
4. **Output Cost**: Generated tokens at base or tiered rate
|
||||
|
||||
### Example: Viewing Cost Breakdown
|
||||
|
||||
You can view the detailed cost breakdown using LiteLLM's cost tracking:
|
||||
|
||||
```python
|
||||
from litellm import completion, completion_cost
|
||||
|
||||
response = completion(
|
||||
model="gemini/gemini-3-pro-preview",
|
||||
messages=[{"role": "user", "content": "Explain prompt caching"}],
|
||||
caching=True # Enable prompt caching
|
||||
)
|
||||
|
||||
# Get total cost
|
||||
total_cost = completion_cost(completion_response=response)
|
||||
print(f"Total cost: ${total_cost:.6f}")
|
||||
|
||||
# Access usage details
|
||||
usage = response.usage
|
||||
print(f"Prompt tokens: {usage.prompt_tokens}")
|
||||
print(f"Completion tokens: {usage.completion_tokens}")
|
||||
|
||||
# Access caching details
|
||||
if usage.prompt_tokens_details:
|
||||
print(f"Cache hit tokens: {usage.prompt_tokens_details.cached_tokens}")
|
||||
print(f"Cache creation tokens: {usage.prompt_tokens_details.cache_creation_tokens}")
|
||||
print(f"Text tokens: {usage.prompt_tokens_details.text_tokens}")
|
||||
```
|
||||
|
||||
### Cost Optimization Tips
|
||||
|
||||
1. **Use Prompt Caching**: For repeated prompt prefixes, enable caching to reduce costs by up to 90% for cached portions
|
||||
2. **Monitor Context Size**: Be aware that prompts above 200k tokens use tiered pricing (2x for input, 1.5x for output)
|
||||
3. **Cache Management**: Cache creation tokens are charged once when writing to cache, then subsequent reads are much cheaper
|
||||
4. **Track Usage**: Use LiteLLM's built-in cost tracking to monitor spending across different token types
|
||||
|
||||
### Integration with LiteLLM Proxy
|
||||
|
||||
When using LiteLLM Proxy, all cost tracking is automatically logged and available through:
|
||||
|
||||
- **Usage Logs**: Detailed token and cost breakdowns in proxy logs
|
||||
- **Budget Management**: Set budgets and alerts based on actual usage
|
||||
- **Analytics Dashboard**: View cost trends and breakdowns by token type
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
model_list:
|
||||
- model_name: gemini-3-pro-preview
|
||||
litellm_params:
|
||||
model: gemini/gemini-3-pro-preview
|
||||
api_key: os.environ/GEMINI_API_KEY
|
||||
|
||||
litellm_settings:
|
||||
# Enable detailed cost tracking
|
||||
success_callback: ["langfuse"] # or your preferred logging service
|
||||
```
|
||||
|
||||
## Using with Claude Code CLI
|
||||
|
||||
You can use `gemini-3-pro-preview` with **Claude Code CLI** - Anthropic's command-line interface. This allows you to use Gemini 3 Pro Preview with Claude Code's native syntax and workflows.
|
||||
@@ -628,6 +753,162 @@ $ claude --model gemini-3-pro-preview
|
||||
- Ensure `GEMINI_API_KEY` is set correctly
|
||||
- Check LiteLLM proxy logs for detailed error messages
|
||||
|
||||
## Responses API Support
|
||||
|
||||
LiteLLM fully supports the OpenAI Responses API for Gemini 3 Pro Preview, including both streaming and non-streaming modes. The Responses API provides a structured way to handle multi-turn conversations with function calling, and LiteLLM automatically preserves thought signatures throughout the conversation.
|
||||
|
||||
### Example: Using Responses API with Gemini 3
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="sdk" label="Non-Streaming">
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
import json
|
||||
|
||||
client = OpenAI()
|
||||
|
||||
# 1. Define a list of callable tools for the model
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"name": "get_horoscope",
|
||||
"description": "Get today's horoscope for an astrological sign.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"sign": {
|
||||
"type": "string",
|
||||
"description": "An astrological sign like Taurus or Aquarius",
|
||||
},
|
||||
},
|
||||
"required": ["sign"],
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
def get_horoscope(sign):
|
||||
return f"{sign}: Next Tuesday you will befriend a baby otter."
|
||||
|
||||
# Create a running input list we will add to over time
|
||||
input_list = [
|
||||
{"role": "user", "content": "What is my horoscope? I am an Aquarius."}
|
||||
]
|
||||
|
||||
# 2. Prompt the model with tools defined
|
||||
response = client.responses.create(
|
||||
model="gemini-3-pro-preview",
|
||||
tools=tools,
|
||||
input=input_list,
|
||||
)
|
||||
|
||||
# Save function call outputs for subsequent requests
|
||||
input_list += response.output
|
||||
|
||||
for item in response.output:
|
||||
if item.type == "function_call":
|
||||
if item.name == "get_horoscope":
|
||||
# 3. Execute the function logic for get_horoscope
|
||||
horoscope = get_horoscope(json.loads(item.arguments))
|
||||
|
||||
# 4. Provide function call results to the model
|
||||
input_list.append({
|
||||
"type": "function_call_output",
|
||||
"call_id": item.call_id,
|
||||
"output": json.dumps({
|
||||
"horoscope": horoscope
|
||||
})
|
||||
})
|
||||
|
||||
print("Final input:")
|
||||
print(input_list)
|
||||
|
||||
response = client.responses.create(
|
||||
model="gemini-3-pro-preview",
|
||||
instructions="Respond only with a horoscope generated by a tool.",
|
||||
tools=tools,
|
||||
input=input_list,
|
||||
)
|
||||
|
||||
# 5. The model should be able to give a response!
|
||||
print("Final output:")
|
||||
print(response.model_dump_json(indent=2))
|
||||
print("\n" + response.output_text)
|
||||
```
|
||||
|
||||
**Key Points:**
|
||||
- ✅ Thought signatures are automatically preserved in function calls
|
||||
- ✅ Works seamlessly with multi-turn conversations
|
||||
- ✅ All Gemini 3-specific features are fully supported
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="streaming" label="Streaming">
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
import json
|
||||
|
||||
client = OpenAI()
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"name": "get_horoscope",
|
||||
"description": "Get today's horoscope for an astrological sign.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"sign": {
|
||||
"type": "string",
|
||||
"description": "An astrological sign like Taurus or Aquarius",
|
||||
},
|
||||
},
|
||||
"required": ["sign"],
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
def get_horoscope(sign):
|
||||
return f"{sign}: Next Tuesday you will befriend a baby otter."
|
||||
|
||||
input_list = [
|
||||
{"role": "user", "content": "What is my horoscope? I am an Aquarius."}
|
||||
]
|
||||
|
||||
# Streaming mode
|
||||
response = client.responses.create(
|
||||
model="gemini-3-pro-preview",
|
||||
tools=tools,
|
||||
input=input_list,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
# Collect all chunks
|
||||
chunks = []
|
||||
for chunk in response:
|
||||
chunks.append(chunk)
|
||||
# Process streaming chunks as they arrive
|
||||
print(chunk)
|
||||
|
||||
# Thought signatures are automatically preserved in streaming mode
|
||||
```
|
||||
|
||||
**Key Points:**
|
||||
- ✅ Streaming mode fully supported
|
||||
- ✅ Thought signatures preserved across streaming chunks
|
||||
- ✅ Real-time processing of function calls and responses
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Responses API Benefits
|
||||
|
||||
- ✅ **Structured Output**: Responses API provides a clear structure for handling function calls and multi-turn conversations
|
||||
- ✅ **Thought Signature Preservation**: LiteLLM automatically preserves thought signatures in both streaming and non-streaming modes
|
||||
- ✅ **Seamless Integration**: Works with existing OpenAI SDK patterns
|
||||
- ✅ **Full Feature Support**: All Gemini 3 features (thought signatures, function calling, reasoning) are fully supported
|
||||
|
||||
|
||||
## Best Practices
|
||||
|
||||
#### 1. Always Include Thought Signatures in Conversation History
|
||||
@@ -665,6 +946,7 @@ When switching from non-Gemini-3 to Gemini-3:
|
||||
- ✅ No manual intervention needed
|
||||
- ✅ Conversation history continues seamlessly
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
#### Issue: Missing Thought Signatures
|
||||
|
||||
@@ -174,6 +174,257 @@ print("list_batches_response=", list_batches_response)
|
||||
</Tabs>
|
||||
|
||||
|
||||
## Multi-Account / Model-Based Routing
|
||||
|
||||
Route batch operations to different provider accounts using model-specific credentials from your `config.yaml`. This eliminates the need for environment variables and enables multi-tenant batch processing.
|
||||
|
||||
### How It Works
|
||||
|
||||
**Priority Order:**
|
||||
1. **Encoded Batch/File ID** (highest) - Model info embedded in the ID
|
||||
2. **Model Parameter** - Via header (`x-litellm-model`), query param, or request body
|
||||
3. **Custom Provider** (fallback) - Uses environment variables
|
||||
|
||||
### Configuration
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gpt-4o-account-1
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: sk-account-1-key
|
||||
api_base: https://api.openai.com/v1
|
||||
|
||||
- model_name: gpt-4o-account-2
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: sk-account-2-key
|
||||
api_base: https://api.openai.com/v1
|
||||
|
||||
- model_name: azure-batches
|
||||
litellm_params:
|
||||
model: azure/gpt-4
|
||||
api_key: azure-key-123
|
||||
api_base: https://my-resource.openai.azure.com
|
||||
api_version: "2024-02-01"
|
||||
```
|
||||
|
||||
### Usage Examples
|
||||
|
||||
#### Scenario 1: Encoded File ID with Model
|
||||
|
||||
When you upload a file with a model parameter, LiteLLM encodes the model information in the file ID. All subsequent operations automatically use those credentials.
|
||||
|
||||
```bash
|
||||
# Step 1: Upload file with model
|
||||
curl http://localhost:4000/v1/files \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-H "x-litellm-model: gpt-4o-account-1" \
|
||||
-F purpose="batch" \
|
||||
-F file="@batch.jsonl"
|
||||
|
||||
# Response includes encoded file ID:
|
||||
# {
|
||||
# "id": "file-bGl0ZWxsbTpmaWxlLUxkaUwzaVYxNGZRVlpYcU5KVEdkSjk7bW9kZWwsZ3B0LTRvLWFjY291bnQtMQ",
|
||||
# ...
|
||||
# }
|
||||
|
||||
# Step 2: Create batch - automatically routes to gpt-4o-account-1
|
||||
curl http://localhost:4000/v1/batches \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input_file_id": "file-bGl0ZWxsbTpmaWxlLUxkaUwzaVYxNGZRVlpYcU5KVEdkSjk7bW9kZWwsZ3B0LTRvLWFjY291bnQtMQ",
|
||||
"endpoint": "/v1/chat/completions",
|
||||
"completion_window": "24h"
|
||||
}'
|
||||
|
||||
# Batch ID is also encoded with model:
|
||||
# {
|
||||
# "id": "batch_bGl0ZWxsbTpiYXRjaF82OTIwM2IzNjg0MDQ4MTkwYTA3ODQ5NDY3YTFjMDJkYTttb2RlbCxncHQtNG8tYWNjb3VudC0x",
|
||||
# "input_file_id": "file-bGl0ZWxsbTpmaWxlLUxkaUwzaVYxNGZRVlpYcU5KVEdkSjk7bW9kZWwsZ3B0LTRvLWFjY291bnQtMQ",
|
||||
# ...
|
||||
# }
|
||||
|
||||
# Step 3: Retrieve batch - automatically routes to gpt-4o-account-1
|
||||
curl http://localhost:4000/v1/batches/batch_bGl0ZWxsbTpiYXRjaF82OTIwM2IzNjg0MDQ4MTkwYTA3ODQ5NDY3YTFjMDJkYTttb2RlbCxncHQtNG8tYWNjb3VudC0x \
|
||||
-H "Authorization: Bearer sk-1234"
|
||||
```
|
||||
|
||||
**✅ Benefits:**
|
||||
- No need to specify model on every request
|
||||
- File and batch IDs "remember" which account created them
|
||||
- Automatic routing for retrieve, cancel, and file content operations
|
||||
|
||||
#### Scenario 2: Model via Header/Query Parameter
|
||||
|
||||
Specify the model for each request without encoding it in the ID.
|
||||
|
||||
```bash
|
||||
# Create batch with model header
|
||||
curl http://localhost:4000/v1/batches \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-H "x-litellm-model: gpt-4o-account-2" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input_file_id": "file-abc123",
|
||||
"endpoint": "/v1/chat/completions",
|
||||
"completion_window": "24h"
|
||||
}'
|
||||
|
||||
# Or use query parameter
|
||||
curl "http://localhost:4000/v1/batches?model=gpt-4o-account-2" \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input_file_id": "file-abc123",
|
||||
"endpoint": "/v1/chat/completions",
|
||||
"completion_window": "24h"
|
||||
}'
|
||||
|
||||
# List batches for specific model
|
||||
curl "http://localhost:4000/v1/batches?model=gpt-4o-account-2" \
|
||||
-H "Authorization: Bearer sk-1234"
|
||||
```
|
||||
|
||||
**✅ Use Case:**
|
||||
- One-off batch operations
|
||||
- Different models for different operations
|
||||
- Explicit control over routing
|
||||
|
||||
#### Scenario 3: Environment Variables (Fallback)
|
||||
|
||||
Traditional approach using environment variables when no model is specified.
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY="sk-env-key"
|
||||
|
||||
curl http://localhost:4000/v1/batches \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input_file_id": "file-abc123",
|
||||
"endpoint": "/v1/chat/completions",
|
||||
"completion_window": "24h"
|
||||
}'
|
||||
```
|
||||
|
||||
**✅ Use Case:**
|
||||
- Backward compatibility
|
||||
- Simple single-account setups
|
||||
- Quick prototyping
|
||||
|
||||
### Complete Multi-Account Example
|
||||
|
||||
```bash
|
||||
# Upload file to Account 1
|
||||
FILE_1=$(curl -s http://localhost:4000/v1/files \
|
||||
-H "x-litellm-model: gpt-4o-account-1" \
|
||||
-F purpose="batch" \
|
||||
-F file="@batch1.jsonl" | jq -r '.id')
|
||||
|
||||
# Upload file to Account 2
|
||||
FILE_2=$(curl -s http://localhost:4000/v1/files \
|
||||
-H "x-litellm-model: gpt-4o-account-2" \
|
||||
-F purpose="batch" \
|
||||
-F file="@batch2.jsonl" | jq -r '.id')
|
||||
|
||||
# Create batch on Account 1 (auto-routed via encoded file ID)
|
||||
BATCH_1=$(curl -s http://localhost:4000/v1/batches \
|
||||
-d "{\"input_file_id\": \"$FILE_1\", \"endpoint\": \"/v1/chat/completions\", \"completion_window\": \"24h\"}" | jq -r '.id')
|
||||
|
||||
# Create batch on Account 2 (auto-routed via encoded file ID)
|
||||
BATCH_2=$(curl -s http://localhost:4000/v1/batches \
|
||||
-d "{\"input_file_id\": \"$FILE_2\", \"endpoint\": \"/v1/chat/completions\", \"completion_window\": \"24h\"}" | jq -r '.id')
|
||||
|
||||
# Retrieve both batches (auto-routed to correct accounts)
|
||||
curl http://localhost:4000/v1/batches/$BATCH_1
|
||||
curl http://localhost:4000/v1/batches/$BATCH_2
|
||||
|
||||
# List batches per account
|
||||
curl "http://localhost:4000/v1/batches?model=gpt-4o-account-1"
|
||||
curl "http://localhost:4000/v1/batches?model=gpt-4o-account-2"
|
||||
```
|
||||
|
||||
### SDK Usage with Model Routing
|
||||
|
||||
```python
|
||||
import litellm
|
||||
import asyncio
|
||||
|
||||
# Upload file with model routing
|
||||
file_obj = await litellm.acreate_file(
|
||||
file=open("batch.jsonl", "rb"),
|
||||
purpose="batch",
|
||||
model="gpt-4o-account-1", # Route to specific account
|
||||
)
|
||||
|
||||
print(f"File ID: {file_obj.id}")
|
||||
# File ID is encoded with model info
|
||||
|
||||
# Create batch - automatically uses gpt-4o-account-1 credentials
|
||||
batch = await litellm.acreate_batch(
|
||||
completion_window="24h",
|
||||
endpoint="/v1/chat/completions",
|
||||
input_file_id=file_obj.id, # Model info embedded in ID
|
||||
)
|
||||
|
||||
print(f"Batch ID: {batch.id}")
|
||||
# Batch ID is also encoded
|
||||
|
||||
# Retrieve batch - automatically routes to correct account
|
||||
retrieved = await litellm.aretrieve_batch(
|
||||
batch_id=batch.id, # Model info embedded in ID
|
||||
)
|
||||
|
||||
print(f"Batch status: {retrieved.status}")
|
||||
|
||||
# Or explicitly specify model
|
||||
batch2 = await litellm.acreate_batch(
|
||||
completion_window="24h",
|
||||
endpoint="/v1/chat/completions",
|
||||
input_file_id="file-regular-id",
|
||||
model="gpt-4o-account-2", # Explicit routing
|
||||
)
|
||||
```
|
||||
|
||||
### How ID Encoding Works
|
||||
|
||||
LiteLLM encodes model information into file and batch IDs using base64:
|
||||
|
||||
```
|
||||
Original: file-abc123
|
||||
Encoded: file-bGl0ZWxsbTpmaWxlLWFiYzEyMzttb2RlbCxncHQtNG8tdGVzdA
|
||||
└─┬─┘ └──────────────────┬──────────────────────┘
|
||||
prefix base64(litellm:file-abc123;model,gpt-4o-test)
|
||||
|
||||
Original: batch_xyz789
|
||||
Encoded: batch_bGl0ZWxsbTpiYXRjaF94eXo3ODk7bW9kZWwsZ3B0LTRvLXRlc3Q
|
||||
└──┬──┘ └──────────────────┬──────────────────────┘
|
||||
prefix base64(litellm:batch_xyz789;model,gpt-4o-test)
|
||||
```
|
||||
|
||||
The encoding:
|
||||
- ✅ Preserves OpenAI-compatible prefixes (`file-`, `batch_`)
|
||||
- ✅ Is transparent to clients
|
||||
- ✅ Enables automatic routing without additional parameters
|
||||
- ✅ Works across all batch and file endpoints
|
||||
|
||||
### Supported Endpoints
|
||||
|
||||
All batch and file endpoints support model-based routing:
|
||||
|
||||
| Endpoint | Method | Model Routing |
|
||||
|----------|--------|---------------|
|
||||
| `/v1/files` | POST | ✅ Via header/query/body |
|
||||
| `/v1/files/{file_id}` | GET | ✅ Auto from encoded ID + header/query |
|
||||
| `/v1/files/{file_id}/content` | GET | ✅ Auto from encoded ID + header/query |
|
||||
| `/v1/files/{file_id}` | DELETE | ✅ Auto from encoded ID |
|
||||
| `/v1/batches` | POST | ✅ Auto from file ID + header/query/body |
|
||||
| `/v1/batches` | GET | ✅ Via header/query |
|
||||
| `/v1/batches/{batch_id}` | GET | ✅ Auto from encoded ID |
|
||||
| `/v1/batches/{batch_id}/cancel` | POST | ✅ Auto from encoded ID |
|
||||
|
||||
## **Supported Providers**:
|
||||
### [Azure OpenAI](./providers/azure#azure-batches-api)
|
||||
### [OpenAI](#quick-start)
|
||||
|
||||
@@ -16,7 +16,137 @@ Use this to call the provider's `/files` endpoints directly, in the OpenAI forma
|
||||
- Delete File
|
||||
- Get File Content
|
||||
|
||||
## Multi-Account Support (Multiple OpenAI Keys)
|
||||
|
||||
Use different OpenAI API keys for files and batches by specifying a `model` parameter that references entries in your `model_list`. This approach works **without requiring a database** and allows you to route files/batches to different OpenAI accounts.
|
||||
|
||||
### How It Works
|
||||
|
||||
1. Define models in `model_list` with different API keys
|
||||
2. Pass `model` parameter when creating files
|
||||
3. LiteLLM returns encoded IDs that contain routing information
|
||||
4. Use encoded IDs for all subsequent operations (retrieve, delete, batches)
|
||||
5. No need to specify model again - routing info is in the ID
|
||||
|
||||
### Setup
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
# litellm OpenAI Account
|
||||
- model_name: "gpt-4o-litellm"
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: os.environ/OPENAI_LITELLM_API_KEY
|
||||
|
||||
# Free OpenAI Account
|
||||
- model_name: "gpt-4o-free"
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: os.environ/OPENAI_FREE_API_KEY
|
||||
```
|
||||
|
||||
### Usage Example
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="sk-1234", # Your LiteLLM proxy key
|
||||
base_url="http://0.0.0.0:4000"
|
||||
)
|
||||
|
||||
# Create file using litellm account
|
||||
file_response = client.files.create(
|
||||
file=open("batch_data.jsonl", "rb"),
|
||||
purpose="batch",
|
||||
extra_body={"model": "gpt-4o-litellm"} # Routes to litellm key
|
||||
)
|
||||
print(f"File ID: {file_response.id}")
|
||||
# Returns encoded ID like: file-bGl0ZWxsbTpmaWxlLWFiYzEyMzttb2RlbCxncHQtNG8taWZvb2Q
|
||||
|
||||
# Create batch using the encoded file ID
|
||||
# No need to specify model again - it's embedded in the file ID
|
||||
batch_response = client.batches.create(
|
||||
input_file_id=file_response.id, # Encoded ID
|
||||
endpoint="/v1/chat/completions",
|
||||
completion_window="24h"
|
||||
)
|
||||
print(f"Batch ID: {batch_response.id}")
|
||||
# Returns encoded batch ID with routing info
|
||||
|
||||
# Retrieve batch - routing happens automatically
|
||||
batch_status = client.batches.retrieve(batch_response.id)
|
||||
print(f"Status: {batch_status.status}")
|
||||
|
||||
# List files for a specific account
|
||||
files = client.files.list(
|
||||
extra_body={"model": "gpt-4o-free"} # List free files
|
||||
)
|
||||
|
||||
# List batches for a specific account
|
||||
batches = client.batches.list(
|
||||
extra_query={"model": "gpt-4o-litellm"} # List litellm batches
|
||||
)
|
||||
```
|
||||
|
||||
### Parameter Options
|
||||
|
||||
You can pass the `model` parameter via:
|
||||
- **Request body**: `extra_body={"model": "gpt-4o-litellm"}`
|
||||
- **Query parameter**: `?model=gpt-4o-litellm`
|
||||
- **Header**: `x-litellm-model: gpt-4o-litellm`
|
||||
|
||||
### How Encoded IDs Work
|
||||
|
||||
- When you create a file/batch with a `model` parameter, LiteLLM encodes the model name into the returned ID
|
||||
- The encoded ID is base64-encoded and looks like: `file-bGl0ZWxsbTpmaWxlLWFiYzEyMzttb2RlbCxncHQtNG8taWZvb2Q`
|
||||
- When you use this ID in subsequent operations (retrieve, delete, batch create), LiteLLM automatically:
|
||||
1. Decodes the ID
|
||||
2. Extracts the model name
|
||||
3. Looks up the credentials
|
||||
4. Routes the request to the correct OpenAI account
|
||||
- The original provider file/batch ID is preserved internally
|
||||
|
||||
### Benefits
|
||||
|
||||
✅ **No Database Required** - All routing info stored in the ID
|
||||
✅ **Stateless** - Works across proxy restarts
|
||||
✅ **Simple** - Just pass the ID around like normal
|
||||
✅ **Backward Compatible** - Existing `custom_llm_provider` and `files_settings` still work
|
||||
✅ **Future-Proof** - Aligns with managed batches approach
|
||||
|
||||
### Migration from files_settings
|
||||
|
||||
**Old approach (still works):**
|
||||
```yaml
|
||||
files_settings:
|
||||
- custom_llm_provider: openai
|
||||
api_key: os.environ/OPENAI_KEY
|
||||
```
|
||||
|
||||
```python
|
||||
# Had to specify provider on every call
|
||||
client.files.create(..., extra_headers={"custom-llm-provider": "openai"})
|
||||
client.files.retrieve(file_id, extra_headers={"custom-llm-provider": "openai"})
|
||||
```
|
||||
|
||||
**New approach (recommended):**
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: "gpt-4o-account1"
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: os.environ/OPENAI_KEY
|
||||
```
|
||||
|
||||
```python
|
||||
# Specify model once on create
|
||||
file = client.files.create(..., extra_body={"model": "gpt-4o-account1"})
|
||||
|
||||
# Then just use the ID - routing is automatic
|
||||
client.files.retrieve(file.id) # No need to specify account
|
||||
client.batches.create(input_file_id=file.id) # Routes correctly
|
||||
```
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="proxy" label="LiteLLM PROXY Server">
|
||||
|
||||
@@ -29,6 +29,18 @@ response = completion(
|
||||
)
|
||||
```
|
||||
|
||||
:::info Metadata passthrough (preview)
|
||||
When `litellm.enable_preview_features = True`, LiteLLM forwards only the values inside `metadata` to OpenAI.
|
||||
|
||||
```python
|
||||
completion(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "hi"}],
|
||||
metadata= {"custom_meta_key": "value"},
|
||||
)
|
||||
```
|
||||
:::
|
||||
|
||||
### Usage - LiteLLM Proxy Server
|
||||
|
||||
Here's how to call OpenAI models with the LiteLLM Proxy Server
|
||||
|
||||
@@ -93,6 +93,35 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
|
||||
choice = Choices(message=msg, finish_reason="stop", index=index)
|
||||
return choice, index + 1
|
||||
|
||||
# Handle function_call items (e.g., from GPT-5 Codex format)
|
||||
if item_type == "function_call":
|
||||
# Extract provider_specific_fields if present and pass through as-is
|
||||
provider_specific_fields = item.get("provider_specific_fields")
|
||||
if provider_specific_fields and not isinstance(provider_specific_fields, dict):
|
||||
provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {}
|
||||
|
||||
tool_call_dict = {
|
||||
"id": item.get("call_id") or item.get("id", ""),
|
||||
"function": {
|
||||
"name": item.get("name", ""),
|
||||
"arguments": item.get("arguments", ""),
|
||||
},
|
||||
"type": "function",
|
||||
}
|
||||
|
||||
# Pass through provider_specific_fields as-is if present
|
||||
if provider_specific_fields:
|
||||
tool_call_dict["provider_specific_fields"] = provider_specific_fields
|
||||
# Also add to function's provider_specific_fields for consistency
|
||||
tool_call_dict["function"]["provider_specific_fields"] = provider_specific_fields
|
||||
|
||||
msg = Message(
|
||||
content=None,
|
||||
tool_calls=[tool_call_dict],
|
||||
)
|
||||
choice = Choices(message=msg, finish_reason="tool_calls", index=index)
|
||||
return choice, index + 1
|
||||
|
||||
# Unknown or unsupported type
|
||||
return None, index
|
||||
|
||||
@@ -257,7 +286,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
|
||||
|
||||
return request_data
|
||||
|
||||
def transform_response(
|
||||
def transform_response( # noqa: PLR0915
|
||||
self,
|
||||
model: str,
|
||||
raw_response: "BaseModel",
|
||||
@@ -321,18 +350,37 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
|
||||
reasoning_content = None # flush reasoning content
|
||||
index += 1
|
||||
elif isinstance(item, ResponseFunctionToolCall):
|
||||
|
||||
provider_specific_fields = None
|
||||
if hasattr(item, "provider_specific_fields") and item.provider_specific_fields:
|
||||
provider_specific_fields = item.provider_specific_fields
|
||||
if not isinstance(provider_specific_fields, dict):
|
||||
provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {}
|
||||
elif hasattr(item, "get") and callable(item.get):
|
||||
provider_fields = item.get("provider_specific_fields")
|
||||
if provider_fields:
|
||||
provider_specific_fields = provider_fields if isinstance(provider_fields, dict) else (dict(provider_fields) if hasattr(provider_fields, "__dict__") else {})
|
||||
|
||||
function_dict: Dict[str, Any] = {
|
||||
"name": item.name,
|
||||
"arguments": item.arguments,
|
||||
}
|
||||
|
||||
if provider_specific_fields:
|
||||
function_dict["provider_specific_fields"] = provider_specific_fields
|
||||
|
||||
tool_call_dict: Dict[str, Any] = {
|
||||
"id": item.call_id,
|
||||
"function": function_dict,
|
||||
"type": "function",
|
||||
}
|
||||
|
||||
if provider_specific_fields:
|
||||
tool_call_dict["provider_specific_fields"] = provider_specific_fields
|
||||
|
||||
msg = Message(
|
||||
content=None,
|
||||
tool_calls=[
|
||||
{
|
||||
"id": item.call_id,
|
||||
"function": {
|
||||
"name": item.name,
|
||||
"arguments": item.arguments,
|
||||
},
|
||||
"type": "function",
|
||||
}
|
||||
],
|
||||
tool_calls=[tool_call_dict],
|
||||
reasoning_content=reasoning_content,
|
||||
)
|
||||
|
||||
@@ -630,7 +678,7 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
|
||||
|
||||
return self.chunk_parser(json.loads(str_line))
|
||||
|
||||
def chunk_parser(
|
||||
def chunk_parser( # noqa: PLR0915
|
||||
self, chunk: dict
|
||||
) -> Union["GenericStreamingChunk", "ModelResponseStream"]:
|
||||
# Transform responses API streaming chunk to chat completion format
|
||||
@@ -667,17 +715,33 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
|
||||
# New output item added
|
||||
output_item = parsed_chunk.get("item", {})
|
||||
if output_item.get("type") == "function_call":
|
||||
# Extract provider_specific_fields if present
|
||||
provider_specific_fields = output_item.get("provider_specific_fields")
|
||||
if provider_specific_fields and not isinstance(provider_specific_fields, dict):
|
||||
provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {}
|
||||
|
||||
function_chunk = ChatCompletionToolCallFunctionChunk(
|
||||
name=output_item.get("name", None),
|
||||
arguments=parsed_chunk.get("arguments", ""),
|
||||
)
|
||||
|
||||
if provider_specific_fields:
|
||||
function_chunk["provider_specific_fields"] = provider_specific_fields
|
||||
|
||||
tool_call_chunk = ChatCompletionToolCallChunk(
|
||||
id=output_item.get("call_id"),
|
||||
index=0,
|
||||
type="function",
|
||||
function=function_chunk,
|
||||
)
|
||||
|
||||
# Add provider_specific_fields if present
|
||||
if provider_specific_fields:
|
||||
tool_call_chunk.provider_specific_fields = provider_specific_fields # type: ignore
|
||||
|
||||
return GenericStreamingChunk(
|
||||
text="",
|
||||
tool_use=ChatCompletionToolCallChunk(
|
||||
id=output_item.get("call_id"),
|
||||
index=0,
|
||||
type="function",
|
||||
function=ChatCompletionToolCallFunctionChunk(
|
||||
name=output_item.get("name", None),
|
||||
arguments=parsed_chunk.get("arguments", ""),
|
||||
),
|
||||
),
|
||||
tool_use=tool_call_chunk,
|
||||
is_finished=False,
|
||||
finish_reason="",
|
||||
usage=None,
|
||||
@@ -713,17 +777,34 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
|
||||
# New output item added
|
||||
output_item = parsed_chunk.get("item", {})
|
||||
if output_item.get("type") == "function_call":
|
||||
# Extract provider_specific_fields if present
|
||||
provider_specific_fields = output_item.get("provider_specific_fields")
|
||||
if provider_specific_fields and not isinstance(provider_specific_fields, dict):
|
||||
provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {}
|
||||
|
||||
function_chunk = ChatCompletionToolCallFunctionChunk(
|
||||
name=output_item.get("name", None),
|
||||
arguments="", # responses API sends everything again, we don't
|
||||
)
|
||||
|
||||
# Add provider_specific_fields to function if present
|
||||
if provider_specific_fields:
|
||||
function_chunk["provider_specific_fields"] = provider_specific_fields
|
||||
|
||||
tool_call_chunk = ChatCompletionToolCallChunk(
|
||||
id=output_item.get("call_id"),
|
||||
index=0,
|
||||
type="function",
|
||||
function=function_chunk,
|
||||
)
|
||||
|
||||
# Add provider_specific_fields if present
|
||||
if provider_specific_fields:
|
||||
tool_call_chunk.provider_specific_fields = provider_specific_fields # type: ignore
|
||||
|
||||
return GenericStreamingChunk(
|
||||
text="",
|
||||
tool_use=ChatCompletionToolCallChunk(
|
||||
id=output_item.get("call_id"),
|
||||
index=0,
|
||||
type="function",
|
||||
function=ChatCompletionToolCallFunctionChunk(
|
||||
name=output_item.get("name", None),
|
||||
arguments="", # responses API sends everything again, we don't
|
||||
),
|
||||
),
|
||||
tool_use=tool_call_chunk,
|
||||
is_finished=True,
|
||||
finish_reason="tool_calls",
|
||||
usage=None,
|
||||
|
||||
@@ -6,7 +6,7 @@ import mimetypes
|
||||
import re
|
||||
import xml.etree.ElementTree as ET
|
||||
from enum import Enum
|
||||
from typing import Any, List, Optional, Tuple, cast, overload
|
||||
from typing import Any, List, Optional, Tuple, Union, cast, overload
|
||||
|
||||
from jinja2.sandbox import ImmutableSandboxedEnvironment
|
||||
|
||||
@@ -910,6 +910,64 @@ def convert_to_anthropic_image_obj(
|
||||
)
|
||||
|
||||
|
||||
def create_anthropic_image_param(
|
||||
image_url_input: Union[str, dict],
|
||||
format: Optional[str] = None,
|
||||
is_bedrock_invoke: bool = False
|
||||
) -> AnthropicMessagesImageParam:
|
||||
"""
|
||||
Create an AnthropicMessagesImageParam from an image URL input.
|
||||
|
||||
Supports both URL references (for HTTP/HTTPS URLs) and base64 encoding.
|
||||
"""
|
||||
# Extract URL and format from input
|
||||
if isinstance(image_url_input, str):
|
||||
image_url = image_url_input
|
||||
else:
|
||||
image_url = image_url_input.get("url", "")
|
||||
if format is None:
|
||||
format = image_url_input.get("format")
|
||||
|
||||
# Check if the image URL is an HTTP/HTTPS URL
|
||||
if image_url.startswith("http://") or image_url.startswith("https://"):
|
||||
# For Bedrock invoke, always convert URLs to base64 (Bedrock invoke doesn't support URLs)
|
||||
if is_bedrock_invoke or image_url.startswith("http://"):
|
||||
base64_url = convert_url_to_base64(url=image_url)
|
||||
image_chunk = convert_to_anthropic_image_obj(
|
||||
openai_image_url=base64_url, format=format
|
||||
)
|
||||
return AnthropicMessagesImageParam(
|
||||
type="image",
|
||||
source=AnthropicContentParamSource(
|
||||
type="base64",
|
||||
media_type=image_chunk["media_type"],
|
||||
data=image_chunk["data"],
|
||||
),
|
||||
)
|
||||
else:
|
||||
# HTTPS URL - pass directly for regular Anthropic
|
||||
return AnthropicMessagesImageParam(
|
||||
type="image",
|
||||
source=AnthropicContentParamSourceUrl(
|
||||
type="url",
|
||||
url=image_url,
|
||||
),
|
||||
)
|
||||
else:
|
||||
# Convert to base64 for data URIs or other formats
|
||||
image_chunk = convert_to_anthropic_image_obj(
|
||||
openai_image_url=image_url, format=format
|
||||
)
|
||||
return AnthropicMessagesImageParam(
|
||||
type="image",
|
||||
source=AnthropicContentParamSource(
|
||||
type="base64",
|
||||
media_type=image_chunk["media_type"],
|
||||
data=image_chunk["data"],
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
# The following XML functions will be deprecated once JSON schema support is available on Bedrock and Vertex
|
||||
# ------------------------------------------------------------------------------
|
||||
def convert_to_anthropic_tool_result_xml(message: dict) -> str:
|
||||
@@ -1012,15 +1070,35 @@ def anthropic_messages_pt_xml(messages: list):
|
||||
if isinstance(messages[msg_i]["content"], list):
|
||||
for m in messages[msg_i]["content"]:
|
||||
if m.get("type", "") == "image_url":
|
||||
format = m["image_url"].get("format")
|
||||
user_content.append(
|
||||
{
|
||||
"type": "image",
|
||||
"source": convert_to_anthropic_image_obj(
|
||||
m["image_url"]["url"], format=format
|
||||
),
|
||||
}
|
||||
)
|
||||
format = m["image_url"].get("format") if isinstance(m["image_url"], dict) else None
|
||||
image_param = create_anthropic_image_param(m["image_url"], format=format)
|
||||
# Convert to dict format for XML version
|
||||
source = image_param["source"]
|
||||
if isinstance(source, dict) and source.get("type") == "url":
|
||||
# Type narrowing for URL source
|
||||
url_source = cast(AnthropicContentParamSourceUrl, source)
|
||||
user_content.append(
|
||||
{
|
||||
"type": "image",
|
||||
"source": {
|
||||
"type": "url",
|
||||
"url": url_source["url"],
|
||||
},
|
||||
}
|
||||
)
|
||||
else:
|
||||
# Type narrowing for base64 source
|
||||
base64_source = cast(AnthropicContentParamSource, source)
|
||||
user_content.append(
|
||||
{
|
||||
"type": "image",
|
||||
"source": {
|
||||
"type": "base64",
|
||||
"media_type": base64_source["media_type"],
|
||||
"data": base64_source["data"],
|
||||
},
|
||||
}
|
||||
)
|
||||
elif m.get("type", "") == "text":
|
||||
user_content.append({"type": "text", "text": m["text"]})
|
||||
else:
|
||||
@@ -1491,24 +1569,9 @@ def convert_to_anthropic_tool_result(
|
||||
)
|
||||
)
|
||||
elif content["type"] == "image_url":
|
||||
if isinstance(content["image_url"], str):
|
||||
image_chunk = convert_to_anthropic_image_obj(
|
||||
content["image_url"], format=None
|
||||
)
|
||||
else:
|
||||
format = content["image_url"].get("format")
|
||||
image_chunk = convert_to_anthropic_image_obj(
|
||||
content["image_url"]["url"], format=format
|
||||
)
|
||||
format = content["image_url"].get("format") if isinstance(content["image_url"], dict) else None
|
||||
anthropic_content_list.append(
|
||||
AnthropicMessagesImageParam(
|
||||
type="image",
|
||||
source=AnthropicContentParamSource(
|
||||
type="base64",
|
||||
media_type=image_chunk["media_type"],
|
||||
data=image_chunk["data"],
|
||||
),
|
||||
)
|
||||
create_anthropic_image_param(content["image_url"], format=format)
|
||||
)
|
||||
|
||||
anthropic_content = anthropic_content_list
|
||||
@@ -1839,21 +1902,22 @@ def anthropic_messages_pt( # noqa: PLR0915
|
||||
for m in user_message_types_block["content"]:
|
||||
if m.get("type", "") == "image_url":
|
||||
m = cast(ChatCompletionImageObject, m)
|
||||
format: Optional[str] = None
|
||||
if isinstance(m["image_url"], str):
|
||||
image_chunk = convert_to_anthropic_image_obj(
|
||||
openai_image_url=m["image_url"], format=None
|
||||
)
|
||||
format = m["image_url"].get("format") if isinstance(m["image_url"], dict) else None
|
||||
# Convert ChatCompletionImageUrlObject to dict if needed
|
||||
image_url_value = m["image_url"]
|
||||
if isinstance(image_url_value, str):
|
||||
image_url_input: Union[str, dict[str, Any]] = image_url_value
|
||||
else:
|
||||
format = m["image_url"].get("format")
|
||||
image_chunk = convert_to_anthropic_image_obj(
|
||||
openai_image_url=m["image_url"]["url"],
|
||||
format=format,
|
||||
)
|
||||
|
||||
_anthropic_content_element = (
|
||||
_anthropic_content_element_factory(image_chunk)
|
||||
)
|
||||
# ChatCompletionImageUrlObject or dict case - convert to dict
|
||||
image_url_input = {
|
||||
"url": image_url_value["url"],
|
||||
"format": image_url_value.get("format"),
|
||||
}
|
||||
# Bedrock invoke models have format: invoke/...
|
||||
is_bedrock_invoke = model.lower().startswith("invoke/")
|
||||
_anthropic_content_element = create_anthropic_image_param(
|
||||
image_url_input, format=format, is_bedrock_invoke=is_bedrock_invoke
|
||||
)
|
||||
_content_element = add_cache_control_to_content(
|
||||
anthropic_content_element=_anthropic_content_element,
|
||||
original_content_element=dict(m),
|
||||
|
||||
@@ -237,6 +237,9 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
|
||||
return False
|
||||
|
||||
def _supports_penalty_parameters(self, model: str) -> bool:
|
||||
# Gemini 3 models do not support penalty parameters
|
||||
if VertexGeminiConfig._is_gemini_3_or_newer(model):
|
||||
return False
|
||||
unsupported_models = ["gemini-2.5-pro-preview-06-05"]
|
||||
if model in unsupported_models:
|
||||
return False
|
||||
@@ -1344,7 +1347,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def _calculate_usage(
|
||||
def _calculate_usage( # noqa: PLR0915
|
||||
completion_response: Union[
|
||||
GenerateContentResponseBody, BidiGenerateContentServerMessage
|
||||
],
|
||||
|
||||
+7
-5
@@ -105,7 +105,7 @@ from litellm.utils import (
|
||||
ProviderConfigManager,
|
||||
Usage,
|
||||
_get_model_info_helper,
|
||||
add_openai_metadata,
|
||||
get_requester_metadata,
|
||||
add_provider_specific_params_to_optional_params,
|
||||
async_mock_completion_streaming_obj,
|
||||
convert_to_model_response_object,
|
||||
@@ -2086,10 +2086,12 @@ def completion( # type: ignore # noqa: PLR0915
|
||||
if extra_headers is not None:
|
||||
optional_params["extra_headers"] = extra_headers
|
||||
|
||||
if litellm.enable_preview_features:
|
||||
metadata_payload = add_openai_metadata(metadata)
|
||||
if metadata_payload is not None:
|
||||
optional_params["metadata"] = metadata_payload
|
||||
if (
|
||||
litellm.enable_preview_features and metadata is not None
|
||||
): # [PREVIEW] allow metadata to be passed to OPENAI
|
||||
openai_metadata = get_requester_metadata(metadata)
|
||||
if openai_metadata is not None:
|
||||
optional_params["metadata"] = openai_metadata
|
||||
|
||||
## LOAD CONFIG - if set
|
||||
config = litellm.OpenAIConfig.get_config()
|
||||
|
||||
@@ -22,10 +22,16 @@ from litellm.proxy.common_utils.openai_endpoint_utils import (
|
||||
)
|
||||
from litellm.proxy.openai_files_endpoints.common_utils import (
|
||||
_is_base64_encoded_unified_file_id,
|
||||
decode_model_from_file_id,
|
||||
encode_file_id_with_model,
|
||||
get_batch_id_from_unified_batch_id,
|
||||
get_credentials_for_model,
|
||||
get_model_id_from_unified_batch_id,
|
||||
get_models_from_unified_file_id,
|
||||
get_original_file_id,
|
||||
prepare_data_with_credentials,
|
||||
)
|
||||
|
||||
from litellm.proxy.utils import handle_exception_on_proxy, is_known_model
|
||||
from litellm.types.llms.openai import LiteLLMBatchCreateRequest
|
||||
|
||||
@@ -47,7 +53,7 @@ router = APIRouter()
|
||||
dependencies=[Depends(user_api_key_auth)],
|
||||
tags=["batch"],
|
||||
)
|
||||
async def create_batch(
|
||||
async def create_batch( # noqa: PLR0915
|
||||
request: Request,
|
||||
fastapi_response: Response,
|
||||
provider: Optional[str] = None,
|
||||
@@ -111,9 +117,59 @@ async def create_batch(
|
||||
_create_batch_data = LiteLLMBatchCreateRequest(**data)
|
||||
input_file_id = _create_batch_data.get("input_file_id", None)
|
||||
unified_file_id: Union[str, Literal[False]] = False
|
||||
|
||||
model_from_file_id = None
|
||||
if input_file_id:
|
||||
model_from_file_id = decode_model_from_file_id(input_file_id)
|
||||
unified_file_id = _is_base64_encoded_unified_file_id(input_file_id)
|
||||
if (
|
||||
|
||||
# SCENARIO 1: File ID is encoded with model info
|
||||
if model_from_file_id is not None and input_file_id:
|
||||
credentials = get_credentials_for_model(
|
||||
llm_router=llm_router,
|
||||
model_id=model_from_file_id,
|
||||
operation_context="batch creation (file created with model)",
|
||||
)
|
||||
|
||||
original_file_id = get_original_file_id(input_file_id)
|
||||
_create_batch_data["input_file_id"] = original_file_id
|
||||
prepare_data_with_credentials(
|
||||
data=_create_batch_data, # type: ignore
|
||||
credentials=credentials,
|
||||
)
|
||||
|
||||
# Create batch using model credentials
|
||||
response = await litellm.acreate_batch(
|
||||
custom_llm_provider=credentials["custom_llm_provider"],
|
||||
**_create_batch_data # type: ignore
|
||||
)
|
||||
|
||||
# Encode the batch ID and related file IDs with model information
|
||||
if response and hasattr(response, "id") and response.id:
|
||||
original_batch_id = response.id
|
||||
encoded_batch_id = encode_file_id_with_model(
|
||||
file_id=original_batch_id, model=model_from_file_id
|
||||
)
|
||||
response.id = encoded_batch_id
|
||||
|
||||
if hasattr(response, "output_file_id") and response.output_file_id:
|
||||
response.output_file_id = encode_file_id_with_model(
|
||||
file_id=response.output_file_id, model=model_from_file_id
|
||||
)
|
||||
|
||||
if hasattr(response, "error_file_id") and response.error_file_id:
|
||||
response.error_file_id = encode_file_id_with_model(
|
||||
file_id=response.error_file_id, model=model_from_file_id
|
||||
)
|
||||
|
||||
verbose_proxy_logger.debug(
|
||||
f"Created batch using model: {model_from_file_id}, "
|
||||
f"original_batch_id: {original_batch_id}, encoded: {encoded_batch_id}"
|
||||
)
|
||||
|
||||
response.input_file_id = input_file_id
|
||||
|
||||
elif (
|
||||
litellm.enable_loadbalancing_on_batch_endpoints is True
|
||||
and is_router_model
|
||||
and router_model is not None
|
||||
@@ -155,9 +211,39 @@ async def create_batch(
|
||||
response.input_file_id = input_file_id
|
||||
response._hidden_params["unified_file_id"] = unified_file_id
|
||||
else:
|
||||
response = await litellm.acreate_batch(
|
||||
custom_llm_provider=custom_llm_provider, **_create_batch_data # type: ignore
|
||||
# Check if model specified via header/query/body param
|
||||
model_param = (
|
||||
data.get("model")
|
||||
or request.query_params.get("model")
|
||||
or request.headers.get("x-litellm-model")
|
||||
)
|
||||
|
||||
# SCENARIO 2 & 3: Model from header/query OR custom_llm_provider fallback
|
||||
if model_param:
|
||||
# SCENARIO 2: Use model-based routing from header/query/body
|
||||
credentials = get_credentials_for_model(
|
||||
llm_router=llm_router,
|
||||
model_id=model_param,
|
||||
operation_context="batch creation",
|
||||
)
|
||||
|
||||
prepare_data_with_credentials(
|
||||
data=_create_batch_data, # type: ignore
|
||||
credentials=credentials,
|
||||
)
|
||||
|
||||
# Create batch using model credentials
|
||||
response = await litellm.acreate_batch(
|
||||
custom_llm_provider=credentials["custom_llm_provider"],
|
||||
**_create_batch_data # type: ignore
|
||||
)
|
||||
|
||||
verbose_proxy_logger.debug(f"Created batch using model: {model_param}")
|
||||
else:
|
||||
# SCENARIO 3: Fallback to custom_llm_provider (uses env variables)
|
||||
response = await litellm.acreate_batch(
|
||||
custom_llm_provider=custom_llm_provider, **_create_batch_data # type: ignore
|
||||
)
|
||||
|
||||
### CALL HOOKS ### - modify outgoing data
|
||||
response = await proxy_logging_obj.post_call_success_hook(
|
||||
@@ -249,7 +335,7 @@ async def retrieve_batch(
|
||||
|
||||
data: Dict = {}
|
||||
try:
|
||||
## check if model is a loadbalanced model
|
||||
model_from_id = decode_model_from_file_id(batch_id)
|
||||
_retrieve_batch_request = RetrieveBatchRequest(
|
||||
batch_id=batch_id,
|
||||
)
|
||||
@@ -271,7 +357,54 @@ async def retrieve_batch(
|
||||
route_type="aretrieve_batch",
|
||||
)
|
||||
|
||||
if litellm.enable_loadbalancing_on_batch_endpoints is True or unified_batch_id:
|
||||
# SCENARIO 1: Batch ID is encoded with model info
|
||||
if model_from_id is not None:
|
||||
credentials = get_credentials_for_model(
|
||||
llm_router=llm_router,
|
||||
model_id=model_from_id,
|
||||
operation_context="batch retrieval (batch created with model)",
|
||||
)
|
||||
|
||||
original_batch_id = get_original_file_id(batch_id)
|
||||
prepare_data_with_credentials(
|
||||
data=data,
|
||||
credentials=credentials,
|
||||
file_id=original_batch_id, # Sets data["batch_id"] = original_batch_id
|
||||
)
|
||||
# Fix: The helper sets "file_id" but we need "batch_id"
|
||||
data["batch_id"] = data.pop("file_id", original_batch_id)
|
||||
|
||||
# Retrieve batch using model credentials
|
||||
response = await litellm.aretrieve_batch(
|
||||
custom_llm_provider=credentials["custom_llm_provider"],
|
||||
**data # type: ignore
|
||||
)
|
||||
|
||||
# Re-encode all IDs in the response
|
||||
if response:
|
||||
if hasattr(response, "id") and response.id:
|
||||
response.id = batch_id # Keep the encoded batch ID
|
||||
|
||||
if hasattr(response, "input_file_id") and response.input_file_id:
|
||||
response.input_file_id = encode_file_id_with_model(
|
||||
file_id=response.input_file_id, model=model_from_id
|
||||
)
|
||||
|
||||
if hasattr(response, "output_file_id") and response.output_file_id:
|
||||
response.output_file_id = encode_file_id_with_model(
|
||||
file_id=response.output_file_id, model=model_from_id
|
||||
)
|
||||
|
||||
if hasattr(response, "error_file_id") and response.error_file_id:
|
||||
response.error_file_id = encode_file_id_with_model(
|
||||
file_id=response.error_file_id, model=model_from_id
|
||||
)
|
||||
|
||||
verbose_proxy_logger.debug(
|
||||
f"Retrieved batch using model: {model_from_id}, original_id: {original_batch_id}"
|
||||
)
|
||||
|
||||
elif litellm.enable_loadbalancing_on_batch_endpoints is True or unified_batch_id:
|
||||
if llm_router is None:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
@@ -282,6 +415,8 @@ async def retrieve_batch(
|
||||
|
||||
response = await llm_router.aretrieve_batch(**data) # type: ignore
|
||||
response._hidden_params["unified_batch_id"] = unified_batch_id
|
||||
|
||||
# SCENARIO 3: Fallback to custom_llm_provider (uses env variables)
|
||||
else:
|
||||
custom_llm_provider = (
|
||||
provider
|
||||
@@ -405,9 +540,36 @@ async def list_batches(
|
||||
route_type="alist_batches",
|
||||
)
|
||||
|
||||
## check for target model names
|
||||
target_model_names = target_model_names or data.get("target_model_names", None)
|
||||
if target_model_names:
|
||||
model_param = (
|
||||
data.get("model")
|
||||
or request.query_params.get("model")
|
||||
or request.headers.get("x-litellm-model")
|
||||
)
|
||||
|
||||
# SCENARIO 2: Use model-based routing from header/query/body
|
||||
if model_param:
|
||||
credentials = get_credentials_for_model(
|
||||
llm_router=llm_router,
|
||||
model_id=model_param,
|
||||
operation_context="batch listing",
|
||||
)
|
||||
|
||||
data.update(credentials)
|
||||
|
||||
response = await litellm.alist_batches(
|
||||
custom_llm_provider=credentials["custom_llm_provider"],
|
||||
after=after,
|
||||
limit=limit,
|
||||
**data # type: ignore
|
||||
)
|
||||
|
||||
verbose_proxy_logger.debug(f"Listed batches using model: {model_param}")
|
||||
|
||||
# SCENARIO 2 (alternative): target_model_names based routing
|
||||
elif target_model_names or data.get("target_model_names", None):
|
||||
target_model_names = target_model_names or data.get("target_model_names", None)
|
||||
if target_model_names is None:
|
||||
raise ValueError("target_model_names is required for this routing scenario")
|
||||
model = target_model_names.split(",")[0]
|
||||
response = await llm_router.alist_batches(
|
||||
model=model,
|
||||
@@ -415,6 +577,8 @@ async def list_batches(
|
||||
limit=limit,
|
||||
**data,
|
||||
)
|
||||
|
||||
# SCENARIO 3: Fallback to custom_llm_provider (uses env variables)
|
||||
else:
|
||||
custom_llm_provider = (
|
||||
provider
|
||||
@@ -520,6 +684,9 @@ async def cancel_batch(
|
||||
verbose_proxy_logger.debug(
|
||||
"Request received by LiteLLM:\n{}".format(json.dumps(data, indent=4)),
|
||||
)
|
||||
|
||||
# Check for encoded batch ID with model info
|
||||
model_from_id = decode_model_from_file_id(batch_id)
|
||||
unified_batch_id = _is_base64_encoded_unified_file_id(batch_id)
|
||||
|
||||
# Include original request and headers in the data
|
||||
@@ -532,7 +699,35 @@ async def cancel_batch(
|
||||
proxy_config=proxy_config,
|
||||
)
|
||||
|
||||
if unified_batch_id:
|
||||
# SCENARIO 1: Batch ID is encoded with model info
|
||||
if model_from_id is not None:
|
||||
credentials = get_credentials_for_model(
|
||||
llm_router=llm_router,
|
||||
model_id=model_from_id,
|
||||
operation_context="batch cancellation (batch created with model)",
|
||||
)
|
||||
|
||||
original_batch_id = get_original_file_id(batch_id)
|
||||
prepare_data_with_credentials(
|
||||
data=data,
|
||||
credentials=credentials,
|
||||
file_id=original_batch_id,
|
||||
)
|
||||
# Fix: The helper sets "file_id" but we need "batch_id"
|
||||
data["batch_id"] = data.pop("file_id", original_batch_id)
|
||||
|
||||
# Cancel batch using model credentials
|
||||
response = await litellm.acancel_batch(
|
||||
custom_llm_provider=credentials["custom_llm_provider"],
|
||||
**data # type: ignore
|
||||
)
|
||||
|
||||
verbose_proxy_logger.debug(
|
||||
f"Cancelled batch using model: {model_from_id}, original_id: {original_batch_id}"
|
||||
)
|
||||
|
||||
# SCENARIO 2: target_model_names based routing
|
||||
elif unified_batch_id:
|
||||
if llm_router is None:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
@@ -552,6 +747,8 @@ async def cancel_batch(
|
||||
data["batch_id"] = model_batch_id
|
||||
|
||||
response = await llm_router.acancel_batch(model=model, **data) # type: ignore
|
||||
|
||||
# SCENARIO 3: Fallback to custom_llm_provider (uses env variables)
|
||||
else:
|
||||
|
||||
custom_llm_provider = (
|
||||
|
||||
@@ -894,6 +894,22 @@ async def add_litellm_data_to_request( # noqa: PLR0915
|
||||
data["metadata"]
|
||||
)
|
||||
|
||||
# Parse litellm_metadata if it's a string (e.g., from multipart/form-data or extra_body)
|
||||
if "litellm_metadata" in data and data["litellm_metadata"] is not None:
|
||||
if isinstance(data["litellm_metadata"], str):
|
||||
parsed_litellm_metadata = safe_json_loads(data["litellm_metadata"])
|
||||
if not isinstance(parsed_litellm_metadata, dict):
|
||||
verbose_proxy_logger.warning(
|
||||
f"Failed to parse 'litellm_metadata' as JSON dict. Received value: {data['litellm_metadata']}"
|
||||
)
|
||||
else:
|
||||
data["litellm_metadata"] = parsed_litellm_metadata
|
||||
# Merge litellm_metadata into the metadata variable (preserving existing values)
|
||||
if isinstance(data["litellm_metadata"], dict):
|
||||
for key, value in data["litellm_metadata"].items():
|
||||
if key not in data[_metadata_variable_name]:
|
||||
data[_metadata_variable_name][key] = value
|
||||
|
||||
data = LiteLLMProxyRequestSetup.add_user_api_key_auth_to_request_metadata(
|
||||
data=data,
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
|
||||
@@ -76,3 +76,266 @@ def get_batch_id_from_unified_batch_id(file_id: str) -> str:
|
||||
return file_id.split("llm_batch_id:")[1].split(",")[0]
|
||||
else:
|
||||
return file_id.split("generic_response_id:")[1].split(",")[0]
|
||||
|
||||
|
||||
def encode_file_id_with_model(file_id: str, model: str) -> str:
|
||||
"""
|
||||
Encode a file/batch ID with model routing information.
|
||||
|
||||
Format: <prefix>-<base64(litellm:<original_id>;model,<model_name>)>
|
||||
The result preserves the original prefix (file-, batch_, etc.) for OpenAI compliance.
|
||||
|
||||
Args:
|
||||
file_id: Original file/batch ID from the provider (e.g., "file-abc123", "batch_xyz")
|
||||
model: Model name from model_list (e.g., "gpt-4o-litellm")
|
||||
|
||||
Returns:
|
||||
Encoded ID starting with appropriate prefix and containing routing information
|
||||
|
||||
Examples:
|
||||
encode_file_id_with_model("file-abc123", "gpt-4o-litellm")
|
||||
-> "file-bGl0ZWxsbTpmaWxlLWFiYzEyMzttb2RlbCxncHQtNG8taWZvb2Q"
|
||||
|
||||
encode_file_id_with_model("batch_abc123", "gpt-4o-test")
|
||||
-> "batch_bGl0ZWxsbTpiYXRjaF9hYmMxMjM7bW9kZWwsZ3B0LTRvLXRlc3Q"
|
||||
"""
|
||||
encoded_str = f"litellm:{file_id};model,{model}"
|
||||
encoded_bytes = base64.urlsafe_b64encode(encoded_str.encode())
|
||||
encoded_b64 = encoded_bytes.decode().rstrip("=")
|
||||
|
||||
# Detect the prefix from the original ID (file-, batch_, etc.)
|
||||
# Default to "file-" if no recognizable prefix
|
||||
if file_id.startswith("batch_"):
|
||||
prefix = "batch_"
|
||||
elif file_id.startswith("file-"):
|
||||
prefix = "file-"
|
||||
else:
|
||||
# Default to file- for backward compatibility
|
||||
prefix = "file-"
|
||||
|
||||
return f"{prefix}{encoded_b64}"
|
||||
|
||||
|
||||
def decode_model_from_file_id(encoded_id: str) -> Optional[str]:
|
||||
"""
|
||||
Extract model name from an encoded file/batch ID.
|
||||
Handles IDs that start with "file-" or "batch_" prefix.
|
||||
"""
|
||||
try:
|
||||
if not isinstance(encoded_id, str):
|
||||
return None
|
||||
|
||||
# Remove prefix if present (file-, batch_, etc.)
|
||||
if encoded_id.startswith("file-"):
|
||||
b64_part = encoded_id[5:] # Remove "file-"
|
||||
elif encoded_id.startswith("batch_"):
|
||||
b64_part = encoded_id[6:] # Remove "batch_"
|
||||
else:
|
||||
b64_part = encoded_id
|
||||
|
||||
padded = b64_part + "=" * (-len(b64_part) % 4)
|
||||
decoded = base64.urlsafe_b64decode(padded).decode()
|
||||
if decoded.startswith("litellm:") and ";model," in decoded:
|
||||
match = re.search(r";model,([^;]+)", decoded)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
|
||||
return None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def get_original_file_id(encoded_id: str) -> str:
|
||||
"""
|
||||
Extract the original provider file/batch ID from an encoded ID.
|
||||
Handles IDs that start with "file-" or "batch_" prefix.
|
||||
"""
|
||||
try:
|
||||
if not isinstance(encoded_id, str):
|
||||
return encoded_id
|
||||
|
||||
# Remove prefix if present (file-, batch_, etc.)
|
||||
if encoded_id.startswith("file-"):
|
||||
b64_part = encoded_id[5:] # Remove "file-"
|
||||
elif encoded_id.startswith("batch_"):
|
||||
b64_part = encoded_id[6:] # Remove "batch_"
|
||||
else:
|
||||
b64_part = encoded_id
|
||||
|
||||
padded = b64_part + "=" * (-len(b64_part) % 4)
|
||||
decoded = base64.urlsafe_b64decode(padded).decode()
|
||||
|
||||
if decoded.startswith("litellm:") and ";model," in decoded:
|
||||
match = re.search(r"litellm:([^;]+);model,", decoded)
|
||||
if match:
|
||||
return match.group(1)
|
||||
|
||||
return encoded_id
|
||||
except Exception:
|
||||
return encoded_id
|
||||
|
||||
|
||||
def is_model_embedded_id(file_id: str) -> bool:
|
||||
"""
|
||||
Check if a file/batch ID has model routing information embedded.
|
||||
"""
|
||||
return decode_model_from_file_id(file_id) is not None
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# MODEL-BASED CREDENTIAL ROUTING HELPERS
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def extract_model_from_sources(
|
||||
file_id: str,
|
||||
request, # FastAPI Request object
|
||||
data: Optional[dict] = None,
|
||||
) -> tuple[Optional[str], Optional[str]]:
|
||||
"""
|
||||
Extract model information from multiple sources in priority order:
|
||||
1. Embedded in file_id (highest priority)
|
||||
2. Request headers (x-litellm-model)
|
||||
3. Query parameters (?model=)
|
||||
4. Request body/data dict
|
||||
|
||||
Args:
|
||||
file_id: File ID that may contain embedded model info
|
||||
request: FastAPI request object
|
||||
data: Optional request data dictionary
|
||||
|
||||
Returns:
|
||||
Tuple of (model_from_id, model_from_param)
|
||||
- model_from_id: Model decoded from file ID (if embedded)
|
||||
- model_from_param: Model from header/query/body
|
||||
"""
|
||||
if data is None:
|
||||
data = {}
|
||||
|
||||
# Check if file_id has embedded model info
|
||||
model_from_id = decode_model_from_file_id(file_id)
|
||||
|
||||
# Check other sources for model parameter
|
||||
model_from_param = (
|
||||
data.get("model")
|
||||
or request.query_params.get("model")
|
||||
or request.headers.get("x-litellm-model")
|
||||
)
|
||||
|
||||
return model_from_id, model_from_param
|
||||
|
||||
|
||||
def get_credentials_for_model(
|
||||
llm_router, # Router instance
|
||||
model_id: str,
|
||||
operation_context: str = "file operation",
|
||||
):
|
||||
"""
|
||||
Retrieve API credentials for a model from the LLM Router.
|
||||
|
||||
Args:
|
||||
llm_router: LiteLLM Router instance
|
||||
model_id: Model name or deployment ID
|
||||
operation_context: Description for error messages (e.g., "file upload", "batch creation")
|
||||
|
||||
Returns:
|
||||
Dictionary with credentials (api_key, api_base, custom_llm_provider, etc.)
|
||||
|
||||
Raises:
|
||||
HTTPException: If router not initialized or model not found
|
||||
"""
|
||||
from fastapi import HTTPException
|
||||
|
||||
if llm_router is None:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail={"error": "Router not initialized. Cannot use model-based routing."},
|
||||
)
|
||||
|
||||
credentials = llm_router.get_deployment_credentials_with_provider(model_id=model_id)
|
||||
|
||||
if credentials is None:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={
|
||||
"error": f"Model '{model_id}' not found in model_list. Please check your config.yaml."
|
||||
},
|
||||
)
|
||||
|
||||
return credentials
|
||||
|
||||
|
||||
def prepare_data_with_credentials(
|
||||
data: dict,
|
||||
credentials: dict,
|
||||
file_id: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Update data dictionary with model credentials (in-place).
|
||||
|
||||
Args:
|
||||
data: Data dictionary to update
|
||||
credentials: Credentials from router
|
||||
file_id: Optional original file_id to set (for decoded file IDs)
|
||||
"""
|
||||
data.update(credentials)
|
||||
data.pop("custom_llm_provider", None)
|
||||
|
||||
if file_id is not None:
|
||||
data["file_id"] = file_id
|
||||
|
||||
|
||||
def handle_model_based_routing(
|
||||
file_id: str,
|
||||
request, # FastAPI Request object
|
||||
llm_router, # Router instance
|
||||
data: dict,
|
||||
check_file_id_encoding: bool = True,
|
||||
) -> tuple[bool, Optional[str], Optional[str], Optional[dict]]:
|
||||
"""
|
||||
Orchestrate model-based credential routing for file operations.
|
||||
|
||||
Args:
|
||||
file_id: File ID (may contain embedded model info)
|
||||
request: FastAPI request object
|
||||
llm_router: LiteLLM Router instance
|
||||
data: Request data dictionary
|
||||
check_file_id_encoding: Whether to check for embedded model in file_id
|
||||
|
||||
Returns:
|
||||
Tuple of (should_use_model_routing, model_used, original_file_id, credentials)
|
||||
- should_use_model_routing: True if model-based routing should be used
|
||||
- model_used: The model name being used
|
||||
- original_file_id: Decoded file ID (if it was encoded)
|
||||
- credentials: Model credentials dict
|
||||
|
||||
Raises:
|
||||
HTTPException: If router unavailable or model not found
|
||||
"""
|
||||
model_from_id, model_from_param = extract_model_from_sources(
|
||||
file_id=file_id,
|
||||
request=request,
|
||||
data=data,
|
||||
)
|
||||
|
||||
# Priority 1: Model embedded in file_id
|
||||
if check_file_id_encoding and model_from_id is not None:
|
||||
credentials = get_credentials_for_model(
|
||||
llm_router=llm_router,
|
||||
model_id=model_from_id,
|
||||
operation_context=f"file operation (file created with model '{model_from_id}')",
|
||||
)
|
||||
original_file_id = get_original_file_id(file_id)
|
||||
return True, model_from_id, original_file_id, credentials
|
||||
|
||||
# Priority 2: Model from header/query/body
|
||||
elif model_from_param is not None:
|
||||
credentials = get_credentials_for_model(
|
||||
llm_router=llm_router,
|
||||
model_id=model_from_param,
|
||||
operation_context="file operation",
|
||||
)
|
||||
return True, model_from_param, None, credentials
|
||||
|
||||
# No model-based routing needed
|
||||
return False, None, None, None
|
||||
|
||||
@@ -29,6 +29,7 @@ from litellm.llms.base_llm.files.transformation import BaseFileEndpoints
|
||||
from litellm.proxy._types import *
|
||||
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
|
||||
from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing
|
||||
from litellm.proxy.common_utils.http_parsing_utils import _read_request_body
|
||||
from litellm.proxy.common_utils.openai_endpoint_utils import (
|
||||
get_custom_llm_provider_from_request_body,
|
||||
get_custom_llm_provider_from_request_headers,
|
||||
@@ -42,7 +43,13 @@ from litellm.types.llms.openai import (
|
||||
OpenAIFilesPurpose,
|
||||
)
|
||||
|
||||
from .common_utils import _is_base64_encoded_unified_file_id
|
||||
from .common_utils import (
|
||||
_is_base64_encoded_unified_file_id,
|
||||
encode_file_id_with_model,
|
||||
get_credentials_for_model,
|
||||
handle_model_based_routing,
|
||||
prepare_data_with_credentials,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -127,7 +134,51 @@ async def route_create_file(
|
||||
is_router_model: bool,
|
||||
router_model: Optional[str],
|
||||
custom_llm_provider: str,
|
||||
model: Optional[str] = None,
|
||||
) -> OpenAIFileObject:
|
||||
"""
|
||||
Route file creation request to the appropriate provider.
|
||||
|
||||
Priority:
|
||||
1. If model parameter provided -> use model credentials and encode ID
|
||||
2. If enable_loadbalancing_on_batch_endpoints -> deprecated loadbalancing
|
||||
3. If target_model_names_list -> managed files (requires DB)
|
||||
4. Else -> use custom_llm_provider with files_settings
|
||||
"""
|
||||
|
||||
# NEW: Handle model-based routing (no DB required)
|
||||
if model is not None:
|
||||
# Get credentials from model_list via router
|
||||
credentials = get_credentials_for_model(
|
||||
llm_router=llm_router,
|
||||
model_id=model,
|
||||
operation_context="file upload",
|
||||
)
|
||||
|
||||
# Merge credentials into the request
|
||||
prepare_data_with_credentials(
|
||||
data=_create_file_request, # type: ignore
|
||||
credentials=credentials,
|
||||
)
|
||||
|
||||
# Create the file with model credentials
|
||||
response = await litellm.acreate_file(
|
||||
**_create_file_request,
|
||||
custom_llm_provider=credentials["custom_llm_provider"]
|
||||
) # type: ignore
|
||||
|
||||
# Encode the file ID with model information
|
||||
if response and hasattr(response, "id") and response.id:
|
||||
original_id = response.id
|
||||
encoded_id = encode_file_id_with_model(file_id=original_id, model=model)
|
||||
response.id = encoded_id
|
||||
verbose_proxy_logger.debug(
|
||||
f"Encoded file ID: {original_id} -> {encoded_id} (model: {model})"
|
||||
)
|
||||
|
||||
return response
|
||||
|
||||
# EXISTING: Deprecated loadbalancing approach
|
||||
if (
|
||||
litellm.enable_loadbalancing_on_batch_endpoints is True
|
||||
and is_router_model
|
||||
@@ -206,6 +257,7 @@ async def create_file(
|
||||
provider: Optional[str] = None,
|
||||
custom_llm_provider: str = Form(default="openai"),
|
||||
file: UploadFile = File(...),
|
||||
litellm_metadata: Optional[str] = Form(default=None),
|
||||
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
|
||||
):
|
||||
"""
|
||||
@@ -245,6 +297,14 @@ async def create_file(
|
||||
or "openai"
|
||||
)
|
||||
|
||||
# NEW: Extract model parameter for multi-account routing
|
||||
request_body = await _read_request_body(request=request) or {}
|
||||
model_param = (
|
||||
request_body.get("model")
|
||||
or request.query_params.get("model")
|
||||
or request.headers.get("x-litellm-model")
|
||||
)
|
||||
|
||||
target_model_names_list = (
|
||||
target_model_names.split(",") if target_model_names else []
|
||||
)
|
||||
@@ -264,6 +324,10 @@ async def create_file(
|
||||
purpose = cast(OpenAIFilesPurpose, purpose)
|
||||
|
||||
data = {}
|
||||
|
||||
# Add litellm_metadata to data if provided (from form field)
|
||||
if litellm_metadata is not None:
|
||||
data["litellm_metadata"] = litellm_metadata
|
||||
|
||||
# Include original request and headers in the data
|
||||
data = await add_litellm_data_to_request(
|
||||
@@ -303,6 +367,7 @@ async def create_file(
|
||||
is_router_model=is_router_model,
|
||||
router_model=router_model,
|
||||
custom_llm_provider=custom_llm_provider,
|
||||
model=model_param,
|
||||
)
|
||||
|
||||
if response is None:
|
||||
@@ -480,13 +545,41 @@ async def get_file_content(
|
||||
}
|
||||
)
|
||||
else:
|
||||
response = await litellm.afile_content(
|
||||
**{
|
||||
"custom_llm_provider": custom_llm_provider,
|
||||
"file_id": file_id,
|
||||
**data,
|
||||
} # type: ignore
|
||||
# Check for model-based credential routing
|
||||
should_route, model_used, original_file_id, credentials = handle_model_based_routing(
|
||||
file_id=file_id,
|
||||
request=request,
|
||||
llm_router=llm_router,
|
||||
data=data,
|
||||
check_file_id_encoding=True,
|
||||
)
|
||||
|
||||
if should_route:
|
||||
# Use model-based routing with credentials from config
|
||||
prepare_data_with_credentials(
|
||||
data=data,
|
||||
credentials=credentials, # type: ignore
|
||||
file_id=original_file_id, # Use decoded file ID if from encoded ID
|
||||
)
|
||||
|
||||
response = await litellm.afile_content(
|
||||
custom_llm_provider=credentials["custom_llm_provider"], # type: ignore
|
||||
**data
|
||||
) # type: ignore
|
||||
|
||||
verbose_proxy_logger.debug(
|
||||
f"Retrieved file content using model: {model_used}"
|
||||
+ (f", file_id: {file_id} -> {original_file_id}" if original_file_id else "")
|
||||
)
|
||||
else:
|
||||
# Fallback to default behavior (uses env variables or provider-based routing)
|
||||
response = await litellm.afile_content(
|
||||
**{
|
||||
"custom_llm_provider": custom_llm_provider,
|
||||
"file_id": file_id,
|
||||
**data,
|
||||
} # type: ignore
|
||||
)
|
||||
|
||||
### ALERTING ###
|
||||
asyncio.create_task(
|
||||
@@ -618,10 +711,38 @@ async def get_file(
|
||||
route_type="afile_retrieve",
|
||||
)
|
||||
|
||||
## check if file_id is a litellm managed file
|
||||
is_base64_unified_file_id = _is_base64_encoded_unified_file_id(file_id)
|
||||
## Check for model-based credential routing
|
||||
from litellm.proxy.proxy_server import llm_router
|
||||
|
||||
should_route, model_used, original_file_id, credentials = handle_model_based_routing(
|
||||
file_id=file_id,
|
||||
request=request,
|
||||
llm_router=llm_router,
|
||||
data=data,
|
||||
check_file_id_encoding=True,
|
||||
)
|
||||
|
||||
if should_route:
|
||||
# Use model-based routing with credentials from config
|
||||
prepare_data_with_credentials(
|
||||
data=data,
|
||||
credentials=credentials, # type: ignore
|
||||
file_id=original_file_id,
|
||||
)
|
||||
|
||||
if is_base64_unified_file_id:
|
||||
response = await litellm.afile_retrieve(**data) # type: ignore
|
||||
|
||||
# Keep the encoded ID in response if it was originally encoded
|
||||
if original_file_id and response and hasattr(response, "id") and response.id:
|
||||
response.id = file_id
|
||||
|
||||
verbose_proxy_logger.debug(
|
||||
f"Retrieved file using model: {model_used}"
|
||||
+ (f", original_id: {original_file_id}" if original_file_id else "")
|
||||
)
|
||||
|
||||
## EXISTING: check if file_id is a litellm managed file
|
||||
elif _is_base64_encoded_unified_file_id(file_id):
|
||||
managed_files_obj = proxy_logging_obj.get_proxy_hook("managed_files")
|
||||
if managed_files_obj is None:
|
||||
raise ProxyException(
|
||||
@@ -762,10 +883,32 @@ async def delete_file(
|
||||
proxy_config=proxy_config,
|
||||
)
|
||||
|
||||
## check if file_id is a litellm managed file
|
||||
is_base64_unified_file_id = _is_base64_encoded_unified_file_id(file_id)
|
||||
|
||||
if is_base64_unified_file_id:
|
||||
# Check for model-based credential routing
|
||||
should_route, model_used, original_file_id, credentials = handle_model_based_routing(
|
||||
file_id=file_id,
|
||||
request=request,
|
||||
llm_router=llm_router,
|
||||
data=data,
|
||||
check_file_id_encoding=True,
|
||||
)
|
||||
|
||||
if should_route:
|
||||
# Use model-based routing with credentials from config
|
||||
prepare_data_with_credentials(
|
||||
data=data,
|
||||
credentials=credentials, # type: ignore
|
||||
file_id=original_file_id,
|
||||
)
|
||||
|
||||
response = await litellm.afile_delete(**data) # type: ignore
|
||||
|
||||
verbose_proxy_logger.debug(
|
||||
f"Deleted file using model: {model_used}"
|
||||
+ (f", original_id: {original_file_id}" if original_file_id else "")
|
||||
)
|
||||
|
||||
## EXISTING: check if file_id is a litellm managed file
|
||||
elif _is_base64_encoded_unified_file_id(file_id):
|
||||
managed_files_obj = proxy_logging_obj.get_proxy_hook("managed_files")
|
||||
if managed_files_obj is None:
|
||||
raise ProxyException(
|
||||
@@ -913,7 +1056,28 @@ async def list_files(
|
||||
)
|
||||
|
||||
response: Optional[Any] = None
|
||||
if target_model_names and isinstance(target_model_names, str):
|
||||
|
||||
# Check for model-based credential routing (no file_id encoding check for list)
|
||||
should_route, model_used, _, credentials = handle_model_based_routing(
|
||||
file_id="", # No file_id for list endpoint
|
||||
request=request,
|
||||
llm_router=llm_router,
|
||||
data=data,
|
||||
check_file_id_encoding=False,
|
||||
)
|
||||
|
||||
if should_route:
|
||||
# Use model-based routing with credentials from config
|
||||
data.update(credentials) # type: ignore
|
||||
response = await litellm.afile_list(
|
||||
custom_llm_provider=credentials["custom_llm_provider"], # type: ignore
|
||||
purpose=purpose,
|
||||
**data # type: ignore
|
||||
)
|
||||
|
||||
verbose_proxy_logger.debug(f"Listed files using model: {model_used}")
|
||||
|
||||
elif target_model_names and isinstance(target_model_names, str):
|
||||
target_model_names_list = target_model_names.split(",")
|
||||
if len(target_model_names_list) != 1:
|
||||
raise HTTPException(
|
||||
|
||||
@@ -618,16 +618,30 @@ class LiteLLMCompletionResponsesConfig:
|
||||
for tool in all_chat_completion_tools:
|
||||
if tool.type == "function":
|
||||
function_definition = tool.function
|
||||
responses_tools.append(
|
||||
OutputFunctionToolCall(
|
||||
name=function_definition.name or "",
|
||||
arguments=function_definition.get("arguments") or "",
|
||||
call_id=tool.id or "",
|
||||
id=tool.id or "",
|
||||
type="function_call", # critical this is "function_call" to work with tools like openai codex
|
||||
status=function_definition.get("status") or "completed",
|
||||
)
|
||||
provider_specific_fields: Optional[Dict[str, Any]] = None
|
||||
if hasattr(tool, "provider_specific_fields") and getattr(tool, "provider_specific_fields", None):
|
||||
provider_specific_fields = getattr(tool, "provider_specific_fields")
|
||||
if not isinstance(provider_specific_fields, dict):
|
||||
provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {}
|
||||
elif hasattr(function_definition, "provider_specific_fields") and getattr(function_definition, "provider_specific_fields", None):
|
||||
provider_specific_fields = getattr(function_definition, "provider_specific_fields")
|
||||
if not isinstance(provider_specific_fields, dict):
|
||||
provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {}
|
||||
|
||||
output_tool_call: OutputFunctionToolCall = OutputFunctionToolCall(
|
||||
name=function_definition.name or "",
|
||||
arguments=function_definition.get("arguments") or "",
|
||||
call_id=tool.id or "",
|
||||
id=tool.id or "",
|
||||
type="function_call", # critical this is "function_call" to work with tools like openai codex
|
||||
status=function_definition.get("status") or "completed",
|
||||
)
|
||||
|
||||
# Pass through provider_specific_fields as-is if present
|
||||
if provider_specific_fields:
|
||||
setattr(output_tool_call, "provider_specific_fields", provider_specific_fields) # type: ignore
|
||||
|
||||
responses_tools.append(output_tool_call)
|
||||
return responses_tools
|
||||
|
||||
@staticmethod
|
||||
|
||||
@@ -5914,6 +5914,57 @@ class Router:
|
||||
raise Exception("Model Name invalid - {}".format(type(model)))
|
||||
return None
|
||||
|
||||
def get_deployment_credentials_with_provider(
|
||||
self, model_id: str
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get API credentials and provider info from a model name in model_list.
|
||||
Useful for passthrough endpoints (files, batches, etc.) that need credentials.
|
||||
|
||||
This method tries to find a deployment by model_id first, and if not found,
|
||||
it tries to find by model_group_name (model_name).
|
||||
|
||||
Args:
|
||||
model_id: Model ID or model name from model_list (e.g., "gpt-4o-litellm")
|
||||
|
||||
Returns:
|
||||
Dictionary containing api_key, api_base, custom_llm_provider, etc.
|
||||
Returns None if model not found.
|
||||
|
||||
Example:
|
||||
credentials = router.get_deployment_credentials_with_provider("gpt-4o-litellm")
|
||||
# Returns: {"api_key": "sk-...", "custom_llm_provider": "openai", ...}
|
||||
"""
|
||||
# Try to get deployment by model_id first
|
||||
deployment = self.get_deployment(model_id=model_id)
|
||||
|
||||
# If not found, try by model_group_name
|
||||
if deployment is None:
|
||||
deployment = self.get_deployment_by_model_group_name(model_group_name=model_id)
|
||||
|
||||
if deployment is None:
|
||||
return None
|
||||
|
||||
# Get basic credentials
|
||||
credentials = CredentialLiteLLMParams(
|
||||
**deployment.litellm_params.model_dump(exclude_none=True)
|
||||
).model_dump(exclude_none=True)
|
||||
|
||||
# Add custom_llm_provider
|
||||
if deployment.litellm_params.custom_llm_provider:
|
||||
credentials["custom_llm_provider"] = (
|
||||
deployment.litellm_params.custom_llm_provider
|
||||
)
|
||||
elif "/" in deployment.litellm_params.model:
|
||||
# Extract provider from "provider/model" format
|
||||
credentials["custom_llm_provider"] = deployment.litellm_params.model.split(
|
||||
"/"
|
||||
)[0]
|
||||
else:
|
||||
credentials["custom_llm_provider"] = "openai" # default
|
||||
|
||||
return credentials
|
||||
|
||||
@overload
|
||||
def get_router_model_info(
|
||||
self, deployment: dict, received_model_name: str, id: None = None
|
||||
|
||||
@@ -166,7 +166,11 @@ class AnthropicMessagesContainerUploadParam(TypedDict, total=False):
|
||||
class AnthropicMessagesImageParam(TypedDict, total=False):
|
||||
type: Required[Literal["image"]]
|
||||
source: Required[
|
||||
Union[AnthropicContentParamSource, AnthropicContentParamSourceFileId]
|
||||
Union[
|
||||
AnthropicContentParamSource,
|
||||
AnthropicContentParamSourceFileId,
|
||||
AnthropicContentParamSourceUrl,
|
||||
]
|
||||
]
|
||||
cache_control: Optional[Union[dict, ChatCompletionCachedContent]]
|
||||
|
||||
|
||||
@@ -261,6 +261,7 @@ class UsageMetadata(TypedDict, total=False):
|
||||
promptTokensDetails: List[PromptTokensDetails]
|
||||
thoughtsTokenCount: int
|
||||
responseTokensDetails: List[PromptTokensDetails]
|
||||
candidatesTokensDetails: List[PromptTokensDetails] # Alternative key name used in some responses
|
||||
|
||||
|
||||
class TokenCountDetailsResponse(TypedDict):
|
||||
|
||||
@@ -8093,6 +8093,21 @@ def add_openai_metadata(metadata: Optional[Mapping[str, Any]]) -> Optional[Dict[
|
||||
|
||||
return visible_metadata.copy()
|
||||
|
||||
def get_requester_metadata(metadata: dict):
|
||||
if not metadata:
|
||||
return None
|
||||
|
||||
requester_metadata = metadata.get("requester_metadata")
|
||||
if isinstance(requester_metadata, dict):
|
||||
cleaned_metadata = add_openai_metadata(requester_metadata)
|
||||
if cleaned_metadata:
|
||||
return cleaned_metadata
|
||||
|
||||
cleaned_metadata = add_openai_metadata(metadata)
|
||||
if cleaned_metadata:
|
||||
return cleaned_metadata
|
||||
|
||||
return None
|
||||
|
||||
def return_raw_request(endpoint: CallTypes, kwargs: dict) -> RawRequestTypedDict:
|
||||
"""
|
||||
|
||||
@@ -85,6 +85,170 @@ async def test_mock_basic_google_ai_studio_responses_api_with_tools():
|
||||
assert call_kwargs["messages"][0]["content"] == "what is the latest version of supabase python package and when was it released?"
|
||||
assert call_kwargs["tools"] == [] # web search tools are converted to web_search_options, not kept as tools
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gemini_3_responses_api_with_thought_signatures():
|
||||
"""
|
||||
Test that Gemini 3 Responses API preserves thought signatures in function calls.
|
||||
This test verifies that provider_specific_fields with thought_signature are correctly
|
||||
preserved when using the Responses API with Gemini 3.
|
||||
"""
|
||||
if not os.getenv("GEMINI_API_KEY"):
|
||||
pytest.skip("GEMINI_API_KEY not set")
|
||||
|
||||
litellm.set_verbose = False
|
||||
request_model = "gemini/gemini-3-pro-preview"
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "City and country e.g. Mumbai, India"
|
||||
},
|
||||
"units": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Units the temperature will be returned in."
|
||||
}
|
||||
},
|
||||
"required": ["location", "units"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
"strict": True
|
||||
}
|
||||
]
|
||||
|
||||
# Step 1: Initial request with tools
|
||||
response = await litellm.aresponses(
|
||||
model=request_model,
|
||||
input="What is the weather in Mumbai?",
|
||||
tools=tools,
|
||||
reasoning_effort="low"
|
||||
)
|
||||
|
||||
# Validate response structure
|
||||
from litellm.types.llms.openai import ResponsesAPIResponse
|
||||
assert isinstance(response, ResponsesAPIResponse), "Response should be a ResponsesAPIResponse"
|
||||
assert hasattr(response, "output") or "output" in response, "Response should have 'output' field"
|
||||
assert isinstance(response.output, list), "Output should be a list"
|
||||
|
||||
# Find function call in output
|
||||
function_call_item = None
|
||||
for item in response.output:
|
||||
# Convert to dict if it's a Pydantic model for easier access
|
||||
if hasattr(item, "model_dump"):
|
||||
item_dict = item.model_dump()
|
||||
elif hasattr(item, "__dict__"):
|
||||
item_dict = dict(item) if not isinstance(item, dict) else item
|
||||
else:
|
||||
item_dict = item if isinstance(item, dict) else {}
|
||||
|
||||
if isinstance(item_dict, dict) and item_dict.get("type") == "function_call":
|
||||
function_call_item = item_dict
|
||||
break
|
||||
|
||||
# Verify function call exists
|
||||
assert function_call_item is not None, "Response should contain a function_call item"
|
||||
assert function_call_item.get("name") == "get_weather", "Function call should be for get_weather"
|
||||
|
||||
# Verify thought signature is present in provider_specific_fields
|
||||
provider_specific_fields = function_call_item.get("provider_specific_fields")
|
||||
assert provider_specific_fields is not None, "Function call should have provider_specific_fields"
|
||||
assert "thought_signature" in provider_specific_fields, "provider_specific_fields should contain thought_signature"
|
||||
assert isinstance(provider_specific_fields["thought_signature"], str), "thought_signature should be a string"
|
||||
assert len(provider_specific_fields["thought_signature"]) > 0, "thought_signature should not be empty"
|
||||
|
||||
print(f"✅ Thought signature preserved: {provider_specific_fields['thought_signature'][:50]}...")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gemini_3_responses_api_streaming_with_thought_signatures():
|
||||
"""
|
||||
Test that Gemini 3 Responses API preserves thought signatures in streaming mode.
|
||||
This test verifies that provider_specific_fields with thought_signature are correctly
|
||||
preserved when using streaming Responses API with Gemini 3.
|
||||
"""
|
||||
if not os.getenv("GEMINI_API_KEY"):
|
||||
pytest.skip("GEMINI_API_KEY not set")
|
||||
|
||||
litellm.set_verbose = False
|
||||
request_model = "gemini/gemini-3-pro-preview"
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "City and country e.g. Mumbai, India"
|
||||
},
|
||||
"units": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Units the temperature will be returned in."
|
||||
}
|
||||
},
|
||||
"required": ["location", "units"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
"strict": True
|
||||
}
|
||||
]
|
||||
|
||||
# Step 1: Streaming request with tools
|
||||
response_stream = await litellm.aresponses(
|
||||
model=request_model,
|
||||
input="What is the weather in Mumbai?",
|
||||
tools=tools,
|
||||
stream=True,
|
||||
reasoning_effort="low"
|
||||
)
|
||||
|
||||
# Collect all chunks
|
||||
chunks = []
|
||||
completed_response = None
|
||||
|
||||
async for chunk in response_stream:
|
||||
chunks.append(chunk)
|
||||
# Check if this is the completed response event
|
||||
if hasattr(chunk, "type") and chunk.type == "response.completed":
|
||||
completed_response = chunk.response
|
||||
elif isinstance(chunk, dict) and chunk.get("type") == "response.completed":
|
||||
completed_response = chunk.get("response")
|
||||
|
||||
# Verify we got chunks
|
||||
assert len(chunks) > 0, "Should receive at least one chunk"
|
||||
|
||||
# If we have a completed response, check for thought signatures
|
||||
if completed_response:
|
||||
output = completed_response.get("output", [])
|
||||
function_call_item = None
|
||||
for item in output:
|
||||
if isinstance(item, dict) and item.get("type") == "function_call":
|
||||
function_call_item = item
|
||||
break
|
||||
|
||||
if function_call_item:
|
||||
provider_specific_fields = function_call_item.get("provider_specific_fields")
|
||||
if provider_specific_fields:
|
||||
thought_signature = provider_specific_fields.get("thought_signature")
|
||||
if thought_signature:
|
||||
assert isinstance(thought_signature, str), "thought_signature should be a string"
|
||||
assert len(thought_signature) > 0, "thought_signature should not be empty"
|
||||
print(f"✅ Streaming thought signature preserved: {thought_signature[:50]}...")
|
||||
|
||||
print(f"✅ Collected {len(chunks)} streaming chunks")
|
||||
|
||||
|
||||
class TestGoogleAIStudioResponsesAPITest(BaseResponsesAPITest):
|
||||
def get_base_completion_call_args(self):
|
||||
#litellm._turn_on_debug()
|
||||
|
||||
@@ -23,6 +23,7 @@ from litellm.utils import (
|
||||
get_optional_params,
|
||||
get_optional_params_embeddings,
|
||||
get_optional_params_image_gen,
|
||||
get_requester_metadata,
|
||||
)
|
||||
|
||||
## get_optional_params_embeddings
|
||||
@@ -67,6 +68,41 @@ def test_anthropic_optional_params(stop_sequence, expected_count):
|
||||
assert len(optional_params) == expected_count
|
||||
|
||||
|
||||
def test_get_requester_metadata_returns_none_for_empty():
|
||||
metadata = {"requester_metadata": {}}
|
||||
assert get_requester_metadata(metadata) is None
|
||||
|
||||
|
||||
@patch("litellm.main.openai_chat_completions.completion")
|
||||
def test_requester_metadata_forwarded_to_openai(mock_completion):
|
||||
mock_completion.return_value = MagicMock()
|
||||
metadata = {
|
||||
"requester_metadata": {
|
||||
"custom_meta_key": "value",
|
||||
"hidden_params": "secret",
|
||||
"int_value": 123,
|
||||
}
|
||||
}
|
||||
|
||||
original_api_key = litellm.api_key
|
||||
litellm.api_key = "sk-test"
|
||||
original_preview_flag = litellm.enable_preview_features
|
||||
litellm.enable_preview_features = True
|
||||
|
||||
try:
|
||||
litellm.completion(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "hi"}],
|
||||
metadata=metadata,
|
||||
)
|
||||
finally:
|
||||
litellm.api_key = original_api_key
|
||||
litellm.enable_preview_features = original_preview_flag
|
||||
|
||||
sent_metadata = mock_completion.call_args.kwargs["optional_params"]["metadata"]
|
||||
assert sent_metadata == {"custom_meta_key": "value"}
|
||||
|
||||
|
||||
def test_get_optional_params_with_allowed_openai_params():
|
||||
"""
|
||||
Test if use can dynamically pass in allowed_openai_params to override default behavior
|
||||
|
||||
@@ -19,6 +19,7 @@ from litellm.litellm_core_utils.prompt_templates.factory import (
|
||||
claude_2_1_pt,
|
||||
convert_to_anthropic_image_obj,
|
||||
convert_url_to_base64,
|
||||
create_anthropic_image_param,
|
||||
llama_2_chat_pt,
|
||||
prompt_factory,
|
||||
)
|
||||
@@ -207,6 +208,125 @@ def test_base64_image_input(url, expected_media_type):
|
||||
assert response["media_type"] == expected_media_type
|
||||
|
||||
|
||||
def test_create_anthropic_image_param_with_http_url():
|
||||
"""Test that HTTP/HTTPS URLs are passed as URL references, not base64."""
|
||||
image_param = create_anthropic_image_param(
|
||||
"https://example.com/image.jpg", format=None
|
||||
)
|
||||
|
||||
assert image_param["type"] == "image"
|
||||
assert image_param["source"]["type"] == "url"
|
||||
assert image_param["source"]["url"] == "https://example.com/image.jpg"
|
||||
|
||||
|
||||
def test_create_anthropic_image_param_with_https_url():
|
||||
"""Test that HTTPS URLs are passed as URL references."""
|
||||
image_param = create_anthropic_image_param(
|
||||
"https://example.com/image.png", format=None
|
||||
)
|
||||
|
||||
assert image_param["type"] == "image"
|
||||
assert image_param["source"]["type"] == "url"
|
||||
assert image_param["source"]["url"] == "https://example.com/image.png"
|
||||
|
||||
|
||||
def test_create_anthropic_image_param_with_dict_input():
|
||||
"""Test that dict input with URL is handled correctly."""
|
||||
image_param = create_anthropic_image_param(
|
||||
{"url": "https://example.com/image.jpg", "format": "image/jpeg"}, format=None
|
||||
)
|
||||
|
||||
assert image_param["type"] == "image"
|
||||
assert image_param["source"]["type"] == "url"
|
||||
assert image_param["source"]["url"] == "https://example.com/image.jpg"
|
||||
|
||||
|
||||
def test_create_anthropic_image_param_with_base64_data_uri():
|
||||
"""Test that data URIs are converted to base64."""
|
||||
image_param = create_anthropic_image_param(
|
||||
"data:image/jpeg;base64,/9j/4AAQSkZJRg==", format=None
|
||||
)
|
||||
|
||||
assert image_param["type"] == "image"
|
||||
assert image_param["source"]["type"] == "base64"
|
||||
assert image_param["source"]["media_type"] == "image/jpeg"
|
||||
assert image_param["source"]["data"] == "/9j/4AAQSkZJRg=="
|
||||
|
||||
|
||||
def test_create_anthropic_image_param_with_format_override():
|
||||
"""Test that format parameter can override media type."""
|
||||
image_param = create_anthropic_image_param(
|
||||
"data:image/jpeg;base64,1234", format="image/png"
|
||||
)
|
||||
|
||||
assert image_param["type"] == "image"
|
||||
assert image_param["source"]["type"] == "base64"
|
||||
assert image_param["source"]["media_type"] == "image/png"
|
||||
|
||||
|
||||
def test_anthropic_messages_pt_with_url_image():
|
||||
"""Test that anthropic_messages_pt correctly handles HTTP/HTTPS URLs as URL references."""
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What's in this image?"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": "https://example.com/image.jpg",
|
||||
},
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
result = anthropic_messages_pt(
|
||||
messages=messages, model="claude-3-5-sonnet", llm_provider="anthropic"
|
||||
)
|
||||
|
||||
assert len(result) == 1
|
||||
assert result[0]["role"] == "user"
|
||||
assert isinstance(result[0]["content"], list)
|
||||
assert len(result[0]["content"]) == 2
|
||||
|
||||
# Check text content
|
||||
assert result[0]["content"][0]["type"] == "text"
|
||||
|
||||
# Check image content - should be URL reference, not base64
|
||||
assert result[0]["content"][1]["type"] == "image"
|
||||
assert result[0]["content"][1]["source"]["type"] == "url"
|
||||
assert result[0]["content"][1]["source"]["url"] == "https://example.com/image.jpg"
|
||||
|
||||
|
||||
def test_anthropic_messages_pt_with_base64_image():
|
||||
"""Test that anthropic_messages_pt correctly handles data URIs as base64."""
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What's in this image?"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": "data:image/jpeg;base64,/9j/4AAQSkZJRg==",
|
||||
},
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
result = anthropic_messages_pt(
|
||||
messages=messages, model="claude-3-5-sonnet", llm_provider="anthropic"
|
||||
)
|
||||
|
||||
assert len(result) == 1
|
||||
assert result[0]["role"] == "user"
|
||||
assert isinstance(result[0]["content"], list)
|
||||
assert len(result[0]["content"]) == 2
|
||||
|
||||
# Check image content - should be base64, not URL
|
||||
assert result[0]["content"][1]["type"] == "image"
|
||||
assert result[0]["content"][1]["source"]["type"] == "base64"
|
||||
assert result[0]["content"][1]["source"]["media_type"] == "image/jpeg"
|
||||
|
||||
|
||||
def test_anthropic_messages_tool_call():
|
||||
messages = [
|
||||
{
|
||||
|
||||
@@ -291,4 +291,4 @@ async def test_list_batches_with_target_model_names():
|
||||
|
||||
# Verify the response structure
|
||||
assert response["object"] == "list"
|
||||
assert len(response["data"]) > 0
|
||||
assert len(response["data"]) > 0
|
||||
@@ -1339,6 +1339,78 @@ def test_vertex_ai_penalty_parameters_validation():
|
||||
assert result["max_output_tokens"] == 100
|
||||
|
||||
|
||||
def test_vertex_ai_gemini_3_penalty_parameters_unsupported():
|
||||
"""
|
||||
Test that penalty parameters are not supported for Gemini 3 models.
|
||||
|
||||
This test ensures that:
|
||||
1. Gemini 3 models do not support penalty parameters
|
||||
2. Penalty parameters are excluded from supported params list for Gemini 3 models
|
||||
3. Penalty parameters are filtered out when mapping params for Gemini 3 models
|
||||
"""
|
||||
v = VertexGeminiConfig()
|
||||
|
||||
# Test Gemini 3 models
|
||||
gemini_3_models = [
|
||||
"gemini-3-pro-preview",
|
||||
"vertex_ai/gemini-3-pro-preview",
|
||||
"gemini/gemini-3-pro-preview",
|
||||
]
|
||||
|
||||
for model in gemini_3_models:
|
||||
# Test _supports_penalty_parameters method
|
||||
assert v._supports_penalty_parameters(model) == False, \
|
||||
f"Gemini 3 model {model} should not support penalty parameters"
|
||||
|
||||
# Test get_supported_openai_params method
|
||||
supported_params = v.get_supported_openai_params(model)
|
||||
assert "frequency_penalty" not in supported_params, \
|
||||
f"frequency_penalty should not be in supported params for {model}"
|
||||
assert "presence_penalty" not in supported_params, \
|
||||
f"presence_penalty should not be in supported params for {model}"
|
||||
|
||||
# Test parameter mapping - penalty params should be filtered out
|
||||
non_default_params = {
|
||||
"temperature": 0.7,
|
||||
"frequency_penalty": 0.5,
|
||||
"presence_penalty": 0.3,
|
||||
"max_tokens": 100
|
||||
}
|
||||
|
||||
optional_params = {}
|
||||
result = v.map_openai_params(
|
||||
non_default_params=non_default_params,
|
||||
optional_params=optional_params,
|
||||
model=model,
|
||||
drop_params=False
|
||||
)
|
||||
|
||||
# Penalty parameters should be filtered out for Gemini 3 models
|
||||
assert "frequency_penalty" not in result, \
|
||||
f"frequency_penalty should be filtered out for Gemini 3 model {model}"
|
||||
assert "presence_penalty" not in result, \
|
||||
f"presence_penalty should be filtered out for Gemini 3 model {model}"
|
||||
|
||||
# Other parameters should still be included
|
||||
assert "temperature" in result, \
|
||||
f"temperature should still be included for Gemini 3 model {model}"
|
||||
assert "max_output_tokens" in result, \
|
||||
f"max_output_tokens should still be included for Gemini 3 model {model}"
|
||||
assert result["temperature"] == 0.7
|
||||
assert result["max_output_tokens"] == 100
|
||||
|
||||
# Test that non-Gemini 3 models still support penalty parameters (if they're not in the unsupported list)
|
||||
non_gemini_3_model = "gemini-2.5-pro"
|
||||
assert v._supports_penalty_parameters(non_gemini_3_model) == True, \
|
||||
f"Non-Gemini 3 model {non_gemini_3_model} should support penalty parameters"
|
||||
|
||||
supported_params = v.get_supported_openai_params(non_gemini_3_model)
|
||||
assert "frequency_penalty" in supported_params, \
|
||||
f"frequency_penalty should be in supported params for {non_gemini_3_model}"
|
||||
assert "presence_penalty" in supported_params, \
|
||||
f"presence_penalty should be in supported params for {non_gemini_3_model}"
|
||||
|
||||
|
||||
def test_vertex_ai_annotation_streaming_events():
|
||||
"""
|
||||
Test that annotation events are properly emitted during streaming for Vertex AI Gemini.
|
||||
|
||||
@@ -192,8 +192,33 @@ async def test_url_with_format_param(model, sync_mode, monkeypatch):
|
||||
json_str = json_str.decode("utf-8")
|
||||
|
||||
print(f"type of json_str: {type(json_str)}")
|
||||
assert "png" in json_str
|
||||
assert "jpeg" not in json_str
|
||||
|
||||
# Bedrock models convert URLs to base64, while direct Anthropic models support URLs
|
||||
# bedrock/invoke models use Anthropic messages API which supports URLs
|
||||
if model.startswith("bedrock/invoke/"):
|
||||
# bedrock/invoke should convert URLs to base64 (doesn't support URL references)
|
||||
# URL should NOT be in the JSON (it should be converted to base64)
|
||||
assert "https://upload.wikimedia.org" not in json_str
|
||||
# Should have base64 data in the source (type="base64", not type="url")
|
||||
assert '"type":"base64"' in json_str or '"type": "base64"' in json_str
|
||||
# Should have "data" field containing base64 content
|
||||
assert '"data"' in json_str
|
||||
elif model.startswith("bedrock/"):
|
||||
# Regular Bedrock models should convert URLs to base64 (uses "bytes" field)
|
||||
# URL should NOT be in the JSON (it should be converted to base64)
|
||||
assert "https://upload.wikimedia.org" not in json_str
|
||||
# Should have "bytes" field (Bedrock uses "bytes" not "base64" in the field name)
|
||||
assert '"bytes"' in json_str or '"bytes":' in json_str
|
||||
elif model.startswith("anthropic/"):
|
||||
# Direct Anthropic models should pass HTTPS URLs directly (HTTP URLs are converted to base64)
|
||||
# Since we're using HTTPS URL, it should be passed as-is
|
||||
assert "https://upload.wikimedia.org" in json_str
|
||||
# For Anthropic, URL references use "url" type, not base64
|
||||
assert '"type":"url"' in json_str or '"type": "url"' in json_str
|
||||
else:
|
||||
# For other models, check format parameter is respected
|
||||
assert "png" in json_str
|
||||
assert "jpeg" not in json_str
|
||||
|
||||
|
||||
@pytest.mark.parametrize("model", ["gpt-4o-mini"])
|
||||
|
||||
Reference in New Issue
Block a user