diff --git a/docs/my-website/blog/gemini_3/index.md b/docs/my-website/blog/gemini_3/index.md
index 2f4cc5bb0f..1b9ff359f3 100644
--- a/docs/my-website/blog/gemini_3/index.md
+++ b/docs/my-website/blog/gemini_3/index.md
@@ -4,8 +4,8 @@ title: "DAY 0 Support: Gemini 3 on LiteLLM"
date: 2025-11-19T10:00:00
authors:
- name: Sameer Kankute
- title: "SWE @ LiteLLM (LLM Translation)"
- url: https://in.linkedin.com/in/sameer-kankute
+ title: SWE @ LiteLLM (LLM Translation)
+ url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://media.licdn.com/dms/image/v2/D4D03AQHB_loQYd5gjg/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1719137160975?e=1765411200&v=beta&t=c8396f--_lH6Fb_pVvx_jGholPfcl0bvwmNynbNdnII
- name: Krrish Dholakia
title: "CEO, LiteLLM"
@@ -88,9 +88,11 @@ curl http://0.0.0.0:4000/v1/chat/completions \
LiteLLM provides **full end-to-end support** for Gemini 3 Pro Preview on:
- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
+- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
+- ✅ `/v1/generateContent` – [Google Gemini API](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini#rest) compatible endpoint (for code, see: `client.models.generate_content(...)`)
-Both endpoints support:
+All endpoints support:
- Streaming and non-streaming responses
- Function calling with thought signatures
- Multi-turn conversations
@@ -548,6 +550,129 @@ curl http://localhost:4000/v1/chat/completions \
3. **Automatic Defaults**: If you don't specify `reasoning_effort`, LiteLLM automatically sets `thinking_level="low"` for optimal performance.
+## Cost Tracking: Prompt Caching & Context Window
+
+LiteLLM provides comprehensive cost tracking for Gemini 3 Pro Preview, including support for prompt caching and tiered pricing based on context window size.
+
+### Prompt Caching Cost Tracking
+
+Gemini 3 supports prompt caching, which allows you to cache frequently used prompt prefixes to reduce costs. LiteLLM automatically tracks and calculates costs for:
+
+- **Cache Hit Tokens**: Tokens that are read from cache (charged at a lower rate)
+- **Cache Creation Tokens**: Tokens that are written to cache (one-time cost)
+- **Text Tokens**: Regular prompt tokens that are processed normally
+
+#### How It Works
+
+LiteLLM extracts caching information from the `prompt_tokens_details` field in the usage object:
+
+```python
+{
+ "usage": {
+ "prompt_tokens": 50000,
+ "completion_tokens": 1000,
+ "total_tokens": 51000,
+ "prompt_tokens_details": {
+ "cached_tokens": 30000, # Cache hit tokens
+ "cache_creation_tokens": 5000, # Tokens written to cache
+ "text_tokens": 15000 # Regular processed tokens
+ }
+ }
+}
+```
+
+### Context Window Tiered Pricing
+
+Gemini 3 Pro Preview supports up to 1M tokens of context, with tiered pricing that automatically applies when your prompt exceeds 200k tokens.
+
+#### Automatic Tier Detection
+
+LiteLLM automatically detects when your prompt exceeds the 200k token threshold and applies the appropriate tiered pricing:
+
+```python
+from litellm import completion_cost
+
+# Example: Small prompt (< 200k tokens)
+response_small = completion(
+ model="gemini/gemini-3-pro-preview",
+ messages=[{"role": "user", "content": "Hello!"}]
+)
+# Uses base pricing: $0.000002/input token, $0.000012/output token
+
+# Example: Large prompt (> 200k tokens)
+response_large = completion(
+ model="gemini/gemini-3-pro-preview",
+ messages=[{"role": "user", "content": "..." * 250000}] # 250k tokens
+)
+# Automatically uses tiered pricing: $0.000004/input token, $0.000018/output token
+```
+
+#### Cost Breakdown
+
+The cost calculation includes:
+
+1. **Text Processing Cost**: Regular tokens processed at base or tiered rate
+2. **Cache Read Cost**: Cached tokens read at discounted rate
+3. **Cache Creation Cost**: One-time cost for writing tokens to cache (applies tiered rate if above 200k)
+4. **Output Cost**: Generated tokens at base or tiered rate
+
+### Example: Viewing Cost Breakdown
+
+You can view the detailed cost breakdown using LiteLLM's cost tracking:
+
+```python
+from litellm import completion, completion_cost
+
+response = completion(
+ model="gemini/gemini-3-pro-preview",
+ messages=[{"role": "user", "content": "Explain prompt caching"}],
+ caching=True # Enable prompt caching
+)
+
+# Get total cost
+total_cost = completion_cost(completion_response=response)
+print(f"Total cost: ${total_cost:.6f}")
+
+# Access usage details
+usage = response.usage
+print(f"Prompt tokens: {usage.prompt_tokens}")
+print(f"Completion tokens: {usage.completion_tokens}")
+
+# Access caching details
+if usage.prompt_tokens_details:
+ print(f"Cache hit tokens: {usage.prompt_tokens_details.cached_tokens}")
+ print(f"Cache creation tokens: {usage.prompt_tokens_details.cache_creation_tokens}")
+ print(f"Text tokens: {usage.prompt_tokens_details.text_tokens}")
+```
+
+### Cost Optimization Tips
+
+1. **Use Prompt Caching**: For repeated prompt prefixes, enable caching to reduce costs by up to 90% for cached portions
+2. **Monitor Context Size**: Be aware that prompts above 200k tokens use tiered pricing (2x for input, 1.5x for output)
+3. **Cache Management**: Cache creation tokens are charged once when writing to cache, then subsequent reads are much cheaper
+4. **Track Usage**: Use LiteLLM's built-in cost tracking to monitor spending across different token types
+
+### Integration with LiteLLM Proxy
+
+When using LiteLLM Proxy, all cost tracking is automatically logged and available through:
+
+- **Usage Logs**: Detailed token and cost breakdowns in proxy logs
+- **Budget Management**: Set budgets and alerts based on actual usage
+- **Analytics Dashboard**: View cost trends and breakdowns by token type
+
+```yaml
+# config.yaml
+model_list:
+ - model_name: gemini-3-pro-preview
+ litellm_params:
+ model: gemini/gemini-3-pro-preview
+ api_key: os.environ/GEMINI_API_KEY
+
+litellm_settings:
+ # Enable detailed cost tracking
+ success_callback: ["langfuse"] # or your preferred logging service
+```
+
## Using with Claude Code CLI
You can use `gemini-3-pro-preview` with **Claude Code CLI** - Anthropic's command-line interface. This allows you to use Gemini 3 Pro Preview with Claude Code's native syntax and workflows.
@@ -628,6 +753,162 @@ $ claude --model gemini-3-pro-preview
- Ensure `GEMINI_API_KEY` is set correctly
- Check LiteLLM proxy logs for detailed error messages
+## Responses API Support
+
+LiteLLM fully supports the OpenAI Responses API for Gemini 3 Pro Preview, including both streaming and non-streaming modes. The Responses API provides a structured way to handle multi-turn conversations with function calling, and LiteLLM automatically preserves thought signatures throughout the conversation.
+
+### Example: Using Responses API with Gemini 3
+
+
+
+
+```python
+from openai import OpenAI
+import json
+
+client = OpenAI()
+
+# 1. Define a list of callable tools for the model
+tools = [
+ {
+ "type": "function",
+ "name": "get_horoscope",
+ "description": "Get today's horoscope for an astrological sign.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "sign": {
+ "type": "string",
+ "description": "An astrological sign like Taurus or Aquarius",
+ },
+ },
+ "required": ["sign"],
+ },
+ },
+]
+
+def get_horoscope(sign):
+ return f"{sign}: Next Tuesday you will befriend a baby otter."
+
+# Create a running input list we will add to over time
+input_list = [
+ {"role": "user", "content": "What is my horoscope? I am an Aquarius."}
+]
+
+# 2. Prompt the model with tools defined
+response = client.responses.create(
+ model="gemini-3-pro-preview",
+ tools=tools,
+ input=input_list,
+)
+
+# Save function call outputs for subsequent requests
+input_list += response.output
+
+for item in response.output:
+ if item.type == "function_call":
+ if item.name == "get_horoscope":
+ # 3. Execute the function logic for get_horoscope
+ horoscope = get_horoscope(json.loads(item.arguments))
+
+ # 4. Provide function call results to the model
+ input_list.append({
+ "type": "function_call_output",
+ "call_id": item.call_id,
+ "output": json.dumps({
+ "horoscope": horoscope
+ })
+ })
+
+print("Final input:")
+print(input_list)
+
+response = client.responses.create(
+ model="gemini-3-pro-preview",
+ instructions="Respond only with a horoscope generated by a tool.",
+ tools=tools,
+ input=input_list,
+)
+
+# 5. The model should be able to give a response!
+print("Final output:")
+print(response.model_dump_json(indent=2))
+print("\n" + response.output_text)
+```
+
+**Key Points:**
+- ✅ Thought signatures are automatically preserved in function calls
+- ✅ Works seamlessly with multi-turn conversations
+- ✅ All Gemini 3-specific features are fully supported
+
+
+
+
+```python
+from openai import OpenAI
+import json
+
+client = OpenAI()
+
+tools = [
+ {
+ "type": "function",
+ "name": "get_horoscope",
+ "description": "Get today's horoscope for an astrological sign.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "sign": {
+ "type": "string",
+ "description": "An astrological sign like Taurus or Aquarius",
+ },
+ },
+ "required": ["sign"],
+ },
+ },
+]
+
+def get_horoscope(sign):
+ return f"{sign}: Next Tuesday you will befriend a baby otter."
+
+input_list = [
+ {"role": "user", "content": "What is my horoscope? I am an Aquarius."}
+]
+
+# Streaming mode
+response = client.responses.create(
+ model="gemini-3-pro-preview",
+ tools=tools,
+ input=input_list,
+ stream=True,
+)
+
+# Collect all chunks
+chunks = []
+for chunk in response:
+ chunks.append(chunk)
+ # Process streaming chunks as they arrive
+ print(chunk)
+
+# Thought signatures are automatically preserved in streaming mode
+```
+
+**Key Points:**
+- ✅ Streaming mode fully supported
+- ✅ Thought signatures preserved across streaming chunks
+- ✅ Real-time processing of function calls and responses
+
+
+
+
+### Responses API Benefits
+
+- ✅ **Structured Output**: Responses API provides a clear structure for handling function calls and multi-turn conversations
+- ✅ **Thought Signature Preservation**: LiteLLM automatically preserves thought signatures in both streaming and non-streaming modes
+- ✅ **Seamless Integration**: Works with existing OpenAI SDK patterns
+- ✅ **Full Feature Support**: All Gemini 3 features (thought signatures, function calling, reasoning) are fully supported
+
+
## Best Practices
#### 1. Always Include Thought Signatures in Conversation History
@@ -665,6 +946,7 @@ When switching from non-Gemini-3 to Gemini-3:
- ✅ No manual intervention needed
- ✅ Conversation history continues seamlessly
+
## Troubleshooting
#### Issue: Missing Thought Signatures
diff --git a/docs/my-website/docs/batches.md b/docs/my-website/docs/batches.md
index 1bd4c700ae..269fee0310 100644
--- a/docs/my-website/docs/batches.md
+++ b/docs/my-website/docs/batches.md
@@ -174,6 +174,257 @@ print("list_batches_response=", list_batches_response)
+## Multi-Account / Model-Based Routing
+
+Route batch operations to different provider accounts using model-specific credentials from your `config.yaml`. This eliminates the need for environment variables and enables multi-tenant batch processing.
+
+### How It Works
+
+**Priority Order:**
+1. **Encoded Batch/File ID** (highest) - Model info embedded in the ID
+2. **Model Parameter** - Via header (`x-litellm-model`), query param, or request body
+3. **Custom Provider** (fallback) - Uses environment variables
+
+### Configuration
+
+```yaml
+model_list:
+ - model_name: gpt-4o-account-1
+ litellm_params:
+ model: openai/gpt-4o
+ api_key: sk-account-1-key
+ api_base: https://api.openai.com/v1
+
+ - model_name: gpt-4o-account-2
+ litellm_params:
+ model: openai/gpt-4o
+ api_key: sk-account-2-key
+ api_base: https://api.openai.com/v1
+
+ - model_name: azure-batches
+ litellm_params:
+ model: azure/gpt-4
+ api_key: azure-key-123
+ api_base: https://my-resource.openai.azure.com
+ api_version: "2024-02-01"
+```
+
+### Usage Examples
+
+#### Scenario 1: Encoded File ID with Model
+
+When you upload a file with a model parameter, LiteLLM encodes the model information in the file ID. All subsequent operations automatically use those credentials.
+
+```bash
+# Step 1: Upload file with model
+curl http://localhost:4000/v1/files \
+ -H "Authorization: Bearer sk-1234" \
+ -H "x-litellm-model: gpt-4o-account-1" \
+ -F purpose="batch" \
+ -F file="@batch.jsonl"
+
+# Response includes encoded file ID:
+# {
+# "id": "file-bGl0ZWxsbTpmaWxlLUxkaUwzaVYxNGZRVlpYcU5KVEdkSjk7bW9kZWwsZ3B0LTRvLWFjY291bnQtMQ",
+# ...
+# }
+
+# Step 2: Create batch - automatically routes to gpt-4o-account-1
+curl http://localhost:4000/v1/batches \
+ -H "Authorization: Bearer sk-1234" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "input_file_id": "file-bGl0ZWxsbTpmaWxlLUxkaUwzaVYxNGZRVlpYcU5KVEdkSjk7bW9kZWwsZ3B0LTRvLWFjY291bnQtMQ",
+ "endpoint": "/v1/chat/completions",
+ "completion_window": "24h"
+ }'
+
+# Batch ID is also encoded with model:
+# {
+# "id": "batch_bGl0ZWxsbTpiYXRjaF82OTIwM2IzNjg0MDQ4MTkwYTA3ODQ5NDY3YTFjMDJkYTttb2RlbCxncHQtNG8tYWNjb3VudC0x",
+# "input_file_id": "file-bGl0ZWxsbTpmaWxlLUxkaUwzaVYxNGZRVlpYcU5KVEdkSjk7bW9kZWwsZ3B0LTRvLWFjY291bnQtMQ",
+# ...
+# }
+
+# Step 3: Retrieve batch - automatically routes to gpt-4o-account-1
+curl http://localhost:4000/v1/batches/batch_bGl0ZWxsbTpiYXRjaF82OTIwM2IzNjg0MDQ4MTkwYTA3ODQ5NDY3YTFjMDJkYTttb2RlbCxncHQtNG8tYWNjb3VudC0x \
+ -H "Authorization: Bearer sk-1234"
+```
+
+**✅ Benefits:**
+- No need to specify model on every request
+- File and batch IDs "remember" which account created them
+- Automatic routing for retrieve, cancel, and file content operations
+
+#### Scenario 2: Model via Header/Query Parameter
+
+Specify the model for each request without encoding it in the ID.
+
+```bash
+# Create batch with model header
+curl http://localhost:4000/v1/batches \
+ -H "Authorization: Bearer sk-1234" \
+ -H "x-litellm-model: gpt-4o-account-2" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "input_file_id": "file-abc123",
+ "endpoint": "/v1/chat/completions",
+ "completion_window": "24h"
+ }'
+
+# Or use query parameter
+curl "http://localhost:4000/v1/batches?model=gpt-4o-account-2" \
+ -H "Authorization: Bearer sk-1234" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "input_file_id": "file-abc123",
+ "endpoint": "/v1/chat/completions",
+ "completion_window": "24h"
+ }'
+
+# List batches for specific model
+curl "http://localhost:4000/v1/batches?model=gpt-4o-account-2" \
+ -H "Authorization: Bearer sk-1234"
+```
+
+**✅ Use Case:**
+- One-off batch operations
+- Different models for different operations
+- Explicit control over routing
+
+#### Scenario 3: Environment Variables (Fallback)
+
+Traditional approach using environment variables when no model is specified.
+
+```bash
+export OPENAI_API_KEY="sk-env-key"
+
+curl http://localhost:4000/v1/batches \
+ -H "Authorization: Bearer sk-1234" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "input_file_id": "file-abc123",
+ "endpoint": "/v1/chat/completions",
+ "completion_window": "24h"
+ }'
+```
+
+**✅ Use Case:**
+- Backward compatibility
+- Simple single-account setups
+- Quick prototyping
+
+### Complete Multi-Account Example
+
+```bash
+# Upload file to Account 1
+FILE_1=$(curl -s http://localhost:4000/v1/files \
+ -H "x-litellm-model: gpt-4o-account-1" \
+ -F purpose="batch" \
+ -F file="@batch1.jsonl" | jq -r '.id')
+
+# Upload file to Account 2
+FILE_2=$(curl -s http://localhost:4000/v1/files \
+ -H "x-litellm-model: gpt-4o-account-2" \
+ -F purpose="batch" \
+ -F file="@batch2.jsonl" | jq -r '.id')
+
+# Create batch on Account 1 (auto-routed via encoded file ID)
+BATCH_1=$(curl -s http://localhost:4000/v1/batches \
+ -d "{\"input_file_id\": \"$FILE_1\", \"endpoint\": \"/v1/chat/completions\", \"completion_window\": \"24h\"}" | jq -r '.id')
+
+# Create batch on Account 2 (auto-routed via encoded file ID)
+BATCH_2=$(curl -s http://localhost:4000/v1/batches \
+ -d "{\"input_file_id\": \"$FILE_2\", \"endpoint\": \"/v1/chat/completions\", \"completion_window\": \"24h\"}" | jq -r '.id')
+
+# Retrieve both batches (auto-routed to correct accounts)
+curl http://localhost:4000/v1/batches/$BATCH_1
+curl http://localhost:4000/v1/batches/$BATCH_2
+
+# List batches per account
+curl "http://localhost:4000/v1/batches?model=gpt-4o-account-1"
+curl "http://localhost:4000/v1/batches?model=gpt-4o-account-2"
+```
+
+### SDK Usage with Model Routing
+
+```python
+import litellm
+import asyncio
+
+# Upload file with model routing
+file_obj = await litellm.acreate_file(
+ file=open("batch.jsonl", "rb"),
+ purpose="batch",
+ model="gpt-4o-account-1", # Route to specific account
+)
+
+print(f"File ID: {file_obj.id}")
+# File ID is encoded with model info
+
+# Create batch - automatically uses gpt-4o-account-1 credentials
+batch = await litellm.acreate_batch(
+ completion_window="24h",
+ endpoint="/v1/chat/completions",
+ input_file_id=file_obj.id, # Model info embedded in ID
+)
+
+print(f"Batch ID: {batch.id}")
+# Batch ID is also encoded
+
+# Retrieve batch - automatically routes to correct account
+retrieved = await litellm.aretrieve_batch(
+ batch_id=batch.id, # Model info embedded in ID
+)
+
+print(f"Batch status: {retrieved.status}")
+
+# Or explicitly specify model
+batch2 = await litellm.acreate_batch(
+ completion_window="24h",
+ endpoint="/v1/chat/completions",
+ input_file_id="file-regular-id",
+ model="gpt-4o-account-2", # Explicit routing
+)
+```
+
+### How ID Encoding Works
+
+LiteLLM encodes model information into file and batch IDs using base64:
+
+```
+Original: file-abc123
+Encoded: file-bGl0ZWxsbTpmaWxlLWFiYzEyMzttb2RlbCxncHQtNG8tdGVzdA
+ └─┬─┘ └──────────────────┬──────────────────────┘
+ prefix base64(litellm:file-abc123;model,gpt-4o-test)
+
+Original: batch_xyz789
+Encoded: batch_bGl0ZWxsbTpiYXRjaF94eXo3ODk7bW9kZWwsZ3B0LTRvLXRlc3Q
+ └──┬──┘ └──────────────────┬──────────────────────┘
+ prefix base64(litellm:batch_xyz789;model,gpt-4o-test)
+```
+
+The encoding:
+- ✅ Preserves OpenAI-compatible prefixes (`file-`, `batch_`)
+- ✅ Is transparent to clients
+- ✅ Enables automatic routing without additional parameters
+- ✅ Works across all batch and file endpoints
+
+### Supported Endpoints
+
+All batch and file endpoints support model-based routing:
+
+| Endpoint | Method | Model Routing |
+|----------|--------|---------------|
+| `/v1/files` | POST | ✅ Via header/query/body |
+| `/v1/files/{file_id}` | GET | ✅ Auto from encoded ID + header/query |
+| `/v1/files/{file_id}/content` | GET | ✅ Auto from encoded ID + header/query |
+| `/v1/files/{file_id}` | DELETE | ✅ Auto from encoded ID |
+| `/v1/batches` | POST | ✅ Auto from file ID + header/query/body |
+| `/v1/batches` | GET | ✅ Via header/query |
+| `/v1/batches/{batch_id}` | GET | ✅ Auto from encoded ID |
+| `/v1/batches/{batch_id}/cancel` | POST | ✅ Auto from encoded ID |
+
## **Supported Providers**:
### [Azure OpenAI](./providers/azure#azure-batches-api)
### [OpenAI](#quick-start)
diff --git a/docs/my-website/docs/files_endpoints.md b/docs/my-website/docs/files_endpoints.md
index 88493fe0bb..fc0484e921 100644
--- a/docs/my-website/docs/files_endpoints.md
+++ b/docs/my-website/docs/files_endpoints.md
@@ -16,7 +16,137 @@ Use this to call the provider's `/files` endpoints directly, in the OpenAI forma
- Delete File
- Get File Content
+## Multi-Account Support (Multiple OpenAI Keys)
+Use different OpenAI API keys for files and batches by specifying a `model` parameter that references entries in your `model_list`. This approach works **without requiring a database** and allows you to route files/batches to different OpenAI accounts.
+
+### How It Works
+
+1. Define models in `model_list` with different API keys
+2. Pass `model` parameter when creating files
+3. LiteLLM returns encoded IDs that contain routing information
+4. Use encoded IDs for all subsequent operations (retrieve, delete, batches)
+5. No need to specify model again - routing info is in the ID
+
+### Setup
+
+```yaml
+model_list:
+ # litellm OpenAI Account
+ - model_name: "gpt-4o-litellm"
+ litellm_params:
+ model: openai/gpt-4o
+ api_key: os.environ/OPENAI_LITELLM_API_KEY
+
+ # Free OpenAI Account
+ - model_name: "gpt-4o-free"
+ litellm_params:
+ model: openai/gpt-4o
+ api_key: os.environ/OPENAI_FREE_API_KEY
+```
+
+### Usage Example
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+ api_key="sk-1234", # Your LiteLLM proxy key
+ base_url="http://0.0.0.0:4000"
+)
+
+# Create file using litellm account
+file_response = client.files.create(
+ file=open("batch_data.jsonl", "rb"),
+ purpose="batch",
+ extra_body={"model": "gpt-4o-litellm"} # Routes to litellm key
+)
+print(f"File ID: {file_response.id}")
+# Returns encoded ID like: file-bGl0ZWxsbTpmaWxlLWFiYzEyMzttb2RlbCxncHQtNG8taWZvb2Q
+
+# Create batch using the encoded file ID
+# No need to specify model again - it's embedded in the file ID
+batch_response = client.batches.create(
+ input_file_id=file_response.id, # Encoded ID
+ endpoint="/v1/chat/completions",
+ completion_window="24h"
+)
+print(f"Batch ID: {batch_response.id}")
+# Returns encoded batch ID with routing info
+
+# Retrieve batch - routing happens automatically
+batch_status = client.batches.retrieve(batch_response.id)
+print(f"Status: {batch_status.status}")
+
+# List files for a specific account
+files = client.files.list(
+ extra_body={"model": "gpt-4o-free"} # List free files
+)
+
+# List batches for a specific account
+batches = client.batches.list(
+ extra_query={"model": "gpt-4o-litellm"} # List litellm batches
+)
+```
+
+### Parameter Options
+
+You can pass the `model` parameter via:
+- **Request body**: `extra_body={"model": "gpt-4o-litellm"}`
+- **Query parameter**: `?model=gpt-4o-litellm`
+- **Header**: `x-litellm-model: gpt-4o-litellm`
+
+### How Encoded IDs Work
+
+- When you create a file/batch with a `model` parameter, LiteLLM encodes the model name into the returned ID
+- The encoded ID is base64-encoded and looks like: `file-bGl0ZWxsbTpmaWxlLWFiYzEyMzttb2RlbCxncHQtNG8taWZvb2Q`
+- When you use this ID in subsequent operations (retrieve, delete, batch create), LiteLLM automatically:
+ 1. Decodes the ID
+ 2. Extracts the model name
+ 3. Looks up the credentials
+ 4. Routes the request to the correct OpenAI account
+- The original provider file/batch ID is preserved internally
+
+### Benefits
+
+✅ **No Database Required** - All routing info stored in the ID
+✅ **Stateless** - Works across proxy restarts
+✅ **Simple** - Just pass the ID around like normal
+✅ **Backward Compatible** - Existing `custom_llm_provider` and `files_settings` still work
+✅ **Future-Proof** - Aligns with managed batches approach
+
+### Migration from files_settings
+
+**Old approach (still works):**
+```yaml
+files_settings:
+ - custom_llm_provider: openai
+ api_key: os.environ/OPENAI_KEY
+```
+
+```python
+# Had to specify provider on every call
+client.files.create(..., extra_headers={"custom-llm-provider": "openai"})
+client.files.retrieve(file_id, extra_headers={"custom-llm-provider": "openai"})
+```
+
+**New approach (recommended):**
+```yaml
+model_list:
+ - model_name: "gpt-4o-account1"
+ litellm_params:
+ model: openai/gpt-4o
+ api_key: os.environ/OPENAI_KEY
+```
+
+```python
+# Specify model once on create
+file = client.files.create(..., extra_body={"model": "gpt-4o-account1"})
+
+# Then just use the ID - routing is automatic
+client.files.retrieve(file.id) # No need to specify account
+client.batches.create(input_file_id=file.id) # Routes correctly
+```
diff --git a/docs/my-website/docs/providers/openai.md b/docs/my-website/docs/providers/openai.md
index 2163d5e611..9303cdffad 100644
--- a/docs/my-website/docs/providers/openai.md
+++ b/docs/my-website/docs/providers/openai.md
@@ -29,6 +29,18 @@ response = completion(
)
```
+:::info Metadata passthrough (preview)
+When `litellm.enable_preview_features = True`, LiteLLM forwards only the values inside `metadata` to OpenAI.
+
+```python
+completion(
+ model="gpt-4o",
+ messages=[{"role": "user", "content": "hi"}],
+ metadata= {"custom_meta_key": "value"},
+)
+```
+:::
+
### Usage - LiteLLM Proxy Server
Here's how to call OpenAI models with the LiteLLM Proxy Server
diff --git a/litellm/completion_extras/litellm_responses_transformation/transformation.py b/litellm/completion_extras/litellm_responses_transformation/transformation.py
index 3ac5a28f90..f39d1bfb5f 100644
--- a/litellm/completion_extras/litellm_responses_transformation/transformation.py
+++ b/litellm/completion_extras/litellm_responses_transformation/transformation.py
@@ -93,6 +93,35 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
choice = Choices(message=msg, finish_reason="stop", index=index)
return choice, index + 1
+ # Handle function_call items (e.g., from GPT-5 Codex format)
+ if item_type == "function_call":
+ # Extract provider_specific_fields if present and pass through as-is
+ provider_specific_fields = item.get("provider_specific_fields")
+ if provider_specific_fields and not isinstance(provider_specific_fields, dict):
+ provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {}
+
+ tool_call_dict = {
+ "id": item.get("call_id") or item.get("id", ""),
+ "function": {
+ "name": item.get("name", ""),
+ "arguments": item.get("arguments", ""),
+ },
+ "type": "function",
+ }
+
+ # Pass through provider_specific_fields as-is if present
+ if provider_specific_fields:
+ tool_call_dict["provider_specific_fields"] = provider_specific_fields
+ # Also add to function's provider_specific_fields for consistency
+ tool_call_dict["function"]["provider_specific_fields"] = provider_specific_fields
+
+ msg = Message(
+ content=None,
+ tool_calls=[tool_call_dict],
+ )
+ choice = Choices(message=msg, finish_reason="tool_calls", index=index)
+ return choice, index + 1
+
# Unknown or unsupported type
return None, index
@@ -257,7 +286,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
return request_data
- def transform_response(
+ def transform_response( # noqa: PLR0915
self,
model: str,
raw_response: "BaseModel",
@@ -321,18 +350,37 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
reasoning_content = None # flush reasoning content
index += 1
elif isinstance(item, ResponseFunctionToolCall):
+
+ provider_specific_fields = None
+ if hasattr(item, "provider_specific_fields") and item.provider_specific_fields:
+ provider_specific_fields = item.provider_specific_fields
+ if not isinstance(provider_specific_fields, dict):
+ provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {}
+ elif hasattr(item, "get") and callable(item.get):
+ provider_fields = item.get("provider_specific_fields")
+ if provider_fields:
+ provider_specific_fields = provider_fields if isinstance(provider_fields, dict) else (dict(provider_fields) if hasattr(provider_fields, "__dict__") else {})
+
+ function_dict: Dict[str, Any] = {
+ "name": item.name,
+ "arguments": item.arguments,
+ }
+
+ if provider_specific_fields:
+ function_dict["provider_specific_fields"] = provider_specific_fields
+
+ tool_call_dict: Dict[str, Any] = {
+ "id": item.call_id,
+ "function": function_dict,
+ "type": "function",
+ }
+
+ if provider_specific_fields:
+ tool_call_dict["provider_specific_fields"] = provider_specific_fields
+
msg = Message(
content=None,
- tool_calls=[
- {
- "id": item.call_id,
- "function": {
- "name": item.name,
- "arguments": item.arguments,
- },
- "type": "function",
- }
- ],
+ tool_calls=[tool_call_dict],
reasoning_content=reasoning_content,
)
@@ -630,7 +678,7 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
return self.chunk_parser(json.loads(str_line))
- def chunk_parser(
+ def chunk_parser( # noqa: PLR0915
self, chunk: dict
) -> Union["GenericStreamingChunk", "ModelResponseStream"]:
# Transform responses API streaming chunk to chat completion format
@@ -667,17 +715,33 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
# New output item added
output_item = parsed_chunk.get("item", {})
if output_item.get("type") == "function_call":
+ # Extract provider_specific_fields if present
+ provider_specific_fields = output_item.get("provider_specific_fields")
+ if provider_specific_fields and not isinstance(provider_specific_fields, dict):
+ provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {}
+
+ function_chunk = ChatCompletionToolCallFunctionChunk(
+ name=output_item.get("name", None),
+ arguments=parsed_chunk.get("arguments", ""),
+ )
+
+ if provider_specific_fields:
+ function_chunk["provider_specific_fields"] = provider_specific_fields
+
+ tool_call_chunk = ChatCompletionToolCallChunk(
+ id=output_item.get("call_id"),
+ index=0,
+ type="function",
+ function=function_chunk,
+ )
+
+ # Add provider_specific_fields if present
+ if provider_specific_fields:
+ tool_call_chunk.provider_specific_fields = provider_specific_fields # type: ignore
+
return GenericStreamingChunk(
text="",
- tool_use=ChatCompletionToolCallChunk(
- id=output_item.get("call_id"),
- index=0,
- type="function",
- function=ChatCompletionToolCallFunctionChunk(
- name=output_item.get("name", None),
- arguments=parsed_chunk.get("arguments", ""),
- ),
- ),
+ tool_use=tool_call_chunk,
is_finished=False,
finish_reason="",
usage=None,
@@ -713,17 +777,34 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
# New output item added
output_item = parsed_chunk.get("item", {})
if output_item.get("type") == "function_call":
+ # Extract provider_specific_fields if present
+ provider_specific_fields = output_item.get("provider_specific_fields")
+ if provider_specific_fields and not isinstance(provider_specific_fields, dict):
+ provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {}
+
+ function_chunk = ChatCompletionToolCallFunctionChunk(
+ name=output_item.get("name", None),
+ arguments="", # responses API sends everything again, we don't
+ )
+
+ # Add provider_specific_fields to function if present
+ if provider_specific_fields:
+ function_chunk["provider_specific_fields"] = provider_specific_fields
+
+ tool_call_chunk = ChatCompletionToolCallChunk(
+ id=output_item.get("call_id"),
+ index=0,
+ type="function",
+ function=function_chunk,
+ )
+
+ # Add provider_specific_fields if present
+ if provider_specific_fields:
+ tool_call_chunk.provider_specific_fields = provider_specific_fields # type: ignore
+
return GenericStreamingChunk(
text="",
- tool_use=ChatCompletionToolCallChunk(
- id=output_item.get("call_id"),
- index=0,
- type="function",
- function=ChatCompletionToolCallFunctionChunk(
- name=output_item.get("name", None),
- arguments="", # responses API sends everything again, we don't
- ),
- ),
+ tool_use=tool_call_chunk,
is_finished=True,
finish_reason="tool_calls",
usage=None,
diff --git a/litellm/litellm_core_utils/prompt_templates/factory.py b/litellm/litellm_core_utils/prompt_templates/factory.py
index 0f4a159975..262692d6d1 100644
--- a/litellm/litellm_core_utils/prompt_templates/factory.py
+++ b/litellm/litellm_core_utils/prompt_templates/factory.py
@@ -6,7 +6,7 @@ import mimetypes
import re
import xml.etree.ElementTree as ET
from enum import Enum
-from typing import Any, List, Optional, Tuple, cast, overload
+from typing import Any, List, Optional, Tuple, Union, cast, overload
from jinja2.sandbox import ImmutableSandboxedEnvironment
@@ -910,6 +910,64 @@ def convert_to_anthropic_image_obj(
)
+def create_anthropic_image_param(
+ image_url_input: Union[str, dict],
+ format: Optional[str] = None,
+ is_bedrock_invoke: bool = False
+) -> AnthropicMessagesImageParam:
+ """
+ Create an AnthropicMessagesImageParam from an image URL input.
+
+ Supports both URL references (for HTTP/HTTPS URLs) and base64 encoding.
+ """
+ # Extract URL and format from input
+ if isinstance(image_url_input, str):
+ image_url = image_url_input
+ else:
+ image_url = image_url_input.get("url", "")
+ if format is None:
+ format = image_url_input.get("format")
+
+ # Check if the image URL is an HTTP/HTTPS URL
+ if image_url.startswith("http://") or image_url.startswith("https://"):
+ # For Bedrock invoke, always convert URLs to base64 (Bedrock invoke doesn't support URLs)
+ if is_bedrock_invoke or image_url.startswith("http://"):
+ base64_url = convert_url_to_base64(url=image_url)
+ image_chunk = convert_to_anthropic_image_obj(
+ openai_image_url=base64_url, format=format
+ )
+ return AnthropicMessagesImageParam(
+ type="image",
+ source=AnthropicContentParamSource(
+ type="base64",
+ media_type=image_chunk["media_type"],
+ data=image_chunk["data"],
+ ),
+ )
+ else:
+ # HTTPS URL - pass directly for regular Anthropic
+ return AnthropicMessagesImageParam(
+ type="image",
+ source=AnthropicContentParamSourceUrl(
+ type="url",
+ url=image_url,
+ ),
+ )
+ else:
+ # Convert to base64 for data URIs or other formats
+ image_chunk = convert_to_anthropic_image_obj(
+ openai_image_url=image_url, format=format
+ )
+ return AnthropicMessagesImageParam(
+ type="image",
+ source=AnthropicContentParamSource(
+ type="base64",
+ media_type=image_chunk["media_type"],
+ data=image_chunk["data"],
+ ),
+ )
+
+
# The following XML functions will be deprecated once JSON schema support is available on Bedrock and Vertex
# ------------------------------------------------------------------------------
def convert_to_anthropic_tool_result_xml(message: dict) -> str:
@@ -1012,15 +1070,35 @@ def anthropic_messages_pt_xml(messages: list):
if isinstance(messages[msg_i]["content"], list):
for m in messages[msg_i]["content"]:
if m.get("type", "") == "image_url":
- format = m["image_url"].get("format")
- user_content.append(
- {
- "type": "image",
- "source": convert_to_anthropic_image_obj(
- m["image_url"]["url"], format=format
- ),
- }
- )
+ format = m["image_url"].get("format") if isinstance(m["image_url"], dict) else None
+ image_param = create_anthropic_image_param(m["image_url"], format=format)
+ # Convert to dict format for XML version
+ source = image_param["source"]
+ if isinstance(source, dict) and source.get("type") == "url":
+ # Type narrowing for URL source
+ url_source = cast(AnthropicContentParamSourceUrl, source)
+ user_content.append(
+ {
+ "type": "image",
+ "source": {
+ "type": "url",
+ "url": url_source["url"],
+ },
+ }
+ )
+ else:
+ # Type narrowing for base64 source
+ base64_source = cast(AnthropicContentParamSource, source)
+ user_content.append(
+ {
+ "type": "image",
+ "source": {
+ "type": "base64",
+ "media_type": base64_source["media_type"],
+ "data": base64_source["data"],
+ },
+ }
+ )
elif m.get("type", "") == "text":
user_content.append({"type": "text", "text": m["text"]})
else:
@@ -1491,24 +1569,9 @@ def convert_to_anthropic_tool_result(
)
)
elif content["type"] == "image_url":
- if isinstance(content["image_url"], str):
- image_chunk = convert_to_anthropic_image_obj(
- content["image_url"], format=None
- )
- else:
- format = content["image_url"].get("format")
- image_chunk = convert_to_anthropic_image_obj(
- content["image_url"]["url"], format=format
- )
+ format = content["image_url"].get("format") if isinstance(content["image_url"], dict) else None
anthropic_content_list.append(
- AnthropicMessagesImageParam(
- type="image",
- source=AnthropicContentParamSource(
- type="base64",
- media_type=image_chunk["media_type"],
- data=image_chunk["data"],
- ),
- )
+ create_anthropic_image_param(content["image_url"], format=format)
)
anthropic_content = anthropic_content_list
@@ -1839,21 +1902,22 @@ def anthropic_messages_pt( # noqa: PLR0915
for m in user_message_types_block["content"]:
if m.get("type", "") == "image_url":
m = cast(ChatCompletionImageObject, m)
- format: Optional[str] = None
- if isinstance(m["image_url"], str):
- image_chunk = convert_to_anthropic_image_obj(
- openai_image_url=m["image_url"], format=None
- )
+ format = m["image_url"].get("format") if isinstance(m["image_url"], dict) else None
+ # Convert ChatCompletionImageUrlObject to dict if needed
+ image_url_value = m["image_url"]
+ if isinstance(image_url_value, str):
+ image_url_input: Union[str, dict[str, Any]] = image_url_value
else:
- format = m["image_url"].get("format")
- image_chunk = convert_to_anthropic_image_obj(
- openai_image_url=m["image_url"]["url"],
- format=format,
- )
-
- _anthropic_content_element = (
- _anthropic_content_element_factory(image_chunk)
- )
+ # ChatCompletionImageUrlObject or dict case - convert to dict
+ image_url_input = {
+ "url": image_url_value["url"],
+ "format": image_url_value.get("format"),
+ }
+ # Bedrock invoke models have format: invoke/...
+ is_bedrock_invoke = model.lower().startswith("invoke/")
+ _anthropic_content_element = create_anthropic_image_param(
+ image_url_input, format=format, is_bedrock_invoke=is_bedrock_invoke
+ )
_content_element = add_cache_control_to_content(
anthropic_content_element=_anthropic_content_element,
original_content_element=dict(m),
diff --git a/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py b/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
index 02e4682b3a..ab594c79ef 100644
--- a/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
+++ b/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
@@ -237,6 +237,9 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
return False
def _supports_penalty_parameters(self, model: str) -> bool:
+ # Gemini 3 models do not support penalty parameters
+ if VertexGeminiConfig._is_gemini_3_or_newer(model):
+ return False
unsupported_models = ["gemini-2.5-pro-preview-06-05"]
if model in unsupported_models:
return False
@@ -1344,7 +1347,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
return False
@staticmethod
- def _calculate_usage(
+ def _calculate_usage( # noqa: PLR0915
completion_response: Union[
GenerateContentResponseBody, BidiGenerateContentServerMessage
],
diff --git a/litellm/main.py b/litellm/main.py
index 1e3826b9a6..b082b491f2 100644
--- a/litellm/main.py
+++ b/litellm/main.py
@@ -105,7 +105,7 @@ from litellm.utils import (
ProviderConfigManager,
Usage,
_get_model_info_helper,
- add_openai_metadata,
+ get_requester_metadata,
add_provider_specific_params_to_optional_params,
async_mock_completion_streaming_obj,
convert_to_model_response_object,
@@ -2086,10 +2086,12 @@ def completion( # type: ignore # noqa: PLR0915
if extra_headers is not None:
optional_params["extra_headers"] = extra_headers
- if litellm.enable_preview_features:
- metadata_payload = add_openai_metadata(metadata)
- if metadata_payload is not None:
- optional_params["metadata"] = metadata_payload
+ if (
+ litellm.enable_preview_features and metadata is not None
+ ): # [PREVIEW] allow metadata to be passed to OPENAI
+ openai_metadata = get_requester_metadata(metadata)
+ if openai_metadata is not None:
+ optional_params["metadata"] = openai_metadata
## LOAD CONFIG - if set
config = litellm.OpenAIConfig.get_config()
diff --git a/litellm/proxy/batches_endpoints/endpoints.py b/litellm/proxy/batches_endpoints/endpoints.py
index b0f13f6afb..98492bcc2d 100644
--- a/litellm/proxy/batches_endpoints/endpoints.py
+++ b/litellm/proxy/batches_endpoints/endpoints.py
@@ -22,10 +22,16 @@ from litellm.proxy.common_utils.openai_endpoint_utils import (
)
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
+ decode_model_from_file_id,
+ encode_file_id_with_model,
get_batch_id_from_unified_batch_id,
+ get_credentials_for_model,
get_model_id_from_unified_batch_id,
get_models_from_unified_file_id,
+ get_original_file_id,
+ prepare_data_with_credentials,
)
+
from litellm.proxy.utils import handle_exception_on_proxy, is_known_model
from litellm.types.llms.openai import LiteLLMBatchCreateRequest
@@ -47,7 +53,7 @@ router = APIRouter()
dependencies=[Depends(user_api_key_auth)],
tags=["batch"],
)
-async def create_batch(
+async def create_batch( # noqa: PLR0915
request: Request,
fastapi_response: Response,
provider: Optional[str] = None,
@@ -111,9 +117,59 @@ async def create_batch(
_create_batch_data = LiteLLMBatchCreateRequest(**data)
input_file_id = _create_batch_data.get("input_file_id", None)
unified_file_id: Union[str, Literal[False]] = False
+
+ model_from_file_id = None
if input_file_id:
+ model_from_file_id = decode_model_from_file_id(input_file_id)
unified_file_id = _is_base64_encoded_unified_file_id(input_file_id)
- if (
+
+ # SCENARIO 1: File ID is encoded with model info
+ if model_from_file_id is not None and input_file_id:
+ credentials = get_credentials_for_model(
+ llm_router=llm_router,
+ model_id=model_from_file_id,
+ operation_context="batch creation (file created with model)",
+ )
+
+ original_file_id = get_original_file_id(input_file_id)
+ _create_batch_data["input_file_id"] = original_file_id
+ prepare_data_with_credentials(
+ data=_create_batch_data, # type: ignore
+ credentials=credentials,
+ )
+
+ # Create batch using model credentials
+ response = await litellm.acreate_batch(
+ custom_llm_provider=credentials["custom_llm_provider"],
+ **_create_batch_data # type: ignore
+ )
+
+ # Encode the batch ID and related file IDs with model information
+ if response and hasattr(response, "id") and response.id:
+ original_batch_id = response.id
+ encoded_batch_id = encode_file_id_with_model(
+ file_id=original_batch_id, model=model_from_file_id
+ )
+ response.id = encoded_batch_id
+
+ if hasattr(response, "output_file_id") and response.output_file_id:
+ response.output_file_id = encode_file_id_with_model(
+ file_id=response.output_file_id, model=model_from_file_id
+ )
+
+ if hasattr(response, "error_file_id") and response.error_file_id:
+ response.error_file_id = encode_file_id_with_model(
+ file_id=response.error_file_id, model=model_from_file_id
+ )
+
+ verbose_proxy_logger.debug(
+ f"Created batch using model: {model_from_file_id}, "
+ f"original_batch_id: {original_batch_id}, encoded: {encoded_batch_id}"
+ )
+
+ response.input_file_id = input_file_id
+
+ elif (
litellm.enable_loadbalancing_on_batch_endpoints is True
and is_router_model
and router_model is not None
@@ -155,9 +211,39 @@ async def create_batch(
response.input_file_id = input_file_id
response._hidden_params["unified_file_id"] = unified_file_id
else:
- response = await litellm.acreate_batch(
- custom_llm_provider=custom_llm_provider, **_create_batch_data # type: ignore
+ # Check if model specified via header/query/body param
+ model_param = (
+ data.get("model")
+ or request.query_params.get("model")
+ or request.headers.get("x-litellm-model")
)
+
+ # SCENARIO 2 & 3: Model from header/query OR custom_llm_provider fallback
+ if model_param:
+ # SCENARIO 2: Use model-based routing from header/query/body
+ credentials = get_credentials_for_model(
+ llm_router=llm_router,
+ model_id=model_param,
+ operation_context="batch creation",
+ )
+
+ prepare_data_with_credentials(
+ data=_create_batch_data, # type: ignore
+ credentials=credentials,
+ )
+
+ # Create batch using model credentials
+ response = await litellm.acreate_batch(
+ custom_llm_provider=credentials["custom_llm_provider"],
+ **_create_batch_data # type: ignore
+ )
+
+ verbose_proxy_logger.debug(f"Created batch using model: {model_param}")
+ else:
+ # SCENARIO 3: Fallback to custom_llm_provider (uses env variables)
+ response = await litellm.acreate_batch(
+ custom_llm_provider=custom_llm_provider, **_create_batch_data # type: ignore
+ )
### CALL HOOKS ### - modify outgoing data
response = await proxy_logging_obj.post_call_success_hook(
@@ -249,7 +335,7 @@ async def retrieve_batch(
data: Dict = {}
try:
- ## check if model is a loadbalanced model
+ model_from_id = decode_model_from_file_id(batch_id)
_retrieve_batch_request = RetrieveBatchRequest(
batch_id=batch_id,
)
@@ -271,7 +357,54 @@ async def retrieve_batch(
route_type="aretrieve_batch",
)
- if litellm.enable_loadbalancing_on_batch_endpoints is True or unified_batch_id:
+ # SCENARIO 1: Batch ID is encoded with model info
+ if model_from_id is not None:
+ credentials = get_credentials_for_model(
+ llm_router=llm_router,
+ model_id=model_from_id,
+ operation_context="batch retrieval (batch created with model)",
+ )
+
+ original_batch_id = get_original_file_id(batch_id)
+ prepare_data_with_credentials(
+ data=data,
+ credentials=credentials,
+ file_id=original_batch_id, # Sets data["batch_id"] = original_batch_id
+ )
+ # Fix: The helper sets "file_id" but we need "batch_id"
+ data["batch_id"] = data.pop("file_id", original_batch_id)
+
+ # Retrieve batch using model credentials
+ response = await litellm.aretrieve_batch(
+ custom_llm_provider=credentials["custom_llm_provider"],
+ **data # type: ignore
+ )
+
+ # Re-encode all IDs in the response
+ if response:
+ if hasattr(response, "id") and response.id:
+ response.id = batch_id # Keep the encoded batch ID
+
+ if hasattr(response, "input_file_id") and response.input_file_id:
+ response.input_file_id = encode_file_id_with_model(
+ file_id=response.input_file_id, model=model_from_id
+ )
+
+ if hasattr(response, "output_file_id") and response.output_file_id:
+ response.output_file_id = encode_file_id_with_model(
+ file_id=response.output_file_id, model=model_from_id
+ )
+
+ if hasattr(response, "error_file_id") and response.error_file_id:
+ response.error_file_id = encode_file_id_with_model(
+ file_id=response.error_file_id, model=model_from_id
+ )
+
+ verbose_proxy_logger.debug(
+ f"Retrieved batch using model: {model_from_id}, original_id: {original_batch_id}"
+ )
+
+ elif litellm.enable_loadbalancing_on_batch_endpoints is True or unified_batch_id:
if llm_router is None:
raise HTTPException(
status_code=500,
@@ -282,6 +415,8 @@ async def retrieve_batch(
response = await llm_router.aretrieve_batch(**data) # type: ignore
response._hidden_params["unified_batch_id"] = unified_batch_id
+
+ # SCENARIO 3: Fallback to custom_llm_provider (uses env variables)
else:
custom_llm_provider = (
provider
@@ -405,9 +540,36 @@ async def list_batches(
route_type="alist_batches",
)
- ## check for target model names
- target_model_names = target_model_names or data.get("target_model_names", None)
- if target_model_names:
+ model_param = (
+ data.get("model")
+ or request.query_params.get("model")
+ or request.headers.get("x-litellm-model")
+ )
+
+ # SCENARIO 2: Use model-based routing from header/query/body
+ if model_param:
+ credentials = get_credentials_for_model(
+ llm_router=llm_router,
+ model_id=model_param,
+ operation_context="batch listing",
+ )
+
+ data.update(credentials)
+
+ response = await litellm.alist_batches(
+ custom_llm_provider=credentials["custom_llm_provider"],
+ after=after,
+ limit=limit,
+ **data # type: ignore
+ )
+
+ verbose_proxy_logger.debug(f"Listed batches using model: {model_param}")
+
+ # SCENARIO 2 (alternative): target_model_names based routing
+ elif target_model_names or data.get("target_model_names", None):
+ target_model_names = target_model_names or data.get("target_model_names", None)
+ if target_model_names is None:
+ raise ValueError("target_model_names is required for this routing scenario")
model = target_model_names.split(",")[0]
response = await llm_router.alist_batches(
model=model,
@@ -415,6 +577,8 @@ async def list_batches(
limit=limit,
**data,
)
+
+ # SCENARIO 3: Fallback to custom_llm_provider (uses env variables)
else:
custom_llm_provider = (
provider
@@ -520,6 +684,9 @@ async def cancel_batch(
verbose_proxy_logger.debug(
"Request received by LiteLLM:\n{}".format(json.dumps(data, indent=4)),
)
+
+ # Check for encoded batch ID with model info
+ model_from_id = decode_model_from_file_id(batch_id)
unified_batch_id = _is_base64_encoded_unified_file_id(batch_id)
# Include original request and headers in the data
@@ -532,7 +699,35 @@ async def cancel_batch(
proxy_config=proxy_config,
)
- if unified_batch_id:
+ # SCENARIO 1: Batch ID is encoded with model info
+ if model_from_id is not None:
+ credentials = get_credentials_for_model(
+ llm_router=llm_router,
+ model_id=model_from_id,
+ operation_context="batch cancellation (batch created with model)",
+ )
+
+ original_batch_id = get_original_file_id(batch_id)
+ prepare_data_with_credentials(
+ data=data,
+ credentials=credentials,
+ file_id=original_batch_id,
+ )
+ # Fix: The helper sets "file_id" but we need "batch_id"
+ data["batch_id"] = data.pop("file_id", original_batch_id)
+
+ # Cancel batch using model credentials
+ response = await litellm.acancel_batch(
+ custom_llm_provider=credentials["custom_llm_provider"],
+ **data # type: ignore
+ )
+
+ verbose_proxy_logger.debug(
+ f"Cancelled batch using model: {model_from_id}, original_id: {original_batch_id}"
+ )
+
+ # SCENARIO 2: target_model_names based routing
+ elif unified_batch_id:
if llm_router is None:
raise HTTPException(
status_code=500,
@@ -552,6 +747,8 @@ async def cancel_batch(
data["batch_id"] = model_batch_id
response = await llm_router.acancel_batch(model=model, **data) # type: ignore
+
+ # SCENARIO 3: Fallback to custom_llm_provider (uses env variables)
else:
custom_llm_provider = (
diff --git a/litellm/proxy/litellm_pre_call_utils.py b/litellm/proxy/litellm_pre_call_utils.py
index 0553c36e3c..89c16ce824 100644
--- a/litellm/proxy/litellm_pre_call_utils.py
+++ b/litellm/proxy/litellm_pre_call_utils.py
@@ -894,6 +894,22 @@ async def add_litellm_data_to_request( # noqa: PLR0915
data["metadata"]
)
+ # Parse litellm_metadata if it's a string (e.g., from multipart/form-data or extra_body)
+ if "litellm_metadata" in data and data["litellm_metadata"] is not None:
+ if isinstance(data["litellm_metadata"], str):
+ parsed_litellm_metadata = safe_json_loads(data["litellm_metadata"])
+ if not isinstance(parsed_litellm_metadata, dict):
+ verbose_proxy_logger.warning(
+ f"Failed to parse 'litellm_metadata' as JSON dict. Received value: {data['litellm_metadata']}"
+ )
+ else:
+ data["litellm_metadata"] = parsed_litellm_metadata
+ # Merge litellm_metadata into the metadata variable (preserving existing values)
+ if isinstance(data["litellm_metadata"], dict):
+ for key, value in data["litellm_metadata"].items():
+ if key not in data[_metadata_variable_name]:
+ data[_metadata_variable_name][key] = value
+
data = LiteLLMProxyRequestSetup.add_user_api_key_auth_to_request_metadata(
data=data,
user_api_key_dict=user_api_key_dict,
diff --git a/litellm/proxy/openai_files_endpoints/common_utils.py b/litellm/proxy/openai_files_endpoints/common_utils.py
index fcbe64409a..c5b58e06d4 100644
--- a/litellm/proxy/openai_files_endpoints/common_utils.py
+++ b/litellm/proxy/openai_files_endpoints/common_utils.py
@@ -76,3 +76,266 @@ def get_batch_id_from_unified_batch_id(file_id: str) -> str:
return file_id.split("llm_batch_id:")[1].split(",")[0]
else:
return file_id.split("generic_response_id:")[1].split(",")[0]
+
+
+def encode_file_id_with_model(file_id: str, model: str) -> str:
+ """
+ Encode a file/batch ID with model routing information.
+
+ Format: -;model,)>
+ The result preserves the original prefix (file-, batch_, etc.) for OpenAI compliance.
+
+ Args:
+ file_id: Original file/batch ID from the provider (e.g., "file-abc123", "batch_xyz")
+ model: Model name from model_list (e.g., "gpt-4o-litellm")
+
+ Returns:
+ Encoded ID starting with appropriate prefix and containing routing information
+
+ Examples:
+ encode_file_id_with_model("file-abc123", "gpt-4o-litellm")
+ -> "file-bGl0ZWxsbTpmaWxlLWFiYzEyMzttb2RlbCxncHQtNG8taWZvb2Q"
+
+ encode_file_id_with_model("batch_abc123", "gpt-4o-test")
+ -> "batch_bGl0ZWxsbTpiYXRjaF9hYmMxMjM7bW9kZWwsZ3B0LTRvLXRlc3Q"
+ """
+ encoded_str = f"litellm:{file_id};model,{model}"
+ encoded_bytes = base64.urlsafe_b64encode(encoded_str.encode())
+ encoded_b64 = encoded_bytes.decode().rstrip("=")
+
+ # Detect the prefix from the original ID (file-, batch_, etc.)
+ # Default to "file-" if no recognizable prefix
+ if file_id.startswith("batch_"):
+ prefix = "batch_"
+ elif file_id.startswith("file-"):
+ prefix = "file-"
+ else:
+ # Default to file- for backward compatibility
+ prefix = "file-"
+
+ return f"{prefix}{encoded_b64}"
+
+
+def decode_model_from_file_id(encoded_id: str) -> Optional[str]:
+ """
+ Extract model name from an encoded file/batch ID.
+ Handles IDs that start with "file-" or "batch_" prefix.
+ """
+ try:
+ if not isinstance(encoded_id, str):
+ return None
+
+ # Remove prefix if present (file-, batch_, etc.)
+ if encoded_id.startswith("file-"):
+ b64_part = encoded_id[5:] # Remove "file-"
+ elif encoded_id.startswith("batch_"):
+ b64_part = encoded_id[6:] # Remove "batch_"
+ else:
+ b64_part = encoded_id
+
+ padded = b64_part + "=" * (-len(b64_part) % 4)
+ decoded = base64.urlsafe_b64decode(padded).decode()
+ if decoded.startswith("litellm:") and ";model," in decoded:
+ match = re.search(r";model,([^;]+)", decoded)
+ if match:
+ return match.group(1).strip()
+
+ return None
+ except Exception:
+ return None
+
+
+def get_original_file_id(encoded_id: str) -> str:
+ """
+ Extract the original provider file/batch ID from an encoded ID.
+ Handles IDs that start with "file-" or "batch_" prefix.
+ """
+ try:
+ if not isinstance(encoded_id, str):
+ return encoded_id
+
+ # Remove prefix if present (file-, batch_, etc.)
+ if encoded_id.startswith("file-"):
+ b64_part = encoded_id[5:] # Remove "file-"
+ elif encoded_id.startswith("batch_"):
+ b64_part = encoded_id[6:] # Remove "batch_"
+ else:
+ b64_part = encoded_id
+
+ padded = b64_part + "=" * (-len(b64_part) % 4)
+ decoded = base64.urlsafe_b64decode(padded).decode()
+
+ if decoded.startswith("litellm:") and ";model," in decoded:
+ match = re.search(r"litellm:([^;]+);model,", decoded)
+ if match:
+ return match.group(1)
+
+ return encoded_id
+ except Exception:
+ return encoded_id
+
+
+def is_model_embedded_id(file_id: str) -> bool:
+ """
+ Check if a file/batch ID has model routing information embedded.
+ """
+ return decode_model_from_file_id(file_id) is not None
+
+
+# ============================================================================
+# MODEL-BASED CREDENTIAL ROUTING HELPERS
+# ============================================================================
+
+
+def extract_model_from_sources(
+ file_id: str,
+ request, # FastAPI Request object
+ data: Optional[dict] = None,
+) -> tuple[Optional[str], Optional[str]]:
+ """
+ Extract model information from multiple sources in priority order:
+ 1. Embedded in file_id (highest priority)
+ 2. Request headers (x-litellm-model)
+ 3. Query parameters (?model=)
+ 4. Request body/data dict
+
+ Args:
+ file_id: File ID that may contain embedded model info
+ request: FastAPI request object
+ data: Optional request data dictionary
+
+ Returns:
+ Tuple of (model_from_id, model_from_param)
+ - model_from_id: Model decoded from file ID (if embedded)
+ - model_from_param: Model from header/query/body
+ """
+ if data is None:
+ data = {}
+
+ # Check if file_id has embedded model info
+ model_from_id = decode_model_from_file_id(file_id)
+
+ # Check other sources for model parameter
+ model_from_param = (
+ data.get("model")
+ or request.query_params.get("model")
+ or request.headers.get("x-litellm-model")
+ )
+
+ return model_from_id, model_from_param
+
+
+def get_credentials_for_model(
+ llm_router, # Router instance
+ model_id: str,
+ operation_context: str = "file operation",
+):
+ """
+ Retrieve API credentials for a model from the LLM Router.
+
+ Args:
+ llm_router: LiteLLM Router instance
+ model_id: Model name or deployment ID
+ operation_context: Description for error messages (e.g., "file upload", "batch creation")
+
+ Returns:
+ Dictionary with credentials (api_key, api_base, custom_llm_provider, etc.)
+
+ Raises:
+ HTTPException: If router not initialized or model not found
+ """
+ from fastapi import HTTPException
+
+ if llm_router is None:
+ raise HTTPException(
+ status_code=500,
+ detail={"error": "Router not initialized. Cannot use model-based routing."},
+ )
+
+ credentials = llm_router.get_deployment_credentials_with_provider(model_id=model_id)
+
+ if credentials is None:
+ raise HTTPException(
+ status_code=400,
+ detail={
+ "error": f"Model '{model_id}' not found in model_list. Please check your config.yaml."
+ },
+ )
+
+ return credentials
+
+
+def prepare_data_with_credentials(
+ data: dict,
+ credentials: dict,
+ file_id: Optional[str] = None,
+) -> None:
+ """
+ Update data dictionary with model credentials (in-place).
+
+ Args:
+ data: Data dictionary to update
+ credentials: Credentials from router
+ file_id: Optional original file_id to set (for decoded file IDs)
+ """
+ data.update(credentials)
+ data.pop("custom_llm_provider", None)
+
+ if file_id is not None:
+ data["file_id"] = file_id
+
+
+def handle_model_based_routing(
+ file_id: str,
+ request, # FastAPI Request object
+ llm_router, # Router instance
+ data: dict,
+ check_file_id_encoding: bool = True,
+) -> tuple[bool, Optional[str], Optional[str], Optional[dict]]:
+ """
+ Orchestrate model-based credential routing for file operations.
+
+ Args:
+ file_id: File ID (may contain embedded model info)
+ request: FastAPI request object
+ llm_router: LiteLLM Router instance
+ data: Request data dictionary
+ check_file_id_encoding: Whether to check for embedded model in file_id
+
+ Returns:
+ Tuple of (should_use_model_routing, model_used, original_file_id, credentials)
+ - should_use_model_routing: True if model-based routing should be used
+ - model_used: The model name being used
+ - original_file_id: Decoded file ID (if it was encoded)
+ - credentials: Model credentials dict
+
+ Raises:
+ HTTPException: If router unavailable or model not found
+ """
+ model_from_id, model_from_param = extract_model_from_sources(
+ file_id=file_id,
+ request=request,
+ data=data,
+ )
+
+ # Priority 1: Model embedded in file_id
+ if check_file_id_encoding and model_from_id is not None:
+ credentials = get_credentials_for_model(
+ llm_router=llm_router,
+ model_id=model_from_id,
+ operation_context=f"file operation (file created with model '{model_from_id}')",
+ )
+ original_file_id = get_original_file_id(file_id)
+ return True, model_from_id, original_file_id, credentials
+
+ # Priority 2: Model from header/query/body
+ elif model_from_param is not None:
+ credentials = get_credentials_for_model(
+ llm_router=llm_router,
+ model_id=model_from_param,
+ operation_context="file operation",
+ )
+ return True, model_from_param, None, credentials
+
+ # No model-based routing needed
+ return False, None, None, None
diff --git a/litellm/proxy/openai_files_endpoints/files_endpoints.py b/litellm/proxy/openai_files_endpoints/files_endpoints.py
index b0be547613..3f08a4ec36 100644
--- a/litellm/proxy/openai_files_endpoints/files_endpoints.py
+++ b/litellm/proxy/openai_files_endpoints/files_endpoints.py
@@ -29,6 +29,7 @@ from litellm.llms.base_llm.files.transformation import BaseFileEndpoints
from litellm.proxy._types import *
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing
+from litellm.proxy.common_utils.http_parsing_utils import _read_request_body
from litellm.proxy.common_utils.openai_endpoint_utils import (
get_custom_llm_provider_from_request_body,
get_custom_llm_provider_from_request_headers,
@@ -42,7 +43,13 @@ from litellm.types.llms.openai import (
OpenAIFilesPurpose,
)
-from .common_utils import _is_base64_encoded_unified_file_id
+from .common_utils import (
+ _is_base64_encoded_unified_file_id,
+ encode_file_id_with_model,
+ get_credentials_for_model,
+ handle_model_based_routing,
+ prepare_data_with_credentials,
+)
router = APIRouter()
@@ -127,7 +134,51 @@ async def route_create_file(
is_router_model: bool,
router_model: Optional[str],
custom_llm_provider: str,
+ model: Optional[str] = None,
) -> OpenAIFileObject:
+ """
+ Route file creation request to the appropriate provider.
+
+ Priority:
+ 1. If model parameter provided -> use model credentials and encode ID
+ 2. If enable_loadbalancing_on_batch_endpoints -> deprecated loadbalancing
+ 3. If target_model_names_list -> managed files (requires DB)
+ 4. Else -> use custom_llm_provider with files_settings
+ """
+
+ # NEW: Handle model-based routing (no DB required)
+ if model is not None:
+ # Get credentials from model_list via router
+ credentials = get_credentials_for_model(
+ llm_router=llm_router,
+ model_id=model,
+ operation_context="file upload",
+ )
+
+ # Merge credentials into the request
+ prepare_data_with_credentials(
+ data=_create_file_request, # type: ignore
+ credentials=credentials,
+ )
+
+ # Create the file with model credentials
+ response = await litellm.acreate_file(
+ **_create_file_request,
+ custom_llm_provider=credentials["custom_llm_provider"]
+ ) # type: ignore
+
+ # Encode the file ID with model information
+ if response and hasattr(response, "id") and response.id:
+ original_id = response.id
+ encoded_id = encode_file_id_with_model(file_id=original_id, model=model)
+ response.id = encoded_id
+ verbose_proxy_logger.debug(
+ f"Encoded file ID: {original_id} -> {encoded_id} (model: {model})"
+ )
+
+ return response
+
+ # EXISTING: Deprecated loadbalancing approach
if (
litellm.enable_loadbalancing_on_batch_endpoints is True
and is_router_model
@@ -206,6 +257,7 @@ async def create_file(
provider: Optional[str] = None,
custom_llm_provider: str = Form(default="openai"),
file: UploadFile = File(...),
+ litellm_metadata: Optional[str] = Form(default=None),
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""
@@ -245,6 +297,14 @@ async def create_file(
or "openai"
)
+ # NEW: Extract model parameter for multi-account routing
+ request_body = await _read_request_body(request=request) or {}
+ model_param = (
+ request_body.get("model")
+ or request.query_params.get("model")
+ or request.headers.get("x-litellm-model")
+ )
+
target_model_names_list = (
target_model_names.split(",") if target_model_names else []
)
@@ -264,6 +324,10 @@ async def create_file(
purpose = cast(OpenAIFilesPurpose, purpose)
data = {}
+
+ # Add litellm_metadata to data if provided (from form field)
+ if litellm_metadata is not None:
+ data["litellm_metadata"] = litellm_metadata
# Include original request and headers in the data
data = await add_litellm_data_to_request(
@@ -303,6 +367,7 @@ async def create_file(
is_router_model=is_router_model,
router_model=router_model,
custom_llm_provider=custom_llm_provider,
+ model=model_param,
)
if response is None:
@@ -480,13 +545,41 @@ async def get_file_content(
}
)
else:
- response = await litellm.afile_content(
- **{
- "custom_llm_provider": custom_llm_provider,
- "file_id": file_id,
- **data,
- } # type: ignore
+ # Check for model-based credential routing
+ should_route, model_used, original_file_id, credentials = handle_model_based_routing(
+ file_id=file_id,
+ request=request,
+ llm_router=llm_router,
+ data=data,
+ check_file_id_encoding=True,
)
+
+ if should_route:
+ # Use model-based routing with credentials from config
+ prepare_data_with_credentials(
+ data=data,
+ credentials=credentials, # type: ignore
+ file_id=original_file_id, # Use decoded file ID if from encoded ID
+ )
+
+ response = await litellm.afile_content(
+ custom_llm_provider=credentials["custom_llm_provider"], # type: ignore
+ **data
+ ) # type: ignore
+
+ verbose_proxy_logger.debug(
+ f"Retrieved file content using model: {model_used}"
+ + (f", file_id: {file_id} -> {original_file_id}" if original_file_id else "")
+ )
+ else:
+ # Fallback to default behavior (uses env variables or provider-based routing)
+ response = await litellm.afile_content(
+ **{
+ "custom_llm_provider": custom_llm_provider,
+ "file_id": file_id,
+ **data,
+ } # type: ignore
+ )
### ALERTING ###
asyncio.create_task(
@@ -618,10 +711,38 @@ async def get_file(
route_type="afile_retrieve",
)
- ## check if file_id is a litellm managed file
- is_base64_unified_file_id = _is_base64_encoded_unified_file_id(file_id)
+ ## Check for model-based credential routing
+ from litellm.proxy.proxy_server import llm_router
+
+ should_route, model_used, original_file_id, credentials = handle_model_based_routing(
+ file_id=file_id,
+ request=request,
+ llm_router=llm_router,
+ data=data,
+ check_file_id_encoding=True,
+ )
+
+ if should_route:
+ # Use model-based routing with credentials from config
+ prepare_data_with_credentials(
+ data=data,
+ credentials=credentials, # type: ignore
+ file_id=original_file_id,
+ )
- if is_base64_unified_file_id:
+ response = await litellm.afile_retrieve(**data) # type: ignore
+
+ # Keep the encoded ID in response if it was originally encoded
+ if original_file_id and response and hasattr(response, "id") and response.id:
+ response.id = file_id
+
+ verbose_proxy_logger.debug(
+ f"Retrieved file using model: {model_used}"
+ + (f", original_id: {original_file_id}" if original_file_id else "")
+ )
+
+ ## EXISTING: check if file_id is a litellm managed file
+ elif _is_base64_encoded_unified_file_id(file_id):
managed_files_obj = proxy_logging_obj.get_proxy_hook("managed_files")
if managed_files_obj is None:
raise ProxyException(
@@ -762,10 +883,32 @@ async def delete_file(
proxy_config=proxy_config,
)
- ## check if file_id is a litellm managed file
- is_base64_unified_file_id = _is_base64_encoded_unified_file_id(file_id)
-
- if is_base64_unified_file_id:
+ # Check for model-based credential routing
+ should_route, model_used, original_file_id, credentials = handle_model_based_routing(
+ file_id=file_id,
+ request=request,
+ llm_router=llm_router,
+ data=data,
+ check_file_id_encoding=True,
+ )
+
+ if should_route:
+ # Use model-based routing with credentials from config
+ prepare_data_with_credentials(
+ data=data,
+ credentials=credentials, # type: ignore
+ file_id=original_file_id,
+ )
+
+ response = await litellm.afile_delete(**data) # type: ignore
+
+ verbose_proxy_logger.debug(
+ f"Deleted file using model: {model_used}"
+ + (f", original_id: {original_file_id}" if original_file_id else "")
+ )
+
+ ## EXISTING: check if file_id is a litellm managed file
+ elif _is_base64_encoded_unified_file_id(file_id):
managed_files_obj = proxy_logging_obj.get_proxy_hook("managed_files")
if managed_files_obj is None:
raise ProxyException(
@@ -913,7 +1056,28 @@ async def list_files(
)
response: Optional[Any] = None
- if target_model_names and isinstance(target_model_names, str):
+
+ # Check for model-based credential routing (no file_id encoding check for list)
+ should_route, model_used, _, credentials = handle_model_based_routing(
+ file_id="", # No file_id for list endpoint
+ request=request,
+ llm_router=llm_router,
+ data=data,
+ check_file_id_encoding=False,
+ )
+
+ if should_route:
+ # Use model-based routing with credentials from config
+ data.update(credentials) # type: ignore
+ response = await litellm.afile_list(
+ custom_llm_provider=credentials["custom_llm_provider"], # type: ignore
+ purpose=purpose,
+ **data # type: ignore
+ )
+
+ verbose_proxy_logger.debug(f"Listed files using model: {model_used}")
+
+ elif target_model_names and isinstance(target_model_names, str):
target_model_names_list = target_model_names.split(",")
if len(target_model_names_list) != 1:
raise HTTPException(
diff --git a/litellm/responses/litellm_completion_transformation/transformation.py b/litellm/responses/litellm_completion_transformation/transformation.py
index e021f4c16d..bd12d0cbb4 100644
--- a/litellm/responses/litellm_completion_transformation/transformation.py
+++ b/litellm/responses/litellm_completion_transformation/transformation.py
@@ -618,16 +618,30 @@ class LiteLLMCompletionResponsesConfig:
for tool in all_chat_completion_tools:
if tool.type == "function":
function_definition = tool.function
- responses_tools.append(
- OutputFunctionToolCall(
- name=function_definition.name or "",
- arguments=function_definition.get("arguments") or "",
- call_id=tool.id or "",
- id=tool.id or "",
- type="function_call", # critical this is "function_call" to work with tools like openai codex
- status=function_definition.get("status") or "completed",
- )
+ provider_specific_fields: Optional[Dict[str, Any]] = None
+ if hasattr(tool, "provider_specific_fields") and getattr(tool, "provider_specific_fields", None):
+ provider_specific_fields = getattr(tool, "provider_specific_fields")
+ if not isinstance(provider_specific_fields, dict):
+ provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {}
+ elif hasattr(function_definition, "provider_specific_fields") and getattr(function_definition, "provider_specific_fields", None):
+ provider_specific_fields = getattr(function_definition, "provider_specific_fields")
+ if not isinstance(provider_specific_fields, dict):
+ provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {}
+
+ output_tool_call: OutputFunctionToolCall = OutputFunctionToolCall(
+ name=function_definition.name or "",
+ arguments=function_definition.get("arguments") or "",
+ call_id=tool.id or "",
+ id=tool.id or "",
+ type="function_call", # critical this is "function_call" to work with tools like openai codex
+ status=function_definition.get("status") or "completed",
)
+
+ # Pass through provider_specific_fields as-is if present
+ if provider_specific_fields:
+ setattr(output_tool_call, "provider_specific_fields", provider_specific_fields) # type: ignore
+
+ responses_tools.append(output_tool_call)
return responses_tools
@staticmethod
diff --git a/litellm/router.py b/litellm/router.py
index 841391653d..6d38d2fc2b 100644
--- a/litellm/router.py
+++ b/litellm/router.py
@@ -5914,6 +5914,57 @@ class Router:
raise Exception("Model Name invalid - {}".format(type(model)))
return None
+ def get_deployment_credentials_with_provider(
+ self, model_id: str
+ ) -> Optional[Dict[str, Any]]:
+ """
+ Get API credentials and provider info from a model name in model_list.
+ Useful for passthrough endpoints (files, batches, etc.) that need credentials.
+
+ This method tries to find a deployment by model_id first, and if not found,
+ it tries to find by model_group_name (model_name).
+
+ Args:
+ model_id: Model ID or model name from model_list (e.g., "gpt-4o-litellm")
+
+ Returns:
+ Dictionary containing api_key, api_base, custom_llm_provider, etc.
+ Returns None if model not found.
+
+ Example:
+ credentials = router.get_deployment_credentials_with_provider("gpt-4o-litellm")
+ # Returns: {"api_key": "sk-...", "custom_llm_provider": "openai", ...}
+ """
+ # Try to get deployment by model_id first
+ deployment = self.get_deployment(model_id=model_id)
+
+ # If not found, try by model_group_name
+ if deployment is None:
+ deployment = self.get_deployment_by_model_group_name(model_group_name=model_id)
+
+ if deployment is None:
+ return None
+
+ # Get basic credentials
+ credentials = CredentialLiteLLMParams(
+ **deployment.litellm_params.model_dump(exclude_none=True)
+ ).model_dump(exclude_none=True)
+
+ # Add custom_llm_provider
+ if deployment.litellm_params.custom_llm_provider:
+ credentials["custom_llm_provider"] = (
+ deployment.litellm_params.custom_llm_provider
+ )
+ elif "/" in deployment.litellm_params.model:
+ # Extract provider from "provider/model" format
+ credentials["custom_llm_provider"] = deployment.litellm_params.model.split(
+ "/"
+ )[0]
+ else:
+ credentials["custom_llm_provider"] = "openai" # default
+
+ return credentials
+
@overload
def get_router_model_info(
self, deployment: dict, received_model_name: str, id: None = None
diff --git a/litellm/types/llms/anthropic.py b/litellm/types/llms/anthropic.py
index d216c2a8d5..a6963c1a4c 100644
--- a/litellm/types/llms/anthropic.py
+++ b/litellm/types/llms/anthropic.py
@@ -166,7 +166,11 @@ class AnthropicMessagesContainerUploadParam(TypedDict, total=False):
class AnthropicMessagesImageParam(TypedDict, total=False):
type: Required[Literal["image"]]
source: Required[
- Union[AnthropicContentParamSource, AnthropicContentParamSourceFileId]
+ Union[
+ AnthropicContentParamSource,
+ AnthropicContentParamSourceFileId,
+ AnthropicContentParamSourceUrl,
+ ]
]
cache_control: Optional[Union[dict, ChatCompletionCachedContent]]
diff --git a/litellm/types/llms/vertex_ai.py b/litellm/types/llms/vertex_ai.py
index 818f0db798..7a4a4723ab 100644
--- a/litellm/types/llms/vertex_ai.py
+++ b/litellm/types/llms/vertex_ai.py
@@ -261,6 +261,7 @@ class UsageMetadata(TypedDict, total=False):
promptTokensDetails: List[PromptTokensDetails]
thoughtsTokenCount: int
responseTokensDetails: List[PromptTokensDetails]
+ candidatesTokensDetails: List[PromptTokensDetails] # Alternative key name used in some responses
class TokenCountDetailsResponse(TypedDict):
diff --git a/litellm/utils.py b/litellm/utils.py
index d7474387cd..1ec2576d35 100644
--- a/litellm/utils.py
+++ b/litellm/utils.py
@@ -8093,6 +8093,21 @@ def add_openai_metadata(metadata: Optional[Mapping[str, Any]]) -> Optional[Dict[
return visible_metadata.copy()
+def get_requester_metadata(metadata: dict):
+ if not metadata:
+ return None
+
+ requester_metadata = metadata.get("requester_metadata")
+ if isinstance(requester_metadata, dict):
+ cleaned_metadata = add_openai_metadata(requester_metadata)
+ if cleaned_metadata:
+ return cleaned_metadata
+
+ cleaned_metadata = add_openai_metadata(metadata)
+ if cleaned_metadata:
+ return cleaned_metadata
+
+ return None
def return_raw_request(endpoint: CallTypes, kwargs: dict) -> RawRequestTypedDict:
"""
diff --git a/tests/llm_responses_api_testing/test_google_ai_studio_responses_api.py b/tests/llm_responses_api_testing/test_google_ai_studio_responses_api.py
index 203ee252b3..13801fce7d 100644
--- a/tests/llm_responses_api_testing/test_google_ai_studio_responses_api.py
+++ b/tests/llm_responses_api_testing/test_google_ai_studio_responses_api.py
@@ -85,6 +85,170 @@ async def test_mock_basic_google_ai_studio_responses_api_with_tools():
assert call_kwargs["messages"][0]["content"] == "what is the latest version of supabase python package and when was it released?"
assert call_kwargs["tools"] == [] # web search tools are converted to web_search_options, not kept as tools
+@pytest.mark.asyncio
+async def test_gemini_3_responses_api_with_thought_signatures():
+ """
+ Test that Gemini 3 Responses API preserves thought signatures in function calls.
+ This test verifies that provider_specific_fields with thought_signature are correctly
+ preserved when using the Responses API with Gemini 3.
+ """
+ if not os.getenv("GEMINI_API_KEY"):
+ pytest.skip("GEMINI_API_KEY not set")
+
+ litellm.set_verbose = False
+ request_model = "gemini/gemini-3-pro-preview"
+
+ tools = [
+ {
+ "type": "function",
+ "name": "get_weather",
+ "description": "Get the current weather for a location",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "location": {
+ "type": "string",
+ "description": "City and country e.g. Mumbai, India"
+ },
+ "units": {
+ "type": "string",
+ "enum": ["celsius", "fahrenheit"],
+ "description": "Units the temperature will be returned in."
+ }
+ },
+ "required": ["location", "units"],
+ "additionalProperties": False,
+ },
+ "strict": True
+ }
+ ]
+
+ # Step 1: Initial request with tools
+ response = await litellm.aresponses(
+ model=request_model,
+ input="What is the weather in Mumbai?",
+ tools=tools,
+ reasoning_effort="low"
+ )
+
+ # Validate response structure
+ from litellm.types.llms.openai import ResponsesAPIResponse
+ assert isinstance(response, ResponsesAPIResponse), "Response should be a ResponsesAPIResponse"
+ assert hasattr(response, "output") or "output" in response, "Response should have 'output' field"
+ assert isinstance(response.output, list), "Output should be a list"
+
+ # Find function call in output
+ function_call_item = None
+ for item in response.output:
+ # Convert to dict if it's a Pydantic model for easier access
+ if hasattr(item, "model_dump"):
+ item_dict = item.model_dump()
+ elif hasattr(item, "__dict__"):
+ item_dict = dict(item) if not isinstance(item, dict) else item
+ else:
+ item_dict = item if isinstance(item, dict) else {}
+
+ if isinstance(item_dict, dict) and item_dict.get("type") == "function_call":
+ function_call_item = item_dict
+ break
+
+ # Verify function call exists
+ assert function_call_item is not None, "Response should contain a function_call item"
+ assert function_call_item.get("name") == "get_weather", "Function call should be for get_weather"
+
+ # Verify thought signature is present in provider_specific_fields
+ provider_specific_fields = function_call_item.get("provider_specific_fields")
+ assert provider_specific_fields is not None, "Function call should have provider_specific_fields"
+ assert "thought_signature" in provider_specific_fields, "provider_specific_fields should contain thought_signature"
+ assert isinstance(provider_specific_fields["thought_signature"], str), "thought_signature should be a string"
+ assert len(provider_specific_fields["thought_signature"]) > 0, "thought_signature should not be empty"
+
+ print(f"✅ Thought signature preserved: {provider_specific_fields['thought_signature'][:50]}...")
+
+
+@pytest.mark.asyncio
+async def test_gemini_3_responses_api_streaming_with_thought_signatures():
+ """
+ Test that Gemini 3 Responses API preserves thought signatures in streaming mode.
+ This test verifies that provider_specific_fields with thought_signature are correctly
+ preserved when using streaming Responses API with Gemini 3.
+ """
+ if not os.getenv("GEMINI_API_KEY"):
+ pytest.skip("GEMINI_API_KEY not set")
+
+ litellm.set_verbose = False
+ request_model = "gemini/gemini-3-pro-preview"
+
+ tools = [
+ {
+ "type": "function",
+ "name": "get_weather",
+ "description": "Get the current weather for a location",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "location": {
+ "type": "string",
+ "description": "City and country e.g. Mumbai, India"
+ },
+ "units": {
+ "type": "string",
+ "enum": ["celsius", "fahrenheit"],
+ "description": "Units the temperature will be returned in."
+ }
+ },
+ "required": ["location", "units"],
+ "additionalProperties": False,
+ },
+ "strict": True
+ }
+ ]
+
+ # Step 1: Streaming request with tools
+ response_stream = await litellm.aresponses(
+ model=request_model,
+ input="What is the weather in Mumbai?",
+ tools=tools,
+ stream=True,
+ reasoning_effort="low"
+ )
+
+ # Collect all chunks
+ chunks = []
+ completed_response = None
+
+ async for chunk in response_stream:
+ chunks.append(chunk)
+ # Check if this is the completed response event
+ if hasattr(chunk, "type") and chunk.type == "response.completed":
+ completed_response = chunk.response
+ elif isinstance(chunk, dict) and chunk.get("type") == "response.completed":
+ completed_response = chunk.get("response")
+
+ # Verify we got chunks
+ assert len(chunks) > 0, "Should receive at least one chunk"
+
+ # If we have a completed response, check for thought signatures
+ if completed_response:
+ output = completed_response.get("output", [])
+ function_call_item = None
+ for item in output:
+ if isinstance(item, dict) and item.get("type") == "function_call":
+ function_call_item = item
+ break
+
+ if function_call_item:
+ provider_specific_fields = function_call_item.get("provider_specific_fields")
+ if provider_specific_fields:
+ thought_signature = provider_specific_fields.get("thought_signature")
+ if thought_signature:
+ assert isinstance(thought_signature, str), "thought_signature should be a string"
+ assert len(thought_signature) > 0, "thought_signature should not be empty"
+ print(f"✅ Streaming thought signature preserved: {thought_signature[:50]}...")
+
+ print(f"✅ Collected {len(chunks)} streaming chunks")
+
+
class TestGoogleAIStudioResponsesAPITest(BaseResponsesAPITest):
def get_base_completion_call_args(self):
#litellm._turn_on_debug()
diff --git a/tests/llm_translation/test_optional_params.py b/tests/llm_translation/test_optional_params.py
index dd42d5e1c5..bc85b99eee 100644
--- a/tests/llm_translation/test_optional_params.py
+++ b/tests/llm_translation/test_optional_params.py
@@ -23,6 +23,7 @@ from litellm.utils import (
get_optional_params,
get_optional_params_embeddings,
get_optional_params_image_gen,
+ get_requester_metadata,
)
## get_optional_params_embeddings
@@ -67,6 +68,41 @@ def test_anthropic_optional_params(stop_sequence, expected_count):
assert len(optional_params) == expected_count
+def test_get_requester_metadata_returns_none_for_empty():
+ metadata = {"requester_metadata": {}}
+ assert get_requester_metadata(metadata) is None
+
+
+@patch("litellm.main.openai_chat_completions.completion")
+def test_requester_metadata_forwarded_to_openai(mock_completion):
+ mock_completion.return_value = MagicMock()
+ metadata = {
+ "requester_metadata": {
+ "custom_meta_key": "value",
+ "hidden_params": "secret",
+ "int_value": 123,
+ }
+ }
+
+ original_api_key = litellm.api_key
+ litellm.api_key = "sk-test"
+ original_preview_flag = litellm.enable_preview_features
+ litellm.enable_preview_features = True
+
+ try:
+ litellm.completion(
+ model="gpt-4o",
+ messages=[{"role": "user", "content": "hi"}],
+ metadata=metadata,
+ )
+ finally:
+ litellm.api_key = original_api_key
+ litellm.enable_preview_features = original_preview_flag
+
+ sent_metadata = mock_completion.call_args.kwargs["optional_params"]["metadata"]
+ assert sent_metadata == {"custom_meta_key": "value"}
+
+
def test_get_optional_params_with_allowed_openai_params():
"""
Test if use can dynamically pass in allowed_openai_params to override default behavior
diff --git a/tests/llm_translation/test_prompt_factory.py b/tests/llm_translation/test_prompt_factory.py
index 2023f12825..1ed1e327ec 100644
--- a/tests/llm_translation/test_prompt_factory.py
+++ b/tests/llm_translation/test_prompt_factory.py
@@ -19,6 +19,7 @@ from litellm.litellm_core_utils.prompt_templates.factory import (
claude_2_1_pt,
convert_to_anthropic_image_obj,
convert_url_to_base64,
+ create_anthropic_image_param,
llama_2_chat_pt,
prompt_factory,
)
@@ -207,6 +208,125 @@ def test_base64_image_input(url, expected_media_type):
assert response["media_type"] == expected_media_type
+def test_create_anthropic_image_param_with_http_url():
+ """Test that HTTP/HTTPS URLs are passed as URL references, not base64."""
+ image_param = create_anthropic_image_param(
+ "https://example.com/image.jpg", format=None
+ )
+
+ assert image_param["type"] == "image"
+ assert image_param["source"]["type"] == "url"
+ assert image_param["source"]["url"] == "https://example.com/image.jpg"
+
+
+def test_create_anthropic_image_param_with_https_url():
+ """Test that HTTPS URLs are passed as URL references."""
+ image_param = create_anthropic_image_param(
+ "https://example.com/image.png", format=None
+ )
+
+ assert image_param["type"] == "image"
+ assert image_param["source"]["type"] == "url"
+ assert image_param["source"]["url"] == "https://example.com/image.png"
+
+
+def test_create_anthropic_image_param_with_dict_input():
+ """Test that dict input with URL is handled correctly."""
+ image_param = create_anthropic_image_param(
+ {"url": "https://example.com/image.jpg", "format": "image/jpeg"}, format=None
+ )
+
+ assert image_param["type"] == "image"
+ assert image_param["source"]["type"] == "url"
+ assert image_param["source"]["url"] == "https://example.com/image.jpg"
+
+
+def test_create_anthropic_image_param_with_base64_data_uri():
+ """Test that data URIs are converted to base64."""
+ image_param = create_anthropic_image_param(
+ "data:image/jpeg;base64,/9j/4AAQSkZJRg==", format=None
+ )
+
+ assert image_param["type"] == "image"
+ assert image_param["source"]["type"] == "base64"
+ assert image_param["source"]["media_type"] == "image/jpeg"
+ assert image_param["source"]["data"] == "/9j/4AAQSkZJRg=="
+
+
+def test_create_anthropic_image_param_with_format_override():
+ """Test that format parameter can override media type."""
+ image_param = create_anthropic_image_param(
+ "data:image/jpeg;base64,1234", format="image/png"
+ )
+
+ assert image_param["type"] == "image"
+ assert image_param["source"]["type"] == "base64"
+ assert image_param["source"]["media_type"] == "image/png"
+
+
+def test_anthropic_messages_pt_with_url_image():
+ """Test that anthropic_messages_pt correctly handles HTTP/HTTPS URLs as URL references."""
+ messages = [
+ {
+ "role": "user",
+ "content": [
+ {"type": "text", "text": "What's in this image?"},
+ {
+ "type": "image_url",
+ "image_url": "https://example.com/image.jpg",
+ },
+ ],
+ }
+ ]
+
+ result = anthropic_messages_pt(
+ messages=messages, model="claude-3-5-sonnet", llm_provider="anthropic"
+ )
+
+ assert len(result) == 1
+ assert result[0]["role"] == "user"
+ assert isinstance(result[0]["content"], list)
+ assert len(result[0]["content"]) == 2
+
+ # Check text content
+ assert result[0]["content"][0]["type"] == "text"
+
+ # Check image content - should be URL reference, not base64
+ assert result[0]["content"][1]["type"] == "image"
+ assert result[0]["content"][1]["source"]["type"] == "url"
+ assert result[0]["content"][1]["source"]["url"] == "https://example.com/image.jpg"
+
+
+def test_anthropic_messages_pt_with_base64_image():
+ """Test that anthropic_messages_pt correctly handles data URIs as base64."""
+ messages = [
+ {
+ "role": "user",
+ "content": [
+ {"type": "text", "text": "What's in this image?"},
+ {
+ "type": "image_url",
+ "image_url": "data:image/jpeg;base64,/9j/4AAQSkZJRg==",
+ },
+ ],
+ }
+ ]
+
+ result = anthropic_messages_pt(
+ messages=messages, model="claude-3-5-sonnet", llm_provider="anthropic"
+ )
+
+ assert len(result) == 1
+ assert result[0]["role"] == "user"
+ assert isinstance(result[0]["content"], list)
+ assert len(result[0]["content"]) == 2
+
+ # Check image content - should be base64, not URL
+ assert result[0]["content"][1]["type"] == "image"
+ assert result[0]["content"][1]["source"]["type"] == "base64"
+ assert result[0]["content"][1]["source"]["media_type"] == "image/jpeg"
+
+
def test_anthropic_messages_tool_call():
messages = [
{
diff --git a/tests/openai_endpoints_tests/test_openai_batches_endpoint.py b/tests/openai_endpoints_tests/test_openai_batches_endpoint.py
index 9fc5962885..5a84900f87 100644
--- a/tests/openai_endpoints_tests/test_openai_batches_endpoint.py
+++ b/tests/openai_endpoints_tests/test_openai_batches_endpoint.py
@@ -291,4 +291,4 @@ async def test_list_batches_with_target_model_names():
# Verify the response structure
assert response["object"] == "list"
- assert len(response["data"]) > 0
+ assert len(response["data"]) > 0
\ No newline at end of file
diff --git a/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py b/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py
index 35e0cd4bec..8942239bb2 100644
--- a/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py
+++ b/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py
@@ -1339,6 +1339,78 @@ def test_vertex_ai_penalty_parameters_validation():
assert result["max_output_tokens"] == 100
+def test_vertex_ai_gemini_3_penalty_parameters_unsupported():
+ """
+ Test that penalty parameters are not supported for Gemini 3 models.
+
+ This test ensures that:
+ 1. Gemini 3 models do not support penalty parameters
+ 2. Penalty parameters are excluded from supported params list for Gemini 3 models
+ 3. Penalty parameters are filtered out when mapping params for Gemini 3 models
+ """
+ v = VertexGeminiConfig()
+
+ # Test Gemini 3 models
+ gemini_3_models = [
+ "gemini-3-pro-preview",
+ "vertex_ai/gemini-3-pro-preview",
+ "gemini/gemini-3-pro-preview",
+ ]
+
+ for model in gemini_3_models:
+ # Test _supports_penalty_parameters method
+ assert v._supports_penalty_parameters(model) == False, \
+ f"Gemini 3 model {model} should not support penalty parameters"
+
+ # Test get_supported_openai_params method
+ supported_params = v.get_supported_openai_params(model)
+ assert "frequency_penalty" not in supported_params, \
+ f"frequency_penalty should not be in supported params for {model}"
+ assert "presence_penalty" not in supported_params, \
+ f"presence_penalty should not be in supported params for {model}"
+
+ # Test parameter mapping - penalty params should be filtered out
+ non_default_params = {
+ "temperature": 0.7,
+ "frequency_penalty": 0.5,
+ "presence_penalty": 0.3,
+ "max_tokens": 100
+ }
+
+ optional_params = {}
+ result = v.map_openai_params(
+ non_default_params=non_default_params,
+ optional_params=optional_params,
+ model=model,
+ drop_params=False
+ )
+
+ # Penalty parameters should be filtered out for Gemini 3 models
+ assert "frequency_penalty" not in result, \
+ f"frequency_penalty should be filtered out for Gemini 3 model {model}"
+ assert "presence_penalty" not in result, \
+ f"presence_penalty should be filtered out for Gemini 3 model {model}"
+
+ # Other parameters should still be included
+ assert "temperature" in result, \
+ f"temperature should still be included for Gemini 3 model {model}"
+ assert "max_output_tokens" in result, \
+ f"max_output_tokens should still be included for Gemini 3 model {model}"
+ assert result["temperature"] == 0.7
+ assert result["max_output_tokens"] == 100
+
+ # Test that non-Gemini 3 models still support penalty parameters (if they're not in the unsupported list)
+ non_gemini_3_model = "gemini-2.5-pro"
+ assert v._supports_penalty_parameters(non_gemini_3_model) == True, \
+ f"Non-Gemini 3 model {non_gemini_3_model} should support penalty parameters"
+
+ supported_params = v.get_supported_openai_params(non_gemini_3_model)
+ assert "frequency_penalty" in supported_params, \
+ f"frequency_penalty should be in supported params for {non_gemini_3_model}"
+ assert "presence_penalty" in supported_params, \
+ f"presence_penalty should be in supported params for {non_gemini_3_model}"
+
+
def test_vertex_ai_annotation_streaming_events():
"""
Test that annotation events are properly emitted during streaming for Vertex AI Gemini.
diff --git a/tests/test_litellm/test_main.py b/tests/test_litellm/test_main.py
index 9f209c1b8d..be4bb4e48a 100644
--- a/tests/test_litellm/test_main.py
+++ b/tests/test_litellm/test_main.py
@@ -192,8 +192,33 @@ async def test_url_with_format_param(model, sync_mode, monkeypatch):
json_str = json_str.decode("utf-8")
print(f"type of json_str: {type(json_str)}")
- assert "png" in json_str
- assert "jpeg" not in json_str
+
+ # Bedrock models convert URLs to base64, while direct Anthropic models support URLs
+ # bedrock/invoke models use Anthropic messages API which supports URLs
+ if model.startswith("bedrock/invoke/"):
+ # bedrock/invoke should convert URLs to base64 (doesn't support URL references)
+ # URL should NOT be in the JSON (it should be converted to base64)
+ assert "https://upload.wikimedia.org" not in json_str
+ # Should have base64 data in the source (type="base64", not type="url")
+ assert '"type":"base64"' in json_str or '"type": "base64"' in json_str
+ # Should have "data" field containing base64 content
+ assert '"data"' in json_str
+ elif model.startswith("bedrock/"):
+ # Regular Bedrock models should convert URLs to base64 (uses "bytes" field)
+ # URL should NOT be in the JSON (it should be converted to base64)
+ assert "https://upload.wikimedia.org" not in json_str
+ # Should have "bytes" field (Bedrock uses "bytes" not "base64" in the field name)
+ assert '"bytes"' in json_str or '"bytes":' in json_str
+ elif model.startswith("anthropic/"):
+ # Direct Anthropic models should pass HTTPS URLs directly (HTTP URLs are converted to base64)
+ # Since we're using HTTPS URL, it should be passed as-is
+ assert "https://upload.wikimedia.org" in json_str
+ # For Anthropic, URL references use "url" type, not base64
+ assert '"type":"url"' in json_str or '"type": "url"' in json_str
+ else:
+ # For other models, check format parameter is respected
+ assert "png" in json_str
+ assert "jpeg" not in json_str
@pytest.mark.parametrize("model", ["gpt-4o-mini"])