diff --git a/docs/my-website/blog/gemini_3/index.md b/docs/my-website/blog/gemini_3/index.md index 2f4cc5bb0f..1b9ff359f3 100644 --- a/docs/my-website/blog/gemini_3/index.md +++ b/docs/my-website/blog/gemini_3/index.md @@ -4,8 +4,8 @@ title: "DAY 0 Support: Gemini 3 on LiteLLM" date: 2025-11-19T10:00:00 authors: - name: Sameer Kankute - title: "SWE @ LiteLLM (LLM Translation)" - url: https://in.linkedin.com/in/sameer-kankute + title: SWE @ LiteLLM (LLM Translation) + url: https://www.linkedin.com/in/sameer-kankute/ image_url: https://media.licdn.com/dms/image/v2/D4D03AQHB_loQYd5gjg/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1719137160975?e=1765411200&v=beta&t=c8396f--_lH6Fb_pVvx_jGholPfcl0bvwmNynbNdnII - name: Krrish Dholakia title: "CEO, LiteLLM" @@ -88,9 +88,11 @@ curl http://0.0.0.0:4000/v1/chat/completions \ LiteLLM provides **full end-to-end support** for Gemini 3 Pro Preview on: - ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint +- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming) - ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint +- ✅ `/v1/generateContent` – [Google Gemini API](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini#rest) compatible endpoint (for code, see: `client.models.generate_content(...)`) -Both endpoints support: +All endpoints support: - Streaming and non-streaming responses - Function calling with thought signatures - Multi-turn conversations @@ -548,6 +550,129 @@ curl http://localhost:4000/v1/chat/completions \ 3. **Automatic Defaults**: If you don't specify `reasoning_effort`, LiteLLM automatically sets `thinking_level="low"` for optimal performance. +## Cost Tracking: Prompt Caching & Context Window + +LiteLLM provides comprehensive cost tracking for Gemini 3 Pro Preview, including support for prompt caching and tiered pricing based on context window size. + +### Prompt Caching Cost Tracking + +Gemini 3 supports prompt caching, which allows you to cache frequently used prompt prefixes to reduce costs. LiteLLM automatically tracks and calculates costs for: + +- **Cache Hit Tokens**: Tokens that are read from cache (charged at a lower rate) +- **Cache Creation Tokens**: Tokens that are written to cache (one-time cost) +- **Text Tokens**: Regular prompt tokens that are processed normally + +#### How It Works + +LiteLLM extracts caching information from the `prompt_tokens_details` field in the usage object: + +```python +{ + "usage": { + "prompt_tokens": 50000, + "completion_tokens": 1000, + "total_tokens": 51000, + "prompt_tokens_details": { + "cached_tokens": 30000, # Cache hit tokens + "cache_creation_tokens": 5000, # Tokens written to cache + "text_tokens": 15000 # Regular processed tokens + } + } +} +``` + +### Context Window Tiered Pricing + +Gemini 3 Pro Preview supports up to 1M tokens of context, with tiered pricing that automatically applies when your prompt exceeds 200k tokens. + +#### Automatic Tier Detection + +LiteLLM automatically detects when your prompt exceeds the 200k token threshold and applies the appropriate tiered pricing: + +```python +from litellm import completion_cost + +# Example: Small prompt (< 200k tokens) +response_small = completion( + model="gemini/gemini-3-pro-preview", + messages=[{"role": "user", "content": "Hello!"}] +) +# Uses base pricing: $0.000002/input token, $0.000012/output token + +# Example: Large prompt (> 200k tokens) +response_large = completion( + model="gemini/gemini-3-pro-preview", + messages=[{"role": "user", "content": "..." * 250000}] # 250k tokens +) +# Automatically uses tiered pricing: $0.000004/input token, $0.000018/output token +``` + +#### Cost Breakdown + +The cost calculation includes: + +1. **Text Processing Cost**: Regular tokens processed at base or tiered rate +2. **Cache Read Cost**: Cached tokens read at discounted rate +3. **Cache Creation Cost**: One-time cost for writing tokens to cache (applies tiered rate if above 200k) +4. **Output Cost**: Generated tokens at base or tiered rate + +### Example: Viewing Cost Breakdown + +You can view the detailed cost breakdown using LiteLLM's cost tracking: + +```python +from litellm import completion, completion_cost + +response = completion( + model="gemini/gemini-3-pro-preview", + messages=[{"role": "user", "content": "Explain prompt caching"}], + caching=True # Enable prompt caching +) + +# Get total cost +total_cost = completion_cost(completion_response=response) +print(f"Total cost: ${total_cost:.6f}") + +# Access usage details +usage = response.usage +print(f"Prompt tokens: {usage.prompt_tokens}") +print(f"Completion tokens: {usage.completion_tokens}") + +# Access caching details +if usage.prompt_tokens_details: + print(f"Cache hit tokens: {usage.prompt_tokens_details.cached_tokens}") + print(f"Cache creation tokens: {usage.prompt_tokens_details.cache_creation_tokens}") + print(f"Text tokens: {usage.prompt_tokens_details.text_tokens}") +``` + +### Cost Optimization Tips + +1. **Use Prompt Caching**: For repeated prompt prefixes, enable caching to reduce costs by up to 90% for cached portions +2. **Monitor Context Size**: Be aware that prompts above 200k tokens use tiered pricing (2x for input, 1.5x for output) +3. **Cache Management**: Cache creation tokens are charged once when writing to cache, then subsequent reads are much cheaper +4. **Track Usage**: Use LiteLLM's built-in cost tracking to monitor spending across different token types + +### Integration with LiteLLM Proxy + +When using LiteLLM Proxy, all cost tracking is automatically logged and available through: + +- **Usage Logs**: Detailed token and cost breakdowns in proxy logs +- **Budget Management**: Set budgets and alerts based on actual usage +- **Analytics Dashboard**: View cost trends and breakdowns by token type + +```yaml +# config.yaml +model_list: + - model_name: gemini-3-pro-preview + litellm_params: + model: gemini/gemini-3-pro-preview + api_key: os.environ/GEMINI_API_KEY + +litellm_settings: + # Enable detailed cost tracking + success_callback: ["langfuse"] # or your preferred logging service +``` + ## Using with Claude Code CLI You can use `gemini-3-pro-preview` with **Claude Code CLI** - Anthropic's command-line interface. This allows you to use Gemini 3 Pro Preview with Claude Code's native syntax and workflows. @@ -628,6 +753,162 @@ $ claude --model gemini-3-pro-preview - Ensure `GEMINI_API_KEY` is set correctly - Check LiteLLM proxy logs for detailed error messages +## Responses API Support + +LiteLLM fully supports the OpenAI Responses API for Gemini 3 Pro Preview, including both streaming and non-streaming modes. The Responses API provides a structured way to handle multi-turn conversations with function calling, and LiteLLM automatically preserves thought signatures throughout the conversation. + +### Example: Using Responses API with Gemini 3 + + + + +```python +from openai import OpenAI +import json + +client = OpenAI() + +# 1. Define a list of callable tools for the model +tools = [ + { + "type": "function", + "name": "get_horoscope", + "description": "Get today's horoscope for an astrological sign.", + "parameters": { + "type": "object", + "properties": { + "sign": { + "type": "string", + "description": "An astrological sign like Taurus or Aquarius", + }, + }, + "required": ["sign"], + }, + }, +] + +def get_horoscope(sign): + return f"{sign}: Next Tuesday you will befriend a baby otter." + +# Create a running input list we will add to over time +input_list = [ + {"role": "user", "content": "What is my horoscope? I am an Aquarius."} +] + +# 2. Prompt the model with tools defined +response = client.responses.create( + model="gemini-3-pro-preview", + tools=tools, + input=input_list, +) + +# Save function call outputs for subsequent requests +input_list += response.output + +for item in response.output: + if item.type == "function_call": + if item.name == "get_horoscope": + # 3. Execute the function logic for get_horoscope + horoscope = get_horoscope(json.loads(item.arguments)) + + # 4. Provide function call results to the model + input_list.append({ + "type": "function_call_output", + "call_id": item.call_id, + "output": json.dumps({ + "horoscope": horoscope + }) + }) + +print("Final input:") +print(input_list) + +response = client.responses.create( + model="gemini-3-pro-preview", + instructions="Respond only with a horoscope generated by a tool.", + tools=tools, + input=input_list, +) + +# 5. The model should be able to give a response! +print("Final output:") +print(response.model_dump_json(indent=2)) +print("\n" + response.output_text) +``` + +**Key Points:** +- ✅ Thought signatures are automatically preserved in function calls +- ✅ Works seamlessly with multi-turn conversations +- ✅ All Gemini 3-specific features are fully supported + + + + +```python +from openai import OpenAI +import json + +client = OpenAI() + +tools = [ + { + "type": "function", + "name": "get_horoscope", + "description": "Get today's horoscope for an astrological sign.", + "parameters": { + "type": "object", + "properties": { + "sign": { + "type": "string", + "description": "An astrological sign like Taurus or Aquarius", + }, + }, + "required": ["sign"], + }, + }, +] + +def get_horoscope(sign): + return f"{sign}: Next Tuesday you will befriend a baby otter." + +input_list = [ + {"role": "user", "content": "What is my horoscope? I am an Aquarius."} +] + +# Streaming mode +response = client.responses.create( + model="gemini-3-pro-preview", + tools=tools, + input=input_list, + stream=True, +) + +# Collect all chunks +chunks = [] +for chunk in response: + chunks.append(chunk) + # Process streaming chunks as they arrive + print(chunk) + +# Thought signatures are automatically preserved in streaming mode +``` + +**Key Points:** +- ✅ Streaming mode fully supported +- ✅ Thought signatures preserved across streaming chunks +- ✅ Real-time processing of function calls and responses + + + + +### Responses API Benefits + +- ✅ **Structured Output**: Responses API provides a clear structure for handling function calls and multi-turn conversations +- ✅ **Thought Signature Preservation**: LiteLLM automatically preserves thought signatures in both streaming and non-streaming modes +- ✅ **Seamless Integration**: Works with existing OpenAI SDK patterns +- ✅ **Full Feature Support**: All Gemini 3 features (thought signatures, function calling, reasoning) are fully supported + + ## Best Practices #### 1. Always Include Thought Signatures in Conversation History @@ -665,6 +946,7 @@ When switching from non-Gemini-3 to Gemini-3: - ✅ No manual intervention needed - ✅ Conversation history continues seamlessly + ## Troubleshooting #### Issue: Missing Thought Signatures diff --git a/docs/my-website/docs/batches.md b/docs/my-website/docs/batches.md index 1bd4c700ae..269fee0310 100644 --- a/docs/my-website/docs/batches.md +++ b/docs/my-website/docs/batches.md @@ -174,6 +174,257 @@ print("list_batches_response=", list_batches_response) +## Multi-Account / Model-Based Routing + +Route batch operations to different provider accounts using model-specific credentials from your `config.yaml`. This eliminates the need for environment variables and enables multi-tenant batch processing. + +### How It Works + +**Priority Order:** +1. **Encoded Batch/File ID** (highest) - Model info embedded in the ID +2. **Model Parameter** - Via header (`x-litellm-model`), query param, or request body +3. **Custom Provider** (fallback) - Uses environment variables + +### Configuration + +```yaml +model_list: + - model_name: gpt-4o-account-1 + litellm_params: + model: openai/gpt-4o + api_key: sk-account-1-key + api_base: https://api.openai.com/v1 + + - model_name: gpt-4o-account-2 + litellm_params: + model: openai/gpt-4o + api_key: sk-account-2-key + api_base: https://api.openai.com/v1 + + - model_name: azure-batches + litellm_params: + model: azure/gpt-4 + api_key: azure-key-123 + api_base: https://my-resource.openai.azure.com + api_version: "2024-02-01" +``` + +### Usage Examples + +#### Scenario 1: Encoded File ID with Model + +When you upload a file with a model parameter, LiteLLM encodes the model information in the file ID. All subsequent operations automatically use those credentials. + +```bash +# Step 1: Upload file with model +curl http://localhost:4000/v1/files \ + -H "Authorization: Bearer sk-1234" \ + -H "x-litellm-model: gpt-4o-account-1" \ + -F purpose="batch" \ + -F file="@batch.jsonl" + +# Response includes encoded file ID: +# { +# "id": "file-bGl0ZWxsbTpmaWxlLUxkaUwzaVYxNGZRVlpYcU5KVEdkSjk7bW9kZWwsZ3B0LTRvLWFjY291bnQtMQ", +# ... +# } + +# Step 2: Create batch - automatically routes to gpt-4o-account-1 +curl http://localhost:4000/v1/batches \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{ + "input_file_id": "file-bGl0ZWxsbTpmaWxlLUxkaUwzaVYxNGZRVlpYcU5KVEdkSjk7bW9kZWwsZ3B0LTRvLWFjY291bnQtMQ", + "endpoint": "/v1/chat/completions", + "completion_window": "24h" + }' + +# Batch ID is also encoded with model: +# { +# "id": "batch_bGl0ZWxsbTpiYXRjaF82OTIwM2IzNjg0MDQ4MTkwYTA3ODQ5NDY3YTFjMDJkYTttb2RlbCxncHQtNG8tYWNjb3VudC0x", +# "input_file_id": "file-bGl0ZWxsbTpmaWxlLUxkaUwzaVYxNGZRVlpYcU5KVEdkSjk7bW9kZWwsZ3B0LTRvLWFjY291bnQtMQ", +# ... +# } + +# Step 3: Retrieve batch - automatically routes to gpt-4o-account-1 +curl http://localhost:4000/v1/batches/batch_bGl0ZWxsbTpiYXRjaF82OTIwM2IzNjg0MDQ4MTkwYTA3ODQ5NDY3YTFjMDJkYTttb2RlbCxncHQtNG8tYWNjb3VudC0x \ + -H "Authorization: Bearer sk-1234" +``` + +**✅ Benefits:** +- No need to specify model on every request +- File and batch IDs "remember" which account created them +- Automatic routing for retrieve, cancel, and file content operations + +#### Scenario 2: Model via Header/Query Parameter + +Specify the model for each request without encoding it in the ID. + +```bash +# Create batch with model header +curl http://localhost:4000/v1/batches \ + -H "Authorization: Bearer sk-1234" \ + -H "x-litellm-model: gpt-4o-account-2" \ + -H "Content-Type: application/json" \ + -d '{ + "input_file_id": "file-abc123", + "endpoint": "/v1/chat/completions", + "completion_window": "24h" + }' + +# Or use query parameter +curl "http://localhost:4000/v1/batches?model=gpt-4o-account-2" \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{ + "input_file_id": "file-abc123", + "endpoint": "/v1/chat/completions", + "completion_window": "24h" + }' + +# List batches for specific model +curl "http://localhost:4000/v1/batches?model=gpt-4o-account-2" \ + -H "Authorization: Bearer sk-1234" +``` + +**✅ Use Case:** +- One-off batch operations +- Different models for different operations +- Explicit control over routing + +#### Scenario 3: Environment Variables (Fallback) + +Traditional approach using environment variables when no model is specified. + +```bash +export OPENAI_API_KEY="sk-env-key" + +curl http://localhost:4000/v1/batches \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{ + "input_file_id": "file-abc123", + "endpoint": "/v1/chat/completions", + "completion_window": "24h" + }' +``` + +**✅ Use Case:** +- Backward compatibility +- Simple single-account setups +- Quick prototyping + +### Complete Multi-Account Example + +```bash +# Upload file to Account 1 +FILE_1=$(curl -s http://localhost:4000/v1/files \ + -H "x-litellm-model: gpt-4o-account-1" \ + -F purpose="batch" \ + -F file="@batch1.jsonl" | jq -r '.id') + +# Upload file to Account 2 +FILE_2=$(curl -s http://localhost:4000/v1/files \ + -H "x-litellm-model: gpt-4o-account-2" \ + -F purpose="batch" \ + -F file="@batch2.jsonl" | jq -r '.id') + +# Create batch on Account 1 (auto-routed via encoded file ID) +BATCH_1=$(curl -s http://localhost:4000/v1/batches \ + -d "{\"input_file_id\": \"$FILE_1\", \"endpoint\": \"/v1/chat/completions\", \"completion_window\": \"24h\"}" | jq -r '.id') + +# Create batch on Account 2 (auto-routed via encoded file ID) +BATCH_2=$(curl -s http://localhost:4000/v1/batches \ + -d "{\"input_file_id\": \"$FILE_2\", \"endpoint\": \"/v1/chat/completions\", \"completion_window\": \"24h\"}" | jq -r '.id') + +# Retrieve both batches (auto-routed to correct accounts) +curl http://localhost:4000/v1/batches/$BATCH_1 +curl http://localhost:4000/v1/batches/$BATCH_2 + +# List batches per account +curl "http://localhost:4000/v1/batches?model=gpt-4o-account-1" +curl "http://localhost:4000/v1/batches?model=gpt-4o-account-2" +``` + +### SDK Usage with Model Routing + +```python +import litellm +import asyncio + +# Upload file with model routing +file_obj = await litellm.acreate_file( + file=open("batch.jsonl", "rb"), + purpose="batch", + model="gpt-4o-account-1", # Route to specific account +) + +print(f"File ID: {file_obj.id}") +# File ID is encoded with model info + +# Create batch - automatically uses gpt-4o-account-1 credentials +batch = await litellm.acreate_batch( + completion_window="24h", + endpoint="/v1/chat/completions", + input_file_id=file_obj.id, # Model info embedded in ID +) + +print(f"Batch ID: {batch.id}") +# Batch ID is also encoded + +# Retrieve batch - automatically routes to correct account +retrieved = await litellm.aretrieve_batch( + batch_id=batch.id, # Model info embedded in ID +) + +print(f"Batch status: {retrieved.status}") + +# Or explicitly specify model +batch2 = await litellm.acreate_batch( + completion_window="24h", + endpoint="/v1/chat/completions", + input_file_id="file-regular-id", + model="gpt-4o-account-2", # Explicit routing +) +``` + +### How ID Encoding Works + +LiteLLM encodes model information into file and batch IDs using base64: + +``` +Original: file-abc123 +Encoded: file-bGl0ZWxsbTpmaWxlLWFiYzEyMzttb2RlbCxncHQtNG8tdGVzdA + └─┬─┘ └──────────────────┬──────────────────────┘ + prefix base64(litellm:file-abc123;model,gpt-4o-test) + +Original: batch_xyz789 +Encoded: batch_bGl0ZWxsbTpiYXRjaF94eXo3ODk7bW9kZWwsZ3B0LTRvLXRlc3Q + └──┬──┘ └──────────────────┬──────────────────────┘ + prefix base64(litellm:batch_xyz789;model,gpt-4o-test) +``` + +The encoding: +- ✅ Preserves OpenAI-compatible prefixes (`file-`, `batch_`) +- ✅ Is transparent to clients +- ✅ Enables automatic routing without additional parameters +- ✅ Works across all batch and file endpoints + +### Supported Endpoints + +All batch and file endpoints support model-based routing: + +| Endpoint | Method | Model Routing | +|----------|--------|---------------| +| `/v1/files` | POST | ✅ Via header/query/body | +| `/v1/files/{file_id}` | GET | ✅ Auto from encoded ID + header/query | +| `/v1/files/{file_id}/content` | GET | ✅ Auto from encoded ID + header/query | +| `/v1/files/{file_id}` | DELETE | ✅ Auto from encoded ID | +| `/v1/batches` | POST | ✅ Auto from file ID + header/query/body | +| `/v1/batches` | GET | ✅ Via header/query | +| `/v1/batches/{batch_id}` | GET | ✅ Auto from encoded ID | +| `/v1/batches/{batch_id}/cancel` | POST | ✅ Auto from encoded ID | + ## **Supported Providers**: ### [Azure OpenAI](./providers/azure#azure-batches-api) ### [OpenAI](#quick-start) diff --git a/docs/my-website/docs/files_endpoints.md b/docs/my-website/docs/files_endpoints.md index 88493fe0bb..fc0484e921 100644 --- a/docs/my-website/docs/files_endpoints.md +++ b/docs/my-website/docs/files_endpoints.md @@ -16,7 +16,137 @@ Use this to call the provider's `/files` endpoints directly, in the OpenAI forma - Delete File - Get File Content +## Multi-Account Support (Multiple OpenAI Keys) +Use different OpenAI API keys for files and batches by specifying a `model` parameter that references entries in your `model_list`. This approach works **without requiring a database** and allows you to route files/batches to different OpenAI accounts. + +### How It Works + +1. Define models in `model_list` with different API keys +2. Pass `model` parameter when creating files +3. LiteLLM returns encoded IDs that contain routing information +4. Use encoded IDs for all subsequent operations (retrieve, delete, batches) +5. No need to specify model again - routing info is in the ID + +### Setup + +```yaml +model_list: + # litellm OpenAI Account + - model_name: "gpt-4o-litellm" + litellm_params: + model: openai/gpt-4o + api_key: os.environ/OPENAI_LITELLM_API_KEY + + # Free OpenAI Account + - model_name: "gpt-4o-free" + litellm_params: + model: openai/gpt-4o + api_key: os.environ/OPENAI_FREE_API_KEY +``` + +### Usage Example + +```python +from openai import OpenAI + +client = OpenAI( + api_key="sk-1234", # Your LiteLLM proxy key + base_url="http://0.0.0.0:4000" +) + +# Create file using litellm account +file_response = client.files.create( + file=open("batch_data.jsonl", "rb"), + purpose="batch", + extra_body={"model": "gpt-4o-litellm"} # Routes to litellm key +) +print(f"File ID: {file_response.id}") +# Returns encoded ID like: file-bGl0ZWxsbTpmaWxlLWFiYzEyMzttb2RlbCxncHQtNG8taWZvb2Q + +# Create batch using the encoded file ID +# No need to specify model again - it's embedded in the file ID +batch_response = client.batches.create( + input_file_id=file_response.id, # Encoded ID + endpoint="/v1/chat/completions", + completion_window="24h" +) +print(f"Batch ID: {batch_response.id}") +# Returns encoded batch ID with routing info + +# Retrieve batch - routing happens automatically +batch_status = client.batches.retrieve(batch_response.id) +print(f"Status: {batch_status.status}") + +# List files for a specific account +files = client.files.list( + extra_body={"model": "gpt-4o-free"} # List free files +) + +# List batches for a specific account +batches = client.batches.list( + extra_query={"model": "gpt-4o-litellm"} # List litellm batches +) +``` + +### Parameter Options + +You can pass the `model` parameter via: +- **Request body**: `extra_body={"model": "gpt-4o-litellm"}` +- **Query parameter**: `?model=gpt-4o-litellm` +- **Header**: `x-litellm-model: gpt-4o-litellm` + +### How Encoded IDs Work + +- When you create a file/batch with a `model` parameter, LiteLLM encodes the model name into the returned ID +- The encoded ID is base64-encoded and looks like: `file-bGl0ZWxsbTpmaWxlLWFiYzEyMzttb2RlbCxncHQtNG8taWZvb2Q` +- When you use this ID in subsequent operations (retrieve, delete, batch create), LiteLLM automatically: + 1. Decodes the ID + 2. Extracts the model name + 3. Looks up the credentials + 4. Routes the request to the correct OpenAI account +- The original provider file/batch ID is preserved internally + +### Benefits + +✅ **No Database Required** - All routing info stored in the ID +✅ **Stateless** - Works across proxy restarts +✅ **Simple** - Just pass the ID around like normal +✅ **Backward Compatible** - Existing `custom_llm_provider` and `files_settings` still work +✅ **Future-Proof** - Aligns with managed batches approach + +### Migration from files_settings + +**Old approach (still works):** +```yaml +files_settings: + - custom_llm_provider: openai + api_key: os.environ/OPENAI_KEY +``` + +```python +# Had to specify provider on every call +client.files.create(..., extra_headers={"custom-llm-provider": "openai"}) +client.files.retrieve(file_id, extra_headers={"custom-llm-provider": "openai"}) +``` + +**New approach (recommended):** +```yaml +model_list: + - model_name: "gpt-4o-account1" + litellm_params: + model: openai/gpt-4o + api_key: os.environ/OPENAI_KEY +``` + +```python +# Specify model once on create +file = client.files.create(..., extra_body={"model": "gpt-4o-account1"}) + +# Then just use the ID - routing is automatic +client.files.retrieve(file.id) # No need to specify account +client.batches.create(input_file_id=file.id) # Routes correctly +``` diff --git a/docs/my-website/docs/providers/openai.md b/docs/my-website/docs/providers/openai.md index 2163d5e611..9303cdffad 100644 --- a/docs/my-website/docs/providers/openai.md +++ b/docs/my-website/docs/providers/openai.md @@ -29,6 +29,18 @@ response = completion( ) ``` +:::info Metadata passthrough (preview) +When `litellm.enable_preview_features = True`, LiteLLM forwards only the values inside `metadata` to OpenAI. + +```python +completion( + model="gpt-4o", + messages=[{"role": "user", "content": "hi"}], + metadata= {"custom_meta_key": "value"}, +) +``` +::: + ### Usage - LiteLLM Proxy Server Here's how to call OpenAI models with the LiteLLM Proxy Server diff --git a/litellm/completion_extras/litellm_responses_transformation/transformation.py b/litellm/completion_extras/litellm_responses_transformation/transformation.py index 3ac5a28f90..f39d1bfb5f 100644 --- a/litellm/completion_extras/litellm_responses_transformation/transformation.py +++ b/litellm/completion_extras/litellm_responses_transformation/transformation.py @@ -93,6 +93,35 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge): choice = Choices(message=msg, finish_reason="stop", index=index) return choice, index + 1 + # Handle function_call items (e.g., from GPT-5 Codex format) + if item_type == "function_call": + # Extract provider_specific_fields if present and pass through as-is + provider_specific_fields = item.get("provider_specific_fields") + if provider_specific_fields and not isinstance(provider_specific_fields, dict): + provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {} + + tool_call_dict = { + "id": item.get("call_id") or item.get("id", ""), + "function": { + "name": item.get("name", ""), + "arguments": item.get("arguments", ""), + }, + "type": "function", + } + + # Pass through provider_specific_fields as-is if present + if provider_specific_fields: + tool_call_dict["provider_specific_fields"] = provider_specific_fields + # Also add to function's provider_specific_fields for consistency + tool_call_dict["function"]["provider_specific_fields"] = provider_specific_fields + + msg = Message( + content=None, + tool_calls=[tool_call_dict], + ) + choice = Choices(message=msg, finish_reason="tool_calls", index=index) + return choice, index + 1 + # Unknown or unsupported type return None, index @@ -257,7 +286,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge): return request_data - def transform_response( + def transform_response( # noqa: PLR0915 self, model: str, raw_response: "BaseModel", @@ -321,18 +350,37 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge): reasoning_content = None # flush reasoning content index += 1 elif isinstance(item, ResponseFunctionToolCall): + + provider_specific_fields = None + if hasattr(item, "provider_specific_fields") and item.provider_specific_fields: + provider_specific_fields = item.provider_specific_fields + if not isinstance(provider_specific_fields, dict): + provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {} + elif hasattr(item, "get") and callable(item.get): + provider_fields = item.get("provider_specific_fields") + if provider_fields: + provider_specific_fields = provider_fields if isinstance(provider_fields, dict) else (dict(provider_fields) if hasattr(provider_fields, "__dict__") else {}) + + function_dict: Dict[str, Any] = { + "name": item.name, + "arguments": item.arguments, + } + + if provider_specific_fields: + function_dict["provider_specific_fields"] = provider_specific_fields + + tool_call_dict: Dict[str, Any] = { + "id": item.call_id, + "function": function_dict, + "type": "function", + } + + if provider_specific_fields: + tool_call_dict["provider_specific_fields"] = provider_specific_fields + msg = Message( content=None, - tool_calls=[ - { - "id": item.call_id, - "function": { - "name": item.name, - "arguments": item.arguments, - }, - "type": "function", - } - ], + tool_calls=[tool_call_dict], reasoning_content=reasoning_content, ) @@ -630,7 +678,7 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator): return self.chunk_parser(json.loads(str_line)) - def chunk_parser( + def chunk_parser( # noqa: PLR0915 self, chunk: dict ) -> Union["GenericStreamingChunk", "ModelResponseStream"]: # Transform responses API streaming chunk to chat completion format @@ -667,17 +715,33 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator): # New output item added output_item = parsed_chunk.get("item", {}) if output_item.get("type") == "function_call": + # Extract provider_specific_fields if present + provider_specific_fields = output_item.get("provider_specific_fields") + if provider_specific_fields and not isinstance(provider_specific_fields, dict): + provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {} + + function_chunk = ChatCompletionToolCallFunctionChunk( + name=output_item.get("name", None), + arguments=parsed_chunk.get("arguments", ""), + ) + + if provider_specific_fields: + function_chunk["provider_specific_fields"] = provider_specific_fields + + tool_call_chunk = ChatCompletionToolCallChunk( + id=output_item.get("call_id"), + index=0, + type="function", + function=function_chunk, + ) + + # Add provider_specific_fields if present + if provider_specific_fields: + tool_call_chunk.provider_specific_fields = provider_specific_fields # type: ignore + return GenericStreamingChunk( text="", - tool_use=ChatCompletionToolCallChunk( - id=output_item.get("call_id"), - index=0, - type="function", - function=ChatCompletionToolCallFunctionChunk( - name=output_item.get("name", None), - arguments=parsed_chunk.get("arguments", ""), - ), - ), + tool_use=tool_call_chunk, is_finished=False, finish_reason="", usage=None, @@ -713,17 +777,34 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator): # New output item added output_item = parsed_chunk.get("item", {}) if output_item.get("type") == "function_call": + # Extract provider_specific_fields if present + provider_specific_fields = output_item.get("provider_specific_fields") + if provider_specific_fields and not isinstance(provider_specific_fields, dict): + provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {} + + function_chunk = ChatCompletionToolCallFunctionChunk( + name=output_item.get("name", None), + arguments="", # responses API sends everything again, we don't + ) + + # Add provider_specific_fields to function if present + if provider_specific_fields: + function_chunk["provider_specific_fields"] = provider_specific_fields + + tool_call_chunk = ChatCompletionToolCallChunk( + id=output_item.get("call_id"), + index=0, + type="function", + function=function_chunk, + ) + + # Add provider_specific_fields if present + if provider_specific_fields: + tool_call_chunk.provider_specific_fields = provider_specific_fields # type: ignore + return GenericStreamingChunk( text="", - tool_use=ChatCompletionToolCallChunk( - id=output_item.get("call_id"), - index=0, - type="function", - function=ChatCompletionToolCallFunctionChunk( - name=output_item.get("name", None), - arguments="", # responses API sends everything again, we don't - ), - ), + tool_use=tool_call_chunk, is_finished=True, finish_reason="tool_calls", usage=None, diff --git a/litellm/litellm_core_utils/prompt_templates/factory.py b/litellm/litellm_core_utils/prompt_templates/factory.py index 0f4a159975..262692d6d1 100644 --- a/litellm/litellm_core_utils/prompt_templates/factory.py +++ b/litellm/litellm_core_utils/prompt_templates/factory.py @@ -6,7 +6,7 @@ import mimetypes import re import xml.etree.ElementTree as ET from enum import Enum -from typing import Any, List, Optional, Tuple, cast, overload +from typing import Any, List, Optional, Tuple, Union, cast, overload from jinja2.sandbox import ImmutableSandboxedEnvironment @@ -910,6 +910,64 @@ def convert_to_anthropic_image_obj( ) +def create_anthropic_image_param( + image_url_input: Union[str, dict], + format: Optional[str] = None, + is_bedrock_invoke: bool = False +) -> AnthropicMessagesImageParam: + """ + Create an AnthropicMessagesImageParam from an image URL input. + + Supports both URL references (for HTTP/HTTPS URLs) and base64 encoding. + """ + # Extract URL and format from input + if isinstance(image_url_input, str): + image_url = image_url_input + else: + image_url = image_url_input.get("url", "") + if format is None: + format = image_url_input.get("format") + + # Check if the image URL is an HTTP/HTTPS URL + if image_url.startswith("http://") or image_url.startswith("https://"): + # For Bedrock invoke, always convert URLs to base64 (Bedrock invoke doesn't support URLs) + if is_bedrock_invoke or image_url.startswith("http://"): + base64_url = convert_url_to_base64(url=image_url) + image_chunk = convert_to_anthropic_image_obj( + openai_image_url=base64_url, format=format + ) + return AnthropicMessagesImageParam( + type="image", + source=AnthropicContentParamSource( + type="base64", + media_type=image_chunk["media_type"], + data=image_chunk["data"], + ), + ) + else: + # HTTPS URL - pass directly for regular Anthropic + return AnthropicMessagesImageParam( + type="image", + source=AnthropicContentParamSourceUrl( + type="url", + url=image_url, + ), + ) + else: + # Convert to base64 for data URIs or other formats + image_chunk = convert_to_anthropic_image_obj( + openai_image_url=image_url, format=format + ) + return AnthropicMessagesImageParam( + type="image", + source=AnthropicContentParamSource( + type="base64", + media_type=image_chunk["media_type"], + data=image_chunk["data"], + ), + ) + + # The following XML functions will be deprecated once JSON schema support is available on Bedrock and Vertex # ------------------------------------------------------------------------------ def convert_to_anthropic_tool_result_xml(message: dict) -> str: @@ -1012,15 +1070,35 @@ def anthropic_messages_pt_xml(messages: list): if isinstance(messages[msg_i]["content"], list): for m in messages[msg_i]["content"]: if m.get("type", "") == "image_url": - format = m["image_url"].get("format") - user_content.append( - { - "type": "image", - "source": convert_to_anthropic_image_obj( - m["image_url"]["url"], format=format - ), - } - ) + format = m["image_url"].get("format") if isinstance(m["image_url"], dict) else None + image_param = create_anthropic_image_param(m["image_url"], format=format) + # Convert to dict format for XML version + source = image_param["source"] + if isinstance(source, dict) and source.get("type") == "url": + # Type narrowing for URL source + url_source = cast(AnthropicContentParamSourceUrl, source) + user_content.append( + { + "type": "image", + "source": { + "type": "url", + "url": url_source["url"], + }, + } + ) + else: + # Type narrowing for base64 source + base64_source = cast(AnthropicContentParamSource, source) + user_content.append( + { + "type": "image", + "source": { + "type": "base64", + "media_type": base64_source["media_type"], + "data": base64_source["data"], + }, + } + ) elif m.get("type", "") == "text": user_content.append({"type": "text", "text": m["text"]}) else: @@ -1491,24 +1569,9 @@ def convert_to_anthropic_tool_result( ) ) elif content["type"] == "image_url": - if isinstance(content["image_url"], str): - image_chunk = convert_to_anthropic_image_obj( - content["image_url"], format=None - ) - else: - format = content["image_url"].get("format") - image_chunk = convert_to_anthropic_image_obj( - content["image_url"]["url"], format=format - ) + format = content["image_url"].get("format") if isinstance(content["image_url"], dict) else None anthropic_content_list.append( - AnthropicMessagesImageParam( - type="image", - source=AnthropicContentParamSource( - type="base64", - media_type=image_chunk["media_type"], - data=image_chunk["data"], - ), - ) + create_anthropic_image_param(content["image_url"], format=format) ) anthropic_content = anthropic_content_list @@ -1839,21 +1902,22 @@ def anthropic_messages_pt( # noqa: PLR0915 for m in user_message_types_block["content"]: if m.get("type", "") == "image_url": m = cast(ChatCompletionImageObject, m) - format: Optional[str] = None - if isinstance(m["image_url"], str): - image_chunk = convert_to_anthropic_image_obj( - openai_image_url=m["image_url"], format=None - ) + format = m["image_url"].get("format") if isinstance(m["image_url"], dict) else None + # Convert ChatCompletionImageUrlObject to dict if needed + image_url_value = m["image_url"] + if isinstance(image_url_value, str): + image_url_input: Union[str, dict[str, Any]] = image_url_value else: - format = m["image_url"].get("format") - image_chunk = convert_to_anthropic_image_obj( - openai_image_url=m["image_url"]["url"], - format=format, - ) - - _anthropic_content_element = ( - _anthropic_content_element_factory(image_chunk) - ) + # ChatCompletionImageUrlObject or dict case - convert to dict + image_url_input = { + "url": image_url_value["url"], + "format": image_url_value.get("format"), + } + # Bedrock invoke models have format: invoke/... + is_bedrock_invoke = model.lower().startswith("invoke/") + _anthropic_content_element = create_anthropic_image_param( + image_url_input, format=format, is_bedrock_invoke=is_bedrock_invoke + ) _content_element = add_cache_control_to_content( anthropic_content_element=_anthropic_content_element, original_content_element=dict(m), diff --git a/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py b/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py index 02e4682b3a..ab594c79ef 100644 --- a/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py +++ b/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py @@ -237,6 +237,9 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig): return False def _supports_penalty_parameters(self, model: str) -> bool: + # Gemini 3 models do not support penalty parameters + if VertexGeminiConfig._is_gemini_3_or_newer(model): + return False unsupported_models = ["gemini-2.5-pro-preview-06-05"] if model in unsupported_models: return False @@ -1344,7 +1347,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig): return False @staticmethod - def _calculate_usage( + def _calculate_usage( # noqa: PLR0915 completion_response: Union[ GenerateContentResponseBody, BidiGenerateContentServerMessage ], diff --git a/litellm/main.py b/litellm/main.py index 1e3826b9a6..b082b491f2 100644 --- a/litellm/main.py +++ b/litellm/main.py @@ -105,7 +105,7 @@ from litellm.utils import ( ProviderConfigManager, Usage, _get_model_info_helper, - add_openai_metadata, + get_requester_metadata, add_provider_specific_params_to_optional_params, async_mock_completion_streaming_obj, convert_to_model_response_object, @@ -2086,10 +2086,12 @@ def completion( # type: ignore # noqa: PLR0915 if extra_headers is not None: optional_params["extra_headers"] = extra_headers - if litellm.enable_preview_features: - metadata_payload = add_openai_metadata(metadata) - if metadata_payload is not None: - optional_params["metadata"] = metadata_payload + if ( + litellm.enable_preview_features and metadata is not None + ): # [PREVIEW] allow metadata to be passed to OPENAI + openai_metadata = get_requester_metadata(metadata) + if openai_metadata is not None: + optional_params["metadata"] = openai_metadata ## LOAD CONFIG - if set config = litellm.OpenAIConfig.get_config() diff --git a/litellm/proxy/batches_endpoints/endpoints.py b/litellm/proxy/batches_endpoints/endpoints.py index b0f13f6afb..98492bcc2d 100644 --- a/litellm/proxy/batches_endpoints/endpoints.py +++ b/litellm/proxy/batches_endpoints/endpoints.py @@ -22,10 +22,16 @@ from litellm.proxy.common_utils.openai_endpoint_utils import ( ) from litellm.proxy.openai_files_endpoints.common_utils import ( _is_base64_encoded_unified_file_id, + decode_model_from_file_id, + encode_file_id_with_model, get_batch_id_from_unified_batch_id, + get_credentials_for_model, get_model_id_from_unified_batch_id, get_models_from_unified_file_id, + get_original_file_id, + prepare_data_with_credentials, ) + from litellm.proxy.utils import handle_exception_on_proxy, is_known_model from litellm.types.llms.openai import LiteLLMBatchCreateRequest @@ -47,7 +53,7 @@ router = APIRouter() dependencies=[Depends(user_api_key_auth)], tags=["batch"], ) -async def create_batch( +async def create_batch( # noqa: PLR0915 request: Request, fastapi_response: Response, provider: Optional[str] = None, @@ -111,9 +117,59 @@ async def create_batch( _create_batch_data = LiteLLMBatchCreateRequest(**data) input_file_id = _create_batch_data.get("input_file_id", None) unified_file_id: Union[str, Literal[False]] = False + + model_from_file_id = None if input_file_id: + model_from_file_id = decode_model_from_file_id(input_file_id) unified_file_id = _is_base64_encoded_unified_file_id(input_file_id) - if ( + + # SCENARIO 1: File ID is encoded with model info + if model_from_file_id is not None and input_file_id: + credentials = get_credentials_for_model( + llm_router=llm_router, + model_id=model_from_file_id, + operation_context="batch creation (file created with model)", + ) + + original_file_id = get_original_file_id(input_file_id) + _create_batch_data["input_file_id"] = original_file_id + prepare_data_with_credentials( + data=_create_batch_data, # type: ignore + credentials=credentials, + ) + + # Create batch using model credentials + response = await litellm.acreate_batch( + custom_llm_provider=credentials["custom_llm_provider"], + **_create_batch_data # type: ignore + ) + + # Encode the batch ID and related file IDs with model information + if response and hasattr(response, "id") and response.id: + original_batch_id = response.id + encoded_batch_id = encode_file_id_with_model( + file_id=original_batch_id, model=model_from_file_id + ) + response.id = encoded_batch_id + + if hasattr(response, "output_file_id") and response.output_file_id: + response.output_file_id = encode_file_id_with_model( + file_id=response.output_file_id, model=model_from_file_id + ) + + if hasattr(response, "error_file_id") and response.error_file_id: + response.error_file_id = encode_file_id_with_model( + file_id=response.error_file_id, model=model_from_file_id + ) + + verbose_proxy_logger.debug( + f"Created batch using model: {model_from_file_id}, " + f"original_batch_id: {original_batch_id}, encoded: {encoded_batch_id}" + ) + + response.input_file_id = input_file_id + + elif ( litellm.enable_loadbalancing_on_batch_endpoints is True and is_router_model and router_model is not None @@ -155,9 +211,39 @@ async def create_batch( response.input_file_id = input_file_id response._hidden_params["unified_file_id"] = unified_file_id else: - response = await litellm.acreate_batch( - custom_llm_provider=custom_llm_provider, **_create_batch_data # type: ignore + # Check if model specified via header/query/body param + model_param = ( + data.get("model") + or request.query_params.get("model") + or request.headers.get("x-litellm-model") ) + + # SCENARIO 2 & 3: Model from header/query OR custom_llm_provider fallback + if model_param: + # SCENARIO 2: Use model-based routing from header/query/body + credentials = get_credentials_for_model( + llm_router=llm_router, + model_id=model_param, + operation_context="batch creation", + ) + + prepare_data_with_credentials( + data=_create_batch_data, # type: ignore + credentials=credentials, + ) + + # Create batch using model credentials + response = await litellm.acreate_batch( + custom_llm_provider=credentials["custom_llm_provider"], + **_create_batch_data # type: ignore + ) + + verbose_proxy_logger.debug(f"Created batch using model: {model_param}") + else: + # SCENARIO 3: Fallback to custom_llm_provider (uses env variables) + response = await litellm.acreate_batch( + custom_llm_provider=custom_llm_provider, **_create_batch_data # type: ignore + ) ### CALL HOOKS ### - modify outgoing data response = await proxy_logging_obj.post_call_success_hook( @@ -249,7 +335,7 @@ async def retrieve_batch( data: Dict = {} try: - ## check if model is a loadbalanced model + model_from_id = decode_model_from_file_id(batch_id) _retrieve_batch_request = RetrieveBatchRequest( batch_id=batch_id, ) @@ -271,7 +357,54 @@ async def retrieve_batch( route_type="aretrieve_batch", ) - if litellm.enable_loadbalancing_on_batch_endpoints is True or unified_batch_id: + # SCENARIO 1: Batch ID is encoded with model info + if model_from_id is not None: + credentials = get_credentials_for_model( + llm_router=llm_router, + model_id=model_from_id, + operation_context="batch retrieval (batch created with model)", + ) + + original_batch_id = get_original_file_id(batch_id) + prepare_data_with_credentials( + data=data, + credentials=credentials, + file_id=original_batch_id, # Sets data["batch_id"] = original_batch_id + ) + # Fix: The helper sets "file_id" but we need "batch_id" + data["batch_id"] = data.pop("file_id", original_batch_id) + + # Retrieve batch using model credentials + response = await litellm.aretrieve_batch( + custom_llm_provider=credentials["custom_llm_provider"], + **data # type: ignore + ) + + # Re-encode all IDs in the response + if response: + if hasattr(response, "id") and response.id: + response.id = batch_id # Keep the encoded batch ID + + if hasattr(response, "input_file_id") and response.input_file_id: + response.input_file_id = encode_file_id_with_model( + file_id=response.input_file_id, model=model_from_id + ) + + if hasattr(response, "output_file_id") and response.output_file_id: + response.output_file_id = encode_file_id_with_model( + file_id=response.output_file_id, model=model_from_id + ) + + if hasattr(response, "error_file_id") and response.error_file_id: + response.error_file_id = encode_file_id_with_model( + file_id=response.error_file_id, model=model_from_id + ) + + verbose_proxy_logger.debug( + f"Retrieved batch using model: {model_from_id}, original_id: {original_batch_id}" + ) + + elif litellm.enable_loadbalancing_on_batch_endpoints is True or unified_batch_id: if llm_router is None: raise HTTPException( status_code=500, @@ -282,6 +415,8 @@ async def retrieve_batch( response = await llm_router.aretrieve_batch(**data) # type: ignore response._hidden_params["unified_batch_id"] = unified_batch_id + + # SCENARIO 3: Fallback to custom_llm_provider (uses env variables) else: custom_llm_provider = ( provider @@ -405,9 +540,36 @@ async def list_batches( route_type="alist_batches", ) - ## check for target model names - target_model_names = target_model_names or data.get("target_model_names", None) - if target_model_names: + model_param = ( + data.get("model") + or request.query_params.get("model") + or request.headers.get("x-litellm-model") + ) + + # SCENARIO 2: Use model-based routing from header/query/body + if model_param: + credentials = get_credentials_for_model( + llm_router=llm_router, + model_id=model_param, + operation_context="batch listing", + ) + + data.update(credentials) + + response = await litellm.alist_batches( + custom_llm_provider=credentials["custom_llm_provider"], + after=after, + limit=limit, + **data # type: ignore + ) + + verbose_proxy_logger.debug(f"Listed batches using model: {model_param}") + + # SCENARIO 2 (alternative): target_model_names based routing + elif target_model_names or data.get("target_model_names", None): + target_model_names = target_model_names or data.get("target_model_names", None) + if target_model_names is None: + raise ValueError("target_model_names is required for this routing scenario") model = target_model_names.split(",")[0] response = await llm_router.alist_batches( model=model, @@ -415,6 +577,8 @@ async def list_batches( limit=limit, **data, ) + + # SCENARIO 3: Fallback to custom_llm_provider (uses env variables) else: custom_llm_provider = ( provider @@ -520,6 +684,9 @@ async def cancel_batch( verbose_proxy_logger.debug( "Request received by LiteLLM:\n{}".format(json.dumps(data, indent=4)), ) + + # Check for encoded batch ID with model info + model_from_id = decode_model_from_file_id(batch_id) unified_batch_id = _is_base64_encoded_unified_file_id(batch_id) # Include original request and headers in the data @@ -532,7 +699,35 @@ async def cancel_batch( proxy_config=proxy_config, ) - if unified_batch_id: + # SCENARIO 1: Batch ID is encoded with model info + if model_from_id is not None: + credentials = get_credentials_for_model( + llm_router=llm_router, + model_id=model_from_id, + operation_context="batch cancellation (batch created with model)", + ) + + original_batch_id = get_original_file_id(batch_id) + prepare_data_with_credentials( + data=data, + credentials=credentials, + file_id=original_batch_id, + ) + # Fix: The helper sets "file_id" but we need "batch_id" + data["batch_id"] = data.pop("file_id", original_batch_id) + + # Cancel batch using model credentials + response = await litellm.acancel_batch( + custom_llm_provider=credentials["custom_llm_provider"], + **data # type: ignore + ) + + verbose_proxy_logger.debug( + f"Cancelled batch using model: {model_from_id}, original_id: {original_batch_id}" + ) + + # SCENARIO 2: target_model_names based routing + elif unified_batch_id: if llm_router is None: raise HTTPException( status_code=500, @@ -552,6 +747,8 @@ async def cancel_batch( data["batch_id"] = model_batch_id response = await llm_router.acancel_batch(model=model, **data) # type: ignore + + # SCENARIO 3: Fallback to custom_llm_provider (uses env variables) else: custom_llm_provider = ( diff --git a/litellm/proxy/litellm_pre_call_utils.py b/litellm/proxy/litellm_pre_call_utils.py index 0553c36e3c..89c16ce824 100644 --- a/litellm/proxy/litellm_pre_call_utils.py +++ b/litellm/proxy/litellm_pre_call_utils.py @@ -894,6 +894,22 @@ async def add_litellm_data_to_request( # noqa: PLR0915 data["metadata"] ) + # Parse litellm_metadata if it's a string (e.g., from multipart/form-data or extra_body) + if "litellm_metadata" in data and data["litellm_metadata"] is not None: + if isinstance(data["litellm_metadata"], str): + parsed_litellm_metadata = safe_json_loads(data["litellm_metadata"]) + if not isinstance(parsed_litellm_metadata, dict): + verbose_proxy_logger.warning( + f"Failed to parse 'litellm_metadata' as JSON dict. Received value: {data['litellm_metadata']}" + ) + else: + data["litellm_metadata"] = parsed_litellm_metadata + # Merge litellm_metadata into the metadata variable (preserving existing values) + if isinstance(data["litellm_metadata"], dict): + for key, value in data["litellm_metadata"].items(): + if key not in data[_metadata_variable_name]: + data[_metadata_variable_name][key] = value + data = LiteLLMProxyRequestSetup.add_user_api_key_auth_to_request_metadata( data=data, user_api_key_dict=user_api_key_dict, diff --git a/litellm/proxy/openai_files_endpoints/common_utils.py b/litellm/proxy/openai_files_endpoints/common_utils.py index fcbe64409a..c5b58e06d4 100644 --- a/litellm/proxy/openai_files_endpoints/common_utils.py +++ b/litellm/proxy/openai_files_endpoints/common_utils.py @@ -76,3 +76,266 @@ def get_batch_id_from_unified_batch_id(file_id: str) -> str: return file_id.split("llm_batch_id:")[1].split(",")[0] else: return file_id.split("generic_response_id:")[1].split(",")[0] + + +def encode_file_id_with_model(file_id: str, model: str) -> str: + """ + Encode a file/batch ID with model routing information. + + Format: -;model,)> + The result preserves the original prefix (file-, batch_, etc.) for OpenAI compliance. + + Args: + file_id: Original file/batch ID from the provider (e.g., "file-abc123", "batch_xyz") + model: Model name from model_list (e.g., "gpt-4o-litellm") + + Returns: + Encoded ID starting with appropriate prefix and containing routing information + + Examples: + encode_file_id_with_model("file-abc123", "gpt-4o-litellm") + -> "file-bGl0ZWxsbTpmaWxlLWFiYzEyMzttb2RlbCxncHQtNG8taWZvb2Q" + + encode_file_id_with_model("batch_abc123", "gpt-4o-test") + -> "batch_bGl0ZWxsbTpiYXRjaF9hYmMxMjM7bW9kZWwsZ3B0LTRvLXRlc3Q" + """ + encoded_str = f"litellm:{file_id};model,{model}" + encoded_bytes = base64.urlsafe_b64encode(encoded_str.encode()) + encoded_b64 = encoded_bytes.decode().rstrip("=") + + # Detect the prefix from the original ID (file-, batch_, etc.) + # Default to "file-" if no recognizable prefix + if file_id.startswith("batch_"): + prefix = "batch_" + elif file_id.startswith("file-"): + prefix = "file-" + else: + # Default to file- for backward compatibility + prefix = "file-" + + return f"{prefix}{encoded_b64}" + + +def decode_model_from_file_id(encoded_id: str) -> Optional[str]: + """ + Extract model name from an encoded file/batch ID. + Handles IDs that start with "file-" or "batch_" prefix. + """ + try: + if not isinstance(encoded_id, str): + return None + + # Remove prefix if present (file-, batch_, etc.) + if encoded_id.startswith("file-"): + b64_part = encoded_id[5:] # Remove "file-" + elif encoded_id.startswith("batch_"): + b64_part = encoded_id[6:] # Remove "batch_" + else: + b64_part = encoded_id + + padded = b64_part + "=" * (-len(b64_part) % 4) + decoded = base64.urlsafe_b64decode(padded).decode() + if decoded.startswith("litellm:") and ";model," in decoded: + match = re.search(r";model,([^;]+)", decoded) + if match: + return match.group(1).strip() + + return None + except Exception: + return None + + +def get_original_file_id(encoded_id: str) -> str: + """ + Extract the original provider file/batch ID from an encoded ID. + Handles IDs that start with "file-" or "batch_" prefix. + """ + try: + if not isinstance(encoded_id, str): + return encoded_id + + # Remove prefix if present (file-, batch_, etc.) + if encoded_id.startswith("file-"): + b64_part = encoded_id[5:] # Remove "file-" + elif encoded_id.startswith("batch_"): + b64_part = encoded_id[6:] # Remove "batch_" + else: + b64_part = encoded_id + + padded = b64_part + "=" * (-len(b64_part) % 4) + decoded = base64.urlsafe_b64decode(padded).decode() + + if decoded.startswith("litellm:") and ";model," in decoded: + match = re.search(r"litellm:([^;]+);model,", decoded) + if match: + return match.group(1) + + return encoded_id + except Exception: + return encoded_id + + +def is_model_embedded_id(file_id: str) -> bool: + """ + Check if a file/batch ID has model routing information embedded. + """ + return decode_model_from_file_id(file_id) is not None + + +# ============================================================================ +# MODEL-BASED CREDENTIAL ROUTING HELPERS +# ============================================================================ + + +def extract_model_from_sources( + file_id: str, + request, # FastAPI Request object + data: Optional[dict] = None, +) -> tuple[Optional[str], Optional[str]]: + """ + Extract model information from multiple sources in priority order: + 1. Embedded in file_id (highest priority) + 2. Request headers (x-litellm-model) + 3. Query parameters (?model=) + 4. Request body/data dict + + Args: + file_id: File ID that may contain embedded model info + request: FastAPI request object + data: Optional request data dictionary + + Returns: + Tuple of (model_from_id, model_from_param) + - model_from_id: Model decoded from file ID (if embedded) + - model_from_param: Model from header/query/body + """ + if data is None: + data = {} + + # Check if file_id has embedded model info + model_from_id = decode_model_from_file_id(file_id) + + # Check other sources for model parameter + model_from_param = ( + data.get("model") + or request.query_params.get("model") + or request.headers.get("x-litellm-model") + ) + + return model_from_id, model_from_param + + +def get_credentials_for_model( + llm_router, # Router instance + model_id: str, + operation_context: str = "file operation", +): + """ + Retrieve API credentials for a model from the LLM Router. + + Args: + llm_router: LiteLLM Router instance + model_id: Model name or deployment ID + operation_context: Description for error messages (e.g., "file upload", "batch creation") + + Returns: + Dictionary with credentials (api_key, api_base, custom_llm_provider, etc.) + + Raises: + HTTPException: If router not initialized or model not found + """ + from fastapi import HTTPException + + if llm_router is None: + raise HTTPException( + status_code=500, + detail={"error": "Router not initialized. Cannot use model-based routing."}, + ) + + credentials = llm_router.get_deployment_credentials_with_provider(model_id=model_id) + + if credentials is None: + raise HTTPException( + status_code=400, + detail={ + "error": f"Model '{model_id}' not found in model_list. Please check your config.yaml." + }, + ) + + return credentials + + +def prepare_data_with_credentials( + data: dict, + credentials: dict, + file_id: Optional[str] = None, +) -> None: + """ + Update data dictionary with model credentials (in-place). + + Args: + data: Data dictionary to update + credentials: Credentials from router + file_id: Optional original file_id to set (for decoded file IDs) + """ + data.update(credentials) + data.pop("custom_llm_provider", None) + + if file_id is not None: + data["file_id"] = file_id + + +def handle_model_based_routing( + file_id: str, + request, # FastAPI Request object + llm_router, # Router instance + data: dict, + check_file_id_encoding: bool = True, +) -> tuple[bool, Optional[str], Optional[str], Optional[dict]]: + """ + Orchestrate model-based credential routing for file operations. + + Args: + file_id: File ID (may contain embedded model info) + request: FastAPI request object + llm_router: LiteLLM Router instance + data: Request data dictionary + check_file_id_encoding: Whether to check for embedded model in file_id + + Returns: + Tuple of (should_use_model_routing, model_used, original_file_id, credentials) + - should_use_model_routing: True if model-based routing should be used + - model_used: The model name being used + - original_file_id: Decoded file ID (if it was encoded) + - credentials: Model credentials dict + + Raises: + HTTPException: If router unavailable or model not found + """ + model_from_id, model_from_param = extract_model_from_sources( + file_id=file_id, + request=request, + data=data, + ) + + # Priority 1: Model embedded in file_id + if check_file_id_encoding and model_from_id is not None: + credentials = get_credentials_for_model( + llm_router=llm_router, + model_id=model_from_id, + operation_context=f"file operation (file created with model '{model_from_id}')", + ) + original_file_id = get_original_file_id(file_id) + return True, model_from_id, original_file_id, credentials + + # Priority 2: Model from header/query/body + elif model_from_param is not None: + credentials = get_credentials_for_model( + llm_router=llm_router, + model_id=model_from_param, + operation_context="file operation", + ) + return True, model_from_param, None, credentials + + # No model-based routing needed + return False, None, None, None diff --git a/litellm/proxy/openai_files_endpoints/files_endpoints.py b/litellm/proxy/openai_files_endpoints/files_endpoints.py index b0be547613..3f08a4ec36 100644 --- a/litellm/proxy/openai_files_endpoints/files_endpoints.py +++ b/litellm/proxy/openai_files_endpoints/files_endpoints.py @@ -29,6 +29,7 @@ from litellm.llms.base_llm.files.transformation import BaseFileEndpoints from litellm.proxy._types import * from litellm.proxy.auth.user_api_key_auth import user_api_key_auth from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing +from litellm.proxy.common_utils.http_parsing_utils import _read_request_body from litellm.proxy.common_utils.openai_endpoint_utils import ( get_custom_llm_provider_from_request_body, get_custom_llm_provider_from_request_headers, @@ -42,7 +43,13 @@ from litellm.types.llms.openai import ( OpenAIFilesPurpose, ) -from .common_utils import _is_base64_encoded_unified_file_id +from .common_utils import ( + _is_base64_encoded_unified_file_id, + encode_file_id_with_model, + get_credentials_for_model, + handle_model_based_routing, + prepare_data_with_credentials, +) router = APIRouter() @@ -127,7 +134,51 @@ async def route_create_file( is_router_model: bool, router_model: Optional[str], custom_llm_provider: str, + model: Optional[str] = None, ) -> OpenAIFileObject: + """ + Route file creation request to the appropriate provider. + + Priority: + 1. If model parameter provided -> use model credentials and encode ID + 2. If enable_loadbalancing_on_batch_endpoints -> deprecated loadbalancing + 3. If target_model_names_list -> managed files (requires DB) + 4. Else -> use custom_llm_provider with files_settings + """ + + # NEW: Handle model-based routing (no DB required) + if model is not None: + # Get credentials from model_list via router + credentials = get_credentials_for_model( + llm_router=llm_router, + model_id=model, + operation_context="file upload", + ) + + # Merge credentials into the request + prepare_data_with_credentials( + data=_create_file_request, # type: ignore + credentials=credentials, + ) + + # Create the file with model credentials + response = await litellm.acreate_file( + **_create_file_request, + custom_llm_provider=credentials["custom_llm_provider"] + ) # type: ignore + + # Encode the file ID with model information + if response and hasattr(response, "id") and response.id: + original_id = response.id + encoded_id = encode_file_id_with_model(file_id=original_id, model=model) + response.id = encoded_id + verbose_proxy_logger.debug( + f"Encoded file ID: {original_id} -> {encoded_id} (model: {model})" + ) + + return response + + # EXISTING: Deprecated loadbalancing approach if ( litellm.enable_loadbalancing_on_batch_endpoints is True and is_router_model @@ -206,6 +257,7 @@ async def create_file( provider: Optional[str] = None, custom_llm_provider: str = Form(default="openai"), file: UploadFile = File(...), + litellm_metadata: Optional[str] = Form(default=None), user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth), ): """ @@ -245,6 +297,14 @@ async def create_file( or "openai" ) + # NEW: Extract model parameter for multi-account routing + request_body = await _read_request_body(request=request) or {} + model_param = ( + request_body.get("model") + or request.query_params.get("model") + or request.headers.get("x-litellm-model") + ) + target_model_names_list = ( target_model_names.split(",") if target_model_names else [] ) @@ -264,6 +324,10 @@ async def create_file( purpose = cast(OpenAIFilesPurpose, purpose) data = {} + + # Add litellm_metadata to data if provided (from form field) + if litellm_metadata is not None: + data["litellm_metadata"] = litellm_metadata # Include original request and headers in the data data = await add_litellm_data_to_request( @@ -303,6 +367,7 @@ async def create_file( is_router_model=is_router_model, router_model=router_model, custom_llm_provider=custom_llm_provider, + model=model_param, ) if response is None: @@ -480,13 +545,41 @@ async def get_file_content( } ) else: - response = await litellm.afile_content( - **{ - "custom_llm_provider": custom_llm_provider, - "file_id": file_id, - **data, - } # type: ignore + # Check for model-based credential routing + should_route, model_used, original_file_id, credentials = handle_model_based_routing( + file_id=file_id, + request=request, + llm_router=llm_router, + data=data, + check_file_id_encoding=True, ) + + if should_route: + # Use model-based routing with credentials from config + prepare_data_with_credentials( + data=data, + credentials=credentials, # type: ignore + file_id=original_file_id, # Use decoded file ID if from encoded ID + ) + + response = await litellm.afile_content( + custom_llm_provider=credentials["custom_llm_provider"], # type: ignore + **data + ) # type: ignore + + verbose_proxy_logger.debug( + f"Retrieved file content using model: {model_used}" + + (f", file_id: {file_id} -> {original_file_id}" if original_file_id else "") + ) + else: + # Fallback to default behavior (uses env variables or provider-based routing) + response = await litellm.afile_content( + **{ + "custom_llm_provider": custom_llm_provider, + "file_id": file_id, + **data, + } # type: ignore + ) ### ALERTING ### asyncio.create_task( @@ -618,10 +711,38 @@ async def get_file( route_type="afile_retrieve", ) - ## check if file_id is a litellm managed file - is_base64_unified_file_id = _is_base64_encoded_unified_file_id(file_id) + ## Check for model-based credential routing + from litellm.proxy.proxy_server import llm_router + + should_route, model_used, original_file_id, credentials = handle_model_based_routing( + file_id=file_id, + request=request, + llm_router=llm_router, + data=data, + check_file_id_encoding=True, + ) + + if should_route: + # Use model-based routing with credentials from config + prepare_data_with_credentials( + data=data, + credentials=credentials, # type: ignore + file_id=original_file_id, + ) - if is_base64_unified_file_id: + response = await litellm.afile_retrieve(**data) # type: ignore + + # Keep the encoded ID in response if it was originally encoded + if original_file_id and response and hasattr(response, "id") and response.id: + response.id = file_id + + verbose_proxy_logger.debug( + f"Retrieved file using model: {model_used}" + + (f", original_id: {original_file_id}" if original_file_id else "") + ) + + ## EXISTING: check if file_id is a litellm managed file + elif _is_base64_encoded_unified_file_id(file_id): managed_files_obj = proxy_logging_obj.get_proxy_hook("managed_files") if managed_files_obj is None: raise ProxyException( @@ -762,10 +883,32 @@ async def delete_file( proxy_config=proxy_config, ) - ## check if file_id is a litellm managed file - is_base64_unified_file_id = _is_base64_encoded_unified_file_id(file_id) - - if is_base64_unified_file_id: + # Check for model-based credential routing + should_route, model_used, original_file_id, credentials = handle_model_based_routing( + file_id=file_id, + request=request, + llm_router=llm_router, + data=data, + check_file_id_encoding=True, + ) + + if should_route: + # Use model-based routing with credentials from config + prepare_data_with_credentials( + data=data, + credentials=credentials, # type: ignore + file_id=original_file_id, + ) + + response = await litellm.afile_delete(**data) # type: ignore + + verbose_proxy_logger.debug( + f"Deleted file using model: {model_used}" + + (f", original_id: {original_file_id}" if original_file_id else "") + ) + + ## EXISTING: check if file_id is a litellm managed file + elif _is_base64_encoded_unified_file_id(file_id): managed_files_obj = proxy_logging_obj.get_proxy_hook("managed_files") if managed_files_obj is None: raise ProxyException( @@ -913,7 +1056,28 @@ async def list_files( ) response: Optional[Any] = None - if target_model_names and isinstance(target_model_names, str): + + # Check for model-based credential routing (no file_id encoding check for list) + should_route, model_used, _, credentials = handle_model_based_routing( + file_id="", # No file_id for list endpoint + request=request, + llm_router=llm_router, + data=data, + check_file_id_encoding=False, + ) + + if should_route: + # Use model-based routing with credentials from config + data.update(credentials) # type: ignore + response = await litellm.afile_list( + custom_llm_provider=credentials["custom_llm_provider"], # type: ignore + purpose=purpose, + **data # type: ignore + ) + + verbose_proxy_logger.debug(f"Listed files using model: {model_used}") + + elif target_model_names and isinstance(target_model_names, str): target_model_names_list = target_model_names.split(",") if len(target_model_names_list) != 1: raise HTTPException( diff --git a/litellm/responses/litellm_completion_transformation/transformation.py b/litellm/responses/litellm_completion_transformation/transformation.py index e021f4c16d..bd12d0cbb4 100644 --- a/litellm/responses/litellm_completion_transformation/transformation.py +++ b/litellm/responses/litellm_completion_transformation/transformation.py @@ -618,16 +618,30 @@ class LiteLLMCompletionResponsesConfig: for tool in all_chat_completion_tools: if tool.type == "function": function_definition = tool.function - responses_tools.append( - OutputFunctionToolCall( - name=function_definition.name or "", - arguments=function_definition.get("arguments") or "", - call_id=tool.id or "", - id=tool.id or "", - type="function_call", # critical this is "function_call" to work with tools like openai codex - status=function_definition.get("status") or "completed", - ) + provider_specific_fields: Optional[Dict[str, Any]] = None + if hasattr(tool, "provider_specific_fields") and getattr(tool, "provider_specific_fields", None): + provider_specific_fields = getattr(tool, "provider_specific_fields") + if not isinstance(provider_specific_fields, dict): + provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {} + elif hasattr(function_definition, "provider_specific_fields") and getattr(function_definition, "provider_specific_fields", None): + provider_specific_fields = getattr(function_definition, "provider_specific_fields") + if not isinstance(provider_specific_fields, dict): + provider_specific_fields = dict(provider_specific_fields) if hasattr(provider_specific_fields, "__dict__") else {} + + output_tool_call: OutputFunctionToolCall = OutputFunctionToolCall( + name=function_definition.name or "", + arguments=function_definition.get("arguments") or "", + call_id=tool.id or "", + id=tool.id or "", + type="function_call", # critical this is "function_call" to work with tools like openai codex + status=function_definition.get("status") or "completed", ) + + # Pass through provider_specific_fields as-is if present + if provider_specific_fields: + setattr(output_tool_call, "provider_specific_fields", provider_specific_fields) # type: ignore + + responses_tools.append(output_tool_call) return responses_tools @staticmethod diff --git a/litellm/router.py b/litellm/router.py index 841391653d..6d38d2fc2b 100644 --- a/litellm/router.py +++ b/litellm/router.py @@ -5914,6 +5914,57 @@ class Router: raise Exception("Model Name invalid - {}".format(type(model))) return None + def get_deployment_credentials_with_provider( + self, model_id: str + ) -> Optional[Dict[str, Any]]: + """ + Get API credentials and provider info from a model name in model_list. + Useful for passthrough endpoints (files, batches, etc.) that need credentials. + + This method tries to find a deployment by model_id first, and if not found, + it tries to find by model_group_name (model_name). + + Args: + model_id: Model ID or model name from model_list (e.g., "gpt-4o-litellm") + + Returns: + Dictionary containing api_key, api_base, custom_llm_provider, etc. + Returns None if model not found. + + Example: + credentials = router.get_deployment_credentials_with_provider("gpt-4o-litellm") + # Returns: {"api_key": "sk-...", "custom_llm_provider": "openai", ...} + """ + # Try to get deployment by model_id first + deployment = self.get_deployment(model_id=model_id) + + # If not found, try by model_group_name + if deployment is None: + deployment = self.get_deployment_by_model_group_name(model_group_name=model_id) + + if deployment is None: + return None + + # Get basic credentials + credentials = CredentialLiteLLMParams( + **deployment.litellm_params.model_dump(exclude_none=True) + ).model_dump(exclude_none=True) + + # Add custom_llm_provider + if deployment.litellm_params.custom_llm_provider: + credentials["custom_llm_provider"] = ( + deployment.litellm_params.custom_llm_provider + ) + elif "/" in deployment.litellm_params.model: + # Extract provider from "provider/model" format + credentials["custom_llm_provider"] = deployment.litellm_params.model.split( + "/" + )[0] + else: + credentials["custom_llm_provider"] = "openai" # default + + return credentials + @overload def get_router_model_info( self, deployment: dict, received_model_name: str, id: None = None diff --git a/litellm/types/llms/anthropic.py b/litellm/types/llms/anthropic.py index d216c2a8d5..a6963c1a4c 100644 --- a/litellm/types/llms/anthropic.py +++ b/litellm/types/llms/anthropic.py @@ -166,7 +166,11 @@ class AnthropicMessagesContainerUploadParam(TypedDict, total=False): class AnthropicMessagesImageParam(TypedDict, total=False): type: Required[Literal["image"]] source: Required[ - Union[AnthropicContentParamSource, AnthropicContentParamSourceFileId] + Union[ + AnthropicContentParamSource, + AnthropicContentParamSourceFileId, + AnthropicContentParamSourceUrl, + ] ] cache_control: Optional[Union[dict, ChatCompletionCachedContent]] diff --git a/litellm/types/llms/vertex_ai.py b/litellm/types/llms/vertex_ai.py index 818f0db798..7a4a4723ab 100644 --- a/litellm/types/llms/vertex_ai.py +++ b/litellm/types/llms/vertex_ai.py @@ -261,6 +261,7 @@ class UsageMetadata(TypedDict, total=False): promptTokensDetails: List[PromptTokensDetails] thoughtsTokenCount: int responseTokensDetails: List[PromptTokensDetails] + candidatesTokensDetails: List[PromptTokensDetails] # Alternative key name used in some responses class TokenCountDetailsResponse(TypedDict): diff --git a/litellm/utils.py b/litellm/utils.py index d7474387cd..1ec2576d35 100644 --- a/litellm/utils.py +++ b/litellm/utils.py @@ -8093,6 +8093,21 @@ def add_openai_metadata(metadata: Optional[Mapping[str, Any]]) -> Optional[Dict[ return visible_metadata.copy() +def get_requester_metadata(metadata: dict): + if not metadata: + return None + + requester_metadata = metadata.get("requester_metadata") + if isinstance(requester_metadata, dict): + cleaned_metadata = add_openai_metadata(requester_metadata) + if cleaned_metadata: + return cleaned_metadata + + cleaned_metadata = add_openai_metadata(metadata) + if cleaned_metadata: + return cleaned_metadata + + return None def return_raw_request(endpoint: CallTypes, kwargs: dict) -> RawRequestTypedDict: """ diff --git a/tests/llm_responses_api_testing/test_google_ai_studio_responses_api.py b/tests/llm_responses_api_testing/test_google_ai_studio_responses_api.py index 203ee252b3..13801fce7d 100644 --- a/tests/llm_responses_api_testing/test_google_ai_studio_responses_api.py +++ b/tests/llm_responses_api_testing/test_google_ai_studio_responses_api.py @@ -85,6 +85,170 @@ async def test_mock_basic_google_ai_studio_responses_api_with_tools(): assert call_kwargs["messages"][0]["content"] == "what is the latest version of supabase python package and when was it released?" assert call_kwargs["tools"] == [] # web search tools are converted to web_search_options, not kept as tools +@pytest.mark.asyncio +async def test_gemini_3_responses_api_with_thought_signatures(): + """ + Test that Gemini 3 Responses API preserves thought signatures in function calls. + This test verifies that provider_specific_fields with thought_signature are correctly + preserved when using the Responses API with Gemini 3. + """ + if not os.getenv("GEMINI_API_KEY"): + pytest.skip("GEMINI_API_KEY not set") + + litellm.set_verbose = False + request_model = "gemini/gemini-3-pro-preview" + + tools = [ + { + "type": "function", + "name": "get_weather", + "description": "Get the current weather for a location", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "City and country e.g. Mumbai, India" + }, + "units": { + "type": "string", + "enum": ["celsius", "fahrenheit"], + "description": "Units the temperature will be returned in." + } + }, + "required": ["location", "units"], + "additionalProperties": False, + }, + "strict": True + } + ] + + # Step 1: Initial request with tools + response = await litellm.aresponses( + model=request_model, + input="What is the weather in Mumbai?", + tools=tools, + reasoning_effort="low" + ) + + # Validate response structure + from litellm.types.llms.openai import ResponsesAPIResponse + assert isinstance(response, ResponsesAPIResponse), "Response should be a ResponsesAPIResponse" + assert hasattr(response, "output") or "output" in response, "Response should have 'output' field" + assert isinstance(response.output, list), "Output should be a list" + + # Find function call in output + function_call_item = None + for item in response.output: + # Convert to dict if it's a Pydantic model for easier access + if hasattr(item, "model_dump"): + item_dict = item.model_dump() + elif hasattr(item, "__dict__"): + item_dict = dict(item) if not isinstance(item, dict) else item + else: + item_dict = item if isinstance(item, dict) else {} + + if isinstance(item_dict, dict) and item_dict.get("type") == "function_call": + function_call_item = item_dict + break + + # Verify function call exists + assert function_call_item is not None, "Response should contain a function_call item" + assert function_call_item.get("name") == "get_weather", "Function call should be for get_weather" + + # Verify thought signature is present in provider_specific_fields + provider_specific_fields = function_call_item.get("provider_specific_fields") + assert provider_specific_fields is not None, "Function call should have provider_specific_fields" + assert "thought_signature" in provider_specific_fields, "provider_specific_fields should contain thought_signature" + assert isinstance(provider_specific_fields["thought_signature"], str), "thought_signature should be a string" + assert len(provider_specific_fields["thought_signature"]) > 0, "thought_signature should not be empty" + + print(f"✅ Thought signature preserved: {provider_specific_fields['thought_signature'][:50]}...") + + +@pytest.mark.asyncio +async def test_gemini_3_responses_api_streaming_with_thought_signatures(): + """ + Test that Gemini 3 Responses API preserves thought signatures in streaming mode. + This test verifies that provider_specific_fields with thought_signature are correctly + preserved when using streaming Responses API with Gemini 3. + """ + if not os.getenv("GEMINI_API_KEY"): + pytest.skip("GEMINI_API_KEY not set") + + litellm.set_verbose = False + request_model = "gemini/gemini-3-pro-preview" + + tools = [ + { + "type": "function", + "name": "get_weather", + "description": "Get the current weather for a location", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "City and country e.g. Mumbai, India" + }, + "units": { + "type": "string", + "enum": ["celsius", "fahrenheit"], + "description": "Units the temperature will be returned in." + } + }, + "required": ["location", "units"], + "additionalProperties": False, + }, + "strict": True + } + ] + + # Step 1: Streaming request with tools + response_stream = await litellm.aresponses( + model=request_model, + input="What is the weather in Mumbai?", + tools=tools, + stream=True, + reasoning_effort="low" + ) + + # Collect all chunks + chunks = [] + completed_response = None + + async for chunk in response_stream: + chunks.append(chunk) + # Check if this is the completed response event + if hasattr(chunk, "type") and chunk.type == "response.completed": + completed_response = chunk.response + elif isinstance(chunk, dict) and chunk.get("type") == "response.completed": + completed_response = chunk.get("response") + + # Verify we got chunks + assert len(chunks) > 0, "Should receive at least one chunk" + + # If we have a completed response, check for thought signatures + if completed_response: + output = completed_response.get("output", []) + function_call_item = None + for item in output: + if isinstance(item, dict) and item.get("type") == "function_call": + function_call_item = item + break + + if function_call_item: + provider_specific_fields = function_call_item.get("provider_specific_fields") + if provider_specific_fields: + thought_signature = provider_specific_fields.get("thought_signature") + if thought_signature: + assert isinstance(thought_signature, str), "thought_signature should be a string" + assert len(thought_signature) > 0, "thought_signature should not be empty" + print(f"✅ Streaming thought signature preserved: {thought_signature[:50]}...") + + print(f"✅ Collected {len(chunks)} streaming chunks") + + class TestGoogleAIStudioResponsesAPITest(BaseResponsesAPITest): def get_base_completion_call_args(self): #litellm._turn_on_debug() diff --git a/tests/llm_translation/test_optional_params.py b/tests/llm_translation/test_optional_params.py index dd42d5e1c5..bc85b99eee 100644 --- a/tests/llm_translation/test_optional_params.py +++ b/tests/llm_translation/test_optional_params.py @@ -23,6 +23,7 @@ from litellm.utils import ( get_optional_params, get_optional_params_embeddings, get_optional_params_image_gen, + get_requester_metadata, ) ## get_optional_params_embeddings @@ -67,6 +68,41 @@ def test_anthropic_optional_params(stop_sequence, expected_count): assert len(optional_params) == expected_count +def test_get_requester_metadata_returns_none_for_empty(): + metadata = {"requester_metadata": {}} + assert get_requester_metadata(metadata) is None + + +@patch("litellm.main.openai_chat_completions.completion") +def test_requester_metadata_forwarded_to_openai(mock_completion): + mock_completion.return_value = MagicMock() + metadata = { + "requester_metadata": { + "custom_meta_key": "value", + "hidden_params": "secret", + "int_value": 123, + } + } + + original_api_key = litellm.api_key + litellm.api_key = "sk-test" + original_preview_flag = litellm.enable_preview_features + litellm.enable_preview_features = True + + try: + litellm.completion( + model="gpt-4o", + messages=[{"role": "user", "content": "hi"}], + metadata=metadata, + ) + finally: + litellm.api_key = original_api_key + litellm.enable_preview_features = original_preview_flag + + sent_metadata = mock_completion.call_args.kwargs["optional_params"]["metadata"] + assert sent_metadata == {"custom_meta_key": "value"} + + def test_get_optional_params_with_allowed_openai_params(): """ Test if use can dynamically pass in allowed_openai_params to override default behavior diff --git a/tests/llm_translation/test_prompt_factory.py b/tests/llm_translation/test_prompt_factory.py index 2023f12825..1ed1e327ec 100644 --- a/tests/llm_translation/test_prompt_factory.py +++ b/tests/llm_translation/test_prompt_factory.py @@ -19,6 +19,7 @@ from litellm.litellm_core_utils.prompt_templates.factory import ( claude_2_1_pt, convert_to_anthropic_image_obj, convert_url_to_base64, + create_anthropic_image_param, llama_2_chat_pt, prompt_factory, ) @@ -207,6 +208,125 @@ def test_base64_image_input(url, expected_media_type): assert response["media_type"] == expected_media_type +def test_create_anthropic_image_param_with_http_url(): + """Test that HTTP/HTTPS URLs are passed as URL references, not base64.""" + image_param = create_anthropic_image_param( + "https://example.com/image.jpg", format=None + ) + + assert image_param["type"] == "image" + assert image_param["source"]["type"] == "url" + assert image_param["source"]["url"] == "https://example.com/image.jpg" + + +def test_create_anthropic_image_param_with_https_url(): + """Test that HTTPS URLs are passed as URL references.""" + image_param = create_anthropic_image_param( + "https://example.com/image.png", format=None + ) + + assert image_param["type"] == "image" + assert image_param["source"]["type"] == "url" + assert image_param["source"]["url"] == "https://example.com/image.png" + + +def test_create_anthropic_image_param_with_dict_input(): + """Test that dict input with URL is handled correctly.""" + image_param = create_anthropic_image_param( + {"url": "https://example.com/image.jpg", "format": "image/jpeg"}, format=None + ) + + assert image_param["type"] == "image" + assert image_param["source"]["type"] == "url" + assert image_param["source"]["url"] == "https://example.com/image.jpg" + + +def test_create_anthropic_image_param_with_base64_data_uri(): + """Test that data URIs are converted to base64.""" + image_param = create_anthropic_image_param( + "data:image/jpeg;base64,/9j/4AAQSkZJRg==", format=None + ) + + assert image_param["type"] == "image" + assert image_param["source"]["type"] == "base64" + assert image_param["source"]["media_type"] == "image/jpeg" + assert image_param["source"]["data"] == "/9j/4AAQSkZJRg==" + + +def test_create_anthropic_image_param_with_format_override(): + """Test that format parameter can override media type.""" + image_param = create_anthropic_image_param( + "data:image/jpeg;base64,1234", format="image/png" + ) + + assert image_param["type"] == "image" + assert image_param["source"]["type"] == "base64" + assert image_param["source"]["media_type"] == "image/png" + + +def test_anthropic_messages_pt_with_url_image(): + """Test that anthropic_messages_pt correctly handles HTTP/HTTPS URLs as URL references.""" + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's in this image?"}, + { + "type": "image_url", + "image_url": "https://example.com/image.jpg", + }, + ], + } + ] + + result = anthropic_messages_pt( + messages=messages, model="claude-3-5-sonnet", llm_provider="anthropic" + ) + + assert len(result) == 1 + assert result[0]["role"] == "user" + assert isinstance(result[0]["content"], list) + assert len(result[0]["content"]) == 2 + + # Check text content + assert result[0]["content"][0]["type"] == "text" + + # Check image content - should be URL reference, not base64 + assert result[0]["content"][1]["type"] == "image" + assert result[0]["content"][1]["source"]["type"] == "url" + assert result[0]["content"][1]["source"]["url"] == "https://example.com/image.jpg" + + +def test_anthropic_messages_pt_with_base64_image(): + """Test that anthropic_messages_pt correctly handles data URIs as base64.""" + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's in this image?"}, + { + "type": "image_url", + "image_url": "data:image/jpeg;base64,/9j/4AAQSkZJRg==", + }, + ], + } + ] + + result = anthropic_messages_pt( + messages=messages, model="claude-3-5-sonnet", llm_provider="anthropic" + ) + + assert len(result) == 1 + assert result[0]["role"] == "user" + assert isinstance(result[0]["content"], list) + assert len(result[0]["content"]) == 2 + + # Check image content - should be base64, not URL + assert result[0]["content"][1]["type"] == "image" + assert result[0]["content"][1]["source"]["type"] == "base64" + assert result[0]["content"][1]["source"]["media_type"] == "image/jpeg" + + def test_anthropic_messages_tool_call(): messages = [ { diff --git a/tests/openai_endpoints_tests/test_openai_batches_endpoint.py b/tests/openai_endpoints_tests/test_openai_batches_endpoint.py index 9fc5962885..5a84900f87 100644 --- a/tests/openai_endpoints_tests/test_openai_batches_endpoint.py +++ b/tests/openai_endpoints_tests/test_openai_batches_endpoint.py @@ -291,4 +291,4 @@ async def test_list_batches_with_target_model_names(): # Verify the response structure assert response["object"] == "list" - assert len(response["data"]) > 0 + assert len(response["data"]) > 0 \ No newline at end of file diff --git a/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py b/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py index 35e0cd4bec..8942239bb2 100644 --- a/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py +++ b/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py @@ -1339,6 +1339,78 @@ def test_vertex_ai_penalty_parameters_validation(): assert result["max_output_tokens"] == 100 +def test_vertex_ai_gemini_3_penalty_parameters_unsupported(): + """ + Test that penalty parameters are not supported for Gemini 3 models. + + This test ensures that: + 1. Gemini 3 models do not support penalty parameters + 2. Penalty parameters are excluded from supported params list for Gemini 3 models + 3. Penalty parameters are filtered out when mapping params for Gemini 3 models + """ + v = VertexGeminiConfig() + + # Test Gemini 3 models + gemini_3_models = [ + "gemini-3-pro-preview", + "vertex_ai/gemini-3-pro-preview", + "gemini/gemini-3-pro-preview", + ] + + for model in gemini_3_models: + # Test _supports_penalty_parameters method + assert v._supports_penalty_parameters(model) == False, \ + f"Gemini 3 model {model} should not support penalty parameters" + + # Test get_supported_openai_params method + supported_params = v.get_supported_openai_params(model) + assert "frequency_penalty" not in supported_params, \ + f"frequency_penalty should not be in supported params for {model}" + assert "presence_penalty" not in supported_params, \ + f"presence_penalty should not be in supported params for {model}" + + # Test parameter mapping - penalty params should be filtered out + non_default_params = { + "temperature": 0.7, + "frequency_penalty": 0.5, + "presence_penalty": 0.3, + "max_tokens": 100 + } + + optional_params = {} + result = v.map_openai_params( + non_default_params=non_default_params, + optional_params=optional_params, + model=model, + drop_params=False + ) + + # Penalty parameters should be filtered out for Gemini 3 models + assert "frequency_penalty" not in result, \ + f"frequency_penalty should be filtered out for Gemini 3 model {model}" + assert "presence_penalty" not in result, \ + f"presence_penalty should be filtered out for Gemini 3 model {model}" + + # Other parameters should still be included + assert "temperature" in result, \ + f"temperature should still be included for Gemini 3 model {model}" + assert "max_output_tokens" in result, \ + f"max_output_tokens should still be included for Gemini 3 model {model}" + assert result["temperature"] == 0.7 + assert result["max_output_tokens"] == 100 + + # Test that non-Gemini 3 models still support penalty parameters (if they're not in the unsupported list) + non_gemini_3_model = "gemini-2.5-pro" + assert v._supports_penalty_parameters(non_gemini_3_model) == True, \ + f"Non-Gemini 3 model {non_gemini_3_model} should support penalty parameters" + + supported_params = v.get_supported_openai_params(non_gemini_3_model) + assert "frequency_penalty" in supported_params, \ + f"frequency_penalty should be in supported params for {non_gemini_3_model}" + assert "presence_penalty" in supported_params, \ + f"presence_penalty should be in supported params for {non_gemini_3_model}" + + def test_vertex_ai_annotation_streaming_events(): """ Test that annotation events are properly emitted during streaming for Vertex AI Gemini. diff --git a/tests/test_litellm/test_main.py b/tests/test_litellm/test_main.py index 9f209c1b8d..be4bb4e48a 100644 --- a/tests/test_litellm/test_main.py +++ b/tests/test_litellm/test_main.py @@ -192,8 +192,33 @@ async def test_url_with_format_param(model, sync_mode, monkeypatch): json_str = json_str.decode("utf-8") print(f"type of json_str: {type(json_str)}") - assert "png" in json_str - assert "jpeg" not in json_str + + # Bedrock models convert URLs to base64, while direct Anthropic models support URLs + # bedrock/invoke models use Anthropic messages API which supports URLs + if model.startswith("bedrock/invoke/"): + # bedrock/invoke should convert URLs to base64 (doesn't support URL references) + # URL should NOT be in the JSON (it should be converted to base64) + assert "https://upload.wikimedia.org" not in json_str + # Should have base64 data in the source (type="base64", not type="url") + assert '"type":"base64"' in json_str or '"type": "base64"' in json_str + # Should have "data" field containing base64 content + assert '"data"' in json_str + elif model.startswith("bedrock/"): + # Regular Bedrock models should convert URLs to base64 (uses "bytes" field) + # URL should NOT be in the JSON (it should be converted to base64) + assert "https://upload.wikimedia.org" not in json_str + # Should have "bytes" field (Bedrock uses "bytes" not "base64" in the field name) + assert '"bytes"' in json_str or '"bytes":' in json_str + elif model.startswith("anthropic/"): + # Direct Anthropic models should pass HTTPS URLs directly (HTTP URLs are converted to base64) + # Since we're using HTTPS URL, it should be passed as-is + assert "https://upload.wikimedia.org" in json_str + # For Anthropic, URL references use "url" type, not base64 + assert '"type":"url"' in json_str or '"type": "url"' in json_str + else: + # For other models, check format parameter is respected + assert "png" in json_str + assert "jpeg" not in json_str @pytest.mark.parametrize("model", ["gpt-4o-mini"])