tiennm99/litellm
1
0
Fork 0
mirror of https://github.com/tiennm99/litellm.git synced 2026-08-05 16:24:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #17142 from BerriAI/litellm_anthropic_update_new_feat

Browse Source 
Update new anthropic feats as reviewed
This commit is contained in:
Sameer Kankute
2025-11-28 21:06:34 +05:30
committed by GitHub
parent 205a563b65 cf6dda5e29
commit 6c326ceb87
14 changed files with 330 additions and 338 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/my-website/blog/anthropic_opus_4_5_and_advanced_features/index.md
+7 -294
View File
@@ -972,14 +972,13 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
## Effort Parameter: Control Token Usage {#effort-parameter}
Controls aspects like how much effort the model puts into its response, via `output_config={"effort": ..}`.
Control how much effort Claude puts into its response using the `reasoning_effort` parameter. This allows you to trade off between response thoroughness and token efficiency.
:::info
Soon, we will map OpenAI's `reasoning_effort` parameter to this.
LiteLLM automatically maps `reasoning_effort` to Anthropic's `output_config` format and adds the required `effort-2025-11-24` beta header for Claude Opus 4.5.
:::
Potential Values for `effort` parameter: `"high"`, `"medium"`, `"low"`.
Potential values for `reasoning_effort` parameter: `"high"`, `"medium"`, `"low"`.
### Usage Example
@@ -995,7 +994,7 @@ message = "Analyze the trade-offs between microservices and monolithic architect
response_high = litellm.completion(
model="anthropic/claude-opus-4-5-20251101",
messages=[{"role": "user", "content": message}],
output_config={"effort": "high"}
reasoning_effort="high"
)
print("High effort response:")
@@ -1006,7 +1005,7 @@ print(f"Tokens used: {response_high.usage.completion_tokens}\n")
response_medium = litellm.completion(
model="anthropic/claude-opus-4-5-20251101",
messages=[{"role": "user", "content": message}],
output_config={"effort": "medium"}
reasoning_effort="medium"
)
print("Medium effort response:")
@@ -1017,7 +1016,7 @@ print(f"Tokens used: {response_medium.usage.completion_tokens}\n")
response_low = litellm.completion(
model="anthropic/claude-opus-4-5-20251101",
messages=[{"role": "user", "content": message}],
output_config={"effort": "low"}
reasoning_effort="low"
)
print("Low effort response:")
@@ -1062,295 +1061,9 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
"role": "user",
"content": "Analyze the trade-offs between microservices and monolithic architectures"
}],
"output_config": {
"effort": "high"
}
"reasoning_effort": "high"
}
'
```
</TabItem>
</Tabs>
## Cost Tracking: Monitor Tool Search Usage {#cost-tracking}
### Understanding Tool Search Costs
Tool search operations are tracked separately in the usage object, allowing you to monitor and optimize costs.
It is available in the `usage` object, under `server_tool_use.tool_search_requests`.
Anthropic charges $0.0001 per tool search request.
### Tracking Example
<Tabs>
<TabItem value="sdk" label="LiteLLM Python SDK">
```python
import litellm
tools = [
{
"type": "tool_search_tool_regex_20251119",
"name": "tool_search_tool_regex"
},
# ... 100 deferred tools
]
response = litellm.completion(
model="anthropic/claude-sonnet-4-5-20250929",
messages=[{
"role": "user",
"content": "Find and use the weather tool for San Francisco"
}],
tools=tools
)
# Standard token usage
print("Token Usage:")
print(f" Input tokens: {response.usage.prompt_tokens}")
print(f" Output tokens: {response.usage.completion_tokens}")
print(f" Total tokens: {response.usage.total_tokens}")
# Tool search specific usage
if hasattr(response.usage, 'server_tool_use') and response.usage.server_tool_use:
print(f"\nTool Search Usage:")
print(f" Search requests: {response.usage.server_tool_use.tool_search_requests}")
# Calculate cost (example pricing)
input_cost = response.usage.prompt_tokens * 0.000003 # $3 per 1M tokens
output_cost = response.usage.completion_tokens * 0.000015 # $15 per 1M tokens
search_cost = response.usage.server_tool_use.tool_search_requests * 0.0001 # Example
total_cost = input_cost + output_cost + search_cost
print(f"\nCost Breakdown:")
print(f" Input tokens: ${input_cost:.6f}")
print(f" Output tokens: ${output_cost:.6f}")
print(f" Tool searches: ${search_cost:.6f}")
print(f" Total: ${total_cost:.6f}")
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
1. Setup config.yaml
```yaml
model_list:
- model_name: claude-4
litellm_params:
model: anthropic/claude-opus-4-5-20251101
api_key: os.environ/ANTHROPIC_API_KEY
```
2. Start the proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data ' {
"model": "claude-4",
"messages": [{
"role": "user",
"content": "Find and use the weather tool for San Francisco"
}],
"tools": [
{
"type": "tool_search_tool_regex_20251119",
"name": "tool_search_tool_regex"
},
# ... 100 deferred tools
]
}
'
```
Expected Response:
```json
{
...,
"usage": {
...,
"server_tool_use": {
"tool_search_requests": 1
}
}
}
```
</TabItem>
</Tabs>
### Cost Optimization Tips
1. **Keep frequently used tools non-deferred** (3-5 tools)
2. **Use tool search for large catalogs** (10+ tools)
3. **Monitor search requests** to identify optimization opportunities
4. **Combine with effort parameter** for maximum efficiency
---
## Combining Features {#combining-features}
### The Power of Integration
These features work together seamlessly. Here's a real-world example combining all of them:
<Tabs>
<TabItem value="sdk" label="LiteLLM Python SDK">
```python
import litellm
import json
# Large tool catalog with search, programmatic calling, and examples
tools = [
# Enable tool search
{
"type": "tool_search_tool_regex_20251119",
"name": "tool_search_tool_regex"
},
# Enable programmatic calling
{
"type": "code_execution_20250825",
"name": "code_execution"
},
# Database tool with all features
{
"type": "function",
"function": {
"name": "query_database",
"description": "Execute SQL queries against the analytics database. Returns JSON array of results.",
"parameters": {
"type": "object",
"properties": {
"sql": {
"type": "string",
"description": "SQL SELECT statement"
},
"limit": {
"type": "integer",
"description": "Maximum rows to return"
}
},
"required": ["sql"]
}
},
"defer_loading": True, # Tool search
"allowed_callers": ["code_execution_20250825"], # Programmatic calling
"input_examples": [ # Input examples
{
"sql": "SELECT region, SUM(revenue) as total FROM sales GROUP BY region",
"limit": 100
}
]
},
# ... 50 more tools with defer_loading
]
# Make request with effort control
response = litellm.completion(
model="anthropic/claude-opus-4-5-20251101",
messages=[{
"role": "user",
"content": "Analyze sales by region for the last quarter and identify top performers"
}],
tools=tools,
output_config={"effort": "medium"} # Balanced efficiency
)
# Track comprehensive usage
print("Complete Usage Metrics:")
print(f" Input tokens: {response.usage.prompt_tokens}")
print(f" Output tokens: {response.usage.completion_tokens}")
print(f" Total tokens: {response.usage.total_tokens}")
if hasattr(response.usage, 'server_tool_use') and response.usage.server_tool_use:
print(f" Tool searches: {response.usage.server_tool_use.tool_search_requests}")
print(f"\nResponse: {response.choices[0].message.content}")
```
</TabItem>
<TabItem value="proxy" label="LiteLLM Proxy">
1. Setup config.yaml
```yaml
model_list:
- model_name: claude-4
litellm_params:
model: anthropic/claude-opus-4-5-20251101
api_key: os.environ/ANTHROPIC_API_KEY
```
2. Start the proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data ' {
"model": "claude-4",
"messages": [{
"role": "user",
"content": "Analyze sales by region for the last quarter and identify top performers"
}],
"tools": [
{
"type": "tool_search_tool_regex_20251119",
"name": "tool_search_tool_regex"
},
# ... 100 deferred tools
],
"output_config": {
"effort": "medium"
}
}
'
```
Expected Response:
```json
{
...,
"usage": {
...,
"server_tool_use": {
"tool_search_requests": 1
}
}
}
```
</TabItem>
</Tabs>
### Real-World Benefits
This combination enables:
1. **Massive scale** - Handle 1000+ tools efficiently
2. **Low latency** - Programmatic calling reduces round trips
3. **High accuracy** - Input examples ensure correct tool usage
4. **Cost control** - Effort parameter optimizes token spend
5. **Full visibility** - Track all usage metrics
docs/my-website/docs/providers/anthropic.md
+3 -1
View File
@@ -41,7 +41,8 @@ Check this in code, [here](../completion/input.md#translated-openai-params)
"extra_headers",
"parallel_tool_calls",
"response_format",
"user"
"user",
"reasoning_effort",
```
:::info
@@ -49,6 +50,7 @@ Check this in code, [here](../completion/input.md#translated-openai-params)
**Notes:**
- Anthropic API fails requests when `max_tokens` are not passed. Due to this litellm passes `max_tokens=4096` when no `max_tokens` are passed.
- `response_format` is fully supported for Claude Sonnet 4.5 and Opus 4.1 models (see [Structured Outputs](#structured-outputs) section)
- `reasoning_effort` is automatically mapped to `output_config={"effort": ...}` for Claude Opus 4.5 models (see [Effort Parameter](./anthropic_effort.md))
:::
docs/my-website/docs/providers/anthropic_effort.md
+17 -10
View File
@@ -9,7 +9,10 @@ Control how many tokens Claude uses when responding with the `effort` parameter,
The `effort` parameter allows you to control how eager Claude is about spending tokens when responding to requests. This gives you the ability to trade off between response thoroughness and token efficiency, all with a single model.
**Note**: The effort parameter is currently in beta and only supported by Claude Opus 4.5. You must include the beta header `effort-2025-11-24` when using this feature (LiteLLM automatically adds this header when `output_config` with `effort` is detected).
**Note**: The effort parameter is currently in beta and only supported by Claude Opus 4.5. LiteLLM automatically adds the `effort-2025-11-24` beta header when:
- `reasoning_effort` parameter is provided (for Claude Opus 4.5 only)
For Claude Opus 4.5, `reasoning_effort="medium"`—both are automatically mapped to the correct format.
## How Effort Works
@@ -52,9 +55,7 @@ response = litellm.completion(
"role": "user",
"content": "Analyze the trade-offs between microservices and monolithic architectures"
}],
output_config={
"effort": "medium"
}
reasoning_effort="medium" # Automatically mapped to output_config for Opus 4.5
)
print(response.choices[0].message.content)
@@ -217,11 +218,14 @@ response = litellm.completion(
The effort parameter is supported across all Anthropic-compatible providers:
- **Standard Anthropic**: ✅ Supported (Claude Opus 4.5)
- **Azure Anthropic**: ✅ Supported (Claude Opus 4.5)
- **Vertex AI Anthropic**: ✅ Supported (Claude Opus 4.5)
- **Standard Anthropic API**: ✅ Supported (Claude Opus 4.5)
- **Azure Anthropic / Microsoft Foundry**: ✅ Supported (Claude Opus 4.5)
- **Amazon Bedrock**: ✅ Supported (Claude Opus 4.5)
- **Google Cloud Vertex AI**: ✅ Supported (Claude Opus 4.5)
LiteLLM automatically handles the beta header injection for all providers.
LiteLLM automatically handles:
- Beta header injection (`effort-2025-11-24`) for all providers
- Parameter mapping: `reasoning_effort` → `output_config={"effort": ...}` for Claude Opus 4.5
## Usage and Pricing
@@ -242,9 +246,12 @@ print(f"Total tokens: {response.usage.total_tokens}")
### Beta header not being added
LiteLLM automatically adds the `effort-2025-11-24` beta header when `output_config` with `effort` is detected. If you're not seeing the header:
LiteLLM automatically adds the `effort-2025-11-24` beta header when:
- `reasoning_effort` parameter is provided (for Claude Opus 4.5 only)
1. Ensure you're using `output_config` with an `effort` field
If you're not seeing the header:
1. Ensure you're using `reasoning_effort` parameter
2. Verify the model is Claude Opus 4.5
3. Check that LiteLLM version supports this feature
docs/my-website/docs/providers/anthropic_programmatic_tool_calling.md
+11 -6
View File
@@ -3,7 +3,11 @@
Programmatic tool calling allows Claude to write code that calls your tools programmatically within a code execution container, rather than requiring round trips through the model for each tool invocation. This reduces latency for multi-tool workflows and decreases token consumption by allowing Claude to filter or process data before it reaches the model's context window.
:::info
Programmatic tool calling is currently in public beta. LiteLLM automatically adds the required `advanced-tool-use-2025-11-20` beta header when it detects tools with the `allowed_callers` field.
Programmatic tool calling is currently in public beta. LiteLLM automatically detects tools with the `allowed_callers` field and adds the appropriate beta header based on your provider:
- **Anthropic API & Microsoft Foundry**: `advanced-tool-use-2025-11-20`
- **Amazon Bedrock**: `advanced-tool-use-2025-11-20`
- **Google Cloud Vertex AI**: Not supported
This feature requires the code execution tool to be enabled.
:::
@@ -380,13 +384,14 @@ For example, calling 10 tools directly uses ~10x the tokens of calling them prog
## Provider Support
LiteLLM supports programmatic tool calling across all Anthropic-compatible providers:
LiteLLM supports programmatic tool calling across the following Anthropic-compatible providers:
- **Standard Anthropic API** (`anthropic/claude-sonnet-4-5-20250929`)
- **Azure Anthropic** (`azure/claude-sonnet-4-5-20250929`)
- **Vertex AI Anthropic** (`vertex_ai/claude-sonnet-4-5-20250929`)
- **Standard Anthropic API** (`anthropic/claude-sonnet-4-5-20250929`) ✅
- **Azure Anthropic / Microsoft Foundry** (`azure/claude-sonnet-4-5-20250929`) ✅
- **Amazon Bedrock** (`bedrock/invoke/anthropic.claude-sonnet-4-5-20250929-v1:0`) ✅
- **Google Cloud Vertex AI** (`vertex_ai/claude-sonnet-4-5-20250929`) ❌ Not supported
The beta header is automatically added when LiteLLM detects tools with `allowed_callers` field.
The beta header (`advanced-tool-use-2025-11-20`) is automatically added when LiteLLM detects tools with the `allowed_callers` field.
## Limitations
docs/my-website/docs/providers/anthropic_tool_input_examples.md
+13 -6
View File
@@ -3,7 +3,13 @@
Provide concrete examples of valid tool inputs to help Claude understand how to use your tools more effectively. This is particularly useful for complex tools with nested objects, optional parameters, or format-sensitive inputs.
:::info
Tool input examples is a beta feature. LiteLLM automatically adds the required `advanced-tool-use-2025-11-20` beta header when it detects tools with the `input_examples` field.
Tool input examples is a beta feature. LiteLLM automatically detects tools with the `input_examples` field and adds the appropriate beta header based on your provider:
- **Anthropic API & Microsoft Foundry**: `advanced-tool-use-2025-11-20`
- **Amazon Bedrock**: `advanced-tool-use-2025-11-20` (Claude Opus 4.5 only)
- **Google Cloud Vertex AI**: Not supported
You don't need to manually specify beta headers—LiteLLM handles this automatically.
:::
## When to Use Input Examples
@@ -378,13 +384,14 @@ Input examples work seamlessly with other Anthropic tool features:
## Provider Support
LiteLLM supports input examples across all Anthropic-compatible providers:
LiteLLM supports input examples across the following Anthropic-compatible providers:
- **Standard Anthropic API** (`anthropic/claude-sonnet-4-5-20250929`)
- **Azure Anthropic** (`azure/claude-sonnet-4-5-20250929`)
- **Vertex AI Anthropic** (`vertex_ai/claude-sonnet-4-5-20250929`)
- **Standard Anthropic API** (`anthropic/claude-sonnet-4-5-20250929`) ✅
- **Azure Anthropic / Microsoft Foundry** (`azure/claude-sonnet-4-5-20250929`) ✅
- **Amazon Bedrock** (`bedrock/invoke/anthropic.claude-opus-4-5-20251101-v1:0`) ✅ (Opus 4.5 only)
- **Google Cloud Vertex AI** (`vertex_ai/claude-sonnet-4-5-20250929`) ❌ Not supported
The beta header is automatically added when LiteLLM detects tools with `input_examples` field.
The beta header (`advanced-tool-use-2025-11-20`) is automatically added when LiteLLM detects tools with the `input_examples` field.
## Troubleshooting
docs/my-website/docs/providers/anthropic_tool_search.md
+16 -1
View File
@@ -290,7 +290,13 @@ response = client.chat.completions.create(
### Beta Header
LiteLLM automatically adds the `advanced-tool-use-2025-11-20` beta header when tool search tools are detected. You don't need to manually specify it.
LiteLLM automatically detects tool search tools and adds the appropriate beta header based on your provider:
- **Anthropic API & Microsoft Foundry**: `advanced-tool-use-2025-11-20`
- **Google Cloud Vertex AI**: `tool-search-tool-2025-10-19`
- **Amazon Bedrock** (Invoke API, Opus 4.5 only): `tool-search-tool-2025-10-19`
You don't need to manually specify beta headers—LiteLLM handles this automatically.
### Deferred Loading
@@ -387,9 +393,18 @@ If Claude references a tool that isn't in your deferred tools list, you'll get a
- Not compatible with tool use examples
- Requires Claude Opus 4.5 or Sonnet 4.5
- On Bedrock, only available via invoke API (not converse API)
- On Bedrock, only supported for Claude Opus 4.5 (not Sonnet 4.5)
- BM25 variant (`tool_search_tool_bm25_20251119`) is not supported on Bedrock
- Maximum 10,000 tools in catalog
- Returns 3-5 most relevant tools per search
### Bedrock-Specific Notes
When using Bedrock's Invoke API:
- The regex variant (`tool_search_tool_regex_20251119`) is automatically normalized to `tool_search_tool_regex`
- The BM25 variant (`tool_search_tool_bm25_20251119`) is automatically filtered out as it's not supported
- Tool search is only available for Claude Opus 4.5 models
## Additional Resources
- [Anthropic Tool Search Documentation](https://docs.anthropic.com/en/docs/build-with-claude/tool-use/tool-search)
litellm/llms/anthropic/chat/transformation.py
+13 -4
View File
@@ -119,6 +119,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
def get_config(cls):
return super().get_config()
def _is_claude_opus_4_5(self, model: str) -> bool:
"""Check if the model is Claude Opus 4.5."""
return "opus-4-5" in model.lower() or "opus_4_5" in model.lower()
def get_supported_openai_params(self, model: str):
params = [
"stream",
@@ -626,7 +630,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
return hosted_web_search_tool
def map_openai_params(
def map_openai_params( # noqa: PLR0915
self,
non_default_params: dict,
optional_params: dict,
@@ -712,9 +716,14 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if param == "thinking":
optional_params["thinking"] = value
elif param == "reasoning_effort" and isinstance(value, str):
optional_params["thinking"] = AnthropicConfig._map_reasoning_effort(
value
)
# For Claude Opus 4.5, map reasoning_effort to output_config
if self._is_claude_opus_4_5(model):
optional_params["output_config"] = {"effort": value}
else:
# For other models, map to thinking parameter
optional_params["thinking"] = AnthropicConfig._map_reasoning_effort(
value
)
elif param == "web_search_options" and isinstance(value, dict):
hosted_web_search_tool = self.map_web_search_tool(
cast(OpenAIWebSearchOptions, value)
litellm/llms/anthropic/common_utils.py
+54 -4
View File
@@ -151,15 +151,22 @@ class AnthropicModelInfo(BaseLLMModelInfo):
return False
def is_effort_used(self, optional_params: Optional[dict]) -> bool:
def is_effort_used(self, optional_params: Optional[dict], model: Optional[str] = None) -> bool:
"""
Check if effort parameter is being used via output_config.
Check if effort parameter is being used.
Returns True if output_config with effort field is present.
Returns True if effort-related parameters are present.
"""
if not optional_params:
return False
# Check if reasoning_effort is provided for Claude Opus 4.5
if model and ("opus-4-5" in model.lower() or "opus_4_5" in model.lower()):
reasoning_effort = optional_params.get("reasoning_effort")
if reasoning_effort and isinstance(reasoning_effort, str):
return True
# Check if output_config is directly provided
output_config = optional_params.get("output_config")
if output_config and isinstance(output_config, dict):
effort = output_config.get("effort")
@@ -193,6 +200,49 @@ class AnthropicModelInfo(BaseLLMModelInfo):
computer_tool_version, "computer-use-2024-10-22" # Default fallback
)
def get_anthropic_beta_list(
self,
model: str,
optional_params: Optional[dict] = None,
computer_tool_used: Optional[str] = None,
prompt_caching_set: bool = False,
file_id_used: bool = False,
mcp_server_used: bool = False,
) -> List[str]:
"""
Get list of common beta headers based on the features that are active.
Returns:
List of beta header strings
"""
from litellm.types.llms.anthropic import (
ANTHROPIC_EFFORT_BETA_HEADER,
)
betas = []
# Detect features
effort_used = self.is_effort_used(optional_params, model)
if effort_used:
betas.append(ANTHROPIC_EFFORT_BETA_HEADER) # effort-2025-11-24
if computer_tool_used:
beta_header = self.get_computer_tool_beta_header(computer_tool_used)
betas.append(beta_header)
if prompt_caching_set:
betas.append("prompt-caching-2024-07-31")
if file_id_used:
betas.append("files-api-2025-04-14")
betas.append("code-execution-2025-05-22")
if mcp_server_used:
betas.append("mcp-client-2025-04-04")
return list(set(betas))
def get_anthropic_headers(
self,
api_key: str,
@@ -278,7 +328,7 @@ class AnthropicModelInfo(BaseLLMModelInfo):
tool_search_used = self.is_tool_search_used(tools=tools)
programmatic_tool_calling_used = self.is_programmatic_tool_calling_used(tools=tools)
input_examples_used = self.is_input_examples_used(tools=tools)
effort_used = self.is_effort_used(optional_params=optional_params)
effort_used = self.is_effort_used(optional_params=optional_params, model=model)
user_anthropic_beta_headers = self._get_user_anthropic_beta_headers(
anthropic_beta_header=headers.get("anthropic-beta")
)
litellm/llms/bedrock/chat/converse_transformation.py
+13 -3
View File
@@ -829,11 +829,21 @@ class AmazonConverseConfig(BaseConfig):
user_betas = get_anthropic_beta_from_headers(headers)
anthropic_beta_list.extend(user_betas)
# Filter out tool search tools - Bedrock Converse API doesn't support them
filtered_tools = []
if original_tools:
for tool in original_tools:
tool_type = tool.get("type", "")
if tool_type in ("tool_search_tool_regex_20251119", "tool_search_tool_bm25_20251119"):
# Tool search not supported in Converse API - skip it
continue
filtered_tools.append(tool)
# Only separate tools if computer use tools are actually present
if original_tools and self.is_computer_use_tool_used(original_tools, model):
if filtered_tools and self.is_computer_use_tool_used(filtered_tools, model):
# Separate computer use tools from regular function tools
computer_use_tools, regular_tools = self._separate_computer_use_tools(
original_tools, model
filtered_tools, model
)
# Process regular function tools using existing logic
@@ -849,7 +859,7 @@ class AmazonConverseConfig(BaseConfig):
additional_request_params["tools"] = transformed_computer_tools
else:
# No computer use tools, process all tools as regular tools
bedrock_tools = _bedrock_tools_pt(original_tools)
bedrock_tools = _bedrock_tools_pt(filtered_tools)
# Set anthropic_beta in additional_request_params if we have any beta features
if anthropic_beta_list:
litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py
+55 -4
View File
@@ -7,6 +7,7 @@ from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation
AmazonInvokeConfig,
)
from litellm.llms.bedrock.common_utils import get_anthropic_beta_from_headers
from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse
@@ -76,6 +77,7 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
for k, v in optional_params.items()
if k not in self.aws_authentication_params
}
filtered_params = self._normalize_bedrock_tool_search_tools(filtered_params)
_anthropic_request = AnthropicConfig.transform_request(
self,
@@ -91,13 +93,62 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
if "anthropic_version" not in _anthropic_request:
_anthropic_request["anthropic_version"] = self.anthropic_version
# Handle anthropic_beta from user headers
anthropic_beta_list = get_anthropic_beta_from_headers(headers)
if anthropic_beta_list:
_anthropic_request["anthropic_beta"] = anthropic_beta_list
tools = optional_params.get("tools")
tool_search_used = self.is_tool_search_used(tools)
programmatic_tool_calling_used = self.is_programmatic_tool_calling_used(tools)
input_examples_used = self.is_input_examples_used(tools)
beta_set = set(get_anthropic_beta_from_headers(headers))
auto_betas = self.get_anthropic_beta_list(
model=model,
optional_params=optional_params,
computer_tool_used=self.is_computer_tool_used(tools),
prompt_caching_set=self.is_cache_control_set(messages),
file_id_used=self.is_file_id_used(messages),
mcp_server_used=self.is_mcp_server_used(optional_params.get("mcp_servers")),
)
beta_set.update(auto_betas)
if (
tool_search_used
and not (programmatic_tool_calling_used or input_examples_used)
):
beta_set.discard(ANTHROPIC_TOOL_SEARCH_BETA_HEADER)
if "opus-4" in model.lower() or "opus_4" in model.lower():
beta_set.add("tool-search-tool-2025-10-19")
if beta_set:
_anthropic_request["anthropic_beta"] = list(beta_set)
return _anthropic_request
def _normalize_bedrock_tool_search_tools(self, optional_params: dict) -> dict:
"""
Convert tool search entries to the format supported by the Bedrock Invoke API.
"""
tools = optional_params.get("tools")
if not tools or not isinstance(tools, list):
return optional_params
normalized_tools = []
for tool in tools:
tool_type = tool.get("type")
if tool_type == "tool_search_tool_bm25_20251119":
# Bedrock Invoke does not support the BM25 variant, so skip it.
continue
if tool_type == "tool_search_tool_regex_20251119":
normalized_tool = tool.copy()
normalized_tool["type"] = "tool_search_tool_regex"
normalized_tool["name"] = normalized_tool.get(
"name", "tool_search_tool_regex"
)
normalized_tools.append(normalized_tool)
continue
normalized_tools.append(tool)
optional_params["tools"] = normalized_tools
return optional_params
def transform_response(
self,
model: str,
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
+47 -5
View File
@@ -1,7 +1,18 @@
from typing import TYPE_CHECKING, Any, AsyncIterator, Dict, List, Optional, Tuple, Union
from typing import (
TYPE_CHECKING,
Any,
AsyncIterator,
Dict,
List,
Optional,
Tuple,
Union,
cast,
)
import httpx
from litellm.llms.anthropic.common_utils import AnthropicModelInfo
from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
AnthropicMessagesConfig,
)
@@ -13,6 +24,8 @@ from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation
AmazonInvokeConfig,
)
from litellm.llms.bedrock.common_utils import get_anthropic_beta_from_headers
from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER
from litellm.types.llms.openai import AllMessageValues
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import GenericStreamingChunk
from litellm.types.utils import GenericStreamingChunk as GChunk
@@ -129,10 +142,39 @@ class AmazonAnthropicClaudeMessagesConfig(
if "model" in anthropic_messages_request:
anthropic_messages_request.pop("model", None)
# 4. Handle anthropic_beta from user headers
anthropic_beta_list = get_anthropic_beta_from_headers(headers)
if anthropic_beta_list:
anthropic_messages_request["anthropic_beta"] = anthropic_beta_list
# 4. AUTO-INJECT beta headers based on features used
anthropic_model_info = AnthropicModelInfo()
tools = anthropic_messages_optional_request_params.get("tools")
messages_typed = cast(List[AllMessageValues], messages)
tool_search_used = anthropic_model_info.is_tool_search_used(tools)
programmatic_tool_calling_used = anthropic_model_info.is_programmatic_tool_calling_used(
tools
)
input_examples_used = anthropic_model_info.is_input_examples_used(tools)
beta_set = set(get_anthropic_beta_from_headers(headers))
auto_betas = anthropic_model_info.get_anthropic_beta_list(
model=model,
optional_params=anthropic_messages_optional_request_params,
computer_tool_used=anthropic_model_info.is_computer_tool_used(tools),
prompt_caching_set=anthropic_model_info.is_cache_control_set(messages_typed),
file_id_used=anthropic_model_info.is_file_id_used(messages_typed),
mcp_server_used=anthropic_model_info.is_mcp_server_used(
anthropic_messages_optional_request_params.get("mcp_servers")
),
)
beta_set.update(auto_betas)
if (
tool_search_used
and not (programmatic_tool_calling_used or input_examples_used)
):
beta_set.discard(ANTHROPIC_TOOL_SEARCH_BETA_HEADER)
if "opus-4" in model.lower() or "opus_4" in model.lower():
beta_set.add("tool-search-tool-2025-10-19")
if beta_set:
anthropic_messages_request["anthropic_beta"] = list(beta_set)
return anthropic_messages_request
litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/transformation.py
+19
View File
@@ -68,6 +68,25 @@ class VertexAIAnthropicConfig(AnthropicConfig):
)
data.pop("model", None) # vertex anthropic doesn't accept 'model' parameter
tools = optional_params.get("tools")
tool_search_used = self.is_tool_search_used(tools)
auto_betas = self.get_anthropic_beta_list(
model=model,
optional_params=optional_params,
computer_tool_used=self.is_computer_tool_used(tools),
prompt_caching_set=self.is_cache_control_set(messages),
file_id_used=self.is_file_id_used(messages),
mcp_server_used=self.is_mcp_server_used(optional_params.get("mcp_servers")),
)
beta_set = set(auto_betas)
if tool_search_used:
beta_set.add("tool-search-tool-2025-10-19") # Vertex requires this header for tool search
if beta_set:
data["anthropic_beta"] = list(beta_set)
return data
def transform_response(
litellm/main.py
+36
View File
@@ -942,6 +942,37 @@ def responses_api_bridge_check(
return model_info, model
def _should_allow_input_examples(custom_llm_provider: Optional[str], model: str) -> bool:
if custom_llm_provider == "anthropic":
return True
if custom_llm_provider == "azure_ai" or custom_llm_provider == "bedrock" or custom_llm_provider == "vertex_ai":
return "claude" in model.lower()
return False
def _drop_input_examples_from_tool(tool: dict) -> dict:
tool_copy = tool.copy()
tool_copy.pop("input_examples", None)
function = tool_copy.get("function")
if isinstance(function, dict):
function = function.copy()
function.pop("input_examples", None)
tool_copy["function"] = function
return tool_copy
def _drop_input_examples_from_tools(tools: Optional[List[dict]]) -> Optional[List[dict]]:
if tools is None:
return None
cleaned_tools: List[dict] = []
for tool in tools:
if isinstance(tool, dict):
cleaned_tools.append(_drop_input_examples_from_tool(tool))
else:
cleaned_tools.append(tool)
return cleaned_tools
@tracer.wrap()
@client
def completion( # type: ignore # noqa: PLR0915
@@ -1187,6 +1218,11 @@ def completion( # type: ignore # noqa: PLR0915
api_key=api_key,
)
if not _should_allow_input_examples(
custom_llm_provider=custom_llm_provider, model=model
):
tools = _drop_input_examples_from_tools(tools=tools)
if provider_specific_header is not None:
headers.update(
ProviderSpecificHeaderUtils.get_provider_specific_headers(
tests/test_litellm/test_main.py
+26
View File
@@ -16,6 +16,8 @@ from unittest.mock import MagicMock, patch
import litellm
from litellm import main as litellm_main
@pytest.fixture(autouse=True)
def add_api_keys_to_env(monkeypatch):
@@ -293,6 +295,30 @@ def test_bedrock_latency_optimized_inference():
assert json_data["performanceConfig"]["latency"] == "optimized"
def test_strip_input_examples_for_non_anthropic_providers():
tools = [
{
"type": "function",
"name": "example_tool",
"input_examples": [{"foo": "bar"}],
"function": {
"name": "example_tool",
"input_examples": [{"foo": "bar"}],
},
}
]
assert not litellm_main._should_allow_input_examples(
custom_llm_provider="openai", model="gpt-4o-mini"
)
cleaned = litellm_main._drop_input_examples_from_tools(tools=tools)
assert isinstance(cleaned, list)
assert "input_examples" not in cleaned[0]
assert "input_examples" not in cleaned[0]["function"]
def test_custom_provider_with_extra_headers():
from litellm.llms.custom_httpx.http_handler import HTTPHandler