mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-21 08:26:34 +00:00
Merge branch 'main' into fix/mcp-call-tool-context
This commit is contained in:
Krish Dholakia
+309
-1
@@ -246,8 +246,203 @@ litellm_settings:
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## MCP Tool Filtering
|
||||
## Converting OpenAPI Specs to MCP Servers
|
||||
|
||||
LiteLLM can automatically convert OpenAPI specifications into MCP servers, allowing you to expose any REST API as MCP tools. This is useful when you have existing APIs with OpenAPI/Swagger documentation and want to make them available as MCP tools.
|
||||
|
||||
### Benefits
|
||||
|
||||
- **Rapid Integration**: Convert existing APIs to MCP tools without writing custom MCP server code
|
||||
- **Automatic Tool Generation**: LiteLLM automatically generates MCP tools from your OpenAPI spec
|
||||
- **Unified Interface**: Use the same MCP interface for both native MCP servers and OpenAPI-based APIs
|
||||
- **Easy Testing**: Test and iterate on API integrations quickly
|
||||
|
||||
### Configuration
|
||||
|
||||
Add your OpenAPI-based MCP server to your `config.yaml`:
|
||||
|
||||
```yaml title="config.yaml - OpenAPI to MCP" showLineNumbers
|
||||
model_list:
|
||||
- model_name: gpt-4o
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: sk-xxxxxxx
|
||||
|
||||
mcp_servers:
|
||||
# OpenAPI Spec Example - Petstore API
|
||||
petstore_mcp:
|
||||
url: "https://petstore.swagger.io/v2"
|
||||
spec_path: "/path/to/openapi.json"
|
||||
auth_type: "none"
|
||||
|
||||
# OpenAPI Spec with API Key Authentication
|
||||
my_api_mcp:
|
||||
url: "http://0.0.0.0:8090"
|
||||
spec_path: "/path/to/openapi.json"
|
||||
auth_type: "api_key"
|
||||
auth_value: "your-api-key-here"
|
||||
|
||||
# OpenAPI Spec with Bearer Token
|
||||
secured_api_mcp:
|
||||
url: "https://api.example.com"
|
||||
spec_path: "/path/to/openapi.json"
|
||||
auth_type: "bearer_token"
|
||||
auth_value: "your-bearer-token"
|
||||
```
|
||||
|
||||
### Configuration Parameters
|
||||
|
||||
| Parameter | Required | Description |
|
||||
|-----------|----------|-------------|
|
||||
| `url` | Yes | The base URL of your API endpoint |
|
||||
| `spec_path` | Yes | Path or URL to your OpenAPI specification file (JSON or YAML) |
|
||||
| `auth_type` | No | Authentication type: `none`, `api_key`, `bearer_token`, `basic`, `authorization` |
|
||||
| `auth_value` | No | Authentication value (required if `auth_type` is set) |
|
||||
| `description` | No | Optional description for the MCP server |
|
||||
| `allowed_tools` | No | List of specific tools to allow (see [MCP Tool Filtering](#mcp-tool-filtering)) |
|
||||
| `disallowed_tools` | No | List of specific tools to block (see [MCP Tool Filtering](#mcp-tool-filtering)) |
|
||||
|
||||
### Usage Example
|
||||
|
||||
Once configured, you can use the OpenAPI-based MCP server just like any other MCP server:
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="fastmcp" label="Python FastMCP">
|
||||
|
||||
```python title="Using OpenAPI-based MCP Server" showLineNumbers
|
||||
from fastmcp import Client
|
||||
import asyncio
|
||||
|
||||
# Standard MCP configuration
|
||||
config = {
|
||||
"mcpServers": {
|
||||
"petstore": {
|
||||
"url": "http://localhost:4000/petstore_mcp/mcp",
|
||||
"headers": {
|
||||
"x-litellm-api-key": "Bearer sk-1234"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Create a client that connects to the server
|
||||
client = Client(config)
|
||||
|
||||
async def main():
|
||||
async with client:
|
||||
# List available tools generated from OpenAPI spec
|
||||
tools = await client.list_tools()
|
||||
print(f"Available tools: {[tool.name for tool in tools]}")
|
||||
|
||||
# Example: Get a pet by ID (from Petstore API)
|
||||
response = await client.call_tool(
|
||||
name="getpetbyid",
|
||||
arguments={"petId": "1"}
|
||||
)
|
||||
print(f"Response:\n{response}\n")
|
||||
|
||||
# Example: Find pets by status
|
||||
response = await client.call_tool(
|
||||
name="findpetsbystatus",
|
||||
arguments={"status": "available"}
|
||||
)
|
||||
print(f"Response:\n{response}\n")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="cursor" label="Cursor IDE">
|
||||
|
||||
```json title="Cursor MCP Configuration for OpenAPI Server" showLineNumbers
|
||||
{
|
||||
"mcpServers": {
|
||||
"Petstore": {
|
||||
"url": "http://localhost:4000/petstore_mcp/mcp",
|
||||
"headers": {
|
||||
"x-litellm-api-key": "Bearer $LITELLM_API_KEY"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="openai" label="OpenAI Responses API">
|
||||
|
||||
```bash title="Using OpenAPI MCP Server with OpenAI" showLineNumbers
|
||||
curl --location 'https://api.openai.com/v1/responses' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header "Authorization: Bearer $OPENAI_API_KEY" \
|
||||
--data '{
|
||||
"model": "gpt-4o",
|
||||
"tools": [
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "petstore",
|
||||
"server_url": "http://localhost:4000/petstore_mcp/mcp",
|
||||
"require_approval": "never",
|
||||
"headers": {
|
||||
"x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY"
|
||||
}
|
||||
}
|
||||
],
|
||||
"input": "Find all available pets in the petstore",
|
||||
"tool_choice": "required"
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **Spec Loading**: LiteLLM loads your OpenAPI specification from the provided `spec_path`
|
||||
2. **Tool Generation**: Each API endpoint in the spec becomes an MCP tool
|
||||
3. **Parameter Mapping**: OpenAPI parameters are automatically mapped to MCP tool parameters
|
||||
4. **Request Handling**: When a tool is called, LiteLLM converts the MCP request to the appropriate HTTP request
|
||||
5. **Response Translation**: API responses are converted back to MCP format
|
||||
|
||||
### OpenAPI Spec Requirements
|
||||
|
||||
Your OpenAPI specification should follow standard OpenAPI/Swagger conventions:
|
||||
- **Supported versions**: OpenAPI 3.0.x, OpenAPI 3.1.x, Swagger 2.0
|
||||
- **Required fields**: `paths`, `info` sections should be properly defined
|
||||
- **Operation IDs**: Each operation should have a unique `operationId` (this becomes the tool name)
|
||||
- **Parameters**: Request parameters should be properly documented with types and descriptions
|
||||
|
||||
### Example OpenAPI Spec Structure
|
||||
|
||||
```yaml title="sample-openapi.yaml" showLineNumbers
|
||||
openapi: 3.0.0
|
||||
info:
|
||||
title: My API
|
||||
version: 1.0.0
|
||||
paths:
|
||||
/pets/{petId}:
|
||||
get:
|
||||
operationId: getPetById
|
||||
summary: Get a pet by ID
|
||||
parameters:
|
||||
- name: petId
|
||||
in: path
|
||||
required: true
|
||||
schema:
|
||||
type: integer
|
||||
responses:
|
||||
'200':
|
||||
description: Successful response
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
```
|
||||
|
||||
## Allow/Disallow MCP Tools
|
||||
|
||||
Control which tools are available from your MCP servers. You can either allow only specific tools or block dangerous ones.
|
||||
|
||||
<Tabs>
|
||||
@@ -306,6 +501,119 @@ mcp_servers:
|
||||
- If you specify both `allowed_tools` and `disallowed_tools`, the allowed list takes priority
|
||||
- Tool names are case-sensitive
|
||||
|
||||
---
|
||||
|
||||
## Allow/Disallow MCP Tool Parameters
|
||||
|
||||
Control which parameters are allowed for specific MCP tools using the `allowed_params` configuration. This provides fine-grained control over tool usage by restricting the parameters that can be passed to each tool.
|
||||
|
||||
### Configuration
|
||||
|
||||
`allowed_params` is a dictionary that maps tool names to lists of allowed parameter names. When configured, only the specified parameters will be accepted for that tool - any other parameters will be rejected with a 403 error.
|
||||
|
||||
```yaml title="config.yaml with allowed_params" showLineNumbers
|
||||
mcp_servers:
|
||||
deepwiki_mcp:
|
||||
url: https://mcp.deepwiki.com/mcp
|
||||
transport: "http"
|
||||
auth_type: "none"
|
||||
allowed_params:
|
||||
# Tool name: list of allowed parameters
|
||||
read_wiki_contents: ["status"]
|
||||
|
||||
my_api_mcp:
|
||||
url: "https://my-api-server.com"
|
||||
auth_type: "api_key"
|
||||
auth_value: "my-key"
|
||||
allowed_params:
|
||||
# Using unprefixed tool name
|
||||
getpetbyid: ["status"]
|
||||
# Using prefixed tool name (both formats work)
|
||||
my_api_mcp-findpetsbystatus: ["status", "limit"]
|
||||
# Another tool with multiple allowed params
|
||||
create_issue: ["title", "body", "labels"]
|
||||
```
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **Tool-specific filtering**: Each tool can have its own list of allowed parameters
|
||||
2. **Flexible naming**: Tool names can be specified with or without the server prefix (e.g., both `"getpetbyid"` and `"my_api_mcp-getpetbyid"` work)
|
||||
3. **Whitelist approach**: Only parameters in the allowed list are permitted
|
||||
4. **Unlisted tools**: If `allowed_params` is not set, all parameters are allowed
|
||||
5. **Error handling**: Requests with disallowed parameters receive a 403 error with details about which parameters are allowed
|
||||
|
||||
### Example Request Behavior
|
||||
|
||||
With the configuration above, here's how requests would be handled:
|
||||
|
||||
**✅ Allowed Request:**
|
||||
```json
|
||||
{
|
||||
"tool": "read_wiki_contents",
|
||||
"arguments": {
|
||||
"status": "active"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**❌ Rejected Request:**
|
||||
```json
|
||||
{
|
||||
"tool": "read_wiki_contents",
|
||||
"arguments": {
|
||||
"status": "active",
|
||||
"limit": 10 // This parameter is not allowed
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Error Response:**
|
||||
```json
|
||||
{
|
||||
"error": "Parameters ['limit'] are not allowed for tool read_wiki_contents. Allowed parameters: ['status']. Contact proxy admin to allow these parameters."
|
||||
}
|
||||
```
|
||||
|
||||
### Use Cases
|
||||
|
||||
- **Security**: Prevent users from accessing sensitive parameters or dangerous operations
|
||||
- **Cost control**: Restrict expensive parameters (e.g., limiting result counts)
|
||||
- **Compliance**: Enforce parameter usage policies for regulatory requirements
|
||||
- **Staged rollouts**: Gradually enable parameters as tools are tested
|
||||
- **Multi-tenant isolation**: Different parameter access for different user groups
|
||||
|
||||
### Combining with Tool Filtering
|
||||
|
||||
`allowed_params` works alongside `allowed_tools` and `disallowed_tools` for complete control:
|
||||
|
||||
```yaml title="Combined filtering example" showLineNumbers
|
||||
mcp_servers:
|
||||
github_mcp:
|
||||
url: "https://api.githubcopilot.com/mcp"
|
||||
auth_type: oauth2
|
||||
authorization_url: https://github.com/login/oauth/authorize
|
||||
token_url: https://github.com/login/oauth/access_token
|
||||
client_id: os.environ/GITHUB_OAUTH_CLIENT_ID
|
||||
client_secret: os.environ/GITHUB_OAUTH_CLIENT_SECRET
|
||||
scopes: ["public_repo", "user:email"]
|
||||
# Only allow specific tools
|
||||
allowed_tools: ["create_issue", "list_issues", "search_issues"]
|
||||
# Block dangerous operations
|
||||
disallowed_tools: ["delete_repo"]
|
||||
# Restrict parameters per tool
|
||||
allowed_params:
|
||||
create_issue: ["title", "body", "labels"]
|
||||
list_issues: ["state", "sort", "perPage"]
|
||||
search_issues: ["query", "sort", "order", "perPage"]
|
||||
```
|
||||
|
||||
This configuration ensures that:
|
||||
1. Only the three listed tools are available
|
||||
2. The `delete_repo` tool is explicitly blocked
|
||||
3. Each tool can only use its specified parameters
|
||||
|
||||
---
|
||||
|
||||
## MCP Server Access Control
|
||||
|
||||
LiteLLM Proxy provides two methods for controlling access to specific MCP servers:
|
||||
|
||||
@@ -16,7 +16,6 @@ import TabItem from '@theme/TabItem';
|
||||
| AI21 (Jamba) | `vertex_ai/jamba-*` | [Vertex AI - AI21 Models](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/ai21) |
|
||||
| Qwen | `vertex_ai/qwen/*` | [Vertex AI - Qwen Models](https://cloud.google.com/vertex-ai/generative-ai/docs/maas/qwen) |
|
||||
| OpenAI (GPT-OSS) | `vertex_ai/openai/gpt-oss-*` | [Vertex AI - GPT-OSS Models](https://console.cloud.google.com/vertex-ai/publishers/openai/model-garden/) |
|
||||
| Model Garden | `vertex_ai/openai/{MODEL_ID}` or `vertex_ai/{MODEL_ID}` | [Vertex Model Garden](https://cloud.google.com/model-garden?hl=en) |
|
||||
|
||||
## Vertex AI - Anthropic (Claude)
|
||||
|
||||
@@ -793,112 +792,3 @@ curl http://0.0.0.0:4000/v1/chat/completions \
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Model Garden
|
||||
|
||||
:::tip
|
||||
|
||||
All OpenAI compatible models from Vertex Model Garden are supported.
|
||||
|
||||
:::
|
||||
|
||||
#### Using Model Garden
|
||||
|
||||
**Almost all Vertex Model Garden models are OpenAI compatible.**
|
||||
|
||||
<Tabs>
|
||||
|
||||
<TabItem value="openai" label="OpenAI Compatible Models">
|
||||
|
||||
| Property | Details |
|
||||
|----------|---------|
|
||||
| Provider Route | `vertex_ai/openai/{MODEL_ID}` |
|
||||
| Vertex Documentation | [Model Garden LiteLLM Inference](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/model_garden_litellm_inference.ipynb), [Vertex Model Garden](https://cloud.google.com/model-garden?hl=en) |
|
||||
| Supported Operations | `/chat/completions`, `/embeddings` |
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="sdk" label="SDK">
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
import os
|
||||
|
||||
## set ENV variables
|
||||
os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811"
|
||||
os.environ["VERTEXAI_LOCATION"] = "us-central1"
|
||||
|
||||
response = completion(
|
||||
model="vertex_ai/openai/<your-endpoint-id>",
|
||||
messages=[{ "content": "Hello, how are you?","role": "user"}]
|
||||
)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="proxy" label="Proxy">
|
||||
|
||||
|
||||
**1. Add to config**
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: llama3-1-8b-instruct
|
||||
litellm_params:
|
||||
model: vertex_ai/openai/5464397967697903616
|
||||
vertex_ai_project: "my-test-project"
|
||||
vertex_ai_location: "us-east-1"
|
||||
```
|
||||
|
||||
**2. Start proxy**
|
||||
|
||||
```bash
|
||||
litellm --config /path/to/config.yaml
|
||||
|
||||
# RUNNING at http://0.0.0.0:4000
|
||||
```
|
||||
|
||||
**3. Test it!**
|
||||
|
||||
```bash
|
||||
curl --location 'http://0.0.0.0:4000/chat/completions' \
|
||||
--header 'Authorization: Bearer sk-1234' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"model": "llama3-1-8b-instruct", # 👈 the 'model_name' in config
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "what llm are you"
|
||||
}
|
||||
],
|
||||
}'
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
</TabItem>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="non-openai" label="Non-OpenAI Compatible Models">
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
import os
|
||||
|
||||
## set ENV variables
|
||||
os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811"
|
||||
os.environ["VERTEXAI_LOCATION"] = "us-central1"
|
||||
|
||||
response = completion(
|
||||
model="vertex_ai/<your-endpoint-id>",
|
||||
messages=[{ "content": "Hello, how are you?","role": "user"}]
|
||||
)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
</Tabs>
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# Vertex AI - Self Deployed Models
|
||||
|
||||
Deploy and use your own models on Vertex AI through Model Garden or custom endpoints.
|
||||
|
||||
## Model Garden
|
||||
|
||||
:::tip
|
||||
|
||||
All OpenAI compatible models from Vertex Model Garden are supported.
|
||||
|
||||
:::
|
||||
|
||||
### Using Model Garden
|
||||
|
||||
**Almost all Vertex Model Garden models are OpenAI compatible.**
|
||||
|
||||
<Tabs>
|
||||
|
||||
<TabItem value="openai" label="OpenAI Compatible Models">
|
||||
|
||||
| Property | Details |
|
||||
|----------|---------|
|
||||
| Provider Route | `vertex_ai/openai/{MODEL_ID}` |
|
||||
| Vertex Documentation | [Model Garden LiteLLM Inference](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/model_garden_litellm_inference.ipynb), [Vertex Model Garden](https://cloud.google.com/model-garden?hl=en) |
|
||||
| Supported Operations | `/chat/completions`, `/embeddings` |
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="sdk" label="SDK">
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
import os
|
||||
|
||||
## set ENV variables
|
||||
os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811"
|
||||
os.environ["VERTEXAI_LOCATION"] = "us-central1"
|
||||
|
||||
response = completion(
|
||||
model="vertex_ai/openai/<your-endpoint-id>",
|
||||
messages=[{ "content": "Hello, how are you?","role": "user"}]
|
||||
)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="proxy" label="Proxy">
|
||||
|
||||
|
||||
**1. Add to config**
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: llama3-1-8b-instruct
|
||||
litellm_params:
|
||||
model: vertex_ai/openai/5464397967697903616
|
||||
vertex_ai_project: "my-test-project"
|
||||
vertex_ai_location: "us-east-1"
|
||||
```
|
||||
|
||||
**2. Start proxy**
|
||||
|
||||
```bash
|
||||
litellm --config /path/to/config.yaml
|
||||
|
||||
# RUNNING at http://0.0.0.0:4000
|
||||
```
|
||||
|
||||
**3. Test it!**
|
||||
|
||||
```bash
|
||||
curl --location 'http://0.0.0.0:4000/chat/completions' \
|
||||
--header 'Authorization: Bearer sk-1234' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"model": "llama3-1-8b-instruct", # 👈 the 'model_name' in config
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "what llm are you"
|
||||
}
|
||||
],
|
||||
}'
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
</TabItem>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="non-openai" label="Non-OpenAI Compatible Models">
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
import os
|
||||
|
||||
## set ENV variables
|
||||
os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811"
|
||||
os.environ["VERTEXAI_LOCATION"] = "us-central1"
|
||||
|
||||
response = completion(
|
||||
model="vertex_ai/<your-endpoint-id>",
|
||||
messages=[{ "content": "Hello, how are you?","role": "user"}]
|
||||
)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
</Tabs>
|
||||
|
||||
## Gemma Models (Custom Endpoints)
|
||||
|
||||
Deploy Gemma models on custom Vertex AI prediction endpoints with OpenAI-compatible format.
|
||||
|
||||
| Property | Details |
|
||||
|----------|---------|
|
||||
| Provider Route | `vertex_ai/gemma/{MODEL_NAME}` |
|
||||
| Vertex Documentation | [Vertex AI Prediction](https://cloud.google.com/vertex-ai/docs/predictions/get-predictions) |
|
||||
| Required Parameter | `api_base` - Full prediction endpoint URL |
|
||||
|
||||
### Usage
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="proxy" label="Proxy">
|
||||
|
||||
**1. Add to config.yaml**
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gemma-model
|
||||
litellm_params:
|
||||
model: vertex_ai/gemma/gemma-3-12b-it-1222199011122
|
||||
api_base: https://ENDPOINT.us-central1-PROJECT.prediction.vertexai.goog/v1/projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID:predict
|
||||
vertex_project: "my-project-id"
|
||||
vertex_location: "us-central1"
|
||||
```
|
||||
|
||||
**2. Start proxy**
|
||||
|
||||
```bash
|
||||
litellm --config /path/to/config.yaml
|
||||
```
|
||||
|
||||
**3. Test it**
|
||||
|
||||
```bash
|
||||
curl http://0.0.0.0:4000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-d '{
|
||||
"model": "gemma-model",
|
||||
"messages": [{"role": "user", "content": "What is machine learning?"}],
|
||||
"max_tokens": 100
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="sdk" label="SDK">
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
|
||||
response = completion(
|
||||
model="vertex_ai/gemma/gemma-3-12b-it-1222199011122",
|
||||
messages=[{"role": "user", "content": "What is machine learning?"}],
|
||||
api_base="https://ENDPOINT.us-central1-PROJECT.prediction.vertexai.goog/v1/projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID:predict",
|
||||
vertex_project="my-project-id",
|
||||
vertex_location="us-central1",
|
||||
)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
@@ -353,7 +353,10 @@ router_settings:
|
||||
| AGENTOPS_SERVICE_NAME | Service Name for AgentOps logging integration
|
||||
| AISPEND_ACCOUNT_ID | Account ID for AI Spend
|
||||
| AISPEND_API_KEY | API Key for AI Spend
|
||||
| AIOHTTP_CONNECTOR_LIMIT | Connection limit for aiohttp connector. When set to 0, no limit is applied. **Default is 0**
|
||||
| AIOHTTP_KEEPALIVE_TIMEOUT | Keep-alive timeout for aiohttp connections in seconds. **Default is 120**
|
||||
| AIOHTTP_TRUST_ENV | Flag to enable aiohttp trust environment. When this is set to True, aiohttp will respect HTTP(S)_PROXY env vars. **Default is False**
|
||||
| AIOHTTP_TTL_DNS_CACHE | DNS cache time-to-live for aiohttp in seconds. **Default is 300**
|
||||
| ALLOWED_EMAIL_DOMAINS | List of email domains allowed for access
|
||||
| ARIZE_API_KEY | API key for Arize platform integration
|
||||
| ARIZE_SPACE_KEY | Space key for Arize platform
|
||||
@@ -506,6 +509,8 @@ router_settings:
|
||||
| EMAIL_SIGNATURE | Custom HTML footer/signature for all emails. Can include HTML tags for formatting and links.
|
||||
| EMAIL_SUBJECT_INVITATION | Custom subject template for invitation emails.
|
||||
| EMAIL_SUBJECT_KEY_CREATED | Custom subject template for key creation emails.
|
||||
| ENKRYPTAI_API_BASE | Base URL for EnkryptAI Guardrails API. **Default is https://api.enkryptai.com**
|
||||
| ENKRYPTAI_API_KEY | API key for EnkryptAI Guardrails service
|
||||
| EXPERIMENTAL_MULTI_INSTANCE_RATE_LIMITING | Flag to enable new multi-instance rate limiting. **Default is False**
|
||||
| FIREWORKS_AI_4_B | Size parameter for Fireworks AI 4B model. Default is 4
|
||||
| FIREWORKS_AI_16_B | Size parameter for Fireworks AI 16B model. Default is 16
|
||||
@@ -629,6 +634,7 @@ router_settings:
|
||||
| LITELLM_MODE | Operating mode for LiteLLM (e.g., production, development)
|
||||
| LITELLM_RATE_LIMIT_WINDOW_SIZE | Rate limit window size for LiteLLM. Default is 60
|
||||
| LITELLM_SALT_KEY | Salt key for encryption in LiteLLM
|
||||
| LITELLM_SSL_CIPHERS | SSL/TLS cipher configuration for faster handshakes. Controls cipher suite preferences for OpenSSL connections.
|
||||
| LITELLM_SECRET_AWS_KMS_LITELLM_LICENSE | AWS KMS encrypted license for LiteLLM
|
||||
| LITELLM_TOKEN | Access token for LiteLLM integration
|
||||
| LITELLM_PRINT_STANDARD_LOGGING_PAYLOAD | If true, prints the standard logging payload to the console - useful for debugging
|
||||
|
||||
@@ -0,0 +1,276 @@
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# EnkryptAI Guardrails
|
||||
|
||||
LiteLLM supports EnkryptAI guardrails for content moderation and safety checks on LLM inputs and outputs.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Define Guardrails on your LiteLLM config.yaml
|
||||
|
||||
Define your guardrails under the `guardrails` section:
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gpt-3.5-turbo
|
||||
litellm_params:
|
||||
model: openai/gpt-3.5-turbo
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
|
||||
guardrails:
|
||||
- guardrail_name: "enkryptai-guard"
|
||||
litellm_params:
|
||||
guardrail: enkryptai
|
||||
mode: "pre_call"
|
||||
api_key: os.environ/ENKRYPTAI_API_KEY
|
||||
detectors:
|
||||
toxicity:
|
||||
enabled: true
|
||||
nsfw:
|
||||
enabled: true
|
||||
pii:
|
||||
enabled: true
|
||||
entities: ["email", "phone", "secrets"]
|
||||
injection_attack:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
#### Supported values for `mode`
|
||||
|
||||
- `pre_call` - Run **before** LLM call, on **input**
|
||||
- `post_call` - Run **after** LLM call, on **output**
|
||||
- `during_call` - Run **during** LLM call, on **input**. Same as `pre_call` but runs in parallel as LLM call
|
||||
|
||||
#### Available Detectors
|
||||
|
||||
EnkryptAI supports multiple content detection types:
|
||||
|
||||
- **toxicity** - Detect toxic language
|
||||
- **nsfw** - Detect NSFW (Not Safe For Work) content
|
||||
- **pii** - Detect personally identifiable information
|
||||
- Configure entities: `["pii", "email", "phone", "secrets", "ip_address", "url"]`
|
||||
- **injection_attack** - Detect prompt injection attempts
|
||||
- **keyword_detector** - Detect custom keywords/phrases
|
||||
- **policy_violation** - Detect policy violations
|
||||
- **bias** - Detect biased content
|
||||
- **sponge_attack** - Detect sponge attacks
|
||||
|
||||
### 2. Set Environment Variables
|
||||
|
||||
```bash
|
||||
export ENKRYPTAI_API_KEY="your-api-key"
|
||||
```
|
||||
|
||||
### 3. Start LiteLLM Gateway
|
||||
|
||||
```shell
|
||||
litellm --config config.yaml --detailed_debug
|
||||
```
|
||||
|
||||
### 4. Test Request
|
||||
|
||||
**[Langchain, OpenAI SDK Usage Examples](../proxy/user_keys#request-format)**
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="Successful Call" value="allowed">
|
||||
|
||||
```shell
|
||||
curl -i http://localhost:4000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-d '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Hello, how can you help me today?"}
|
||||
],
|
||||
"guardrails": ["enkryptai-guard"]
|
||||
}'
|
||||
```
|
||||
|
||||
**Response: HTTP 200 Success**
|
||||
|
||||
Content passes all detector checks and is allowed through.
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem label="Unsuccessful Call" value="not-allowed">
|
||||
|
||||
Expect this to fail if content violates detector policies:
|
||||
|
||||
```shell
|
||||
curl -i http://localhost:4000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-d '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"messages": [
|
||||
{"role": "user", "content": "My email is test@example.com and my SSN is 123-45-6789"}
|
||||
],
|
||||
"guardrails": ["enkryptai-guard"]
|
||||
}'
|
||||
```
|
||||
|
||||
**Expected Response on Failure: HTTP 400 Error**
|
||||
|
||||
```json
|
||||
{
|
||||
"error": {
|
||||
"message": {
|
||||
"error": "Content blocked by EnkryptAI guardrail",
|
||||
"detected": true,
|
||||
"violations": ["pii"],
|
||||
"response": {
|
||||
"summary": {
|
||||
"pii": 1
|
||||
},
|
||||
"details": {
|
||||
"pii": {
|
||||
"detected": ["email", "ssn"]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"type": "None",
|
||||
"param": "None",
|
||||
"code": "400"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Video Walkthrough
|
||||
|
||||
<iframe width="840" height="500" src="https://www.loom.com/embed/ff222211e0864937aee4aeef0f28c3b7" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### Using Custom Policies
|
||||
|
||||
You can specify a custom EnkryptAI policy:
|
||||
|
||||
```yaml
|
||||
guardrails:
|
||||
- guardrail_name: "enkryptai-custom"
|
||||
litellm_params:
|
||||
guardrail: enkryptai
|
||||
mode: "pre_call"
|
||||
api_key: os.environ/ENKRYPTAI_API_KEY
|
||||
policy_name: "my-custom-policy" # Sent via x-enkrypt-policy header
|
||||
detectors:
|
||||
toxicity:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
### Using Deployments
|
||||
|
||||
Specify an EnkryptAI deployment:
|
||||
|
||||
```yaml
|
||||
guardrails:
|
||||
- guardrail_name: "enkryptai-deployment"
|
||||
litellm_params:
|
||||
guardrail: enkryptai
|
||||
mode: "pre_call"
|
||||
api_key: os.environ/ENKRYPTAI_API_KEY
|
||||
deployment_name: "production" # Sent via X-Enkrypt-Deployment header
|
||||
detectors:
|
||||
toxicity:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
### Monitor Mode (Logging Without Blocking)
|
||||
|
||||
Set `block_on_violation: false` to log violations without blocking requests:
|
||||
|
||||
```yaml
|
||||
guardrails:
|
||||
- guardrail_name: "enkryptai-monitor"
|
||||
litellm_params:
|
||||
guardrail: enkryptai
|
||||
mode: "pre_call"
|
||||
api_key: os.environ/ENKRYPTAI_API_KEY
|
||||
block_on_violation: false # Log violations but don't block
|
||||
detectors:
|
||||
toxicity:
|
||||
enabled: true
|
||||
nsfw:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
In monitor mode, all violations are logged but requests are never blocked.
|
||||
|
||||
### Input and Output Guardrails
|
||||
|
||||
Configure separate guardrails for input and output:
|
||||
|
||||
```yaml
|
||||
guardrails:
|
||||
# Input guardrail
|
||||
- guardrail_name: "enkryptai-input"
|
||||
litellm_params:
|
||||
guardrail: enkryptai
|
||||
mode: "pre_call"
|
||||
api_key: os.environ/ENKRYPTAI_API_KEY
|
||||
detectors:
|
||||
pii:
|
||||
enabled: true
|
||||
entities: ["email", "phone", "ssn"]
|
||||
injection_attack:
|
||||
enabled: true
|
||||
|
||||
# Output guardrail
|
||||
- guardrail_name: "enkryptai-output"
|
||||
litellm_params:
|
||||
guardrail: enkryptai
|
||||
mode: "post_call"
|
||||
api_key: os.environ/ENKRYPTAI_API_KEY
|
||||
detectors:
|
||||
toxicity:
|
||||
enabled: true
|
||||
nsfw:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
| Parameter | Type | Description | Default |
|
||||
|-----------|------|-------------|---------|
|
||||
| `api_key` | string | EnkryptAI API key | `ENKRYPTAI_API_KEY` env var |
|
||||
| `api_base` | string | EnkryptAI API base URL | `https://api.enkryptai.com` |
|
||||
| `policy_name` | string | Custom policy name (sent via `x-enkrypt-policy` header) | None |
|
||||
| `deployment_name` | string | Deployment name (sent via `X-Enkrypt-Deployment` header) | None |
|
||||
| `detectors` | object | Detector configuration | `{}` |
|
||||
| `block_on_violation` | boolean | Block requests on violations | `true` |
|
||||
| `mode` | string | When to run: `pre_call`, `post_call`, or `during_call` | Required |
|
||||
|
||||
## Observability
|
||||
|
||||
EnkryptAI guardrail logs include:
|
||||
|
||||
- **guardrail_status**: `success`, `guardrail_intervened`, or `guardrail_failed_to_respond`
|
||||
- **guardrail_provider**: `enkryptai`
|
||||
- **guardrail_json_response**: Full API response with detection details
|
||||
- **duration**: Time taken for guardrail check
|
||||
- **start_time** and **end_time**: Timestamps
|
||||
|
||||
These logs are available through your configured LiteLLM logging callbacks.
|
||||
|
||||
## Error Handling
|
||||
|
||||
The guardrail handles errors gracefully:
|
||||
|
||||
- **API Failures**: Logs error and raises exception
|
||||
- **Rate Limits (429)**: Logs error and raises exception
|
||||
- **Invalid Configuration**: Raises `ValueError` on initialization
|
||||
|
||||
Set `block_on_violation: false` to continue processing even when violations are detected (monitor mode).
|
||||
|
||||
## Support
|
||||
|
||||
For more information about EnkryptAI:
|
||||
- Documentation: [https://docs.enkryptai.com](https://docs.enkryptai.com)
|
||||
- Website: [https://enkryptai.com](https://enkryptai.com)
|
||||
|
||||
@@ -36,6 +36,7 @@ const sidebars = {
|
||||
"proxy/guardrails/aporia_api",
|
||||
"proxy/guardrails/azure_content_guardrail",
|
||||
"proxy/guardrails/bedrock",
|
||||
"proxy/guardrails/enkryptai",
|
||||
"proxy/guardrails/lasso_security",
|
||||
"proxy/guardrails/guardrails_ai",
|
||||
"proxy/guardrails/lakera_ai",
|
||||
@@ -538,16 +539,10 @@ const sidebars = {
|
||||
type: "category",
|
||||
label: "Guides",
|
||||
items: [
|
||||
{
|
||||
type: "category",
|
||||
label: "Tools",
|
||||
items: [
|
||||
"completion/computer_use",
|
||||
"completion/web_search",
|
||||
"completion/web_fetch",
|
||||
"completion/function_call",
|
||||
]
|
||||
},
|
||||
"completion/computer_use",
|
||||
"completion/web_search",
|
||||
"completion/web_fetch",
|
||||
"completion/function_call",
|
||||
"completion/audio",
|
||||
"completion/document_understanding",
|
||||
"completion/drop_params",
|
||||
|
||||
@@ -87,6 +87,35 @@ MAX_TOKEN_TRIMMING_ATTEMPTS = int(
|
||||
########## Networking constants ##############################################################
|
||||
_DEFAULT_TTL_FOR_HTTPX_CLIENTS = 3600 # 1 hour, re-use the same httpx client for 1 hour
|
||||
|
||||
# Aiohttp connection pooling constants
|
||||
AIOHTTP_CONNECTOR_LIMIT = int(os.getenv("AIOHTTP_CONNECTOR_LIMIT", 0))
|
||||
AIOHTTP_KEEPALIVE_TIMEOUT = int(os.getenv("AIOHTTP_KEEPALIVE_TIMEOUT", 120))
|
||||
AIOHTTP_TTL_DNS_CACHE = int(os.getenv("AIOHTTP_TTL_DNS_CACHE", 300))
|
||||
|
||||
# SSL/TLS cipher configuration for faster handshakes
|
||||
# Strategy: Strongly prefer fast modern ciphers, but allow fallback to commonly supported ones
|
||||
# This balances performance with broad compatibility
|
||||
DEFAULT_SSL_CIPHERS = os.getenv(
|
||||
"LITELLM_SSL_CIPHERS",
|
||||
# Priority 1: TLS 1.3 ciphers (fastest, ~50ms handshake)
|
||||
"TLS_AES_256_GCM_SHA384:" # Fastest observed in testing
|
||||
"TLS_AES_128_GCM_SHA256:" # Slightly faster than 256-bit
|
||||
"TLS_CHACHA20_POLY1305_SHA256:" # Fast on ARM/mobile
|
||||
# Priority 2: TLS 1.2 ECDHE+GCM (fast, ~100ms handshake, widely supported)
|
||||
"ECDHE-RSA-AES256-GCM-SHA384:"
|
||||
"ECDHE-RSA-AES128-GCM-SHA256:"
|
||||
"ECDHE-ECDSA-AES256-GCM-SHA384:"
|
||||
"ECDHE-ECDSA-AES128-GCM-SHA256:"
|
||||
# Priority 3: Additional modern ciphers (good balance)
|
||||
"ECDHE-RSA-CHACHA20-POLY1305:"
|
||||
"ECDHE-ECDSA-CHACHA20-POLY1305:"
|
||||
# Priority 4: Widely compatible fallbacks (slower but universally supported)
|
||||
"ECDHE-RSA-AES256-SHA384:" # Common fallback
|
||||
"ECDHE-RSA-AES128-SHA256:" # Very widely supported
|
||||
"AES256-GCM-SHA384:" # Non-PFS fallback (compatibility)
|
||||
"AES128-GCM-SHA256", # Last resort (maximum compatibility)
|
||||
)
|
||||
|
||||
########### v2 Architecture constants for managing writing updates to the database ###########
|
||||
REDIS_UPDATE_BUFFER_KEY = "litellm_spend_update_buffer"
|
||||
REDIS_DAILY_SPEND_UPDATE_BUFFER_KEY = "litellm_daily_spend_update_buffer"
|
||||
|
||||
@@ -18,6 +18,7 @@ from litellm import get_secret_str
|
||||
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
|
||||
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
|
||||
from litellm.llms.azure.files.handler import AzureOpenAIFilesAPI
|
||||
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
|
||||
from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
|
||||
from litellm.llms.openai.openai import FileDeleted, FileObject, OpenAIFilesAPI
|
||||
from litellm.llms.vertex_ai.files.handler import VertexAIFilesHandler
|
||||
|
||||
@@ -81,12 +81,12 @@ from litellm.types.llms.openai import (
|
||||
)
|
||||
from litellm.types.mcp import MCPPostCallResponseObject
|
||||
from litellm.types.rerank import RerankResponse
|
||||
from litellm.types.router import CustomPricingLiteLLMParams
|
||||
from litellm.types.utils import (
|
||||
CachingDetails,
|
||||
CallTypes,
|
||||
CostBreakdown,
|
||||
CostResponseTypes,
|
||||
CustomPricingLiteLLMParams,
|
||||
DynamicPromptManagementParamLiteral,
|
||||
EmbeddingResponse,
|
||||
GuardrailStatus,
|
||||
|
||||
@@ -270,7 +270,6 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
|
||||
processed_chunk.get("delta", {}).get("stop_reason")
|
||||
is not None
|
||||
):
|
||||
|
||||
self.holding_stop_reason_chunk = processed_chunk
|
||||
else:
|
||||
self.chunk_queue.append(processed_chunk)
|
||||
@@ -380,4 +379,11 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
|
||||
self.current_content_block_start = content_block_start
|
||||
return True
|
||||
|
||||
# For parallel tool calls, we'll necessarily have a new content block
|
||||
# if we get a function name since it signals a new tool call
|
||||
if block_type == "tool_use" and content_block_start.get("name"):
|
||||
self.current_content_block_type = block_type
|
||||
self.current_content_block_start = content_block_start
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
@@ -12,7 +12,13 @@ from httpx._types import RequestFiles
|
||||
|
||||
import litellm
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.constants import _DEFAULT_TTL_FOR_HTTPX_CLIENTS
|
||||
from litellm.constants import (
|
||||
_DEFAULT_TTL_FOR_HTTPX_CLIENTS,
|
||||
AIOHTTP_CONNECTOR_LIMIT,
|
||||
AIOHTTP_KEEPALIVE_TIMEOUT,
|
||||
AIOHTTP_TTL_DNS_CACHE,
|
||||
DEFAULT_SSL_CIPHERS
|
||||
)
|
||||
from litellm.litellm_core_utils.logging_utils import track_llm_api_timing
|
||||
from litellm.types.llms.custom_http import *
|
||||
|
||||
@@ -94,10 +100,19 @@ def get_ssl_configuration(
|
||||
|
||||
if ssl_verify is not False:
|
||||
custom_ssl_context = ssl.create_default_context(cafile=cafile)
|
||||
# If security level is set, apply it to the SSL context
|
||||
|
||||
# Optimize SSL handshake performance
|
||||
# Set minimum TLS version to 1.2 for better performance
|
||||
custom_ssl_context.minimum_version = ssl.TLSVersion.TLSv1_2
|
||||
|
||||
# Configure cipher suites for optimal performance
|
||||
if ssl_security_level and isinstance(ssl_security_level, str):
|
||||
# Create a custom SSL context with reduced security level
|
||||
# User provided custom cipher configuration (e.g., via SSL_SECURITY_LEVEL env var)
|
||||
custom_ssl_context.set_ciphers(ssl_security_level)
|
||||
else:
|
||||
# Use optimized cipher list that strongly prefers fast ciphers
|
||||
# but falls back to widely compatible ones
|
||||
custom_ssl_context.set_ciphers(DEFAULT_SSL_CIPHERS)
|
||||
|
||||
# Use our custom SSL context instead of the original ssl_verify value
|
||||
return custom_ssl_context
|
||||
@@ -651,7 +666,13 @@ class AsyncHTTPHandler:
|
||||
)
|
||||
return LiteLLMAiohttpTransport(
|
||||
client=lambda: ClientSession(
|
||||
connector=TCPConnector(limit=0, **connector_kwargs), # 0 = unlimited connections per host
|
||||
connector=TCPConnector(
|
||||
limit=AIOHTTP_CONNECTOR_LIMIT,
|
||||
keepalive_timeout=AIOHTTP_KEEPALIVE_TIMEOUT,
|
||||
ttl_dns_cache=AIOHTTP_TTL_DNS_CACHE,
|
||||
enable_cleanup_closed=True,
|
||||
**connector_kwargs
|
||||
),
|
||||
trust_env=trust_env,
|
||||
),
|
||||
)
|
||||
|
||||
@@ -13,13 +13,13 @@ from typing import (
|
||||
cast,
|
||||
)
|
||||
|
||||
from litellm._logging import verbose_logger
|
||||
import httpx # type: ignore
|
||||
|
||||
import litellm
|
||||
import litellm.litellm_core_utils
|
||||
import litellm.types
|
||||
import litellm.types.utils
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.litellm_core_utils.realtime_streaming import RealTimeStreaming
|
||||
from litellm.llms.base_llm.anthropic_messages.transformation import (
|
||||
BaseAnthropicMessagesConfig,
|
||||
@@ -239,7 +239,7 @@ class BaseLLMHTTPHandler:
|
||||
json_mode: bool = False,
|
||||
signed_json_body: Optional[bytes] = None,
|
||||
shared_session: Optional["ClientSession"] = None,
|
||||
):
|
||||
):
|
||||
if client is None:
|
||||
verbose_logger.debug(
|
||||
f"Creating HTTP client with shared_session: {id(shared_session) if shared_session else None}"
|
||||
@@ -426,6 +426,7 @@ class BaseLLMHTTPHandler:
|
||||
),
|
||||
json_mode=json_mode,
|
||||
signed_json_body=signed_json_body,
|
||||
shared_session=shared_session,
|
||||
)
|
||||
|
||||
if stream is True:
|
||||
|
||||
@@ -169,7 +169,7 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
|
||||
raise OpenAIError(
|
||||
message=raw_response.text, status_code=raw_response.status_code
|
||||
)
|
||||
return ResponsesAPIResponse(**raw_response_json)
|
||||
return ResponsesAPIResponse.model_construct(**raw_response_json)
|
||||
|
||||
def validate_environment(
|
||||
self, headers: dict, model: str, litellm_params: Optional[GenericLiteLLMParams]
|
||||
|
||||
@@ -7,7 +7,7 @@ Docs: https://openrouter.ai/docs/parameters
|
||||
"""
|
||||
|
||||
from enum import Enum
|
||||
from typing import Any, AsyncIterator, Iterator, List, Optional, Tuple, Union
|
||||
from typing import Any, AsyncIterator, Iterator, List, Optional, Tuple, Union, cast
|
||||
|
||||
import httpx
|
||||
|
||||
@@ -88,29 +88,31 @@ class OpenrouterConfig(OpenAIGPTConfig):
|
||||
Move cache_control from message level to content blocks.
|
||||
OpenRouter requires cache_control to be inside content blocks, not at message level.
|
||||
|
||||
When cache_control is at message level, it's added to ALL content blocks
|
||||
to cache the entire message content.
|
||||
To avoid exceeding Anthropic's limit of 4 cache breakpoints, cache_control is only
|
||||
added to the LAST content block in each message.
|
||||
"""
|
||||
transformed_messages = []
|
||||
transformed_messages: List[AllMessageValues] = []
|
||||
for message in messages:
|
||||
message_copy = dict(message)
|
||||
cache_control = message_copy.pop("cache_control", None)
|
||||
message_dict = dict(message)
|
||||
cache_control = message_dict.pop("cache_control", None)
|
||||
|
||||
if cache_control is not None:
|
||||
content = message_copy.get("content")
|
||||
content = message_dict.get("content")
|
||||
|
||||
if isinstance(content, list):
|
||||
# Content is already a list, add cache_control to all blocks
|
||||
# Content is already a list, add cache_control only to the last block
|
||||
if len(content) > 0:
|
||||
content_copy = []
|
||||
for block in content:
|
||||
block_copy = dict(block)
|
||||
block_copy["cache_control"] = cache_control
|
||||
content_copy.append(block_copy)
|
||||
message_copy["content"] = content_copy
|
||||
for i, block in enumerate(content):
|
||||
block_dict = dict(block)
|
||||
# Only add cache_control to the last content block
|
||||
if i == len(content) - 1:
|
||||
block_dict["cache_control"] = cache_control
|
||||
content_copy.append(block_dict)
|
||||
message_dict["content"] = content_copy
|
||||
else:
|
||||
# Content is a string, convert to structured format
|
||||
message_copy["content"] = [
|
||||
message_dict["content"] = [
|
||||
{
|
||||
"type": "text",
|
||||
"text": content,
|
||||
@@ -118,7 +120,8 @@ class OpenrouterConfig(OpenAIGPTConfig):
|
||||
}
|
||||
]
|
||||
|
||||
transformed_messages.append(message_copy)
|
||||
# Cast back to AllMessageValues after modification
|
||||
transformed_messages.append(cast(AllMessageValues, message_dict))
|
||||
|
||||
return transformed_messages
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import re
|
||||
from enum import Enum
|
||||
from typing import Any, Dict, List, Literal, Optional, Set, Tuple, Union, get_type_hints
|
||||
|
||||
import httpx
|
||||
@@ -24,6 +25,68 @@ class VertexAIError(BaseLLMException):
|
||||
super().__init__(message=message, status_code=status_code, headers=headers)
|
||||
|
||||
|
||||
class VertexAIModelRoute(str, Enum):
|
||||
"""Enum for Vertex AI model routing"""
|
||||
PARTNER_MODELS = "partner_models"
|
||||
GEMINI = "gemini"
|
||||
GEMMA = "gemma"
|
||||
MODEL_GARDEN = "model_garden"
|
||||
NON_GEMINI = "non_gemini"
|
||||
|
||||
|
||||
def get_vertex_ai_model_route(model: str, litellm_params: Optional[dict] = None) -> VertexAIModelRoute:
|
||||
"""
|
||||
Determine which handler to use for a Vertex AI model based on the model name.
|
||||
|
||||
Args:
|
||||
model: The model name (e.g., "llama3-405b", "gemini-pro", "gemma/gemma-3-12b-it", "openai/gpt-oss-120b")
|
||||
litellm_params: Optional litellm parameters dict that may contain base_model for routing
|
||||
|
||||
Returns:
|
||||
VertexAIModelRoute: The route enum indicating which handler should be used
|
||||
|
||||
Examples:
|
||||
>>> get_vertex_ai_model_route("llama3-405b")
|
||||
VertexAIModelRoute.PARTNER_MODELS
|
||||
|
||||
>>> get_vertex_ai_model_route("gemini-pro")
|
||||
VertexAIModelRoute.GEMINI
|
||||
|
||||
>>> get_vertex_ai_model_route("gemma/gemma-3-12b-it")
|
||||
VertexAIModelRoute.GEMMA
|
||||
|
||||
>>> get_vertex_ai_model_route("openai/gpt-oss-120b")
|
||||
VertexAIModelRoute.MODEL_GARDEN
|
||||
"""
|
||||
from litellm.llms.vertex_ai.vertex_ai_partner_models.main import (
|
||||
VertexAIPartnerModels,
|
||||
)
|
||||
|
||||
# Check base_model in litellm_params for gemini override
|
||||
if litellm_params and litellm_params.get("base_model") is not None:
|
||||
if "gemini" in litellm_params["base_model"]:
|
||||
return VertexAIModelRoute.GEMINI
|
||||
|
||||
# Check for partner models (llama, mistral, claude, etc.)
|
||||
if VertexAIPartnerModels.is_vertex_partner_model(model=model):
|
||||
return VertexAIModelRoute.PARTNER_MODELS
|
||||
|
||||
# Check for gemma models
|
||||
if "gemma/" in model:
|
||||
return VertexAIModelRoute.GEMMA
|
||||
|
||||
# Check for model garden openai models
|
||||
if "openai" in model:
|
||||
return VertexAIModelRoute.MODEL_GARDEN
|
||||
|
||||
# Check for gemini models
|
||||
if "gemini" in model:
|
||||
return VertexAIModelRoute.GEMINI
|
||||
|
||||
# Default to non-gemini (legacy vertex models like chat-bison, text-bison, etc.)
|
||||
return VertexAIModelRoute.NON_GEMINI
|
||||
|
||||
|
||||
def get_supports_system_message(
|
||||
model: str, custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"]
|
||||
) -> bool:
|
||||
|
||||
@@ -44,6 +44,7 @@ def cost_router(
|
||||
or "mistral" in model
|
||||
or "jamba" in model
|
||||
or "codestral" in model
|
||||
or "gemma" in model
|
||||
):
|
||||
return "cost_per_token"
|
||||
elif custom_llm_provider == "vertex_ai" and (
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
"""Vertex AI Gemma-AI Models Handler"""
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
"""
|
||||
API Handler for calling Vertex AI Gemma Models
|
||||
|
||||
These models use a custom prediction endpoint format that wraps messages in 'instances'
|
||||
with @requestFormat: "chatCompletions" and returns responses wrapped in 'predictions'.
|
||||
|
||||
Usage:
|
||||
|
||||
response = litellm.completion(
|
||||
model="vertex_ai/gemma/gemma-3-12b-it-1222199011122",
|
||||
messages=[{"role": "user", "content": "What is machine learning?"}],
|
||||
vertex_project="your-project-id",
|
||||
vertex_location="us-central1",
|
||||
)
|
||||
|
||||
Sent to this route when `model` is in the format `vertex_ai/gemma/{MODEL_NAME}`
|
||||
|
||||
The API expects a custom endpoint URL format:
|
||||
https://{ENDPOINT_NUMBER}.{location}-{REGION_NUMBER}.prediction.vertexai.goog/v1/projects/{PROJECT_ID}/locations/{location}/endpoints/{ENDPOINT_ID}:predict
|
||||
"""
|
||||
|
||||
from typing import Callable, Optional, Union
|
||||
|
||||
import httpx # type: ignore
|
||||
|
||||
from litellm.utils import ModelResponse
|
||||
|
||||
from ..common_utils import VertexAIError
|
||||
from ..vertex_llm_base import VertexBase
|
||||
|
||||
|
||||
class VertexAIGemmaModels(VertexBase):
|
||||
def __init__(self) -> None:
|
||||
pass
|
||||
|
||||
def completion(
|
||||
self,
|
||||
model: str,
|
||||
messages: list,
|
||||
model_response: ModelResponse,
|
||||
print_verbose: Callable,
|
||||
encoding,
|
||||
logging_obj,
|
||||
api_base: Optional[str],
|
||||
optional_params: dict,
|
||||
custom_prompt_dict: dict,
|
||||
headers: Optional[dict],
|
||||
timeout: Union[float, httpx.Timeout],
|
||||
litellm_params: dict,
|
||||
vertex_project=None,
|
||||
vertex_location=None,
|
||||
vertex_credentials=None,
|
||||
logger_fn=None,
|
||||
acompletion: bool = False,
|
||||
client=None,
|
||||
):
|
||||
"""
|
||||
Handles calling Vertex AI Gemma Models
|
||||
|
||||
Sent to this route when `model` is in the format `vertex_ai/gemma/{MODEL_NAME}`
|
||||
"""
|
||||
try:
|
||||
import vertexai
|
||||
|
||||
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
|
||||
VertexLLM,
|
||||
)
|
||||
from litellm.llms.vertex_ai.vertex_gemma_models.transformation import (
|
||||
VertexGemmaConfig,
|
||||
)
|
||||
except Exception as e:
|
||||
raise VertexAIError(
|
||||
status_code=400,
|
||||
message=f"""vertexai import failed please run `pip install -U "google-cloud-aiplatform>=1.38"`. Got error: {e}""",
|
||||
)
|
||||
|
||||
if not (
|
||||
hasattr(vertexai, "preview") or hasattr(vertexai.preview, "language_models")
|
||||
):
|
||||
raise VertexAIError(
|
||||
status_code=400,
|
||||
message="""Upgrade vertex ai. Run `pip install "google-cloud-aiplatform>=1.38"`""",
|
||||
)
|
||||
try:
|
||||
model = model.replace("gemma/", "")
|
||||
vertex_httpx_logic = VertexLLM()
|
||||
|
||||
access_token, project_id = vertex_httpx_logic._ensure_access_token(
|
||||
credentials=vertex_credentials,
|
||||
project_id=vertex_project,
|
||||
custom_llm_provider="vertex_ai",
|
||||
)
|
||||
|
||||
gemma_transformation = VertexGemmaConfig()
|
||||
|
||||
## CONSTRUCT API BASE
|
||||
stream: bool = optional_params.get("stream", False) or False
|
||||
optional_params["stream"] = stream
|
||||
|
||||
# If api_base is not provided, it should be set as an environment variable
|
||||
# or passed explicitly because the endpoint URL is unique per deployment
|
||||
if api_base is None:
|
||||
raise VertexAIError(
|
||||
status_code=400,
|
||||
message="api_base is required for Vertex AI Gemma models. Please provide the full endpoint URL.",
|
||||
)
|
||||
|
||||
# Check if we need to append :predict
|
||||
if not api_base.endswith(":predict"):
|
||||
_, api_base = self._check_custom_proxy(
|
||||
api_base=api_base,
|
||||
custom_llm_provider="vertex_ai",
|
||||
gemini_api_key=None,
|
||||
endpoint="predict",
|
||||
stream=stream,
|
||||
auth_header=None,
|
||||
url=api_base,
|
||||
)
|
||||
# If api_base already ends with :predict, use it as-is
|
||||
|
||||
# Use the custom transformation handler for gemma models
|
||||
return gemma_transformation.completion(
|
||||
model=model,
|
||||
messages=messages,
|
||||
api_base=api_base,
|
||||
api_key=access_token,
|
||||
custom_prompt_dict=custom_prompt_dict,
|
||||
model_response=model_response,
|
||||
print_verbose=print_verbose,
|
||||
logging_obj=logging_obj,
|
||||
optional_params=optional_params,
|
||||
acompletion=acompletion,
|
||||
litellm_params=litellm_params,
|
||||
logger_fn=logger_fn,
|
||||
client=client,
|
||||
timeout=timeout,
|
||||
encoding=encoding,
|
||||
custom_llm_provider="vertex_ai",
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
if hasattr(e, "status_code"):
|
||||
raise e
|
||||
raise VertexAIError(status_code=500, message=str(e))
|
||||
|
||||
@@ -0,0 +1,350 @@
|
||||
"""
|
||||
Transformation logic for Vertex AI Gemma Models
|
||||
|
||||
Handles the custom request/response format:
|
||||
- Request: Wraps messages in 'instances' with @requestFormat: "chatCompletions"
|
||||
- Response: Extracts data from 'predictions' wrapper
|
||||
|
||||
The actual message transformation reuses OpenAIGPTConfig since Gemma uses OpenAI-compatible format.
|
||||
"""
|
||||
|
||||
from typing import Any, Callable, Dict, List, Optional, Union, cast
|
||||
|
||||
import httpx
|
||||
|
||||
from litellm.llms.base_llm.chat.transformation import BaseLLMException
|
||||
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
|
||||
from litellm.types.llms.openai import AllMessageValues
|
||||
from litellm.types.utils import ModelResponse
|
||||
|
||||
|
||||
class VertexGemmaConfig(OpenAIGPTConfig):
|
||||
"""
|
||||
Configuration and transformation class for Vertex AI Gemma models
|
||||
|
||||
Extends OpenAIGPTConfig to wrap/unwrap the instances/predictions format
|
||||
used by Vertex AI's Gemma deployment endpoint.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
|
||||
def transform_request(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
optional_params: dict,
|
||||
litellm_params: dict,
|
||||
headers: dict,
|
||||
) -> dict:
|
||||
"""
|
||||
Transform request to Vertex Gemma format.
|
||||
|
||||
Uses parent class to create OpenAI-compatible request, then wraps it
|
||||
in the Vertex Gemma instances format.
|
||||
"""
|
||||
# Get the base OpenAI request from parent class
|
||||
openai_request = super().transform_request(
|
||||
model=model,
|
||||
messages=messages,
|
||||
optional_params=optional_params,
|
||||
litellm_params=litellm_params,
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
# Remove 'model' from the request as it's not needed in the instance
|
||||
openai_request.pop("model", None)
|
||||
|
||||
# Wrap in Vertex Gemma format
|
||||
return {
|
||||
"instances": [
|
||||
{
|
||||
"@requestFormat": "chatCompletions",
|
||||
**openai_request,
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
async def async_transform_request(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[AllMessageValues],
|
||||
optional_params: dict,
|
||||
litellm_params: dict,
|
||||
headers: dict,
|
||||
) -> dict:
|
||||
"""
|
||||
Async version of transform_request.
|
||||
"""
|
||||
# Get the base OpenAI request from parent class
|
||||
openai_request = await super().async_transform_request(
|
||||
model=model,
|
||||
messages=messages,
|
||||
optional_params=optional_params,
|
||||
litellm_params=litellm_params,
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
# Remove 'model' from the request as it's not needed in the instance
|
||||
openai_request.pop("model", None)
|
||||
|
||||
# Wrap in Vertex Gemma format
|
||||
return {
|
||||
"instances": [
|
||||
{
|
||||
"@requestFormat": "chatCompletions",
|
||||
**openai_request,
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
def _unwrap_predictions_response(
|
||||
self,
|
||||
response_json: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Unwrap the Vertex Gemma predictions format to OpenAI format.
|
||||
|
||||
Vertex Gemma wraps the OpenAI-compatible response in a 'predictions' field.
|
||||
This method extracts it so the parent class can process it normally.
|
||||
"""
|
||||
if "predictions" not in response_json:
|
||||
raise BaseLLMException(
|
||||
status_code=422,
|
||||
message="Invalid response format: missing 'predictions' field",
|
||||
)
|
||||
|
||||
return response_json["predictions"]
|
||||
|
||||
def completion(
|
||||
self,
|
||||
model: str,
|
||||
messages: list,
|
||||
api_base: str,
|
||||
api_key: str,
|
||||
custom_prompt_dict: dict,
|
||||
model_response: ModelResponse,
|
||||
print_verbose: Callable,
|
||||
logging_obj: Any,
|
||||
optional_params: dict,
|
||||
acompletion: bool,
|
||||
litellm_params: dict,
|
||||
logger_fn: Optional[Callable] = None,
|
||||
client: Optional[httpx.Client] = None,
|
||||
timeout: Optional[Union[float, httpx.Timeout]] = None,
|
||||
encoding=None,
|
||||
custom_llm_provider: str = "vertex_ai",
|
||||
):
|
||||
"""
|
||||
Make completion request to Vertex Gemma endpoint.
|
||||
Supports both sync and async requests.
|
||||
"""
|
||||
# Handle streaming
|
||||
stream = optional_params.get("stream", False)
|
||||
if stream:
|
||||
raise BaseLLMException(
|
||||
status_code=400,
|
||||
message="Streaming is not yet supported for Vertex AI Gemma models",
|
||||
)
|
||||
|
||||
if acompletion:
|
||||
return self._async_completion(
|
||||
model=model,
|
||||
messages=messages,
|
||||
api_base=api_base,
|
||||
api_key=api_key,
|
||||
model_response=model_response,
|
||||
print_verbose=print_verbose,
|
||||
logging_obj=logging_obj,
|
||||
optional_params=optional_params,
|
||||
litellm_params=litellm_params,
|
||||
timeout=timeout,
|
||||
encoding=encoding,
|
||||
)
|
||||
else:
|
||||
return self._sync_completion(
|
||||
model=model,
|
||||
messages=messages,
|
||||
api_base=api_base,
|
||||
api_key=api_key,
|
||||
model_response=model_response,
|
||||
print_verbose=print_verbose,
|
||||
logging_obj=logging_obj,
|
||||
optional_params=optional_params,
|
||||
litellm_params=litellm_params,
|
||||
timeout=timeout,
|
||||
encoding=encoding,
|
||||
)
|
||||
|
||||
def _sync_completion(
|
||||
self,
|
||||
model: str,
|
||||
messages: list,
|
||||
api_base: str,
|
||||
api_key: str,
|
||||
model_response: ModelResponse,
|
||||
print_verbose: Callable,
|
||||
logging_obj: Any,
|
||||
optional_params: dict,
|
||||
litellm_params: dict,
|
||||
timeout: Optional[Union[float, httpx.Timeout]],
|
||||
encoding: Any,
|
||||
):
|
||||
"""Synchronous completion request"""
|
||||
from litellm.llms.custom_httpx.http_handler import HTTPHandler
|
||||
from litellm.utils import convert_to_model_response_object
|
||||
|
||||
# Transform the request using parent class methods
|
||||
request_data = self.transform_request(
|
||||
model=model,
|
||||
messages=messages,
|
||||
optional_params=optional_params.copy(),
|
||||
litellm_params=litellm_params,
|
||||
headers={},
|
||||
)
|
||||
|
||||
# Set up headers
|
||||
headers = {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
# Log the request
|
||||
logging_obj.pre_call(
|
||||
input=messages,
|
||||
api_key=api_key,
|
||||
additional_args={
|
||||
"complete_input_dict": request_data,
|
||||
"api_base": api_base,
|
||||
},
|
||||
)
|
||||
|
||||
# Make the HTTP request
|
||||
http_handler = HTTPHandler(concurrent_limit=1)
|
||||
response = http_handler.post(
|
||||
url=api_base,
|
||||
headers=headers,
|
||||
json=request_data,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
raise BaseLLMException(
|
||||
status_code=response.status_code,
|
||||
message=f"Request failed: {response.text}",
|
||||
)
|
||||
|
||||
response_json = response.json()
|
||||
|
||||
# Unwrap predictions to get OpenAI-compatible response
|
||||
openai_response = self._unwrap_predictions_response(response_json)
|
||||
|
||||
# Use litellm's standard response converter
|
||||
model_response = cast(
|
||||
ModelResponse,
|
||||
convert_to_model_response_object(
|
||||
response_object=openai_response,
|
||||
model_response_object=model_response,
|
||||
_response_headers={},
|
||||
),
|
||||
)
|
||||
|
||||
# Ensure model is set correctly
|
||||
model_response.model = model
|
||||
|
||||
# Log the response
|
||||
logging_obj.post_call(
|
||||
input=messages,
|
||||
api_key=api_key,
|
||||
original_response=response_json,
|
||||
additional_args={"complete_input_dict": request_data},
|
||||
)
|
||||
|
||||
return model_response
|
||||
|
||||
async def _async_completion(
|
||||
self,
|
||||
model: str,
|
||||
messages: list,
|
||||
api_base: str,
|
||||
api_key: str,
|
||||
model_response: ModelResponse,
|
||||
print_verbose: Callable,
|
||||
logging_obj: Any,
|
||||
optional_params: dict,
|
||||
litellm_params: dict,
|
||||
timeout: Optional[Union[float, httpx.Timeout]],
|
||||
encoding: Any,
|
||||
):
|
||||
"""Asynchronous completion request"""
|
||||
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
|
||||
from litellm.utils import convert_to_model_response_object
|
||||
|
||||
# Transform the request using parent class async methods
|
||||
request_data = await self.async_transform_request(
|
||||
model=model,
|
||||
messages=messages,
|
||||
optional_params=optional_params.copy(),
|
||||
litellm_params=litellm_params,
|
||||
headers={},
|
||||
)
|
||||
|
||||
# Set up headers
|
||||
headers = {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
# Log the request
|
||||
logging_obj.pre_call(
|
||||
input=messages,
|
||||
api_key=api_key,
|
||||
additional_args={
|
||||
"complete_input_dict": request_data,
|
||||
"api_base": api_base,
|
||||
},
|
||||
)
|
||||
|
||||
# Make the HTTP request
|
||||
http_handler = AsyncHTTPHandler(concurrent_limit=1)
|
||||
response = await http_handler.post(
|
||||
url=api_base,
|
||||
headers=headers,
|
||||
json=request_data,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
raise BaseLLMException(
|
||||
status_code=response.status_code,
|
||||
message=f"Request failed: {response.text}",
|
||||
)
|
||||
|
||||
response_json = response.json()
|
||||
|
||||
# Unwrap predictions to get OpenAI-compatible response
|
||||
openai_response = self._unwrap_predictions_response(response_json)
|
||||
|
||||
# Use litellm's standard response converter
|
||||
model_response = cast(
|
||||
ModelResponse,
|
||||
convert_to_model_response_object(
|
||||
response_object=openai_response,
|
||||
model_response_object=model_response,
|
||||
_response_headers={},
|
||||
),
|
||||
)
|
||||
|
||||
# Ensure model is set correctly
|
||||
model_response.model = model
|
||||
|
||||
# Log the response
|
||||
logging_obj.post_call(
|
||||
input=messages,
|
||||
api_key=api_key,
|
||||
original_response=response_json,
|
||||
additional_args={"complete_input_dict": request_data},
|
||||
)
|
||||
|
||||
return model_response
|
||||
|
||||
+36
-9
@@ -85,6 +85,10 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
|
||||
from litellm.llms.base_llm import BaseConfig, BaseImageGenerationConfig
|
||||
from litellm.llms.bedrock.common_utils import BedrockModelInfo
|
||||
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
|
||||
from litellm.llms.vertex_ai.common_utils import (
|
||||
VertexAIModelRoute,
|
||||
get_vertex_ai_model_route,
|
||||
)
|
||||
from litellm.realtime_api.main import _realtime_health_check
|
||||
from litellm.secret_managers.main import get_secret_bool, get_secret_str
|
||||
from litellm.types.router import GenericLiteLLMParams
|
||||
@@ -150,7 +154,6 @@ from .llms.bedrock.chat import BedrockConverseLLM, BedrockLLM
|
||||
from .llms.bedrock.embed.embedding import BedrockEmbedding
|
||||
from .llms.bedrock.image.image_handler import BedrockImageGeneration
|
||||
from .llms.bytez.chat.transformation import BytezChatConfig
|
||||
from .llms.lemonade.chat.transformation import LemonadeChatConfig
|
||||
from .llms.codestral.completion.handler import CodestralTextCompletion
|
||||
from .llms.cohere.embed import handler as cohere_embed
|
||||
from .llms.custom_httpx.aiohttp_handler import BaseLLMAIOHTTPHandler
|
||||
@@ -162,6 +165,7 @@ from .llms.gemini.common_utils import get_api_key_from_env
|
||||
from .llms.groq.chat.handler import GroqChatCompletion
|
||||
from .llms.heroku.chat.transformation import HerokuChatConfig
|
||||
from .llms.huggingface.embedding.handler import HuggingFaceEmbedding
|
||||
from .llms.lemonade.chat.transformation import LemonadeChatConfig
|
||||
from .llms.nlp_cloud.chat.handler import completion as nlp_cloud_chat_completion
|
||||
from .llms.oci.chat.transformation import OCIChatConfig
|
||||
from .llms.ollama.completion import handler as ollama
|
||||
@@ -192,6 +196,7 @@ from .llms.vertex_ai.multimodal_embeddings.embedding_handler import (
|
||||
from .llms.vertex_ai.text_to_speech.text_to_speech_handler import VertexTextToSpeechAPI
|
||||
from .llms.vertex_ai.vertex_ai_partner_models.main import VertexAIPartnerModels
|
||||
from .llms.vertex_ai.vertex_embeddings.embedding_handler import VertexEmbedding
|
||||
from .llms.vertex_ai.vertex_gemma_models.main import VertexAIGemmaModels
|
||||
from .llms.vertex_ai.vertex_model_garden.main import VertexAIModelGardenModels
|
||||
from .llms.vllm.completion import handler as vllm_handler
|
||||
from .llms.watsonx.chat.handler import WatsonXChatHandler
|
||||
@@ -255,6 +260,7 @@ vertex_multimodal_embedding = VertexMultimodalEmbedding()
|
||||
vertex_image_generation = VertexImageGeneration()
|
||||
google_batch_embeddings = GoogleBatchEmbeddings()
|
||||
vertex_partner_models_chat_completion = VertexAIPartnerModels()
|
||||
vertex_gemma_chat_completion = VertexAIGemmaModels()
|
||||
vertex_model_garden_chat_completion = VertexAIModelGardenModels()
|
||||
vertex_text_to_speech = VertexTextToSpeechAPI()
|
||||
sagemaker_llm = SagemakerLLM()
|
||||
@@ -2875,7 +2881,7 @@ def completion( # type: ignore # noqa: PLR0915
|
||||
extra_headers=headers,
|
||||
)
|
||||
|
||||
elif custom_llm_provider == "vertex_ai":
|
||||
elif custom_llm_provider == "vertex_ai":
|
||||
vertex_ai_project = (
|
||||
optional_params.pop("vertex_project", None)
|
||||
or optional_params.pop("vertex_ai_project", None)
|
||||
@@ -2897,7 +2903,9 @@ def completion( # type: ignore # noqa: PLR0915
|
||||
api_base = api_base or litellm.api_base or get_secret("VERTEXAI_API_BASE")
|
||||
|
||||
new_params = safe_deep_copy(optional_params or {})
|
||||
if vertex_partner_models_chat_completion.is_vertex_partner_model(model):
|
||||
model_route = get_vertex_ai_model_route(model=model, litellm_params=litellm_params)
|
||||
|
||||
if model_route == VertexAIModelRoute.PARTNER_MODELS:
|
||||
model_response = vertex_partner_models_chat_completion.completion(
|
||||
model=model,
|
||||
messages=messages,
|
||||
@@ -2918,10 +2926,7 @@ def completion( # type: ignore # noqa: PLR0915
|
||||
timeout=timeout,
|
||||
client=client,
|
||||
)
|
||||
elif "gemini" in model or (
|
||||
litellm_params.get("base_model") is not None
|
||||
and "gemini" in litellm_params["base_model"]
|
||||
):
|
||||
elif model_route == VertexAIModelRoute.GEMINI:
|
||||
model_response = vertex_chat_completion.completion( # type: ignore
|
||||
model=model,
|
||||
messages=messages,
|
||||
@@ -2943,7 +2948,29 @@ def completion( # type: ignore # noqa: PLR0915
|
||||
api_base=api_base,
|
||||
extra_headers=headers,
|
||||
)
|
||||
elif "openai" in model:
|
||||
elif model_route == VertexAIModelRoute.GEMMA:
|
||||
# Vertex Gemma Models with custom prediction endpoint
|
||||
model_response = vertex_gemma_chat_completion.completion(
|
||||
model=model,
|
||||
messages=messages,
|
||||
model_response=model_response,
|
||||
print_verbose=print_verbose,
|
||||
optional_params=new_params,
|
||||
litellm_params=litellm_params, # type: ignore
|
||||
logger_fn=logger_fn,
|
||||
encoding=encoding,
|
||||
api_base=api_base,
|
||||
vertex_location=vertex_ai_location,
|
||||
vertex_project=vertex_ai_project,
|
||||
vertex_credentials=vertex_credentials,
|
||||
logging_obj=logging,
|
||||
acompletion=acompletion,
|
||||
headers=headers,
|
||||
custom_prompt_dict=custom_prompt_dict,
|
||||
timeout=timeout,
|
||||
client=client,
|
||||
)
|
||||
elif model_route == VertexAIModelRoute.MODEL_GARDEN:
|
||||
# Vertex Model Garden - OpenAI compatible models
|
||||
model_response = vertex_model_garden_chat_completion.completion(
|
||||
model=model,
|
||||
@@ -2965,7 +2992,7 @@ def completion( # type: ignore # noqa: PLR0915
|
||||
timeout=timeout,
|
||||
client=client,
|
||||
)
|
||||
else:
|
||||
else: # VertexAIModelRoute.NON_GEMINI
|
||||
model_response = vertex_ai_non_gemini.completion(
|
||||
model=model,
|
||||
messages=messages,
|
||||
|
||||
@@ -3164,6 +3164,42 @@
|
||||
"supports_function_calling": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"azure_ai/Phi-4-mini-reasoning": {
|
||||
"input_cost_per_token": 8e-08,
|
||||
"litellm_provider": "azure_ai",
|
||||
"max_input_tokens": 131072,
|
||||
"max_output_tokens": 4096,
|
||||
"max_tokens": 4096,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 3.2e-07,
|
||||
"source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/microsoft/",
|
||||
"supports_function_calling": true
|
||||
},
|
||||
"azure_ai/Phi-4-reasoning": {
|
||||
"input_cost_per_token": 1.25e-07,
|
||||
"litellm_provider": "azure_ai",
|
||||
"max_input_tokens": 32768,
|
||||
"max_output_tokens": 4096,
|
||||
"max_tokens": 4096,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 5e-07,
|
||||
"source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/microsoft/",
|
||||
"supports_function_calling": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_reasoning": true
|
||||
},
|
||||
"azure_ai/MAI-DS-R1": {
|
||||
"input_cost_per_token": 1.35e-06,
|
||||
"litellm_provider": "azure_ai",
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 8192,
|
||||
"max_tokens": 8192,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 5.4e-06,
|
||||
"source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/microsoft/",
|
||||
"supports_reasoning": true,
|
||||
"supports_tool_choice": true
|
||||
},
|
||||
"azure_ai/cohere-rerank-v3-english": {
|
||||
"input_cost_per_query": 0.002,
|
||||
"input_cost_per_token": 0.0,
|
||||
|
||||
@@ -109,10 +109,21 @@ class MCPRequestHandler:
|
||||
request.body = mock_body # type: ignore
|
||||
if ".well-known" in str(request.url): # public routes
|
||||
validated_user_api_key_auth = UserAPIKeyAuth()
|
||||
# elif litellm_api_key == "":
|
||||
# from fastapi import HTTPException
|
||||
|
||||
# raise HTTPException(
|
||||
# status_code=401,
|
||||
# detail="LiteLLM API key is missing. Please add it or use OAuth authentication.",
|
||||
# headers={
|
||||
# "WWW-Authenticate": f'Bearer resource_metadata=f"{request.base_url}/.well-known/oauth-protected-resource"',
|
||||
# },
|
||||
# )
|
||||
else:
|
||||
validated_user_api_key_auth = await user_api_key_auth(
|
||||
api_key=litellm_api_key, request=request
|
||||
)
|
||||
|
||||
return (
|
||||
validated_user_api_key_auth,
|
||||
mcp_auth_header,
|
||||
@@ -344,14 +355,14 @@ class MCPRequestHandler:
|
||||
proxy_logging_obj,
|
||||
user_api_key_cache,
|
||||
)
|
||||
|
||||
|
||||
if not user_api_key_auth:
|
||||
return None
|
||||
|
||||
|
||||
# Already loaded
|
||||
if user_api_key_auth.object_permission:
|
||||
return user_api_key_auth.object_permission
|
||||
|
||||
|
||||
# Need to fetch from DB
|
||||
if user_api_key_auth.object_permission_id and prisma_client:
|
||||
return await get_object_permission(
|
||||
@@ -361,7 +372,7 @@ class MCPRequestHandler:
|
||||
parent_otel_span=user_api_key_auth.parent_otel_span,
|
||||
proxy_logging_obj=proxy_logging_obj,
|
||||
)
|
||||
|
||||
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
@@ -369,16 +380,19 @@ class MCPRequestHandler:
|
||||
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
|
||||
):
|
||||
"""Helper to get team object_permission from cache or DB."""
|
||||
from litellm.proxy.auth.auth_checks import get_object_permission, get_team_object
|
||||
from litellm.proxy.auth.auth_checks import (
|
||||
get_object_permission,
|
||||
get_team_object,
|
||||
)
|
||||
from litellm.proxy.proxy_server import (
|
||||
prisma_client,
|
||||
proxy_logging_obj,
|
||||
user_api_key_cache,
|
||||
)
|
||||
|
||||
|
||||
if not user_api_key_auth or not user_api_key_auth.team_id or not prisma_client:
|
||||
return None
|
||||
|
||||
|
||||
# First get the team object (which may have object_permission already loaded)
|
||||
team_obj: Optional[LiteLLM_TeamTable] = await get_team_object(
|
||||
team_id=user_api_key_auth.team_id,
|
||||
@@ -387,14 +401,14 @@ class MCPRequestHandler:
|
||||
parent_otel_span=user_api_key_auth.parent_otel_span,
|
||||
proxy_logging_obj=proxy_logging_obj,
|
||||
)
|
||||
|
||||
|
||||
if not team_obj:
|
||||
return None
|
||||
|
||||
|
||||
# Already loaded
|
||||
if team_obj.object_permission:
|
||||
return team_obj.object_permission
|
||||
|
||||
|
||||
# Need to fetch from DB using object_permission_id
|
||||
if team_obj.object_permission_id:
|
||||
return await get_object_permission(
|
||||
@@ -404,7 +418,7 @@ class MCPRequestHandler:
|
||||
parent_otel_span=user_api_key_auth.parent_otel_span,
|
||||
proxy_logging_obj=proxy_logging_obj,
|
||||
)
|
||||
|
||||
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
@@ -415,26 +429,38 @@ class MCPRequestHandler:
|
||||
"""
|
||||
Get list of allowed tool names for a specific server based on key/team permissions.
|
||||
Follows same inheritance logic as get_allowed_mcp_servers.
|
||||
|
||||
|
||||
Args:
|
||||
server_id: Server ID to check permissions for
|
||||
user_api_key_auth: User auth
|
||||
|
||||
|
||||
Returns:
|
||||
List[str] if restrictions exist, None if no restrictions (allow all)
|
||||
"""
|
||||
if not user_api_key_auth:
|
||||
return None
|
||||
|
||||
|
||||
try:
|
||||
# Get key and team object permissions
|
||||
key_obj_perm = await MCPRequestHandler._get_key_object_permission(user_api_key_auth)
|
||||
team_obj_perm = await MCPRequestHandler._get_team_object_permission(user_api_key_auth)
|
||||
|
||||
key_obj_perm = await MCPRequestHandler._get_key_object_permission(
|
||||
user_api_key_auth
|
||||
)
|
||||
team_obj_perm = await MCPRequestHandler._get_team_object_permission(
|
||||
user_api_key_auth
|
||||
)
|
||||
|
||||
# Extract tool permissions for this server
|
||||
key_tools = key_obj_perm.mcp_tool_permissions.get(server_id) if key_obj_perm and key_obj_perm.mcp_tool_permissions else None
|
||||
team_tools = team_obj_perm.mcp_tool_permissions.get(server_id) if team_obj_perm and team_obj_perm.mcp_tool_permissions else None
|
||||
|
||||
key_tools = (
|
||||
key_obj_perm.mcp_tool_permissions.get(server_id)
|
||||
if key_obj_perm and key_obj_perm.mcp_tool_permissions
|
||||
else None
|
||||
)
|
||||
team_tools = (
|
||||
team_obj_perm.mcp_tool_permissions.get(server_id)
|
||||
if team_obj_perm and team_obj_perm.mcp_tool_permissions
|
||||
else None
|
||||
)
|
||||
|
||||
# Apply same inheritance logic as get_allowed_mcp_servers
|
||||
if team_tools:
|
||||
if key_tools:
|
||||
@@ -446,7 +472,7 @@ class MCPRequestHandler:
|
||||
else:
|
||||
# No team restrictions → use key restrictions
|
||||
return key_tools
|
||||
|
||||
|
||||
except Exception as e:
|
||||
verbose_logger.warning(f"Failed to get allowed tools for server: {str(e)}")
|
||||
return None
|
||||
@@ -459,12 +485,12 @@ class MCPRequestHandler:
|
||||
) -> bool:
|
||||
"""
|
||||
Check if a specific tool is allowed for a server based on key/team permissions.
|
||||
|
||||
|
||||
Args:
|
||||
tool_name: Name of the tool to check
|
||||
server_id: Server ID
|
||||
user_api_key_auth: User auth
|
||||
|
||||
|
||||
Returns:
|
||||
True if allowed, False if blocked
|
||||
"""
|
||||
@@ -472,15 +498,15 @@ class MCPRequestHandler:
|
||||
server_id=server_id,
|
||||
user_api_key_auth=user_api_key_auth,
|
||||
)
|
||||
|
||||
|
||||
# None means no restrictions (allow all)
|
||||
if allowed_tools is None:
|
||||
return True
|
||||
|
||||
|
||||
# Empty list means no tools allowed
|
||||
if not allowed_tools:
|
||||
return False
|
||||
|
||||
|
||||
# Check if tool is in allowed list
|
||||
return tool_name in allowed_tools
|
||||
|
||||
@@ -555,7 +581,7 @@ class MCPRequestHandler:
|
||||
) -> List[str]:
|
||||
"""
|
||||
Get allowed MCP servers for a team.
|
||||
|
||||
|
||||
Uses the helper _get_team_object_permission which:
|
||||
1. First checks if object_permission is already loaded on the team
|
||||
2. If not, fetches from DB using object_permission_id if it exists
|
||||
@@ -571,7 +597,7 @@ class MCPRequestHandler:
|
||||
object_permissions = await MCPRequestHandler._get_team_object_permission(
|
||||
user_api_key_auth
|
||||
)
|
||||
|
||||
|
||||
if object_permissions is None:
|
||||
return []
|
||||
|
||||
|
||||
@@ -216,6 +216,7 @@ class MCPServerManager:
|
||||
extra_headers=server_config.get("extra_headers", None),
|
||||
allowed_tools=server_config.get("allowed_tools", None),
|
||||
disallowed_tools=server_config.get("disallowed_tools", None),
|
||||
allowed_params=server_config.get("allowed_params", None),
|
||||
access_groups=server_config.get("access_groups", None),
|
||||
)
|
||||
self.config_mcp_servers[server_id] = new_server
|
||||
@@ -771,6 +772,58 @@ class MCPServerManager:
|
||||
)
|
||||
return True
|
||||
|
||||
def validate_allowed_params(
|
||||
self, tool_name: str, arguments: Dict[str, Any], server: MCPServer
|
||||
) -> None:
|
||||
"""
|
||||
Filter arguments to only include allowed parameters for the given tool.
|
||||
|
||||
Args:
|
||||
tool_name: Name of the tool (with or without prefix)
|
||||
arguments: Dictionary of arguments to filter
|
||||
server: MCPServer configuration
|
||||
|
||||
Returns:
|
||||
Filtered dictionary containing only allowed parameters
|
||||
|
||||
Raises:
|
||||
HTTPException: If allowed_params is configured for this tool but arguments contain disallowed params
|
||||
"""
|
||||
from litellm.proxy._experimental.mcp_server.utils import (
|
||||
get_server_name_prefix_tool_mcp,
|
||||
)
|
||||
|
||||
# If no allowed_params configured, return all arguments
|
||||
if not server.allowed_params:
|
||||
return
|
||||
|
||||
# Get the unprefixed tool name to match against config
|
||||
unprefixed_tool_name, _ = get_server_name_prefix_tool_mcp(tool_name)
|
||||
|
||||
# Check both prefixed and unprefixed tool names
|
||||
allowed_params_list = server.allowed_params.get(
|
||||
tool_name
|
||||
) or server.allowed_params.get(unprefixed_tool_name)
|
||||
|
||||
# If this tool doesn't have allowed_params specified, allow all params
|
||||
if allowed_params_list is None:
|
||||
return None
|
||||
|
||||
# Filter arguments to only include allowed parameters
|
||||
disallowed_params = [
|
||||
param for param in arguments.keys() if param not in allowed_params_list
|
||||
]
|
||||
|
||||
if disallowed_params:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail={
|
||||
"error": f"Parameters {disallowed_params} are not allowed for tool {tool_name}. "
|
||||
f"Allowed parameters: {allowed_params_list}. "
|
||||
f"Contact proxy admin to allow these parameters."
|
||||
},
|
||||
)
|
||||
|
||||
async def check_tool_permission_for_key_team(
|
||||
self,
|
||||
tool_name: str,
|
||||
@@ -895,6 +948,13 @@ class MCPServerManager:
|
||||
user_api_key_auth=user_api_key_auth,
|
||||
)
|
||||
|
||||
## filter parameters based on allowed_params configuration
|
||||
self.validate_allowed_params(
|
||||
tool_name=name,
|
||||
arguments=arguments,
|
||||
server=server,
|
||||
)
|
||||
|
||||
pre_hook_kwargs = {
|
||||
"name": name,
|
||||
"arguments": arguments,
|
||||
@@ -958,7 +1018,47 @@ class MCPServerManager:
|
||||
verbose_logger.error(f"Guardrail blocked MCP tool call pre call: {str(e)}")
|
||||
raise e
|
||||
|
||||
async def call_tool( # noqa: PLR0915
|
||||
def _create_during_hook_task(
|
||||
self,
|
||||
name: str,
|
||||
arguments: Dict[str, Any],
|
||||
server_name_from_prefix: Optional[str],
|
||||
user_api_key_auth: Optional[UserAPIKeyAuth],
|
||||
proxy_logging_obj: ProxyLogging,
|
||||
start_time: datetime.datetime,
|
||||
):
|
||||
"""Create and return a during hook task for MCP tool calls."""
|
||||
from litellm.types.llms.base import HiddenParams
|
||||
from litellm.types.mcp import MCPDuringCallRequestObject
|
||||
|
||||
request_obj = MCPDuringCallRequestObject(
|
||||
tool_name=name,
|
||||
arguments=arguments,
|
||||
server_name=server_name_from_prefix,
|
||||
start_time=start_time.timestamp() if start_time else None,
|
||||
hidden_params=HiddenParams(),
|
||||
)
|
||||
|
||||
during_hook_kwargs = {
|
||||
"name": name,
|
||||
"arguments": arguments,
|
||||
"server_name": server_name_from_prefix,
|
||||
"user_api_key_auth": user_api_key_auth,
|
||||
}
|
||||
|
||||
synthetic_llm_data = proxy_logging_obj._convert_mcp_to_llm_format(
|
||||
request_obj, during_hook_kwargs
|
||||
)
|
||||
|
||||
return asyncio.create_task(
|
||||
proxy_logging_obj.during_call_hook(
|
||||
user_api_key_dict=user_api_key_auth,
|
||||
data=synthetic_llm_data,
|
||||
call_type="mcp_call", # type: ignore
|
||||
)
|
||||
)
|
||||
|
||||
async def call_tool(
|
||||
self,
|
||||
name: str,
|
||||
arguments: Dict[str, Any],
|
||||
@@ -1024,35 +1124,13 @@ class MCPServerManager:
|
||||
# Prepare tasks for during hooks
|
||||
tasks = []
|
||||
if proxy_logging_obj:
|
||||
# Create synthetic LLM data for during hook processing
|
||||
from litellm.types.llms.base import HiddenParams
|
||||
from litellm.types.mcp import MCPDuringCallRequestObject
|
||||
|
||||
request_obj = MCPDuringCallRequestObject(
|
||||
tool_name=name,
|
||||
during_hook_task = self._create_during_hook_task(
|
||||
name=name,
|
||||
arguments=arguments,
|
||||
server_name=server_name_from_prefix,
|
||||
start_time=start_time.timestamp() if start_time else None,
|
||||
hidden_params=HiddenParams(),
|
||||
)
|
||||
|
||||
during_hook_kwargs = {
|
||||
"name": name,
|
||||
"arguments": arguments,
|
||||
"server_name": server_name_from_prefix,
|
||||
"user_api_key_auth": user_api_key_auth,
|
||||
}
|
||||
|
||||
synthetic_llm_data = proxy_logging_obj._convert_mcp_to_llm_format(
|
||||
request_obj, during_hook_kwargs
|
||||
)
|
||||
|
||||
during_hook_task = asyncio.create_task(
|
||||
proxy_logging_obj.during_call_hook(
|
||||
user_api_key_dict=user_api_key_auth,
|
||||
data=synthetic_llm_data,
|
||||
call_type="mcp_call", # type: ignore
|
||||
)
|
||||
server_name_from_prefix=server_name_from_prefix,
|
||||
user_api_key_auth=user_api_key_auth,
|
||||
proxy_logging_obj=proxy_logging_obj,
|
||||
start_time=start_time,
|
||||
)
|
||||
tasks.append(during_hook_task)
|
||||
|
||||
|
||||
@@ -902,6 +902,7 @@ if MCP_AVAILABLE:
|
||||
|
||||
await session_manager.handle_request(scope, receive, send)
|
||||
except Exception as e:
|
||||
raise e
|
||||
verbose_logger.exception(f"Error handling MCP request: {e}")
|
||||
# Instead of re-raising, try to send a graceful error response
|
||||
try:
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
self.__BUILD_MANIFEST={__rewrites:{afterFiles:[],beforeFiles:[],fallback:[]},"/_error":["static/chunks/pages/_error-28b803cb2479b966.js"],sortedPages:["/_app","/_error"]},self.__BUILD_MANIFEST_CB&&self.__BUILD_MANIFEST_CB();
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
"use strict";(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[3665],{84566:function(e,t,s){s.d(t,{GH$:function(){return l}});var c=s(2265);let l=({color:e="currentColor",size:t=24,className:s,...l})=>c.createElement("svg",{viewBox:"0 0 24 24",xmlns:"http://www.w3.org/2000/svg",width:t,height:t,fill:e,...l,className:"remixicon "+(s||"")},c.createElement("path",{d:"M12 22C6.47715 22 2 17.5228 2 12C2 6.47715 6.47715 2 12 2C17.5228 2 22 6.47715 22 12C22 17.5228 17.5228 22 12 22ZM12 20C16.4183 20 20 16.4183 20 12C20 7.58172 16.4183 4 12 4C7.58172 4 4 7.58172 4 12C4 16.4183 7.58172 20 12 20ZM11.0026 16L6.75999 11.7574L8.17421 10.3431L11.0026 13.1716L16.6595 7.51472L18.0737 8.92893L11.0026 16Z"}))}}]);
|
||||
@@ -1 +0,0 @@
|
||||
"use strict";(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[665],{84566:function(e,t,s){s.d(t,{GH$:function(){return l}});var c=s(2265);let l=({color:e="currentColor",size:t=24,className:s,...l})=>c.createElement("svg",{viewBox:"0 0 24 24",xmlns:"http://www.w3.org/2000/svg",width:t,height:t,fill:e,...l,className:"remixicon "+(s||"")},c.createElement("path",{d:"M12 22C6.47715 22 2 17.5228 2 12C2 6.47715 6.47715 2 12 2C17.5228 2 22 6.47715 22 12C22 17.5228 17.5228 22 12 22ZM12 20C16.4183 20 20 16.4183 20 12C20 7.58172 16.4183 4 12 4C7.58172 4 4 7.58172 4 12C4 16.4183 7.58172 20 12 20ZM11.0026 16L6.75999 11.7574L8.17421 10.3431L11.0026 13.1716L16.6595 7.51472L18.0737 8.92893L11.0026 16Z"}))}}]);
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user