litellm mcp support

This commit is contained in:
Ishaan Jaff
2025-03-20 21:48:55 -07:00
parent 9173685ec2
commit 435fba929e
+29 -2
View File
@@ -29,6 +29,8 @@ When MCP clients connect to LiteLLM they can follow this workflow:
### 1. Define your tools on mcp_tools
LiteLLM allows you to define your tools on the `mcp_tools` section in your config.yaml file. All tools listed here will be available to MCP clients (when they connect to LiteLLM and call `list_tools`).
```yaml
model_list:
- model_name: gpt-4o
@@ -54,13 +56,38 @@ mcp_tools:
handler: "mcp_tools.get_current_time"
```
### 2. Define a handler for your tool
Create a new file called `mcp_tools.py` and add this code. The key method here is `get_current_time` which gets executed when the `get_current_time` tool is called.
```python
# mcp_tools.py
from datetime import datetime
def get_current_time(format: str = "short"):
"""
Simple handler for the 'get_current_time' tool.
Args:
format (str): The format of the time to return ('short').
Returns:
str: The current time formatted as 'HH:MM'.
"""
# Get the current time
current_time = datetime.now()
# Format the time as 'HH:MM'
return current_time.strftime('%H:%M')
```
### 3. Start LiteLLM Gateway
<Tabs>
<TabItem value="docker" label="Docker Run">
Mount your `custom_logger.py` on the LiteLLM Docker container.
Mount your `mcp_tools.py` on the LiteLLM Docker container.
```shell
docker run -d \
@@ -68,7 +95,7 @@ docker run -d \
-e OPENAI_API_KEY=$OPENAI_API_KEY \
--name my-app \
-v $(pwd)/my_config.yaml:/app/config.yaml \
-v $(pwd)/custom_logger.py:/app/custom_logger.py \
-v $(pwd)/mcp_tools.py:/app/mcp_tools.py \
my-app:latest \
--config /app/config.yaml \
--port 4000 \