mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-02 02:21:27 +00:00
Litellm release notes 07 12 2025 (#12563)
* docs(index.md): add new models / updated models to new release note * docs(index.md): add LLM api endpoints to release notes * docs(index.md): refactor mcp into separate section ' * docs: doc improvements * docs(index.md): more cleanup * docs(index.md): more cleanup * docs: update litellm enterprise trial link * docs(index.md): cleanup * docs(datadog.md): add separate datadog doc * docs(index.md): cleanup * docs(index.md): update docker version * docs(index.md): add screenshots for new features * docs(index.md): add doc links
This commit is contained in:
Krish Dholakia
@@ -4,7 +4,7 @@ import Image from '@theme/IdealImage';
|
||||
For companies that need SSO, user management and professional support for LiteLLM Proxy
|
||||
|
||||
:::info
|
||||
Get free 7-day trial key [here](https://www.litellm.ai/#trial)
|
||||
Get free 7-day trial key [here](https://www.litellm.ai/enterprise#trial)
|
||||
:::
|
||||
|
||||
## Enterprise Features
|
||||
|
||||
@@ -42,7 +42,7 @@ LiteLLM supports the following MCP transports:
|
||||
style={{width: '80%', display: 'block', margin: '0'}}
|
||||
/>
|
||||
|
||||
### Adding a stdio MCP Server
|
||||
#### Adding a stdio MCP Server
|
||||
|
||||
For stdio MCP servers, select "Standard Input/Output (stdio)" as the transport type and provide the stdio configuration in JSON format:
|
||||
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
import Image from '@theme/IdealImage';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# DataDog
|
||||
|
||||
LiteLLM Supports logging to the following Datdog Integrations:
|
||||
- `datadog` [Datadog Logs](https://docs.datadoghq.com/logs/)
|
||||
- `datadog_llm_observability` [Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/)
|
||||
- `ddtrace-run` [Datadog Tracing](#datadog-tracing)
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="datadog" label="Datadog Logs">
|
||||
|
||||
We will use the `--config` to set `litellm.callbacks = ["datadog"]` this will log all successful LLM calls to DataDog
|
||||
|
||||
**Step 1**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gpt-3.5-turbo
|
||||
litellm_params:
|
||||
model: gpt-3.5-turbo
|
||||
litellm_settings:
|
||||
callbacks: ["datadog"] # logs llm success + failure logs on datadog
|
||||
service_callback: ["datadog"] # logs redis, postgres failures on datadog
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="datadog_llm_observability" label="Datadog LLM Observability">
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gpt-3.5-turbo
|
||||
litellm_params:
|
||||
model: gpt-3.5-turbo
|
||||
litellm_settings:
|
||||
callbacks: ["datadog_llm_observability"] # logs llm success logs on datadog
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
**Step 2**: Set Required env variables for datadog
|
||||
|
||||
```shell
|
||||
DD_API_KEY="5f2d0f310***********" # your datadog API Key
|
||||
DD_SITE="us5.datadoghq.com" # your datadog base url
|
||||
DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source. use to differentiate dev vs. prod deployments
|
||||
```
|
||||
|
||||
**Step 3**: Start the proxy, make a test request
|
||||
|
||||
Start proxy
|
||||
|
||||
```shell
|
||||
litellm --config config.yaml --debug
|
||||
```
|
||||
|
||||
Test Request
|
||||
|
||||
```shell
|
||||
curl --location 'http://0.0.0.0:4000/chat/completions' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "what llm are you"
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"your-custom-metadata": "custom-field",
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
Expected output on Datadog
|
||||
|
||||
<Image img={require('../../img/dd_small1.png')} />
|
||||
|
||||
#### Datadog Tracing
|
||||
|
||||
Use `ddtrace-run` to enable [Datadog Tracing](https://ddtrace.readthedocs.io/en/stable/installation_quickstart.html) on litellm proxy
|
||||
|
||||
**DD Tracer**
|
||||
Pass `USE_DDTRACE=true` to the docker run command. When `USE_DDTRACE=true`, the proxy will run `ddtrace-run litellm` as the `ENTRYPOINT` instead of just `litellm`
|
||||
|
||||
**DD Profiler**
|
||||
|
||||
Pass `USE_DDPROFILER=true` to the docker run command. When `USE_DDPROFILER=true`, the proxy will activate the [Datadog Profiler](https://docs.datadoghq.com/profiler/enabling/python/). This is useful for debugging CPU% and memory usage.
|
||||
|
||||
We don't recommend using `USE_DDPROFILER` in production. It is only recommended for debugging CPU% and memory usage.
|
||||
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
-v $(pwd)/litellm_config.yaml:/app/config.yaml \
|
||||
-e USE_DDTRACE=true \
|
||||
-e USE_DDPROFILER=true \
|
||||
-p 4000:4000 \
|
||||
ghcr.io/berriai/litellm:main-latest \
|
||||
--config /app/config.yaml --detailed_debug
|
||||
```
|
||||
|
||||
### Set DD variables (`DD_SERVICE` etc)
|
||||
|
||||
LiteLLM supports customizing the following Datadog environment variables
|
||||
|
||||
| Environment Variable | Description | Default Value | Required |
|
||||
|---------------------|-------------|---------------|----------|
|
||||
| `DD_API_KEY` | Your Datadog API key for authentication | None | ✅ Yes |
|
||||
| `DD_SITE` | Your Datadog site (e.g., "us5.datadoghq.com") | None | ✅ Yes |
|
||||
| `DD_ENV` | Environment tag for your logs (e.g., "production", "staging") | "unknown" | ❌ No |
|
||||
| `DD_SERVICE` | Service name for your logs | "litellm-server" | ❌ No |
|
||||
| `DD_SOURCE` | Source name for your logs | "litellm" | ❌ No |
|
||||
| `DD_VERSION` | Version tag for your logs | "unknown" | ❌ No |
|
||||
| `HOSTNAME` | Hostname tag for your logs | "" | ❌ No |
|
||||
| `POD_NAME` | Pod name tag (useful for Kubernetes deployments) | "unknown" | ❌ No |
|
||||
|
||||
@@ -212,6 +212,7 @@ The following parameters can be updated on a continuation of a trace by passing
|
||||
* `parent_observation_id` - Identifier for the parent observation, defaults to `None`
|
||||
* `prompt` - Langfuse prompt object used for the generation, defaults to `None`
|
||||
|
||||
|
||||
Any other key value pairs passed into the metadata not listed in the above spec for a `litellm` completion will be added as a metadata key value pair for the generation.
|
||||
|
||||
#### Disable Logging - Specific Calls
|
||||
|
||||
@@ -10,7 +10,7 @@ import TabItem from '@theme/TabItem';
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Get free 7-day trial key](https://www.litellm.ai/#trial)
|
||||
[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -512,6 +512,7 @@ router_settings:
|
||||
| LAGO_API_KEY | API key for accessing Lago services
|
||||
| LANGFUSE_DEBUG | Toggle debug mode for Langfuse
|
||||
| LANGFUSE_FLUSH_INTERVAL | Interval for flushing Langfuse logs
|
||||
| LANGFUSE_TRACING_ENVIRONMENT | Environment for Langfuse tracing
|
||||
| LANGFUSE_HOST | Host URL for Langfuse service
|
||||
| LANGFUSE_PUBLIC_KEY | Public key for Langfuse authentication
|
||||
| LANGFUSE_RELEASE | Release version of Langfuse integration
|
||||
|
||||
@@ -201,7 +201,7 @@ Follow this simple workflow to implement and tune guardrails:
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/#trial)
|
||||
✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
:::
|
||||
|
||||
@@ -295,7 +295,7 @@ curl -i http://localhost:4000/v1/chat/completions \
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/#trial)
|
||||
✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
:::
|
||||
|
||||
@@ -380,7 +380,7 @@ Monitor which guardrails were executed and whether they passed or failed. e.g. g
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/#trial)
|
||||
✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
:::
|
||||
|
||||
@@ -405,7 +405,7 @@ Monitor which guardrails were executed and whether they passed or failed. e.g. g
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/#trial)
|
||||
✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
:::
|
||||
|
||||
@@ -465,7 +465,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/#trial)
|
||||
✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
:::
|
||||
|
||||
@@ -495,7 +495,7 @@ guardrails:
|
||||
|
||||
:::info
|
||||
|
||||
✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/#trial)
|
||||
✨ This is an Enterprise only feature [Get a free trial](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ import TabItem from '@theme/TabItem';
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Get free 7-day trial key](https://www.litellm.ai/#trial)
|
||||
[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -1537,124 +1537,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
|
||||
[**The standard logging object is logged on Azure Data Lake Storage**](../proxy/logging_spec)
|
||||
|
||||
|
||||
|
||||
## DataDog
|
||||
|
||||
LiteLLM Supports logging to the following Datdog Integrations:
|
||||
- `datadog` [Datadog Logs](https://docs.datadoghq.com/logs/)
|
||||
- `datadog_llm_observability` [Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/)
|
||||
- `ddtrace-run` [Datadog Tracing](#datadog-tracing)
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="datadog" label="Datadog Logs">
|
||||
|
||||
We will use the `--config` to set `litellm.callbacks = ["datadog"]` this will log all successful LLM calls to DataDog
|
||||
|
||||
**Step 1**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gpt-3.5-turbo
|
||||
litellm_params:
|
||||
model: gpt-3.5-turbo
|
||||
litellm_settings:
|
||||
callbacks: ["datadog"] # logs llm success + failure logs on datadog
|
||||
service_callback: ["datadog"] # logs redis, postgres failures on datadog
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="datadog_llm_observability" label="Datadog LLM Observability">
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gpt-3.5-turbo
|
||||
litellm_params:
|
||||
model: gpt-3.5-turbo
|
||||
litellm_settings:
|
||||
callbacks: ["datadog_llm_observability"] # logs llm success logs on datadog
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
**Step 2**: Set Required env variables for datadog
|
||||
|
||||
```shell
|
||||
DD_API_KEY="5f2d0f310***********" # your datadog API Key
|
||||
DD_SITE="us5.datadoghq.com" # your datadog base url
|
||||
DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source. use to differentiate dev vs. prod deployments
|
||||
```
|
||||
|
||||
**Step 3**: Start the proxy, make a test request
|
||||
|
||||
Start proxy
|
||||
|
||||
```shell
|
||||
litellm --config config.yaml --debug
|
||||
```
|
||||
|
||||
Test Request
|
||||
|
||||
```shell
|
||||
curl --location 'http://0.0.0.0:4000/chat/completions' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "what llm are you"
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"your-custom-metadata": "custom-field",
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
Expected output on Datadog
|
||||
|
||||
<Image img={require('../../img/dd_small1.png')} />
|
||||
|
||||
#### Datadog Tracing
|
||||
|
||||
Use `ddtrace-run` to enable [Datadog Tracing](https://ddtrace.readthedocs.io/en/stable/installation_quickstart.html) on litellm proxy
|
||||
|
||||
**DD Tracer**
|
||||
Pass `USE_DDTRACE=true` to the docker run command. When `USE_DDTRACE=true`, the proxy will run `ddtrace-run litellm` as the `ENTRYPOINT` instead of just `litellm`
|
||||
|
||||
**DD Profiler**
|
||||
|
||||
Pass `USE_DDPROFILER=true` to the docker run command. When `USE_DDPROFILER=true`, the proxy will activate the [Datadog Profiler](https://docs.datadoghq.com/profiler/enabling/python/). This is useful for debugging CPU% and memory usage.
|
||||
|
||||
We don't recommend using `USE_DDPROFILER` in production. It is only recommended for debugging CPU% and memory usage.
|
||||
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
-v $(pwd)/litellm_config.yaml:/app/config.yaml \
|
||||
-e USE_DDTRACE=true \
|
||||
-e USE_DDPROFILER=true \
|
||||
-p 4000:4000 \
|
||||
ghcr.io/berriai/litellm:main-latest \
|
||||
--config /app/config.yaml --detailed_debug
|
||||
```
|
||||
|
||||
### Set DD variables (`DD_SERVICE` etc)
|
||||
|
||||
LiteLLM supports customizing the following Datadog environment variables
|
||||
|
||||
| Environment Variable | Description | Default Value | Required |
|
||||
|---------------------|-------------|---------------|----------|
|
||||
| `DD_API_KEY` | Your Datadog API key for authentication | None | ✅ Yes |
|
||||
| `DD_SITE` | Your Datadog site (e.g., "us5.datadoghq.com") | None | ✅ Yes |
|
||||
| `DD_ENV` | Environment tag for your logs (e.g., "production", "staging") | "unknown" | ❌ No |
|
||||
| `DD_SERVICE` | Service name for your logs | "litellm-server" | ❌ No |
|
||||
| `DD_SOURCE` | Source name for your logs | "litellm" | ❌ No |
|
||||
| `DD_VERSION` | Version tag for your logs | "unknown" | ❌ No |
|
||||
| `HOSTNAME` | Hostname tag for your logs | "" | ❌ No |
|
||||
| `POD_NAME` | Pod name tag (useful for Kubernetes deployments) | "unknown" | ❌ No |
|
||||
|
||||
## [Datadog](../observability/datadog)
|
||||
|
||||
## Lunary
|
||||
#### Step1: Install dependencies and set your environment variables
|
||||
|
||||
@@ -8,7 +8,7 @@ import Image from '@theme/IdealImage';
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Get free 7-day trial key](https://www.litellm.ai/#trial)
|
||||
[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ import Image from '@theme/IdealImage';
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Get free 7-day trial key](https://www.litellm.ai/#trial)
|
||||
[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -14,6 +14,8 @@ Special headers that are supported by LiteLLM.
|
||||
|
||||
`anthropic-version` Optional[str]: The version of the Anthropic API to use.
|
||||
`anthropic-beta` Optional[str]: The beta version of the Anthropic API to use.
|
||||
- For `/v1/messages` endpoint, this will always be forward the header to the underlying model.
|
||||
- For `/chat/completions` endpoint, this will only be forwarded if `forward_client_headers_to_llm_api` is true.
|
||||
|
||||
## OpenAI Headers
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ This walks through how to set the maximum retention period for spend logs. This
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Get free 7-day trial key](https://www.litellm.ai/#trial)
|
||||
[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ Set temporary budget increase for a LiteLLM Virtual Key. Use this if you get ask
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Get free 7-day trial key](https://www.litellm.ai/#trial)
|
||||
[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -527,7 +527,7 @@ This is an Enterprise feature.
|
||||
|
||||
[Enterprise Pricing](https://www.litellm.ai/#pricing)
|
||||
|
||||
[Get free 7-day trial key](https://www.litellm.ai/#trial)
|
||||
[Get free 7-day trial key](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
|
||||
:::
|
||||
|
||||
@@ -733,6 +733,68 @@ follow_up = client.responses.create(
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Calling non-Responses API endpoints (`/responses` to `/chat/completions` Bridge)
|
||||
|
||||
LiteLLM allows you to call non-Responses API models via a bridge to LiteLLM's `/chat/completions` endpoint. This is useful for calling Anthropic, Gemini and even non-Responses API OpenAI models.
|
||||
|
||||
|
||||
#### Python SDK Usage
|
||||
|
||||
```python showLineNumbers title="SDK Usage"
|
||||
import litellm
|
||||
import os
|
||||
|
||||
# Set API key
|
||||
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-api-key"
|
||||
|
||||
# Non-streaming response
|
||||
response = litellm.responses(
|
||||
model="anthropic/claude-3-5-sonnet-20240620",
|
||||
input="Tell me a three sentence bedtime story about a unicorn.",
|
||||
max_output_tokens=100
|
||||
)
|
||||
|
||||
print(response)
|
||||
```
|
||||
|
||||
#### LiteLLM Proxy Usage
|
||||
|
||||
**Setup Config:**
|
||||
|
||||
```yaml showLineNumbers title="Example Configuration"
|
||||
model_list:
|
||||
- model_name: anthropic-model
|
||||
litellm_params:
|
||||
model: anthropic/claude-3-5-sonnet-20240620
|
||||
api_key: os.environ/ANTHROPIC_API_KEY
|
||||
```
|
||||
|
||||
**Start Proxy:**
|
||||
|
||||
```bash showLineNumbers title="Start LiteLLM Proxy"
|
||||
litellm --config /path/to/config.yaml
|
||||
|
||||
# RUNNING on http://0.0.0.0:4000
|
||||
```
|
||||
|
||||
**Make Request:**
|
||||
|
||||
```bash showLineNumbers title="non-Responses API Model Request"
|
||||
curl http://localhost:4000/v1/responses \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-d '{
|
||||
"model": "anthropic-model",
|
||||
"input": "who is Michael Jordan"
|
||||
}'
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Session Management - Non-OpenAI Models
|
||||
|
||||
LiteLLM Proxy supports session management for non-OpenAI models. This allows you to store and fetch conversation history (state) in LiteLLM Proxy.
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 632 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 175 KiB |
@@ -28,7 +28,7 @@ import Image from '@theme/IdealImage';
|
||||
|
||||
:::info
|
||||
|
||||
Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/#trial)
|
||||
Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
**No call needed**
|
||||
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
---
|
||||
title: "[PRE-RELEASE] v1.74.3-stable"
|
||||
slug: "v1-74-3-stable"
|
||||
date: 2025-07-12T10:00:00
|
||||
authors:
|
||||
- name: Krrish Dholakia
|
||||
title: CEO, LiteLLM
|
||||
url: https://www.linkedin.com/in/krish-d/
|
||||
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
|
||||
- name: Ishaan Jaffer
|
||||
title: CTO, LiteLLM
|
||||
url: https://www.linkedin.com/in/reffajnaahsi/
|
||||
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
|
||||
|
||||
hide_table_of_contents: false
|
||||
---
|
||||
|
||||
import Image from '@theme/IdealImage';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
## Deploy this version
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="docker" label="Docker">
|
||||
|
||||
``` showLineNumbers title="docker run litellm"
|
||||
docker run \
|
||||
-e STORE_MODEL_IN_DB=True \
|
||||
-p 4000:4000 \
|
||||
ghcr.io/berriai/litellm:v1.74.3.rc.2
|
||||
```
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="pip" label="Pip">
|
||||
|
||||
``` showLineNumbers title="pip install litellm"
|
||||
pip install litellm==1.74.3rc2
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
---
|
||||
|
||||
|
||||
## New Models / Updated Models
|
||||
|
||||
#### Pricing / Context Window Updates
|
||||
|
||||
| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Type |
|
||||
| ----------- | -------------------------------------- | -------------- | ------------------- | -------------------- | ---- |
|
||||
| Xai | `xai/grok-4` | 256k | $3.00 | $15.00 | New |
|
||||
| Xai | `xai/grok-4-0709` | 256k | $3.00 | $15.00 | New |
|
||||
| Xai | `xai/grok-4-latest` | 256k | $3.00 | $15.00 | New |
|
||||
| Mistral | `mistral/devstral-small-2507` | 128k | $0.1 | $0.3 | New |
|
||||
| Mistral | `mistral/devstral-medium-2507` | 128k | $0.4 | $2 | New |
|
||||
| Azure OpenAI | `azure/o3-deep-research` | 200k | $10 | $40 | New |
|
||||
|
||||
|
||||
#### Features
|
||||
- **[Xinference](../../docs/providers/xinference)**
|
||||
- Image generation API support - [PR](https://github.com/BerriAI/litellm/pull/12439)
|
||||
- **[Bedrock](../../docs/providers/bedrock)**
|
||||
- API Key Auth support for AWS Bedrock API - [PR](https://github.com/BerriAI/litellm/pull/12495)
|
||||
- **[🆕 Dashscope](../../docs/providers/dashscope)**
|
||||
- New integration from Alibaba (enables qwen usage) - [PR](https://github.com/BerriAI/litellm/pull/12361)
|
||||
- **[🆕 Bytez](../../docs/providers/bytez)**
|
||||
- New /chat/completion integration - [PR](https://github.com/BerriAI/litellm/pull/12121)
|
||||
|
||||
#### Bugs
|
||||
- **[Github Copilot](../../docs/providers/github_copilot)**
|
||||
- Fix API base url for Github Copilot - [PR](https://github.com/BerriAI/litellm/pull/12418)
|
||||
- **[Bedrock](../../docs/providers/bedrock)**
|
||||
- Ensure supported bedrock/converse/ params = bedrock/ params - [PR](https://github.com/BerriAI/litellm/pull/12466)
|
||||
- Fix cache token cost calculation - [PR](https://github.com/BerriAI/litellm/pull/12488)
|
||||
- **[XAI](../../docs/providers/xai)**
|
||||
- ensure finish_reason includes tool calls when xai responses with tool calls - [PR](https://github.com/BerriAI/litellm/pull/12545)
|
||||
|
||||
---
|
||||
|
||||
## LLM API Endpoints
|
||||
|
||||
#### Features
|
||||
- **[/completions](../../docs/text_completion)**
|
||||
- Return ‘reasoning_content’ on streaming - [PR](https://github.com/BerriAI/litellm/pull/12377)
|
||||
- **[/chat/completions](../../docs/completion/input)**
|
||||
- Add 'thinking blocks' to stream chunk builder - [PR](https://github.com/BerriAI/litellm/pull/12395)
|
||||
- **[/v1/messages](../../docs/anthropic_unified)**
|
||||
- Fallbacks support - [PR](https://github.com/BerriAI/litellm/pull/12440)
|
||||
- tool call handling for non-anthropic models (/v1/messages to /chat/completion bridge) - [PR](https://github.com/BerriAI/litellm/pull/12473)
|
||||
|
||||
---
|
||||
|
||||
## [MCP Gateway](../../docs/mcp)
|
||||
|
||||
<Image
|
||||
img={require('../../img/release_notes/mcp_tool_cost_tracking.png')}
|
||||
style={{width: '100%', display: 'block', margin: '2rem auto'}}
|
||||
/>
|
||||
|
||||
#### Features
|
||||
- **[Cost Tracking](../../docs/mcp#-mcp-cost-tracking)**
|
||||
- Add Cost Tracking - [PR](https://github.com/BerriAI/litellm/pull/12385)
|
||||
- Add usage tracking - [PR](https://github.com/BerriAI/litellm/pull/12397)
|
||||
- Add custom cost configuration for each MCP tool - [PR](https://github.com/BerriAI/litellm/pull/12499)
|
||||
- Add support for editing MCP cost per tool - [PR](https://github.com/BerriAI/litellm/pull/12501)
|
||||
- Allow using custom post call MCP hook for cost tracking - [PR](https://github.com/BerriAI/litellm/pull/12469)
|
||||
- **[Auth](../../docs/mcp#using-your-mcp-with-client-side-credentials)**
|
||||
- Allow customizing what client side auth header to use - [PR](https://github.com/BerriAI/litellm/pull/12460)
|
||||
- Raises error when MCP server header is malformed in the request - [PR](https://github.com/BerriAI/litellm/pull/12494)
|
||||
- **[MCP Server](../../docs/mcp#adding-your-mcp)**
|
||||
- Allow using stdio MCPs with LiteLLM (enables using Circle CI MCP w/ LiteLLM) - [PR](https://github.com/BerriAI/litellm/pull/12530), [Get Started](../../docs/mcp#adding-a-stdio-mcp-server)
|
||||
|
||||
#### Bugs
|
||||
- **General**
|
||||
- Fix task group is not initialized error - [PR](https://github.com/BerriAI/litellm/pull/12411) s/o [@juancarlosm](https://github.com/juancarlosm)
|
||||
- **[MCP Server](../../docs/mcp#adding-your-mcp)**
|
||||
- Fix mcp tool separator to work with Claude code - [PR](https://github.com/BerriAI/litellm/pull/12430), [Get Started](../../docs/mcp#adding-your-mcp)
|
||||
- Add validation to mcp server name to not allow "-" (enables namespaces to work) - [PR](https://github.com/BerriAI/litellm/pull/12515)
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Management Endpoints / UI
|
||||
|
||||
|
||||
<Image
|
||||
img={require('../../img/release_notes/model_hub_v2.png')}
|
||||
style={{width: '100%', display: 'block', margin: '2rem auto'}}
|
||||
/>
|
||||
|
||||
#### Features
|
||||
- **Model Hub**
|
||||
- new model hub table view - [PR](https://github.com/BerriAI/litellm/pull/12468)
|
||||
- new /public/model_hub endpoint - [PR](https://github.com/BerriAI/litellm/pull/12468)
|
||||
- Make Model Hub OSS - [PR](https://github.com/BerriAI/litellm/pull/12553)
|
||||
- New ‘make public’ modal flow for showing proxy models on public model hub - [PR](https://github.com/BerriAI/litellm/pull/12555)
|
||||
- **MCP**
|
||||
- support for internal users to use and manage MCP servers - [PR](https://github.com/BerriAI/litellm/pull/12458)
|
||||
- Adds UI support to add MCP access groups (similar to namespaces) - [PR](https://github.com/BerriAI/litellm/pull/12470)
|
||||
- MCP Tool Testing Playground - [PR](https://github.com/BerriAI/litellm/pull/12520)
|
||||
- Show cost config on root of MCP settings - [PR](https://github.com/BerriAI/litellm/pull/12526)
|
||||
- **Test Key**
|
||||
- Stick sessions - [PR](https://github.com/BerriAI/litellm/pull/12365)
|
||||
- MCP Access Groups - allow mcp access groups - [PR](https://github.com/BerriAI/litellm/pull/12529)
|
||||
- **Usage**
|
||||
- Truncate long labels and improve tooltip in Top API Keys chart - [PR](https://github.com/BerriAI/litellm/pull/12371)
|
||||
- Improve Chart Readability for Tag Usage - [PR](https://github.com/BerriAI/litellm/pull/12378)
|
||||
- **Teams**
|
||||
- Prevent navigation reset after team member operations - [PR](https://github.com/BerriAI/litellm/pull/12424)
|
||||
- Team Members - reset budget, if duration set - [PR](https://github.com/BerriAI/litellm/pull/12534)
|
||||
- Use central team member budget when max_budget_in_team set on UI - [PR](https://github.com/BerriAI/litellm/pull/12533)
|
||||
- **SSO**
|
||||
- Allow users to run a custom sso login handler - [PR](https://github.com/BerriAI/litellm/pull/12465)
|
||||
- **Navbar**
|
||||
- improve user dropdown UI with premium badge and cleaner layout - [PR](https://github.com/BerriAI/litellm/pull/12502)
|
||||
- **General**
|
||||
- Consistent layout for Create and Back buttons on all the pages - [PR](https://github.com/BerriAI/litellm/pull/12542)
|
||||
- Align Show Password with Checkbox - [PR](https://github.com/BerriAI/litellm/pull/12538)
|
||||
- Prevent writing default user setting updates to yaml (causes error in non-root env) - [PR](https://github.com/BerriAI/litellm/pull/12533)
|
||||
|
||||
#### Bugs
|
||||
- **Model Hub**
|
||||
- fix duplicates in /model_group/info - [PR](https://github.com/BerriAI/litellm/pull/12468)
|
||||
- **MCP**
|
||||
- Fix UI not syncing MCP access groups properly with object permissions - [PR](https://github.com/BerriAI/litellm/pull/12523)
|
||||
|
||||
---
|
||||
|
||||
## Logging / Guardrail Integrations
|
||||
|
||||
#### Features
|
||||
- **[Langfuse](../../docs/observability/langfuse_integration)**
|
||||
- Version bump - [PR](https://github.com/BerriAI/litellm/pull/12376)
|
||||
- LANGFUSE_TRACING_ENVIRONMENT support - [PR](https://github.com/BerriAI/litellm/pull/12376)
|
||||
- **[Bedrock Guardrails](../../docs/proxy/guardrails/bedrock)**
|
||||
- Raise Bedrock output text on 'BLOCKED' actions from guardrail - [PR](https://github.com/BerriAI/litellm/pull/12435)
|
||||
- **[OTEL](../../docs/observability/opentelemetry_integration)**
|
||||
- `OTEL_RESOURCE_ATTRIBUTES` support - [PR](https://github.com/BerriAI/litellm/pull/12468)
|
||||
- **[Guardrails AI](../../docs/proxy/guardrails/guardrails_ai)**
|
||||
- pre-call + logging only guardrail (pii detection/competitor names) support - [PR](https://github.com/BerriAI/litellm/pull/12506)
|
||||
- **[Guardrails](../../docs/proxy/guardrails/quick_start)**
|
||||
- [Enterprise] Support tag based mode for guardrails - [PR](https://github.com/BerriAI/litellm/pull/12508), [Get Started](../../docs/proxy/guardrails/quick_start#-tag-based-guardrail-modes)
|
||||
- **[OpenAI Moderations API](../../docs/proxy/guardrails/openai_moderation)**
|
||||
- New guardrail integration - [PR](https://github.com/BerriAI/litellm/pull/12519)
|
||||
- **[Prometheus](../../docs/proxy/prometheus)**
|
||||
- support tag based metrics (enables prometheus metrics for measuring roo-code/cline/claude code engagement) - [PR](https://github.com/BerriAI/litellm/pull/12534), [Get Started](../../docs/proxy/prometheus#custom-tags)
|
||||
- **[Datadog LLM Observability](../../docs/observability/datadog)**
|
||||
- Added `total_cost` field to track costs in DataDog LLM observability metrics - [PR](https://github.com/BerriAI/litellm/pull/12467)
|
||||
|
||||
#### Bugs
|
||||
- **[Prometheus](../../docs/proxy/prometheus)**
|
||||
- Remove experimental `_by_tag` metrics (fixes cardinality issue) - [PR](https://github.com/BerriAI/litellm/pull/12395)
|
||||
- **[Slack Alerting](../../docs/proxy/alerting)**
|
||||
- Fix slack alerting for outage and region outage alerts - [PR](https://github.com/BerriAI/litellm/pull/12464), [Get Started](../../docs/proxy/alerting#region-outage-alerting--enterprise-feature)
|
||||
|
||||
---
|
||||
|
||||
## Performance / Loadbalancing / Reliability improvements
|
||||
|
||||
#### Bugs
|
||||
- **[Responses API Bridge](../../docs/response_api#calling-non-responses-api-endpoints-responses-to-chatcompletions-bridge)**
|
||||
- add image support for Responses API when falling back on Chat Completions - [PR](https://github.com/BerriAI/litellm/pull/12204) s/o [@ryan-castner](https://github.com/ryan-castner)
|
||||
- **aiohttp**
|
||||
- Properly close aiohttp client sessions to prevent resource leaks - [PR](https://github.com/BerriAI/litellm/pull/12251)
|
||||
- **Router**
|
||||
- don't add invalid deployment to router pattern match - [PR](https://github.com/BerriAI/litellm/pull/12459)
|
||||
|
||||
|
||||
---
|
||||
|
||||
## General Proxy Improvements
|
||||
|
||||
#### Bugs
|
||||
- **S3**
|
||||
- s3 config.yaml file - ensure yaml safe load is used - [PR](https://github.com/BerriAI/litellm/pull/12373)
|
||||
- **Audit Logs**
|
||||
- Add audit logs for model updates - [PR](https://github.com/BerriAI/litellm/pull/12396)
|
||||
- **Startup**
|
||||
- Multiple API Keys Created on Startup when max_budget is enabled - [PR](https://github.com/BerriAI/litellm/pull/12436)
|
||||
- **Auth**
|
||||
- Resolve model group alias on Auth (if user has access to underlying model, allow alias request to work) - [PR](https://github.com/BerriAI/litellm/pull/12440)
|
||||
- **config.yaml**
|
||||
- fix parsing environment_variables from config.yaml - [PR](https://github.com/BerriAI/litellm/pull/12482)
|
||||
- **Security**
|
||||
- Log hashed jwt w/ prefix instead of actual value - [PR](https://github.com/BerriAI/litellm/pull/12524)
|
||||
|
||||
#### Features
|
||||
- **MCP**
|
||||
- Bump mcp version on docker img - [PR](https://github.com/BerriAI/litellm/pull/12362)
|
||||
- **Request Headers**
|
||||
- Forward ‘anthropic-beta’ header when forward_client_headers_to_llm_api is true - [PR](https://github.com/BerriAI/litellm/pull/12462)
|
||||
|
||||
---
|
||||
|
||||
## New Contributors
|
||||
* @kanaka made their first contribution in https://github.com/BerriAI/litellm/pull/12418
|
||||
* @juancarlosm made their first contribution in https://github.com/BerriAI/litellm/pull/12411
|
||||
* @DmitriyAlergant made their first contribution in https://github.com/BerriAI/litellm/pull/12356
|
||||
* @Rayshard made their first contribution in https://github.com/BerriAI/litellm/pull/12487
|
||||
* @minghao51 made their first contribution in https://github.com/BerriAI/litellm/pull/12361
|
||||
* @jdietzsch91 made their first contribution in https://github.com/BerriAI/litellm/pull/12488
|
||||
* @iwinux made their first contribution in https://github.com/BerriAI/litellm/pull/12473
|
||||
* @andresC98 made their first contribution in https://github.com/BerriAI/litellm/pull/12413
|
||||
* @EmaSuriano made their first contribution in https://github.com/BerriAI/litellm/pull/12509
|
||||
* @strawgate made their first contribution in https://github.com/BerriAI/litellm/pull/12528
|
||||
* @inf3rnus made their first contribution in https://github.com/BerriAI/litellm/pull/12121
|
||||
|
||||
## **[Git Diff](https://github.com/BerriAI/litellm/compare/v1.74.0-stable...v1.74.3-stable)**
|
||||
|
||||
@@ -3424,7 +3424,7 @@ You can now set custom parameters (like success threshold) for your guardrails i
|
||||
|
||||
info
|
||||
|
||||
Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/#trial)
|
||||
Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
**No call needed**
|
||||
|
||||
@@ -4107,7 +4107,7 @@ Use this to see the changes in the codebase.
|
||||
|
||||
info
|
||||
|
||||
Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/#trial)
|
||||
Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
**No call needed**
|
||||
|
||||
@@ -4966,7 +4966,7 @@ Before adding a model you can test the connection to the LLM provider to verify
|
||||
|
||||
info
|
||||
|
||||
Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/#trial)
|
||||
Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
**No call needed**
|
||||
|
||||
@@ -5815,7 +5815,7 @@ You can now set custom parameters (like success threshold) for your guardrails i
|
||||
|
||||
info
|
||||
|
||||
Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/#trial)
|
||||
Get a free 7-day LiteLLM Enterprise trial here. [Start here](https://www.litellm.ai/enterprise#trial)
|
||||
|
||||
**No call needed**
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user