tiennm99/litellm
1
0
Fork 0
mirror of https://github.com/tiennm99/litellm.git synced 2026-07-13 05:06:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into litellm_perf_skip_throwaway_usage

Browse Source 
# Conflicts:
#	tests/llm_translation/test_llm_response_utils/test_convert_dict_to_chat_completion.py
This commit is contained in:
Ryan Crabbe
2026-02-24 14:51:57 -08:00
parent 7414c09277 5219b1d0c3
commit 6a0dd0a45f
1150 changed files with 70363 additions and 19984 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.circleci/config.yml
+1
View File
@@ -1456,6 +1456,7 @@ jobs:
pip install "respx==0.22.0"
pip install "pydantic==2.10.2"
pip install "boto3==1.36.0"
pip install "semantic_router==0.1.10"
# Run pytest and generate JUnit XML report
- run:
name: Run tests
.github/ISSUE_TEMPLATE/config.yml
+1 -1
View File
@@ -1,7 +1,7 @@
blank_issues_enabled: true
contact_links:
- name: Schedule Demo
url: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
url: https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions
about: Speak directly with Krrish and Ishaan, the founders, to discuss issues, share feedback, or explore improvements for LiteLLM
- name: Discord
url: https://discord.com/invite/wuPM9dRgDw
.github/workflows/check_duplicate_issues.yml
+2 -2
View File
@@ -20,10 +20,10 @@ jobs:
reaction: eyes
comment: |
**⚠️ Potential duplicate detected**
This issue appears similar to existing issue(s):
{{#issues}}
- [#{{number}}]({{html_url}}) - {{title}} ({{accuracy}}% similar)
{{/issues}}
Please review the linked issue(s) to see if they address your concern. If this is not a duplicate, please provide additional context to help us understand the difference.
.github/workflows/interpret_load_test.py
+1 -1
View File
@@ -123,7 +123,7 @@ if __name__ == "__main__":
+ docker_run_command
+ "\n\n"
+ "### Don't want to maintain your internal proxy? get in touch 🎉"
+ "\nHosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat"
+ "\nHosted Proxy Alpha: https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions"
+ "\n\n"
+ "## Load Test LiteLLM Proxy Results"
+ "\n\n"
.github/workflows/regenerate-poetry-lock.yml
+80
View File
@@ -0,0 +1,80 @@
name: Regenerate poetry.lock
# Runs whenever pyproject.toml is merged into main (the most common cause of
# the "pyproject.toml changed significantly since poetry.lock was last generated"
# CI failure). Can also be triggered manually.
on:
push:
branches:
- main
paths:
- pyproject.toml
workflow_dispatch:
permissions:
contents: write # needed to push the auto/regenerate-poetry-lock-* branch
pull-requests: write # needed to open the PR and enable auto-merge
jobs:
regenerate-lock:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install Poetry
run: pip install poetry
- name: Regenerate poetry.lock
run: poetry lock
- name: Check whether poetry.lock actually changed
id: diff
run: |
if git diff --quiet poetry.lock; then
echo "changed=false" >> "$GITHUB_OUTPUT"
else
echo "changed=true" >> "$GITHUB_OUTPUT"
fi
- name: Open PR with the refreshed lock file
if: steps.diff.outputs.changed == 'true'
id: open-pr
run: |
BRANCH="auto/regenerate-poetry-lock-$(date +'%Y%m%d%H%M%S')"
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git checkout -b "$BRANCH"
git add poetry.lock
git commit -m "chore: regenerate poetry.lock to match pyproject.toml"
git push -f origin "$BRANCH"
cat > /tmp/pr-body.md << 'BODY'
Automated regeneration of `poetry.lock` after `pyproject.toml` was updated on `main`.
Fixes the recurring CI failure:
```
pyproject.toml changed significantly since poetry.lock was last generated.
Run `poetry lock` to fix the lock file.
```
BODY
PR_URL=$(gh pr create \
--title "chore: regenerate poetry.lock to match pyproject.toml" \
--body-file /tmp/pr-body.md \
--head "$BRANCH" \
--base main)
echo "pr_url=$PR_URL" >> "$GITHUB_OUTPUT"
env:
GH_TOKEN: ${{ github.token }}
- name: Enable auto-merge
if: steps.diff.outputs.changed == 'true'
run: |
gh pr merge "${{ steps.open-pr.outputs.pr_url }}" --auto --squash
env:
GH_TOKEN: ${{ github.token }}
.github/workflows/test-litellm-matrix.yml
+54 -6
View File
@@ -48,8 +48,19 @@ jobs:
path: "tests/test_litellm/litellm_core_utils"
workers: 2
reruns: 1
- name: "other"
path: "tests/test_litellm/caching tests/test_litellm/responses tests/test_litellm/secret_managers tests/test_litellm/vector_stores tests/test_litellm/a2a_protocol tests/test_litellm/anthropic_interface tests/test_litellm/completion_extras tests/test_litellm/containers tests/test_litellm/enterprise tests/test_litellm/experimental_mcp_client tests/test_litellm/google_genai tests/test_litellm/images tests/test_litellm/interactions tests/test_litellm/passthrough tests/test_litellm/router_strategy tests/test_litellm/router_utils tests/test_litellm/types"
- name: "other-1"
# responses (5942) + caching (1723) + types (819) ≈ 8.5k lines
path: "tests/test_litellm/responses tests/test_litellm/caching tests/test_litellm/types"
workers: 2
reruns: 2
- name: "other-2"
# enterprise (3062) + google_genai (2511) + router_utils (1982) ≈ 7.6k lines
path: "tests/test_litellm/enterprise tests/test_litellm/google_genai tests/test_litellm/router_utils"
workers: 2
reruns: 2
- name: "other-3"
# remaining dirs ≈ 8.0k lines
path: "tests/test_litellm/router_strategy tests/test_litellm/secret_managers tests/test_litellm/a2a_protocol tests/test_litellm/anthropic_interface tests/test_litellm/completion_extras tests/test_litellm/containers tests/test_litellm/experimental_mcp_client tests/test_litellm/images tests/test_litellm/interactions tests/test_litellm/passthrough tests/test_litellm/vector_stores"
workers: 2
reruns: 2
- name: "root"
@@ -57,12 +68,49 @@ jobs:
workers: 2
reruns: 2
# tests/proxy_unit_tests split alphabetically (~48 files total)
- name: "proxy-unit-a"
path: "tests/proxy_unit_tests/test_[a-o]*.py"
- name: "proxy-unit-a1"
# test_[a-j]*.py: jwt (1564) + auth_checks (978) + google_gemini (478) + e2e_pod_lock (437) + rest
path: "tests/proxy_unit_tests/test_[a-j]*.py"
workers: 2
reruns: 1
- name: "proxy-unit-b"
path: "tests/proxy_unit_tests/test_[p-z]*.py"
- name: "proxy-unit-a2"
# test_[k-o]*.py: key_generate_prisma (4346) + key_generate_dynamodb + models_fallback
path: "tests/proxy_unit_tests/test_[k-o]*.py"
workers: 2
reruns: 1
- name: "proxy-unit-b1"
# lighter config/utility proxy tests (prisma, project, prompt, proxy_[c-r]*)
path: "tests/proxy_unit_tests/test_prisma*.py tests/proxy_unit_tests/test_project*.py tests/proxy_unit_tests/test_prompt*.py tests/proxy_unit_tests/test_proxy_[c-r]*.py"
workers: 2
reruns: 1
- name: "proxy-unit-b2"
# proxy_server.py alone (2750 lines) - isolated to avoid blocking smaller tests
path: "tests/proxy_unit_tests/test_proxy_server.py"
workers: 2
reruns: 1
- name: "proxy-unit-b3"
# proxy_server_* (618) + proxy_setting_guardrails (71) - smaller server-related tests
path: "tests/proxy_unit_tests/test_proxy_server_*.py tests/proxy_unit_tests/test_proxy_setting_guardrails.py"
workers: 2
reruns: 1
- name: "proxy-unit-b4"
# proxy_utils.py alone (2339 lines) - isolated to avoid blocking token counter
path: "tests/proxy_unit_tests/test_proxy_utils.py"
workers: 2
reruns: 1
- name: "proxy-unit-b5"
# proxy_token_counter (1279) - runs independently from utils
path: "tests/proxy_unit_tests/test_proxy_token_counter.py"
workers: 2
reruns: 1
- name: "proxy-unit-b6"
# test_[r-t]*.py: response_polling (1399) + search_api_logging (202) + server_root (64) + skills_db (261) + realtime_cache (62)
path: "tests/proxy_unit_tests/test_[r-t]*.py"
workers: 2
reruns: 1
- name: "proxy-unit-b7"
# test_[u-z]*.py: user_api_key_auth (1136) + zero_cost (590) + update_spend (305) + unit_test_* (206) + ui_path (157)
path: "tests/proxy_unit_tests/test_[u-z]*.py"
workers: 2
reruns: 1
Dockerfile
+16 -4
View File
@@ -49,7 +49,7 @@ USER root
# Install runtime dependencies (libsndfile needed for audio processing on ARM64)
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile && \
npm install -g npm@latest tar@7.5.7 glob@11.1.0 @isaacs/brace-expansion@5.0.1 && \
npm install -g npm@latest tar@7.5.8 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.1 diff@8.0.3 && \
# SECURITY FIX: npm bundles tar, glob, and brace-expansion at multiple nested
# levels inside its dependency tree. `npm install -g <pkg>` only creates a
# SEPARATE global package, it does NOT replace npm's internal copies.
@@ -64,6 +64,12 @@ RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile
find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done && \
find "$GLOBAL/npm" -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
done && \
find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done && \
npm cache clean --force
WORKDIR /app
@@ -90,14 +96,20 @@ RUN find /usr/lib -type f -path "*/tornado/test/*" -delete && \
# npm with old vulnerable deps at /usr/lib/python3.*/site-packages/nodejs_wheel/.
# Patch every copy of tar, glob, and brace-expansion inside that tree.
RUN GLOBAL="$(npm root -g)" && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/tar" -type d | while read d; do \
find /usr/lib -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/glob" -type d | while read d; do \
find /usr/lib -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/@isaacs/brace-expansion" -type d | while read d; do \
find /usr/lib -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done && \
find /usr/lib -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
done && \
find /usr/lib -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done
# Install semantic_router and aurelio-sdk using script
README.md
+1 -1
View File
@@ -399,7 +399,7 @@ Support for more providers. Missing a provider or LLM Platform, raise a [feature
# Enterprise
For companies that need better security, user management and professional support
[Talk to founders](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[Talk to founders](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
This covers:
- ✅ **Features under the [LiteLLM Commercial License](https://docs.litellm.ai/docs/proxy/enterprise):**
ci_cd/security_scans.sh
+3
View File
@@ -158,6 +158,9 @@ run_grype_scans() {
"CVE-2025-11468" # No fix available yet
"CVE-2026-1299" # Python 3.13 email module header injection - not applicable, LiteLLM doesn't use BytesGenerator for email serialization
"CVE-2026-0775" # npm cli incorrect permission assignment - no fix available yet, npm is only used at build/prisma-generate time
"GHSA-3ppc-4f35-3m26" # minimatch ReDoS via repeated wildcards - from nodejs_wheel bundled npm, not used in application runtime code
"GHSA-83g3-92jg-28cx" # tar arbitrary file read/write via hardlink - from nodejs_wheel bundled npm, not used in application runtime code
"CVE-2026-25639" # axios - full fix requires 1.x major version bump; pinned to >=0.30.2 to clear other axios CVEs, upgrade to 1.x in follow-up
)
# Build JSON array of allowlisted CVE IDs for jq
cookbook/benchmark/readme.md
+1 -1
View File
@@ -178,4 +178,4 @@ Benchmark Results for 'When will BerriAI IPO?':
```
## Support
**🤝 Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
**🤝 Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
cookbook/gollem_go_agent_framework/README.md
+119
View File
@@ -0,0 +1,119 @@
# Gollem Go Agent Framework with LiteLLM
A working example showing how to use [gollem](https://github.com/fugue-labs/gollem), a production-grade Go agent framework, with LiteLLM as a proxy gateway. This lets Go developers access 100+ LLM providers through a single proxy while keeping compile-time type safety for tools and structured output.
## Quick Start
### 1. Start LiteLLM Proxy
```bash
# Simple start with a single model
litellm --model gpt-4o
# Or with the example config for multi-provider access
litellm --config proxy_config.yaml
```
### 2. Run the examples
```bash
# Install Go dependencies
go mod tidy
# Basic agent
go run ./basic
# Agent with type-safe tools
go run ./tools
# Streaming responses
go run ./streaming
```
## Configuration
The included `proxy_config.yaml` sets up three providers through LiteLLM:
```yaml
model_list:
- model_name: gpt-4o # OpenAI
- model_name: claude-sonnet # Anthropic
- model_name: gemini-pro # Google Vertex AI
```
Switch providers in Go by changing a single string — no code changes needed:
```go
model := openai.NewLiteLLM("http://localhost:4000",
openai.WithModel("gpt-4o"), // OpenAI
// openai.WithModel("claude-sonnet"), // Anthropic
// openai.WithModel("gemini-pro"), // Google
)
```
## Examples
### `basic/` — Basic Agent
Connects gollem to LiteLLM and runs a simple prompt. Demonstrates the `NewLiteLLM` constructor and basic agent creation.
### `tools/` — Type-Safe Tools
Shows gollem's compile-time type-safe tool framework working through LiteLLM's tool-use passthrough. The tool parameters are Go structs with JSON tags — the schema is generated automatically at compile time.
### `streaming/` — Streaming Responses
Real-time token streaming using Go 1.23+ range-over-function iterators, proxied through LiteLLM's SSE passthrough.
## How It Works
Gollem's `openai.NewLiteLLM()` constructor creates an OpenAI-compatible provider pointed at your LiteLLM proxy. Since LiteLLM speaks the OpenAI API protocol, everything works out of the box:
- **Chat completions** — standard request/response
- **Tool use** — LiteLLM passes tool definitions and calls through transparently
- **Streaming** — Server-Sent Events proxied through LiteLLM
- **Structured output** — JSON schema response format works with supporting models
```
Go App (gollem) → LiteLLM Proxy → OpenAI / Anthropic / Google / ...
```
## Why Use This?
- **Type-safe Go**: Compile-time type checking for tools, structured output, and agent configuration — no runtime surprises
- **Single proxy, many models**: Switch between OpenAI, Anthropic, Google, and 100+ other providers by changing a model name string
- **Zero-dependency core**: gollem's core has no external dependencies — just stdlib
- **Single binary deployment**: `go build` produces one binary, no pip/venv/Docker needed
- **Cost tracking & rate limiting**: LiteLLM handles cost tracking, rate limits, and fallbacks at the proxy layer
## Environment Variables
```bash
# Required for providers you want to use (set in LiteLLM config or env)
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
# Optional: point to a non-default LiteLLM proxy
export LITELLM_PROXY_URL="http://localhost:4000"
```
## Troubleshooting
**Connection errors?**
- Make sure LiteLLM is running: `litellm --model gpt-4o`
- Check the URL is correct (default: `http://localhost:4000`)
**Model not found?**
- Verify the model name matches what's configured in LiteLLM
- Run `curl http://localhost:4000/models` to see available models
**Tool calls not working?**
- Ensure the underlying model supports tool use (GPT-4o, Claude, Gemini)
- Check LiteLLM logs for any provider-specific errors
## Learn More
- [gollem GitHub](https://github.com/fugue-labs/gollem)
- [gollem API Reference](https://pkg.go.dev/github.com/fugue-labs/gollem/core)
- [LiteLLM Proxy Docs](https://docs.litellm.ai/docs/simple_proxy)
- [LiteLLM Supported Models](https://docs.litellm.ai/docs/providers)
cookbook/gollem_go_agent_framework/basic/main.go
+41
View File
@@ -0,0 +1,41 @@
// Basic gollem agent connected to a LiteLLM proxy.
//
// Usage:
//
// litellm --model gpt-4o # start proxy in another terminal
// go run ./basic
package main
import (
"context"
"fmt"
"log"
"os"
"github.com/fugue-labs/gollem/core"
"github.com/fugue-labs/gollem/provider/openai"
)
func main() {
proxyURL := "http://localhost:4000"
if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
proxyURL = u
}
// Connect to LiteLLM proxy. NewLiteLLM creates an OpenAI-compatible
// provider pointed at the given URL.
model := openai.NewLiteLLM(proxyURL,
openai.WithModel("gpt-4o"), // any model name configured in LiteLLM
)
// Create and run a simple agent.
agent := core.NewAgent[string](model,
core.WithSystemPrompt[string]("You are a helpful assistant. Be concise."),
)
result, err := agent.Run(context.Background(), "Explain quantum computing in two sentences.")
if err != nil {
log.Fatal(err)
}
fmt.Println(result.Output)
}
cookbook/gollem_go_agent_framework/go.mod
+5
View File
@@ -0,0 +1,5 @@
module github.com/BerriAI/litellm/cookbook/gollem_go_agent_framework
go 1.25.1
require github.com/fugue-labs/gollem v0.1.0
cookbook/gollem_go_agent_framework/go.sum
+2
View File
@@ -0,0 +1,2 @@
github.com/fugue-labs/gollem v0.1.0 h1:QexYnvkb44QZFEljgAePqMIGZjgsbk0Y5GJ2jYYgfa8=
github.com/fugue-labs/gollem v0.1.0/go.mod h1:htW1YO81uysSKVOkYJtxhGCFrzm+36HBFxEWuECoHKQ=
cookbook/gollem_go_agent_framework/proxy_config.yaml
+16
View File
@@ -0,0 +1,16 @@
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: gemini-pro
litellm_params:
model: vertex_ai/gemini-2.0-flash
vertex_project: my-project
vertex_location: us-central1
cookbook/gollem_go_agent_framework/streaming/main.go
+56
View File
@@ -0,0 +1,56 @@
// Streaming responses from gollem through LiteLLM.
//
// Uses Go 1.23+ range-over-function iterators for real-time token
// streaming via LiteLLM's SSE passthrough.
//
// Usage:
//
// litellm --model gpt-4o
// go run ./streaming
package main
import (
"context"
"fmt"
"log"
"os"
"github.com/fugue-labs/gollem/core"
"github.com/fugue-labs/gollem/provider/openai"
)
func main() {
proxyURL := "http://localhost:4000"
if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
proxyURL = u
}
model := openai.NewLiteLLM(proxyURL,
openai.WithModel("gpt-4o"),
)
agent := core.NewAgent[string](model)
// RunStream returns a streaming result that yields tokens as they arrive.
stream, err := agent.RunStream(context.Background(), "Write a haiku about distributed systems")
if err != nil {
log.Fatal(err)
}
// StreamText yields text chunks in real-time.
// The boolean argument controls whether deltas (true) or accumulated
// text (false) is returned.
fmt.Print("Response: ")
for text, err := range stream.StreamText(true) {
if err != nil {
log.Fatal(err)
}
fmt.Print(text)
}
fmt.Println()
// After streaming completes, the final response is available.
resp := stream.Response()
fmt.Printf("\nTokens used: input=%d, output=%d\n",
resp.Usage.InputTokens, resp.Usage.OutputTokens)
}
cookbook/gollem_go_agent_framework/tools/main.go
+64
View File
@@ -0,0 +1,64 @@
// Gollem agent with type-safe tools through LiteLLM.
//
// The tool parameters are Go structs — gollem generates the JSON schema
// automatically at compile time. LiteLLM passes tool definitions through
// transparently to the underlying provider.
//
// Usage:
//
// litellm --model gpt-4o
// go run ./tools
package main
import (
"context"
"fmt"
"log"
"os"
"github.com/fugue-labs/gollem/core"
"github.com/fugue-labs/gollem/provider/openai"
)
// WeatherParams defines the tool's input schema via struct tags.
// The JSON schema is generated at compile time — no runtime reflection needed.
type WeatherParams struct {
City string `json:"city" description:"City name to get weather for"`
Unit string `json:"unit,omitempty" description:"Temperature unit: celsius or fahrenheit"`
}
func main() {
proxyURL := "http://localhost:4000"
if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
proxyURL = u
}
model := openai.NewLiteLLM(proxyURL,
openai.WithModel("gpt-4o"),
)
// Define a type-safe tool. The function signature enforces correct types.
weatherTool := core.FuncTool[WeatherParams](
"get_weather",
"Get current weather for a city",
func(ctx context.Context, p WeatherParams) (string, error) {
unit := p.Unit
if unit == "" {
unit = "fahrenheit"
}
// In production, call a real weather API here.
return fmt.Sprintf("Weather in %s: 72°F (22°C), sunny", p.City), nil
},
)
agent := core.NewAgent[string](model,
core.WithTools[string](weatherTool),
core.WithSystemPrompt[string]("You are a helpful weather assistant. Use the get_weather tool to answer weather questions."),
)
result, err := agent.Run(context.Background(), "What's the weather like in San Francisco and Tokyo?")
if err != nil {
log.Fatal(err)
}
fmt.Println(result.Output)
}
deploy/charts/litellm-helm/README.md
+4
View File
@@ -36,6 +36,10 @@ If `db.useStackgresOperator` is used (not yet implemented):
| `serviceAccount.create` | Whether or not to create a Kubernetes Service Account for this deployment. The default is `false` because LiteLLM has no need to access the Kubernetes API. | `false` |
| `service.type` | Kubernetes Service type (e.g. `LoadBalancer`, `ClusterIP`, etc.) | `ClusterIP` |
| `service.port` | TCP port that the Kubernetes Service will listen on. Also the TCP port within the Pod that the proxy will listen on. | `4000` |
| `livenessProbe.*` | Liveness probe settings for the LiteLLM container (`path`, `periodSeconds`, `timeoutSeconds`, thresholds, and initial delay). | See `values.yaml` |
| `readinessProbe.*` | Readiness probe settings for the LiteLLM container (`path`, `periodSeconds`, `timeoutSeconds`, thresholds, and initial delay). | See `values.yaml` |
| `startupProbe.*` | Startup probe settings for the LiteLLM container (`path`, `periodSeconds`, `timeoutSeconds`, thresholds, and initial delay). | See `values.yaml` |
| `resources.*` | CPU/memory requests and limits for the LiteLLM container. | `{}` |
| `service.loadBalancerClass` | Optional LoadBalancer implementation class (only used when `service.type` is `LoadBalancer`) | `""` |
| `ingress.labels` | Additional labels for the Ingress resource | `{}` |
| `ingress.*` | See [values.yaml](./values.yaml) for example settings | N/A |
deploy/charts/litellm-helm/templates/configmap-litellm.yaml
+1 -1
View File
@@ -6,4 +6,4 @@ metadata:
data:
config.yaml: |
{{ .Values.proxy_config | toYaml | indent 6 }}
{{- end }}
{{- end }}
deploy/charts/litellm-helm/templates/deployment.yaml
+19 -6
View File
@@ -158,18 +158,31 @@ spec:
{{- end }}
livenessProbe:
httpGet:
path: /health/liveliness
path: {{ .Values.livenessProbe.path | quote }}
port: {{ if .Values.separateHealthApp }}"health"{{ else }}"http"{{ end }}
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
successThreshold: {{ .Values.livenessProbe.successThreshold }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
readinessProbe:
httpGet:
path: /health/readiness
path: {{ .Values.readinessProbe.path | quote }}
port: {{ if .Values.separateHealthApp }}"health"{{ else }}"http"{{ end }}
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
successThreshold: {{ .Values.readinessProbe.successThreshold }}
failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
startupProbe:
httpGet:
path: /health/readiness
path: {{ .Values.startupProbe.path | quote }}
port: {{ if .Values.separateHealthApp }}"health"{{ else }}"http"{{ end }}
failureThreshold: 30
periodSeconds: 10
initialDelaySeconds: {{ .Values.startupProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.startupProbe.periodSeconds }}
timeoutSeconds: {{ .Values.startupProbe.timeoutSeconds }}
successThreshold: {{ .Values.startupProbe.successThreshold }}
failureThreshold: {{ .Values.startupProbe.failureThreshold }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
volumeMounts:
@@ -235,4 +248,4 @@ spec:
{{- if .Values.topologySpreadConstraints }}
topologySpreadConstraints:
{{- toYaml .Values.topologySpreadConstraints | nindent 8 }}
{{- end }}
{{- end }}
deploy/charts/litellm-helm/tests/deployment_tests.yaml
+147 -1
View File
@@ -159,4 +159,150 @@ tests:
value: -c
- equal:
path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2]
value: echo "Container stopping"
value: echo "Container stopping"
- it: should render background health check settings from proxy_config.general_settings
template: configmap-litellm.yaml
set:
proxy_config.general_settings.background_health_checks: true
proxy_config.general_settings.health_check_interval: 240
proxy_config.general_settings.health_check_concurrency: 16
proxy_config.general_settings.health_check_details: false
asserts:
- matchRegex:
path: data["config.yaml"]
pattern: '(?m)^\s*background_health_checks:\s*true$'
- matchRegex:
path: data["config.yaml"]
pattern: '(?m)^\s*health_check_interval:\s*240$'
- matchRegex:
path: data["config.yaml"]
pattern: '(?m)^\s*health_check_concurrency:\s*16$'
- matchRegex:
path: data["config.yaml"]
pattern: '(?m)^\s*health_check_details:\s*false$'
- it: should allow overriding liveness, readiness, and startup probes
template: deployment.yaml
set:
livenessProbe:
path: /custom/livez
initialDelaySeconds: 5
periodSeconds: 15
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
path: /custom/readyz
initialDelaySeconds: 10
periodSeconds: 20
timeoutSeconds: 6
successThreshold: 1
failureThreshold: 6
startupProbe:
path: /custom/startupz
initialDelaySeconds: 15
periodSeconds: 25
timeoutSeconds: 7
successThreshold: 1
failureThreshold: 40
asserts:
- equal:
path: spec.template.spec.containers[0].livenessProbe.httpGet.path
value: /custom/livez
- equal:
path: spec.template.spec.containers[0].livenessProbe.timeoutSeconds
value: 5
- equal:
path: spec.template.spec.containers[0].readinessProbe.httpGet.path
value: /custom/readyz
- equal:
path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds
value: 6
- equal:
path: spec.template.spec.containers[0].startupProbe.httpGet.path
value: /custom/startupz
- equal:
path: spec.template.spec.containers[0].startupProbe.failureThreshold
value: 40
- it: should render container resources from values
template: deployment.yaml
set:
resources:
limits:
cpu: 500m
memory: 2Gi
requests:
cpu: 250m
memory: 1Gi
asserts:
- equal:
path: spec.template.spec.containers[0].resources.limits.cpu
value: 500m
- equal:
path: spec.template.spec.containers[0].resources.limits.memory
value: 2Gi
- equal:
path: spec.template.spec.containers[0].resources.requests.cpu
value: 250m
- equal:
path: spec.template.spec.containers[0].resources.requests.memory
value: 1Gi
- it: should keep default probes and empty resources unchanged
template: deployment.yaml
asserts:
- equal:
path: spec.template.spec.containers[0].livenessProbe.httpGet.path
value: /health/liveliness
- equal:
path: spec.template.spec.containers[0].livenessProbe.initialDelaySeconds
value: 0
- equal:
path: spec.template.spec.containers[0].livenessProbe.periodSeconds
value: 10
- equal:
path: spec.template.spec.containers[0].livenessProbe.timeoutSeconds
value: 1
- equal:
path: spec.template.spec.containers[0].livenessProbe.successThreshold
value: 1
- equal:
path: spec.template.spec.containers[0].livenessProbe.failureThreshold
value: 3
- equal:
path: spec.template.spec.containers[0].readinessProbe.httpGet.path
value: /health/readiness
- equal:
path: spec.template.spec.containers[0].readinessProbe.initialDelaySeconds
value: 0
- equal:
path: spec.template.spec.containers[0].readinessProbe.periodSeconds
value: 10
- equal:
path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds
value: 1
- equal:
path: spec.template.spec.containers[0].readinessProbe.successThreshold
value: 1
- equal:
path: spec.template.spec.containers[0].readinessProbe.failureThreshold
value: 3
- equal:
path: spec.template.spec.containers[0].startupProbe.httpGet.path
value: /health/readiness
- equal:
path: spec.template.spec.containers[0].startupProbe.initialDelaySeconds
value: 0
- equal:
path: spec.template.spec.containers[0].startupProbe.periodSeconds
value: 10
- equal:
path: spec.template.spec.containers[0].startupProbe.timeoutSeconds
value: 1
- equal:
path: spec.template.spec.containers[0].startupProbe.successThreshold
value: 1
- equal:
path: spec.template.spec.containers[0].startupProbe.failureThreshold
value: 30
- equal:
path: spec.template.spec.containers[0].resources
value: {}
deploy/charts/litellm-helm/values.yaml
+25
View File
@@ -84,6 +84,31 @@ service:
separateHealthApp: false
separateHealthPort: 8081
# Probe tuning for proxy container
livenessProbe:
path: /health/liveliness
initialDelaySeconds: 0
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 3
readinessProbe:
path: /health/readiness
initialDelaySeconds: 0
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 3
startupProbe:
path: /health/readiness
initialDelaySeconds: 0
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 30
ingress:
enabled: false
className: "nginx"
docker/Dockerfile.custom_ui
+21 -2
View File
@@ -5,8 +5,21 @@ FROM ghcr.io/berriai/litellm:litellm_fwd_server_root_path-dev
WORKDIR /app
# Install Node.js and npm (adjust version as needed)
RUN apt-get update && apt-get install -y nodejs npm && \
npm install -g npm@latest tar@7.5.7 glob@11.1.0 @isaacs/brace-expansion@5.0.1 && \
RUN apt-get update && apt-get upgrade -y \
libxml2 \
libexpat1 \
openssl \
libssl3 \
git \
libkrb5-3 \
libglib2.0-0 \
wget \
libaom3 \
libxslt1.1 \
libgnutls30 \
libc6 && \
apt-get install -y nodejs npm && \
npm install -g npm@latest tar@7.5.8 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.1 diff@8.0.3 && \
GLOBAL="$(npm root -g)" && \
find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
@@ -17,6 +30,12 @@ RUN apt-get update && apt-get install -y nodejs npm && \
find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done && \
find "$GLOBAL/npm" -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
done && \
find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done && \
npm cache clean --force
# Copy the UI source into the container
docker/Dockerfile.database
+16 -4
View File
@@ -50,7 +50,7 @@ USER root
# Install runtime dependencies
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile && \
npm install -g npm@latest tar@7.5.7 glob@11.1.0 @isaacs/brace-expansion@5.0.1 && \
npm install -g npm@latest tar@7.5.8 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.1 diff@8.0.3 && \
GLOBAL="$(npm root -g)" && \
find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
@@ -61,6 +61,12 @@ RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip libsndfile
find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done && \
find "$GLOBAL/npm" -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
done && \
find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done && \
npm cache clean --force
WORKDIR /app
@@ -79,14 +85,20 @@ RUN pip install *.whl /wheels/* --no-index --find-links=/wheels/ && rm -f *.whl
# npm with old vulnerable deps at /usr/lib/python3.*/site-packages/nodejs_wheel/.
# Patch every copy of tar, glob, and brace-expansion inside that tree.
RUN GLOBAL="$(npm root -g)" && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/tar" -type d | while read d; do \
find /usr/lib -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/glob" -type d | while read d; do \
find /usr/lib -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/@isaacs/brace-expansion" -type d | while read d; do \
find /usr/lib -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done && \
find /usr/lib -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
done && \
find /usr/lib -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done
# Install semantic_router and aurelio-sdk using script
docker/Dockerfile.dev
+31 -6
View File
@@ -56,13 +56,26 @@ FROM $LITELLM_RUNTIME_IMAGE AS runtime
USER root
# Install only runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
libssl3 \
RUN apt-get update && apt-get upgrade -y \
libxml2 \
libexpat1 \
openssl \
libssl3 \
git \
libkrb5-3 \
libglib2.0-0 \
wget \
libaom3 \
libxslt1.1 \
libgnutls30 \
libc6 \
&& apt-get install -y --no-install-recommends \
libssl3 \
libatomic1 \
nodejs \
npm \
&& rm -rf /var/lib/apt/lists/* \
&& npm install -g npm@latest tar@7.5.7 glob@11.1.0 @isaacs/brace-expansion@5.0.1 \
&& npm install -g npm@latest tar@7.5.8 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.1 diff@8.0.3 \
&& GLOBAL="$(npm root -g)" \
&& find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
@@ -73,6 +86,12 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done \
&& find "$GLOBAL/npm" -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
done \
&& find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done \
&& npm cache clean --force
WORKDIR /app
@@ -95,14 +114,20 @@ RUN pip install --no-cache-dir *.whl /wheels/* --no-index --find-links=/wheels/
# npm with old vulnerable deps at /usr/lib/python3.*/site-packages/nodejs_wheel/.
# Patch every copy of tar, glob, and brace-expansion inside that tree.
RUN GLOBAL="$(npm root -g)" && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/tar" -type d | while read d; do \
find /usr/lib -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/glob" -type d | while read d; do \
find /usr/lib -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/@isaacs/brace-expansion" -type d | while read d; do \
find /usr/lib -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done && \
find /usr/lib -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
done && \
find /usr/lib -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done
# Generate prisma client and set permissions
docker/Dockerfile.non_root
+18 -5
View File
@@ -80,7 +80,7 @@ ENV PRISMA_BINARY_CACHE_DIR=/app/.cache/prisma-python/binaries \
XDG_CACHE_HOME=/app/.cache \
PATH="/usr/lib/python3.13/site-packages/nodejs/bin:${PATH}"
RUN pip install --no-cache-dir prisma==0.11.0 nodejs-wheel-binaries==24.12.0 \
RUN pip install --no-cache-dir prisma==0.11.0 nodejs-wheel-binaries==24.13.1 \
&& mkdir -p /app/.cache/npm
RUN NPM_CONFIG_CACHE=/app/.cache/npm \
@@ -105,7 +105,8 @@ RUN for i in 1 2 3; do \
&& for i in 1 2 3; do \
apk add --no-cache python3 py3-pip bash openssl tzdata nodejs npm supervisor && break || sleep 5; \
done \
&& npm install -g npm@latest tar@7.5.7 glob@11.1.0 @isaacs/brace-expansion@5.0.1 \
&& apk upgrade --no-cache nodejs \
&& npm install -g npm@latest tar@7.5.8 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.1 diff@8.0.3 \
&& GLOBAL="$(npm root -g)" \
&& find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
@@ -116,6 +117,12 @@ RUN for i in 1 2 3; do \
&& find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done \
&& find "$GLOBAL/npm" -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
done \
&& find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done \
&& npm cache clean --force
# Copy artifacts from builder
@@ -162,14 +169,20 @@ RUN pip install --no-index --find-links=/wheels/ -r requirements.txt && \
# npm with old vulnerable deps at /usr/lib/python3.*/site-packages/nodejs_wheel/.
# Patch every copy of tar, glob, and brace-expansion inside that tree.
RUN GLOBAL="$(npm root -g)" && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/tar" -type d | while read d; do \
find /usr/lib -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/glob" -type d | while read d; do \
find /usr/lib -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
done && \
find /usr/lib -path "*/nodejs_wheel/*/node_modules/@isaacs/brace-expansion" -type d | while read d; do \
find /usr/lib -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
done && \
find /usr/lib -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
done && \
find /usr/lib -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
done
# Permissions, cleanup, and Prisma prep
docs/my-website/blog/anthropic_wildcard_model_access_incident/index.md
+147
View File
@@ -0,0 +1,147 @@
---
slug: anthropic-wildcard-model-access-incident
title: "Incident Report: Wildcard Blocking New Models After Cost Map Reload"
date: 2026-02-23T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
tags: [incident-report, proxy, auth, model-access]
hide_table_of_contents: false
---
**Date:** Feb 23, 2026
**Duration:** ~3 hours
**Severity:** High (for users with provider wildcard access rules)
**Status:** Resolved
## Summary
When a new Anthropic model (e.g. `claude-sonnet-4-6`) was added to the LiteLLM model cost map and a cost map reload was triggered, requests to the new model were rejected with:
```
key not allowed to access model. This key can only access models=['anthropic/*']. Tried to access claude-sonnet-4-6.
```
The reload updated `litellm.model_cost` correctly but never re-ran `add_known_models()`, so `litellm.anthropic_models` (the in-memory set used by the wildcard resolver) remained stale. The new model was invisible to the `anthropic/*` wildcard even though the cost map knew about it.
- **LLM calls:** All requests to newly-added Anthropic models were blocked with a 401.
- **Existing models:** Unaffected — only models missing from the stale provider set were impacted.
- **Other providers:** Same bug class existed for any provider wildcard (e.g. `openai/*`, `gemini/*`).
{/* truncate */}
---
## Background
LiteLLM supports provider-level wildcard access rules. When an admin configures a key or team with `models=['anthropic/*']`, any model whose provider resolves to `anthropic` should be allowed. The resolution happens in `_model_custom_llm_provider_matches_wildcard_pattern`:
```mermaid
flowchart TD
A["1. Request arrives for claude-sonnet-4-6"] --> B["2. Auth check: can this key call this model?
proxy/auth/auth_checks.py"]
B --> C["3. Key has models=['anthropic/*']
→ wildcard match attempted"]
C --> D["4. get_llm_provider('claude-sonnet-4-6')
checks litellm.anthropic_models set"]
D -->|"model IN set"| E["5a. ✅ Provider = 'anthropic'
→ 'anthropic/claude-sonnet-4-6' matches 'anthropic/*'"]
D -->|"model NOT IN set"| F["5b. ❌ Provider unknown
→ exception raised → wildcard returns False"]
E --> G["6. Request allowed"]
F --> H["6. 401: key not allowed to access model"]
style E fill:#d4edda,stroke:#28a745
style F fill:#f8d7da,stroke:#dc3545
style H fill:#f8d7da,stroke:#dc3545
style D fill:#fff3cd,stroke:#ffc107
```
`litellm.anthropic_models` is a Python `set` populated at import time by `add_known_models()`. It is the source `get_llm_provider()` consults to map a bare model name like `claude-sonnet-4-6` to the provider string `"anthropic"`.
---
## Root Cause
`add_known_models()` is called **once** at module import time. Both reload paths in `proxy_server.py` updated `litellm.model_cost` with the fresh map but never called `add_known_models()` again:
```python
# Before the fix — both reload paths looked like this:
new_model_cost_map = get_model_cost_map(url=model_cost_map_url)
litellm.model_cost = new_model_cost_map # ✅ cost map updated
_invalidate_model_cost_lowercase_map() # ✅ cache cleared
# ❌ add_known_models() never called
# → litellm.anthropic_models still has the old set
# → new model not in the set
# → get_llm_provider() raises for the new model
# → wildcard match returns False
# → 401 for every request to the new model
```
The gap existed in two places:
1. `_check_and_reload_model_cost_map` — the periodic automatic reload (every 10 s)
2. The `/reload/model_cost_map` admin endpoint — the manual reload
**Timeline:**
1. New model (`claude-sonnet-4-6`) added to `model_prices_and_context_window.json`
2. Admin triggers cost map reload via UI → `litellm.model_cost` updated
3. Users with `anthropic/*` wildcard keys attempt requests to `claude-sonnet-4-6`
4. `get_llm_provider('claude-sonnet-4-6')` raises → wildcard returns False → 401
5. Admin reloads cost map again — same result (root cause not addressed)
6. ~3 hours of investigation → root cause identified → fix deployed
---
## The Fix
After each reload, `add_known_models()` is called with the freshly fetched map passed explicitly. Passing the map directly (rather than relying on the module-level reference) removes any ambiguity about which dict is iterated:
```python
# After the fix — both reload paths now do:
new_model_cost_map = get_model_cost_map(url=model_cost_map_url)
litellm.model_cost = new_model_cost_map
_invalidate_model_cost_lowercase_map()
litellm.add_known_models(model_cost_map=new_model_cost_map) # ✅ sets repopulated
```
`add_known_models()` was also updated to accept an optional explicit map so callers cannot accidentally iterate a stale module-level reference:
```python
# Before
def add_known_models():
for key, value in model_cost.items(): # reads module global — ambiguous after reload
...
# After
def add_known_models(model_cost_map: Optional[Dict] = None):
_map = model_cost_map if model_cost_map is not None else model_cost
for key, value in _map.items(): # always iterates the map you just fetched
...
```
After the fix, the provider sets (`anthropic_models`, `open_ai_chat_completion_models`, etc.) are always consistent with `litellm.model_cost` immediately after every reload. New models become accessible via wildcard rules without any proxy restart.
---
## Remediation
| # | Action | Status | Code |
|---|---|---|---|
| 1 | Call `add_known_models(model_cost_map=...)` in the periodic reload path | ✅ Done | [`proxy_server.py#L4393`](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/proxy_server.py#L4393) |
| 2 | Call `add_known_models(model_cost_map=...)` in the `/reload/model_cost_map` endpoint | ✅ Done | [`proxy_server.py#L11904`](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/proxy_server.py#L11904) |
| 3 | Update `add_known_models()` to accept an explicit map parameter | ✅ Done | [`__init__.py#L617`](https://github.com/BerriAI/litellm/blob/main/litellm/__init__.py#L617) |
| 4 | Regression test: `add_known_models(model_cost_map=...)` populates provider sets | ✅ Done | [`test_auth_checks.py`](https://github.com/BerriAI/litellm/blob/main/tests/proxy_unit_tests/test_auth_checks.py) |
| 5 | Regression test: `anthropic/*` wildcard grants/denies access correctly after reload | ✅ Done | [`test_auth_checks.py`](https://github.com/BerriAI/litellm/blob/main/tests/proxy_unit_tests/test_auth_checks.py) |
---
docs/my-website/blog/gpt_5_3_codex/index.md
+145
View File
@@ -0,0 +1,145 @@
---
slug: gpt_5_3_codex
title: "Day 0 Support: GPT-5.3-Codex"
date: 2026-02-24T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
description: "Day 0 support for GPT-5.3-Codex on LiteLLM, including phase parameter handling for Responses API."
tags: [openai, gpt-5.3-codex, codex, day 0 support]
hide_table_of_contents: false
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
LiteLLM now supports GPT-5.3-Codex on Day 0, including support for the new assistant `phase` metadata on Responses API output items.
## Why `phase` matters for GPT-5.3-Codex
`phase` appears on assistant output items and helps distinguish preamble/commentary turns from final closeout responses.
Reference: [Phase parameter docs](https://developers.openai.com/api/reference/overview)
Supported values:
- `null`
- `"commentary"`
- `"final_answer"`
Important:
- Persist assistant output items with `phase` exactly as returned.
- Send those assistant items back on the next turn.
- Do **not** add `phase` to user messages.
## Docker Image
```bash
docker pull ghcr.io/berriai/litellm:v1.81.12-stable.gpt-5.3
```
## Usage
<Tabs>
<TabItem value="proxy" label="LiteLLM Proxy">
**1. Setup config.yaml**
```yaml
model_list:
- model_name: gpt-5.3-codex
litellm_params:
model: openai/gpt-5.3-codex
```
**2. Start the proxy**
```bash
docker run -d \
-p 4000:4000 \
-e ANTHROPIC_API_KEY=$OPENAI_API_KEY \
-v $(pwd)/config.yaml:/app/config.yaml \
ghcr.io/berriai/litellm:v1.81.12-stable.gpt-5.3 \
--config /app/config.yaml
```
**3. Test it**
```bash
curl -X POST "http://0.0.0.0:4000/v1/responses" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_KEY" \
-d '{
"model": "gpt-5.3-codex",
"input": "Write a Python script that checks if a number is prime."
}'
```
</TabItem>
</Tabs>
## Python Example: Persist `phase` with OpenAI Client + LiteLLM Base URL
```python
from openai import OpenAI
client = OpenAI(
base_url="http://0.0.0.0:4000/v1", # LiteLLM Proxy
api_key="your-litellm-api-key",
)
items = [] # Persist this per conversation/thread
def _item_get(item, key, default=None):
if isinstance(item, dict):
return item.get(key, default)
return getattr(item, key, default)
def run_turn(user_text: str):
global items
# User message: no phase field
items.append(
{
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": user_text}],
}
)
resp = client.responses.create(
model="gpt-5.3-codex",
input=items,
)
# Persist assistant output items verbatim, including phase
for out_item in (resp.output or []):
items.append(out_item)
# Optional: inspect latest phase for UI/telemetry routing
latest_phase = None
for out_item in reversed(resp.output or []):
if _item_get(out_item, "type") == "output_item.done" and _item_get(out_item, "phase") is not None:
latest_phase = _item_get(out_item, "phase")
break
return resp, latest_phase
```
## Notes
- Use `/v1/responses` for GPT Codex models.
- Preserve full assistant output history for best multi-turn behavior.
- If `phase` metadata is dropped during history reconstruction, output quality can degrade on long-running tasks.
docs/my-website/blog/server_root_path/index.md
+154
View File
@@ -0,0 +1,154 @@
---
slug: server-root-path-incident
title: "Incident Report: SERVER_ROOT_PATH regression broke UI routing"
date: 2026-02-21T10:00:00
authors:
- name: Yuneng Jiang
title: SWE @ LiteLLM (Full Stack)
url: https://www.linkedin.com/in/yunengjiang/
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
tags: [incident-report, ui, stability]
hide_table_of_contents: false
---
**Date:** January 22, 2026
**Duration:** ~4 days (until fix merged January 26, 2026)
**Severity:** High
**Status:** Resolved
> **Note:** This fix is available starting from LiteLLM `v1.81.3.rc.6` or higher.
## Summary
A PR ([`#19467`](https://github.com/BerriAI/litellm/pull/19467)) accidentally removed the `root_path=server_root_path` parameter from the FastAPI app initialization in `proxy_server.py`. This caused the proxy to ignore the `SERVER_ROOT_PATH` environment variable when serving the UI. Users who deploy LiteLLM behind a reverse proxy with a path prefix (e.g., `/api/v1` or `/llmproxy`) found that all UI pages returned 404 Not Found.
- **LLM API calls:** No impact. API routing was unaffected.
- **UI pages:** All UI pages returned 404 for deployments using `SERVER_ROOT_PATH`.
- **Swagger/OpenAPI docs:** Broken when accessed through the configured root path.
{/* truncate */}
---
## Background
Many LiteLLM deployments run behind a reverse proxy (e.g., Nginx, Traefik, AWS ALB) that routes traffic to LiteLLM under a path prefix. FastAPI's `root_path` parameter tells the application about this prefix so it can correctly serve static files, generate URLs, and handle routing.
```mermaid
sequenceDiagram
participant User as User Browser
participant RP as Reverse Proxy
participant LP as LiteLLM Proxy
User->>RP: GET /llmproxy/ui/
RP->>LP: GET /ui/ (X-Forwarded-Prefix: /llmproxy)
Note over LP: Before regression:<br/>FastAPI root_path="/llmproxy"<br/>→ Serves UI correctly
Note over LP: After regression:<br/>FastAPI root_path=""<br/>→ UI assets resolve to wrong paths<br/>→ 404 Not Found
```
The `root_path` parameter was present in `proxy_server.py` since early versions of LiteLLM. It was removed as a side effect of PR [#19467](https://github.com/BerriAI/litellm/pull/19467), which was intended to fix a different UI 404 issue.
---
## Root cause
PR [#19467](https://github.com/BerriAI/litellm/pull/19467) (`73d49f8`) removed the `root_path=server_root_path` line from the `FastAPI()` constructor in `proxy_server.py`:
```diff
app = FastAPI(
docs_url=_get_docs_url(),
redoc_url=_get_redoc_url(),
title=_title,
description=_description,
version=version,
- root_path=server_root_path,
lifespan=proxy_startup_event,
)
```
Without `root_path`, FastAPI treated all requests as if the application was mounted at `/`, causing path mismatches for any deployment using `SERVER_ROOT_PATH`.
The regression went undetected because:
1. **No automated test** verified that `root_path` was set on the FastAPI app.
2. **No manual test procedure** existed for `SERVER_ROOT_PATH` functionality.
3. **Default deployments** (without `SERVER_ROOT_PATH`) were unaffected, so most CI tests passed.
---
## Remediation
| # | Action | Status | Code |
| --- | ------------------------------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- |
| 1 | Restore `root_path=server_root_path` in FastAPI app initialization | ✅ Done | [`#19790`](https://github.com/BerriAI/litellm/pull/19790) (`5426b3c`) |
| 2 | Add unit tests for `get_server_root_path()` and FastAPI app initialization | ✅ Done | [`test_server_root_path.py`](https://github.com/BerriAI/litellm/blob/main/tests/proxy_unit_tests/test_server_root_path.py) |
| 3 | Add CI workflow that builds Docker image and tests UI routing with `SERVER_ROOT_PATH` on every PR | ✅ Done | [`test_server_root_path.yml`](https://github.com/BerriAI/litellm/blob/main/.github/workflows/test_server_root_path.yml) |
| 4 | Document manual test procedure for `SERVER_ROOT_PATH` | ✅ Done | [Discussion #8495](https://github.com/BerriAI/litellm/discussions/8495) |
---
## CI workflow details
The new [`test_server_root_path.yml`](https://github.com/BerriAI/litellm/blob/main/.github/workflows/test_server_root_path.yml) workflow runs on every PR against `main`. It:
1. Builds the LiteLLM Docker image
2. Starts a container with `SERVER_ROOT_PATH` set (tests both `/api/v1` and `/llmproxy`)
3. Verifies the UI returns valid HTML at `{ROOT_PATH}/ui/`
4. Fails the workflow if the UI is unreachable
```mermaid
flowchart TD
A["PR opened/updated"] --> B["Build Docker image"]
B --> C["Start container with SERVER_ROOT_PATH=/api/v1"]
B --> D["Start container with SERVER_ROOT_PATH=/llmproxy"]
C --> E["curl {ROOT_PATH}/ui/ → expect HTML"]
D --> F["curl {ROOT_PATH}/ui/ → expect HTML"]
E -->|"HTML found"| G["✅ Pass"]
E -->|"404 or no HTML"| H["❌ Fail Workflow"]
F -->|"HTML found"| G
F -->|"404 or no HTML"| H
style G fill:#d4edda,stroke:#28a745
style H fill:#f8d7da,stroke:#dc3545
```
This prevents future regressions where changes to `proxy_server.py` accidentally break `SERVER_ROOT_PATH` support.
---
## Timeline
| Time (UTC) | Event |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Jan 22, 2026 04:20 | PR [#19467](https://github.com/BerriAI/litellm/pull/19467) merged, removing `root_path=server_root_path` |
| Jan 22–26 | Users on nightly builds report UI 404 errors when using `SERVER_ROOT_PATH` |
| Jan 26, 2026 17:48 | Fix PR [#19790](https://github.com/BerriAI/litellm/pull/19790) merged, restoring `root_path=server_root_path` |
| Feb 18, 2026 | CI workflow [`test_server_root_path.yml`](https://github.com/BerriAI/litellm/blob/main/.github/workflows/test_server_root_path.yml) added to run on every PR |
---
## Resolution steps for users
For users still experiencing issues, update to the latest LiteLLM version:
```bash
pip install --upgrade litellm
```
Verify your `SERVER_ROOT_PATH` is correctly set:
```bash
# In your environment or docker-compose.yml
SERVER_ROOT_PATH="/your-prefix"
```
Then confirm the UI is accessible at `http://your-host:4000/your-prefix/ui/`.
docs/my-website/docs/benchmarks.md
+38
View File
@@ -5,6 +5,44 @@ import Image from '@theme/IdealImage';
Benchmarks for LiteLLM Gateway (Proxy Server) tested against a fake OpenAI endpoint.
## Setting Up Benchmarking with Network Mock
The fastest way to benchmark proxy overhead is using `network_mock` mode. This intercepts outbound requests at the httpx transport layer and returns canned responses, no need for setting up a mock provider.
**1. Create a proxy config:**
```yaml
model_list:
- model_name: db-openai-endpoint
litellm_params:
model: openai/gpt-4o
api_key: "sk-fake-key"
api_base: "https://api.openai.com"
litellm_settings:
network_mock: true
callbacks: []
num_retries: 0
request_timeout: 30
general_settings:
master_key: "sk-1234"
```
**2. Start the proxy:**
```bash
litellm --config benchmark_config.yaml --port 4000 --num_workers 8
```
**3. Run the benchmark script:**
```bash
python scripts/benchmark_mock.py --requests 2000 --max-concurrent 200 --runs 3
```
This measures pure proxy overhead on the hot path without any network latency to a real or fake provider.
## Setting Up a Fake OpenAI Endpoint
For load testing and benchmarking, you can use a fake OpenAI proxy server. LiteLLM provides:
docs/my-website/docs/caching/all_caches.md
+2
View File
@@ -297,6 +297,7 @@ litellm.cache = Cache(
similarity_threshold=0.7, # similarity threshold for cache hits, 0 == no similarity, 1 = exact matches, 0.5 == 50% similarity
qdrant_quantization_config ="binary", # can be one of 'binary', 'product' or 'scalar' quantizations that is supported by qdrant
qdrant_semantic_cache_embedding_model="text-embedding-ada-002", # this model is passed to litellm.embedding(), any litellm.embedding() model is supported here
qdrant_semantic_cache_vector_size=1536, # vector size for the embedding model, must match the dimensionality of the embedding model used
)
response1 = completion(
@@ -635,6 +636,7 @@ def __init__(
qdrant_quantization_config: Optional[str] = None,
qdrant_semantic_cache_embedding_model="text-embedding-ada-002",
qdrant_semantic_cache_vector_size: Optional[int] = None,
**kwargs
):
```
docs/my-website/docs/contributing.md
+21 -1
View File
@@ -79,7 +79,27 @@ cp -r out/* ../../litellm/proxy/_experimental/out/
Then restart the proxy and access the UI at `http://localhost:4000/ui`
## 4. Submitting a PR
## 4. Pre-PR Checklist
Before submitting your pull request, make sure the following pass locally from `ui/litellm-dashboard/`:
**Run tests related to your changes:**
```bash
npx vitest run src/components/path/to/YourComponent.test.tsx
```
Tests are co-located with components (e.g., `TeamInfo.tsx` → `TeamInfo.test.tsx`). If you add a new component, add a corresponding `.test.tsx` file next to it.
**Run the build:**
```bash
npm run build
```
These map to the `ui_tests` and `ui_build` CI checks.
## 5. Submitting a PR
1. Create a new branch for your changes:
```bash
docs/my-website/docs/enterprise.md
+3 -3
View File
@@ -4,7 +4,7 @@ import Image from '@theme/IdealImage';
:::info
- ✨ SSO is free for up to 5 users. After that, an enterprise license is required. [Get Started with Enterprise here](https://www.litellm.ai/enterprise)
- Who is Enterprise for? Companies giving access to 100+ users **OR** 10+ AI use-cases. If you're not sure, [get in touch with us](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) to discuss your needs.
- Who is Enterprise for? Companies giving access to 100+ users **OR** 10+ AI use-cases. If you're not sure, [get in touch with us](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions) to discuss your needs.
:::
For companies that need SSO, user management and professional support for LiteLLM Proxy
@@ -36,7 +36,7 @@ Manage Yourself - you can deploy our Docker Image or build a custom image from o
### What’s the cost of the Self-Managed Enterprise edition?
Self-Managed Enterprise deployments require our team to understand your exact needs. [Get in touch with us to learn more](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
Self-Managed Enterprise deployments require our team to understand your exact needs. [Get in touch with us to learn more](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
### How does deployment with Enterprise License work?
@@ -106,7 +106,7 @@ Professional Support can assist with LLM/Provider integrations, deployment, upgr
Pricing is based on usage. We can figure out a price that works for your team, on the call.
[**Contact Us to learn more**](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[**Contact Us to learn more**](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
docs/my-website/docs/fine_tuning.md
+1 -1
View File
@@ -6,7 +6,7 @@ import TabItem from '@theme/TabItem';
:::info
This is an Enterprise only endpoint [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
This is an Enterprise only endpoint [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/interactions.md
+8 -10
View File
@@ -130,13 +130,12 @@ Point the Google GenAI SDK to LiteLLM Proxy:
```python showLineNumbers title="Google GenAI SDK with LiteLLM Proxy"
from google import genai
import os
# Point SDK to LiteLLM Proxy
os.environ["GOOGLE_GENAI_BASE_URL"] = "http://localhost:4000"
os.environ["GEMINI_API_KEY"] = "sk-1234" # Your LiteLLM API key
client = genai.Client()
client = genai.Client(
api_key="sk-1234", # Your LiteLLM API key
http_options={"base_url": "http://localhost:4000"},
)
# Create an interaction
interaction = client.interactions.create(
@@ -151,12 +150,11 @@ print(interaction.outputs[-1].text)
```python showLineNumbers title="Google GenAI SDK Streaming"
from google import genai
import os
os.environ["GOOGLE_GENAI_BASE_URL"] = "http://localhost:4000"
os.environ["GEMINI_API_KEY"] = "sk-1234"
client = genai.Client()
client = genai.Client(
api_key="sk-1234", # Your LiteLLM API key
http_options={"base_url": "http://localhost:4000"},
)
for chunk in client.interactions.create_stream(
model="gemini/gemini-2.5-flash",
docs/my-website/docs/observability/gcs_bucket_integration.md
+1 -1
View File
@@ -6,7 +6,7 @@ Log LLM Logs to [Google Cloud Storage Buckets](https://cloud.google.com/storage?
:::info
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/pass_through/google_ai_studio.md
+35 -36
View File
@@ -35,26 +35,25 @@ curl 'http://0.0.0.0:4000/gemini/v1beta/models/gemini-1.5-flash:countTokens?key=
```
</TabItem>
<TabItem value="js" label="Google AI Node.js SDK">
<TabItem value="js" label="Google GenAI JS SDK">
```javascript
const { GoogleGenerativeAI } = require("@google/generative-ai");
const { GoogleGenAI } = require("@google/genai");
const modelParams = {
model: 'gemini-pro',
};
const requestOptions = {
baseUrl: 'http://localhost:4000/gemini', // http://<proxy-base-url>/gemini
};
const genAI = new GoogleGenerativeAI("sk-1234"); // litellm proxy API key
const model = genAI.getGenerativeModel(modelParams, requestOptions);
const ai = new GoogleGenAI({
apiKey: "sk-1234", // litellm proxy API key
httpOptions: {
baseUrl: "http://localhost:4000/gemini", // http://<proxy-base-url>/gemini
},
});
async function main() {
try {
const result = await model.generateContent("Explain how AI works");
console.log(result.response.text());
const response = await ai.models.generateContent({
model: "gemini-2.5-flash",
contents: "Explain how AI works",
});
console.log(response.text);
} catch (error) {
console.error('Error:', error);
}
@@ -63,12 +62,13 @@ async function main() {
// For streaming responses
async function main_streaming() {
try {
const streamingResult = await model.generateContentStream("Explain how AI works");
for await (const chunk of streamingResult.stream) {
console.log('Stream chunk:', JSON.stringify(chunk));
const response = await ai.models.generateContentStream({
model: "gemini-2.5-flash",
contents: "Explain how AI works",
});
for await (const chunk of response) {
process.stdout.write(chunk.text);
}
const aggregatedResponse = await streamingResult.response;
console.log('Aggregated response:', JSON.stringify(aggregatedResponse));
} catch (error) {
console.error('Error:', error);
}
@@ -321,29 +321,28 @@ curl 'http://0.0.0.0:4000/gemini/v1beta/models/gemini-1.5-flash:generateContent?
```
</TabItem>
<TabItem value="js" label="Google AI Node.js SDK">
<TabItem value="js" label="Google GenAI JS SDK">
```javascript
const { GoogleGenerativeAI } = require("@google/generative-ai");
const { GoogleGenAI } = require("@google/genai");
const modelParams = {
model: 'gemini-pro',
};
const requestOptions = {
baseUrl: 'http://localhost:4000/gemini', // http://<proxy-base-url>/gemini
customHeaders: {
"tags": "gemini-js-sdk,pass-through-endpoint"
}
};
const genAI = new GoogleGenerativeAI("sk-1234");
const model = genAI.getGenerativeModel(modelParams, requestOptions);
const ai = new GoogleGenAI({
apiKey: "sk-1234",
httpOptions: {
baseUrl: "http://localhost:4000/gemini", // http://<proxy-base-url>/gemini
headers: {
"tags": "gemini-js-sdk,pass-through-endpoint",
},
},
});
async function main() {
try {
const result = await model.generateContent("Explain how AI works");
console.log(result.response.text());
const response = await ai.models.generateContent({
model: "gemini-2.5-flash",
contents: "Explain how AI works",
});
console.log(response.text);
} catch (error) {
console.error('Error:', error);
}
docs/my-website/docs/providers/groq.md
+1
View File
@@ -159,6 +159,7 @@ We support ALL Groq models, just set `groq/` as a prefix when sending completion
| moonshotai/kimi-k2-instruct-0905 | `completion(model="groq/moonshotai/kimi-k2-instruct-0905", messages)` |
| openai/gpt-oss-120b | `completion(model="groq/openai/gpt-oss-120b", messages)` |
| openai/gpt-oss-20b | `completion(model="groq/openai/gpt-oss-20b", messages)` |
| openai/gpt-oss-safeguard-20b | `completion(model="groq/openai/gpt-oss-safeguard-20b", messages)` |
## Groq - Tool / Function Calling Example
docs/my-website/docs/providers/perplexity.md
+99 -22
View File
@@ -120,7 +120,7 @@ All models listed here https://docs.perplexity.ai/docs/model-cards are supported
## Agentic Research API (Responses API)
## Agent API (Responses API)
Requires v1.72.6+
@@ -196,7 +196,7 @@ import os
os.environ['PERPLEXITY_API_KEY'] = ""
response = responses(
model="perplexity/openai/gpt-4o",
model="perplexity/openai/gpt-5.2",
input="Explain quantum computing in simple terms",
custom_llm_provider="perplexity",
max_output_tokens=500,
@@ -215,7 +215,7 @@ import os
os.environ['PERPLEXITY_API_KEY'] = ""
response = responses(
model="perplexity/anthropic/claude-3-5-sonnet-20241022",
model="perplexity/anthropic/claude-sonnet-4-5",
input="Write a short story about a robot learning to paint",
custom_llm_provider="perplexity",
max_output_tokens=500,
@@ -234,7 +234,7 @@ import os
os.environ['PERPLEXITY_API_KEY'] = ""
response = responses(
model="perplexity/google/gemini-2.0-flash-exp",
model="perplexity/google/gemini-2.5-flash",
input="Explain the concept of neural networks",
custom_llm_provider="perplexity",
max_output_tokens=500,
@@ -253,7 +253,7 @@ import os
os.environ['PERPLEXITY_API_KEY'] = ""
response = responses(
model="perplexity/xai/grok-2-1212",
model="perplexity/xai/grok-4-1-fast-non-reasoning",
input="What makes a good AI assistant?",
custom_llm_provider="perplexity",
max_output_tokens=500,
@@ -276,7 +276,7 @@ import os
os.environ['PERPLEXITY_API_KEY'] = ""
response = responses(
model="perplexity/openai/gpt-4o",
model="perplexity/openai/gpt-5.2",
input="What's the weather in San Francisco today?",
custom_llm_provider="perplexity",
tools=[{"type": "web_search"}],
@@ -286,6 +286,78 @@ response = responses(
print(response.output)
```
### Function Calling
The Agent API supports custom function tools. Pass function tools through unchanged:
```python
from litellm import responses
import os
os.environ['PERPLEXITY_API_KEY'] = ""
response = responses(
model="perplexity/openai/gpt-5.2",
input="What's the weather in San Francisco?",
custom_llm_provider="perplexity",
tools=[
{"type": "web_search"},
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
},
},
},
],
instructions="Use tools when appropriate.",
)
print(response.output)
```
### Structured Outputs
Request JSON schema structured outputs via the `text` parameter:
```python
from litellm import responses
import os
os.environ['PERPLEXITY_API_KEY'] = ""
response = responses(
model="perplexity/preset/pro-search",
input="Extract key facts about the Eiffel Tower",
custom_llm_provider="perplexity",
text={
"format": {
"type": "json_schema",
"name": "facts",
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"height_meters": {"type": "number"},
"year_built": {"type": "integer"},
},
"required": ["name", "height_meters", "year_built"],
},
"strict": True,
}
},
)
print(response.output)
```
### Reasoning Effort (Responses API)
@@ -319,7 +391,7 @@ import os
os.environ['PERPLEXITY_API_KEY'] = ""
response = responses(
model="perplexity/anthropic/claude-3-5-sonnet-20241022",
model="perplexity/anthropic/claude-sonnet-4-5",
input=[
{"type": "message", "role": "system", "content": "You are a helpful assistant."},
{"type": "message", "role": "user", "content": "What are the latest AI developments?"},
@@ -343,7 +415,7 @@ import os
os.environ['PERPLEXITY_API_KEY'] = ""
response = responses(
model="perplexity/openai/gpt-4o",
model="perplexity/openai/gpt-5.2",
input="Tell me a story about space exploration",
custom_llm_provider="perplexity",
stream=True,
@@ -360,23 +432,28 @@ for chunk in response:
| Provider | Model Name | Function Call |
|----------|------------|---------------|
| OpenAI | gpt-4o | `responses(model="perplexity/openai/gpt-4o", ...)` |
| OpenAI | gpt-4o-mini | `responses(model="perplexity/openai/gpt-4o-mini", ...)` |
| OpenAI | gpt-5.2 | `responses(model="perplexity/openai/gpt-5.2", ...)` |
| Anthropic | claude-3-5-sonnet-20241022 | `responses(model="perplexity/anthropic/claude-3-5-sonnet-20241022", ...)` |
| Anthropic | claude-3-5-haiku-20241022 | `responses(model="perplexity/anthropic/claude-3-5-haiku-20241022", ...)` |
| Google | gemini-2.0-flash-exp | `responses(model="perplexity/google/gemini-2.0-flash-exp", ...)` |
| Google | gemini-2.0-flash-thinking-exp | `responses(model="perplexity/google/gemini-2.0-flash-thinking-exp", ...)` |
| xAI | grok-2-1212 | `responses(model="perplexity/xai/grok-2-1212", ...)` |
| xAI | grok-2-vision-1212 | `responses(model="perplexity/xai/grok-2-vision-1212", ...)` |
| OpenAI | gpt-5.1 | `responses(model="perplexity/openai/gpt-5.1", ...)` |
| OpenAI | gpt-5-mini | `responses(model="perplexity/openai/gpt-5-mini", ...)` |
| Anthropic | claude-opus-4-6 | `responses(model="perplexity/anthropic/claude-opus-4-6", ...)` |
| Anthropic | claude-opus-4-5 | `responses(model="perplexity/anthropic/claude-opus-4-5", ...)` |
| Anthropic | claude-sonnet-4-5 | `responses(model="perplexity/anthropic/claude-sonnet-4-5", ...)` |
| Anthropic | claude-haiku-4-5 | `responses(model="perplexity/anthropic/claude-haiku-4-5", ...)` |
| Google | gemini-3-pro-preview | `responses(model="perplexity/google/gemini-3-pro-preview", ...)` |
| Google | gemini-3-flash-preview | `responses(model="perplexity/google/gemini-3-flash-preview", ...)` |
| Google | gemini-2.5-pro | `responses(model="perplexity/google/gemini-2.5-pro", ...)` |
| Google | gemini-2.5-flash | `responses(model="perplexity/google/gemini-2.5-flash", ...)` |
| xAI | grok-4-1-fast-non-reasoning | `responses(model="perplexity/xai/grok-4-1-fast-non-reasoning", ...)` |
| Perplexity | sonar | `responses(model="perplexity/perplexity/sonar", ...)` |
### Available Presets
| Preset Name | Function Call |
|----------------|--------------------------------------------------------|
| fast-search | `responses(model="perplexity/preset/fast-search", ...)`|
| pro-search | `responses(model="perplexity/preset/pro-search", ...)` |
| deep-research | `responses(model="perplexity/preset/deep-research", ...)`|
| Preset Name | Function Call |
|-------------|---------------|
| fast-search | `responses(model="perplexity/preset/fast-search", ...)` |
| pro-search | `responses(model="perplexity/preset/pro-search", ...)` |
| deep-research | `responses(model="perplexity/preset/deep-research", ...)` |
| advanced-deep-research | `responses(model="perplexity/preset/advanced-deep-research", ...)` |
### Complete Example
@@ -388,7 +465,7 @@ os.environ['PERPLEXITY_API_KEY'] = ""
# Comprehensive example with multiple features
response = responses(
model="perplexity/openai/gpt-4o",
model="perplexity/openai/gpt-5.2",
input="Research the latest developments in quantum computing and provide sources",
custom_llm_provider="perplexity",
tools=[
docs/my-website/docs/proxy/alerting.md
+53
View File
@@ -438,6 +438,59 @@ curl -X GET --location 'http://0.0.0.0:4000/health/services?service=webhook' \
- `event_message` *str*: A human-readable description of the event.
### Digest Mode (Reducing Alert Noise)
By default, LiteLLM sends a separate Slack message for **every** alert event. For high-frequency alert types like `llm_requests_hanging` or `llm_too_slow`, this can produce hundreds of duplicate messages per day.
**Digest mode** aggregates duplicate alerts within a configurable time window and emits a single summary message with the total count and time range.
#### Configuration
Use `alert_type_config` in `general_settings` to enable digest mode per alert type:
```yaml
general_settings:
alerting: ["slack"]
alert_type_config:
llm_requests_hanging:
digest: true
digest_interval: 86400 # 24 hours (default)
llm_too_slow:
digest: true
digest_interval: 3600 # 1 hour
llm_exceptions:
digest: true
# uses default interval (86400 seconds / 24 hours)
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `digest` | bool | `false` | Enable digest mode for this alert type |
| `digest_interval` | int | `86400` (24h) | Time window in seconds. Alerts are aggregated within this interval. |
#### How It Works
1. When an alert fires for a digest-enabled type, it is **grouped** by `(alert_type, request_model, api_base)` instead of being sent immediately
2. A counter tracks how many times the alert fires within the interval
3. When the interval expires, a **single summary message** is sent:
```
Alert type: `llm_requests_hanging` (Digest)
Level: `Medium`
Start: `2026-02-19 03:27:39`
End: `2026-02-20 03:27:39`
Count: `847`
Message: `Requests are hanging - 600s+ request time`
Request Model: `gemini-2.5-flash`
API Base: `None`
```
#### Limitations
- **Per-instance**: Digest state is held in memory per proxy instance. If you run multiple instances (e.g., Cloud Run with autoscaling), each instance maintains its own digest and emits its own summary.
- **Not durable**: If an instance is terminated before the digest interval expires, the aggregated alerts for that instance are lost.
## Region-outage alerting (✨ Enterprise feature)
:::info
docs/my-website/docs/proxy/auto_routing.md
+186
View File
@@ -219,3 +219,189 @@ curl -X POST http://localhost:4000/v1/chat/completions \
3. If a route's similarity score exceeds the threshold, the request is routed to that model
4. If no route matches, the request goes to the default model
---
## Complexity Router
The Complexity Router provides an alternative to semantic routing that uses **rule-based scoring** to classify requests by complexity and route them to appropriate models — with **zero external API calls** and **sub-millisecond latency**.
### When to Use
| Feature | Semantic Auto Router | Complexity Router |
|---------|---------------------|-------------------|
| Classification | Embedding-based matching | Rule-based scoring |
| Latency | ~100-500ms (embedding API) | &lt;1ms |
| API Calls | Requires embedding model | None |
| Training | Requires utterance examples | Works out of the box |
| Best For | Intent-based routing | Cost optimization |
Use **Complexity Router** when you want to:
- Route simple queries to cheaper/faster models (e.g., gpt-4o-mini)
- Route complex queries to more capable models (e.g., claude-sonnet-4)
- Minimize latency overhead from routing decisions
- Avoid additional API costs for embeddings
### LiteLLM Python SDK
```python
from litellm import Router
router = Router(
model_list=[
# Target models for each tier
{
"model_name": "gpt-4o-mini",
"litellm_params": {"model": "gpt-4o-mini"},
},
{
"model_name": "gpt-4o",
"litellm_params": {"model": "gpt-4o"},
},
{
"model_name": "claude-sonnet",
"litellm_params": {"model": "claude-sonnet-4-20250514"},
},
{
"model_name": "o1-preview",
"litellm_params": {"model": "o1-preview"},
},
# Complexity router configuration
{
"model_name": "smart-router",
"litellm_params": {
"model": "auto_router/complexity_router",
"complexity_router_config": {
"tiers": {
"SIMPLE": "gpt-4o-mini",
"MEDIUM": "gpt-4o",
"COMPLEX": "claude-sonnet",
"REASONING": "o1-preview",
},
},
"complexity_router_default_model": "gpt-4o",
},
},
],
)
```
#### Usage
```python
# Simple query → routes to gpt-4o-mini
response = await router.acompletion(
model="smart-router",
messages=[{"role": "user", "content": "What is 2+2?"}],
)
# Complex technical query → routes to claude-sonnet or higher
response = await router.acompletion(
model="smart-router",
messages=[{"role": "user", "content": "Design a distributed microservice architecture with Kubernetes orchestration"}],
)
# Reasoning request → routes to o1-preview
response = await router.acompletion(
model="smart-router",
messages=[{"role": "user", "content": "Think step by step and reason through this problem carefully..."}],
)
```
### LiteLLM Proxy Server
Add the complexity router to your `config.yaml`:
```yaml
model_list:
# Target models
- model_name: gpt-4o-mini
litellm_params:
model: gpt-4o-mini
- model_name: gpt-4o
litellm_params:
model: gpt-4o
- model_name: claude-sonnet
litellm_params:
model: claude-sonnet-4-20250514
- model_name: o1-preview
litellm_params:
model: o1-preview
# Complexity router
- model_name: smart-router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: gpt-4o-mini
MEDIUM: gpt-4o
COMPLEX: claude-sonnet
REASONING: o1-preview
complexity_router_default_model: gpt-4o
```
### Configuration Options
#### Tier Boundaries
Customize the score thresholds for each tier:
```yaml
complexity_router_config:
tiers:
SIMPLE: gpt-4o-mini
MEDIUM: gpt-4o
COMPLEX: claude-sonnet
REASONING: o1-preview
tier_boundaries:
simple_medium: 0.15 # Below 0.15 → SIMPLE
medium_complex: 0.35 # 0.15-0.35 → MEDIUM
complex_reasoning: 0.60 # 0.35-0.60 → COMPLEX, above → REASONING
```
#### Token Thresholds
Adjust when prompts are considered "short" or "long":
```yaml
complexity_router_config:
token_thresholds:
simple: 15 # Prompts under 15 tokens are penalized (simple indicator)
complex: 400 # Prompts over 400 tokens get complexity boost
```
#### Dimension Weights
Customize how much each signal contributes to the complexity score:
```yaml
complexity_router_config:
dimension_weights:
tokenCount: 0.10 # Prompt length
codePresence: 0.30 # Code-related keywords
reasoningMarkers: 0.25 # "step by step", "think through", etc.
technicalTerms: 0.25 # Domain-specific complexity
simpleIndicators: 0.05 # "what is", "define", greetings
multiStepPatterns: 0.03 # "first...then", numbered steps
questionComplexity: 0.02 # Multiple questions
```
### How Complexity Routing Works
The router scores each request across 7 dimensions:
| Dimension | What It Detects | Effect |
|-----------|-----------------|--------|
| Token Count | Short (&lt;15) or long (&gt;400) prompts | Short = simple, long = complex |
| Code Presence | "function", "class", "api", "database", etc. | Increases complexity |
| Reasoning Markers | "step by step", "think through", "analyze" | Triggers REASONING tier |
| Technical Terms | "architecture", "distributed", "encryption" | Increases complexity |
| Simple Indicators | "what is", "define", "hello" | Decreases complexity |
| Multi-Step Patterns | "first...then", "1. 2. 3." | Increases complexity |
| Question Complexity | Multiple question marks | Increases complexity |
**Special behavior:** If 2+ reasoning markers are detected in the user message, the request automatically routes to the REASONING tier regardless of the weighted score.
docs/my-website/docs/proxy/budget_reset_and_tz.md
+2
View File
@@ -22,6 +22,8 @@ litellm_settings:
This ensures that all budget resets happen at midnight in your specified timezone rather than in UTC.
If no timezone is specified, UTC will be used by default.
Any valid [IANA timezone string](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) is supported (powered by Python's `zoneinfo` module). DST transitions are handled automatically.
Common timezone values:
- `UTC` - Coordinated Universal Time
docs/my-website/docs/proxy/caching.md
+1
View File
@@ -340,6 +340,7 @@ litellm_settings:
qdrant_semantic_cache_embedding_model: openai-embedding # the model should be defined on the model_list
qdrant_collection_name: test_collection
qdrant_quantization_config: binary
qdrant_semantic_cache_vector_size: 1536 # vector size must match embedding model dimensionality
similarity_threshold: 0.8 # similarity threshold for semantic cache
```
docs/my-website/docs/proxy/config_settings.md
+18
View File
@@ -73,6 +73,7 @@ litellm_settings:
qdrant_semantic_cache_embedding_model: openai-embedding # the model should be defined on the model_list
qdrant_collection_name: test_collection
qdrant_quantization_config: binary
qdrant_semantic_cache_vector_size: 1536 # vector size must match embedding model dimensionality
similarity_threshold: 0.8 # similarity threshold for semantic cache
# Optional - S3 Cache Settings
@@ -485,6 +486,7 @@ router_settings:
| CUSTOM_TIKTOKEN_CACHE_DIR | Custom directory for Tiktoken cache
| CONFIDENT_API_KEY | API key for Confident AI (Deepeval) Logging service
| COHERE_API_BASE | Base URL for Cohere API. Default is https://api.cohere.com
| COMPETITOR_LLM_TEMPERATURE | Temperature setting for the LLM used in competitor discovery. Default is 0.3
| DATABASE_HOST | Hostname for the database server
| DATABASE_NAME | Name of the database
| DATABASE_PASSWORD | Password for the database user
@@ -573,6 +575,8 @@ router_settings:
| DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_PRO | Default minimal reasoning effort thinking budget for Gemini 2.5 Pro. Default is 512
| DEFAULT_REDIS_MAJOR_VERSION | Default Redis major version to assume when version cannot be determined. Default is 7
| DEFAULT_REDIS_SYNC_INTERVAL | Default Redis synchronization interval in seconds. Default is 1
| DEFAULT_SEMANTIC_GUARD_EMBEDDING_MODEL | Default embedding model for Semantic Guard (route-matching guardrail). Default is "text-embedding-3-small"
| DEFAULT_SEMANTIC_GUARD_SIMILARITY_THRESHOLD | Default similarity threshold for Semantic Guard route matching. Default is 0.75
| DEFAULT_REPLICATE_GPU_PRICE_PER_SECOND | Default price per second for Replicate GPU. Default is 0.001400
| DEFAULT_REPLICATE_POLLING_DELAY_SECONDS | Default delay in seconds for Replicate polling. Default is 1
| DEFAULT_REPLICATE_POLLING_RETRIES | Default number of retries for Replicate polling. Default is 5
@@ -752,15 +756,18 @@ router_settings:
| LITELLM_ANTHROPIC_BETA_HEADERS_URL | Custom URL for fetching Anthropic beta headers configuration. Default is the GitHub main branch URL
| LITELLM_ANTHROPIC_DISABLE_URL_SUFFIX | Disable automatic URL suffix appending for Anthropic API base URLs. When set to `true`, prevents LiteLLM from automatically adding `/v1/messages` or `/v1/complete` to custom Anthropic API endpoints
| LITELLM_ASSETS_PATH | Path to directory for UI assets and logos. Used when running with read-only filesystem (e.g., Kubernetes). Default is `/var/lib/litellm/assets` in Docker.
| LITELLM_BLOG_POSTS_URL | Custom URL for fetching LiteLLM blog posts JSON. Default is the GitHub main branch URL
| LITELLM_CLI_JWT_EXPIRATION_HOURS | Expiration time in hours for CLI-generated JWT tokens. Default is 24 hours
| LITELLM_DD_AGENT_HOST | Hostname or IP of DataDog agent for LiteLLM-specific logging. When set, logs are sent to agent instead of direct API
| LITELLM_DEPLOYMENT_ENVIRONMENT | Environment name for the deployment (e.g., "production", "staging"). Used as a fallback when OTEL_ENVIRONMENT_NAME is not set. Sets the `environment` tag in telemetry data
| LITELLM_DETAILED_TIMING | When true, adds detailed per-phase timing headers to responses (`x-litellm-timing-{pre-processing,llm-api,post-processing,message-copy}-ms`). Default is false. See [latency overhead docs](../troubleshoot/latency_overhead.md)
| LITELLM_DD_AGENT_PORT | Port of DataDog agent for LiteLLM-specific log intake. Default is 10518
| LITELLM_DD_LLM_OBS_PORT | Port for Datadog LLM Observability agent. Default is 8126
| LITELLM_DONT_SHOW_FEEDBACK_BOX | Flag to hide feedback box in LiteLLM UI
| LITELLM_DROP_PARAMS | Parameters to drop in LiteLLM requests
| LITELLM_MODIFY_PARAMS | Parameters to modify in LiteLLM requests
| LITELLM_EMAIL | Email associated with LiteLLM account
| LITELLM_FAVICON_URL | Custom URL for the LiteLLM UI favicon. When set, overrides the default favicon
| LITELLM_GLOBAL_MAX_PARALLEL_REQUEST_RETRIES | Maximum retries for parallel requests in LiteLLM
| LITELLM_GLOBAL_MAX_PARALLEL_REQUEST_RETRY_TIMEOUT | Timeout for retries of parallel requests in LiteLLM
| LITELLM_DISABLE_LAZY_LOADING | When set to "1", "true", "yes", or "on", disables lazy loading of attributes (currently only affects encoding/tiktoken). This ensures encoding is initialized before VCR starts recording HTTP requests, fixing VCR cassette creation issues. See [issue #18659](https://github.com/BerriAI/litellm/issues/18659)
@@ -774,6 +781,7 @@ router_settings:
| LITELLM_KEY_ROTATION_GRACE_PERIOD | Duration to keep old key valid after rotation (e.g. "24h", "2d"). Default is empty (immediate revoke). Used for scheduled rotations and as fallback when not specified in regenerate request.
| LITELLM_LICENSE | License key for LiteLLM usage
| LITELLM_LOCAL_ANTHROPIC_BETA_HEADERS | Set to `True` to use the local bundled Anthropic beta headers config only, disabling remote fetching. Default is `False`
| LITELLM_LOCAL_BLOG_POSTS | When set to `True`, uses the local bundled blog posts only, disabling remote fetching from GitHub. Default is `False`
| LITELLM_LOCAL_MODEL_COST_MAP | Local configuration for model cost mapping in LiteLLM
| LITELLM_LOCAL_POLICY_TEMPLATES | When set to "true", uses local backup policy templates instead of fetching from GitHub. Policy templates are fetched from https://raw.githubusercontent.com/BerriAI/litellm/main/policy_templates.json by default, with automatic fallback to local backup on failure
| LITELLM_LOG | Enable detailed logging for LiteLLM
@@ -806,6 +814,8 @@ router_settings:
| LOGGING_WORKER_MAX_QUEUE_SIZE | Maximum size of the logging worker queue. When the queue is full, the worker aggressively clears tasks to make room instead of dropping logs. Default is 50,000
| LOGGING_WORKER_MAX_TIME_PER_COROUTINE | Maximum time in seconds allowed for each coroutine in the logging worker before timing out. Default is 20.0
| LOGGING_WORKER_CLEAR_PERCENTAGE | Percentage of the queue to extract when clearing. Default is 50%
| MAX_BASE64_LENGTH_FOR_LOGGING | Maximum number of base64 characters to keep in logging payloads. Data URIs exceeding this are replaced with a size placeholder. Set to 0 to disable truncation. Default is 64
| MAX_COMPETITOR_NAMES | Maximum number of competitor names allowed in policy template enrichment. Default is 100
| MAX_EXCEPTION_MESSAGE_LENGTH | Maximum length for exception messages. Default is 2000
| MAX_ITERATIONS_TO_CLEAR_QUEUE | Maximum number of iterations to attempt when clearing the logging worker queue during shutdown. Default is 200
| MAX_TIME_TO_CLEAR_QUEUE | Maximum time in seconds to spend clearing the logging worker queue during shutdown. Default is 5.0
@@ -828,6 +838,7 @@ router_settings:
| MAX_LANGFUSE_INITIALIZED_CLIENTS | Maximum number of Langfuse clients to initialize on proxy. Default is 50. This is set since langfuse initializes 1 thread everytime a client is initialized. We've had an incident in the past where we reached 100% cpu utilization because Langfuse was initialized several times.
| MAX_MCP_SEMANTIC_FILTER_TOOLS_HEADER_LENGTH | Maximum header length for MCP semantic filter tools. Default is 150
| MAX_POLICY_ESTIMATE_IMPACT_ROWS | Maximum number of rows returned when estimating the impact of a policy. Default is 1000
| MAX_PAYLOAD_SIZE_FOR_DEBUG_LOG | Maximum payload size in bytes for full DEBUG serialization. Payloads exceeding this will be truncated in logs. Default is 102400 (100 KB)
| MIN_NON_ZERO_TEMPERATURE | Minimum non-zero temperature value. Default is 0.0001
| MINIMUM_PROMPT_CACHE_TOKEN_COUNT | Minimum token count for caching a prompt. Default is 1024
| MISTRAL_API_BASE | Base URL for Mistral API. Default is https://api.mistral.ai
@@ -891,6 +902,13 @@ router_settings:
| POSTHOG_API_URL | Base URL for PostHog API (defaults to https://us.i.posthog.com)
| POSTHOG_MOCK | Enable mock mode for PostHog integration testing. When set to true, intercepts PostHog API calls and returns mock responses without making actual network calls. Default is false
| POSTHOG_MOCK_LATENCY_MS | Mock latency in milliseconds for PostHog API calls when mock mode is enabled. Simulates network round-trip time. Default is 100ms
| PRISMA_AUTH_RECONNECT_LOCK_TIMEOUT_SECONDS | Lock timeout in seconds for Prisma auth reconnection. Default is 0.1
| PRISMA_AUTH_RECONNECT_TIMEOUT_SECONDS | Timeout in seconds for Prisma auth reconnection attempts. Default is 2.0
| PRISMA_HEALTH_WATCHDOG_ENABLED | Enable the Prisma DB health watchdog that monitors and reconnects on connection loss. Default is true
| PRISMA_HEALTH_WATCHDOG_INTERVAL_SECONDS | Interval in seconds for Prisma health watchdog probes. Default is 30
| PRISMA_HEALTH_WATCHDOG_PROBE_TIMEOUT_SECONDS | Timeout in seconds for each Prisma health probe. Default is 5.0
| PRISMA_RECONNECT_COOLDOWN_SECONDS | Cooldown in seconds between Prisma reconnection attempts. Default is 15
| PRISMA_WATCHDOG_RECONNECT_TIMEOUT_SECONDS | Timeout in seconds for Prisma watchdog-initiated reconnection. Default is 30.0
| PREDIBASE_API_BASE | Base URL for Predibase API
| PRESIDIO_ANALYZER_API_BASE | Base URL for Presidio Analyzer service
| PRESIDIO_ANONYMIZER_API_BASE | Base URL for Presidio Anonymizer service
docs/my-website/docs/proxy/cost_tracking.md
+1 -1
View File
@@ -161,7 +161,7 @@ Use this when you want non-proxy admins to access `/spend` endpoints
:::info
Schedule a [meeting with us to get your Enterprise License](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
Schedule a [meeting with us to get your Enterprise License](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/email.md
+1 -1
View File
@@ -203,7 +203,7 @@ After regenerating the key, the user will receive an email notification with:
:::info
Customizing Email Branding is an Enterprise Feature [Get in touch with us for a Free Trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
Customizing Email Branding is an Enterprise Feature [Get in touch with us for a Free Trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/enterprise.md
+1 -1
View File
@@ -5,7 +5,7 @@ import TabItem from '@theme/TabItem';
# ✨ Enterprise Features
:::tip
To get a license, get in touch with us [here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
To get a license, get in touch with us [here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/guardrails/aporia_api.md
+1 -1
View File
@@ -139,7 +139,7 @@ curl -i http://localhost:4000/v1/chat/completions \
:::info
✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/guardrails/custom_guardrail.md
+1 -1
View File
@@ -409,7 +409,7 @@ curl -i -X POST http://localhost:4000/v1/chat/completions \
:::info
✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/guardrails/guardrails_ai.md
+1 -1
View File
@@ -59,7 +59,7 @@ curl -i http://localhost:4000/v1/chat/completions \
:::info
✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/guardrails/noma_security.md
+102
View File
@@ -6,6 +6,108 @@ import TabItem from '@theme/TabItem';
Use [Noma Security](https://noma.security/) to protect your LLM applications with comprehensive AI content moderation and safety guardrails.
:::warning Deprecated: `guardrail: noma` (Legacy)
`guardrail: noma` is deprecated and users should migrate to `guardrail: noma_v2`.
The legacy `guardrail: noma` API will no longer be supported after March 31, 2026.
For easier migration of existing integrations, keep `guardrail: noma` and set `use_v2: true`.
With `use_v2: true`, requests route to `noma_v2`; `monitor_mode` and `block_failures` still apply, while `anonymize_input` is ignored.
:::
## Noma v2 guardrails (Recommended)
### Quick Start
```yaml showLineNumbers title="litellm config.yaml"
guardrails:
- guardrail_name: "noma-v2-guard"
litellm_params:
guardrail: noma_v2
mode: "pre_call"
api_key: os.environ/NOMA_API_KEY
api_base: os.environ/NOMA_API_BASE
```
If you want to migrate gradually without changing guardrail names yet:
```yaml showLineNumbers title="litellm config.yaml"
guardrails:
- guardrail_name: "noma-guard"
litellm_params:
guardrail: noma
use_v2: true
mode: "pre_call"
api_key: os.environ/NOMA_API_KEY
api_base: os.environ/NOMA_API_BASE
```
### Supported Params
- **`guardrail`**: Use `noma_v2` (recommended), or `noma` with `use_v2: true` for migration
- **`mode`**: `pre_call`, `post_call`, `during_call`, `pre_mcp_call`, `during_mcp_call`
- **`api_key`**: Noma API key (required for Noma SaaS, optional for self-managed deployments)
- **`api_base`**: Noma API base URL (defaults to `https://api.noma.security/`)
- **`application_id`**: Application identifier. If omitted, v2 checks dynamic `extra_body.application_id`, then configured/env `application_id`; otherwise it is omitted.
- **`monitor_mode`**: If `true`, runs in monitor-only mode without blocking (defaults to `false`)
- **`block_failures`**: If `true`, fail-closed on guardrail technical failures (defaults to `true`)
- **`use_v2`**: Migration toggle when `guardrail: noma` is used
### Environment Variables
```shell
export NOMA_API_KEY="your-api-key-here"
export NOMA_API_BASE="https://api.noma.security/" # Optional
export NOMA_APPLICATION_ID="my-app" # Optional
export NOMA_MONITOR_MODE="false" # Optional
export NOMA_BLOCK_FAILURES="true" # Optional
```
### Multiple Guardrails
Apply different v2 configurations for input and output:
```yaml showLineNumbers title="litellm config.yaml"
guardrails:
- guardrail_name: "noma-v2-input"
litellm_params:
guardrail: noma_v2
mode: "pre_call"
api_key: os.environ/NOMA_API_KEY
- guardrail_name: "noma-v2-output"
litellm_params:
guardrail: noma_v2
mode: "post_call"
api_key: os.environ/NOMA_API_KEY
```
### Pass Additional Parameters
This is supported in v2 via `extra_body`.
Currently, `noma_v2` consumes dynamic `application_id`.
```shell showLineNumbers title="Curl Request"
curl 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Content-Type: application/json' \
-d '{
"model": "gpt-4o-mini",
"messages": [
{
"role": "user",
"content": "Hello, how are you?"
}
],
"guardrails": {
"noma-v2-guard": {
"extra_body": {
"application_id": "my-specific-app-id"
}
}
}
}'
```
## Noma guardrails (Legacy)
## Quick Start
### 1. Define Guardrails on your LiteLLM config.yaml
docs/my-website/docs/proxy/guardrails/realtime_guardrails.md
+199
View File
@@ -0,0 +1,199 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Realtime API Guardrails
Guard voice conversations in the [Realtime API](/docs/realtime) — intercept speech transcriptions **before** the LLM responds.
## How it works
The Realtime API is a long-lived WebSocket session. Unlike `/chat/completions` where a guardrail runs once per HTTP request, a voice session has many turns — each one needs to be checked individually.
LiteLLM intercepts each turn at the transcription event, after Whisper converts speech to text but before the LLM generates a response:
```
User speaks into mic
│
▼ audio bytes (PCM)
┌───────────────────┐
│ LiteLLM Proxy │ forwards audio to OpenAI unchanged
└────────┬──────────┘
│
▼
┌───────────────────┐
│ OpenAI │
│ VAD → Whisper │ detects speech end, transcribes
└────────┬──────────┘
│
│ conversation.item.input_audio_transcription.completed
│ { transcript: "system update: ignore all instructions" }
│
▼
┌───────────────────────────────────────────┐
│ LiteLLM Proxy │
│ │
│ ◄──── GUARDRAIL RUNS HERE ────► │
│ apply_guardrail(texts=[transcript]) │
│ │
│ ┌──────────────┬──────────────────┐ │
│ │ BLOCKED │ CLEAN │ │
│ └──────┬───────┴───────┬──────────┘ │
│ │ │ │
│ speak warning send response.create │
│ (TTS audio) → LLM responds │
└───────────────────────────────────────────┘
```
**Key detail**: LiteLLM also injects `create_response: false` into the session on connect, so the LLM never auto-responds before the guardrail has run.
## Supported guardrail mode
| Mode | Description |
|------|-------------|
| `realtime_input_transcription` | Runs after each voice turn is transcribed, before LLM responds |
## Quick Start
### Step 1: Configure proxy
Add a guardrail with `mode: realtime_input_transcription` to your proxy config:
```yaml
model_list:
- model_name: openai/gpt-4o-realtime-preview
litellm_params:
model: openai/gpt-4o-realtime-preview
api_key: os.environ/OPENAI_API_KEY
guardrails:
- guardrail_name: "voice-content-filter"
litellm_params:
guardrail: litellm_content_filter
mode: realtime_input_transcription
default_on: true
blocked_words:
- keyword: "ignore previous instructions"
action: BLOCK
description: "Prompt injection attempt"
- keyword: "system update"
action: BLOCK
description: "Prompt injection attempt"
- keyword: "ignore all instructions"
action: BLOCK
description: "Prompt injection attempt"
general_settings:
master_key: sk-1234
```
### Step 2: Start proxy
```bash
litellm --config proxy_config.yaml --port 4000
```
### Step 3: Connect a Realtime client
Connect your client to the proxy instead of directly to OpenAI:
<Tabs>
<TabItem value="js" label="JavaScript">
```javascript
const ws = new WebSocket(
"ws://localhost:4000/v1/realtime?model=openai/gpt-4o-realtime-preview",
[],
{ headers: { Authorization: "Bearer sk-1234" } }
)
ws.onopen = () => {
ws.send(JSON.stringify({
type: "session.update",
session: {
modalities: ["audio", "text"],
input_audio_transcription: { model: "whisper-1" },
turn_detection: { type: "server_vad" },
},
}))
}
ws.onmessage = (e) => {
const event = JSON.parse(e.data)
if (event.type === "response.audio.delta") {
// play audio...
}
}
```
</TabItem>
<TabItem value="python" label="Python">
```python
import asyncio
import json
import websockets
async def main():
async with websockets.connect(
"ws://localhost:4000/v1/realtime?model=openai/gpt-4o-realtime-preview",
additional_headers={"Authorization": "Bearer sk-1234"},
) as ws:
await ws.recv() # session.created
await ws.send(json.dumps({
"type": "session.update",
"session": {
"modalities": ["audio", "text"],
"input_audio_transcription": {"model": "whisper-1"},
"turn_detection": {"type": "server_vad"},
},
}))
async for raw in ws:
event = json.loads(raw)
print(event["type"])
asyncio.run(main())
```
</TabItem>
</Tabs>
### What happens when a turn is blocked
When the guardrail fires, the proxy:
1. Sends `response.cancel` to kill any in-flight LLM response
2. Sends `response.create` with the block message as forced instructions
3. OpenAI's TTS **speaks the warning** back to the user — e.g. *"Content blocked: keyword 'system update' detected (Prompt injection attempt)"*
The LLM never processes the injected instruction.
## Using with any guardrail provider
`realtime_input_transcription` mode works with any guardrail that implements `apply_guardrail`. Just swap `litellm_content_filter` for your provider:
```yaml
guardrails:
- guardrail_name: "voice-lakera"
litellm_params:
guardrail: lakera_ai
mode: realtime_input_transcription
default_on: true
api_key: os.environ/LAKERA_API_KEY
```
## Per-key guardrail control
To enable realtime guardrails only for specific API keys, set `default_on: false` and pass the guardrail name in the request metadata:
```yaml
guardrails:
- guardrail_name: "voice-content-filter"
litellm_params:
guardrail: litellm_content_filter
mode: realtime_input_transcription
default_on: false # off by default
```
Then the client opts in per-connection by passing it in the initial metadata (enterprise feature).
docs/my-website/docs/proxy/ip_address.md
+1 -1
View File
@@ -3,7 +3,7 @@
:::info
You need a LiteLLM License to unlock this feature. [Grab time](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat), to get one today!
You need a LiteLLM License to unlock this feature. [Grab time](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions), to get one today!
:::
docs/my-website/docs/proxy/logging.md
+3 -3
View File
@@ -1109,7 +1109,7 @@ Log LLM Logs to [Google Cloud Storage Buckets](https://cloud.google.com/storage?
:::info
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
@@ -1194,7 +1194,7 @@ Log LLM Logs/SpendLogs to [Google Cloud Storage PubSub Topic](https://cloud.goog
:::info
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
@@ -1497,7 +1497,7 @@ Log LLM Logs to [Azure Data Lake Storage](https://learn.microsoft.com/en-us/azur
:::info
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/multiple_admins.md
+1 -1
View File
@@ -20,7 +20,7 @@ LiteLLM tracks changes to the following entities and actions:
:::tip
Requires Enterprise License, Get in touch with us [here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
Requires Enterprise License, Get in touch with us [here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/oauth2.md
+1 -1
View File
@@ -4,7 +4,7 @@ Use this if you want to use an Oauth2.0 token to make `/chat`, `/embeddings` req
:::info
This is an Enterprise Feature - [get in touch with us if you want a free trial to test if this feature meets your needs]((https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat))
This is an Enterprise Feature - [get in touch with us if you want a free trial to test if this feature meets your needs]((https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions))
:::
docs/my-website/docs/proxy/prod.md
+1 -1
View File
@@ -47,7 +47,7 @@ export LITELLM_LOG="ERROR"
:::info
Need Help or want dedicated support ? Talk to a founder [here]: (https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
Need Help or want dedicated support ? Talk to a founder [here]: (https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/prometheus.md
+24 -2
View File
@@ -122,7 +122,7 @@ Use this to track overall LiteLLM Proxy usage.
| Metric Name | Description |
|----------------------|--------------------------------------|
| `litellm_proxy_failed_requests_metric` | Total number of failed responses from proxy - the client did not get a success response from litellm proxy. Labels: `"end_user", "hashed_api_key", "api_key_alias", "requested_model", "team", "team_alias", "user", "user_email", "exception_status", "exception_class", "route", "model_id"` |
| `litellm_proxy_total_requests_metric` | Total number of requests made to the proxy server - track number of client side requests. Labels: `"end_user", "hashed_api_key", "api_key_alias", "requested_model", "team", "team_alias", "user", "status_code", "user_email", "route", "model_id"` |
| `litellm_proxy_total_requests_metric` | Total number of requests made to the proxy server - track number of client side requests. Labels: `"end_user", "hashed_api_key", "api_key_alias", "requested_model", "team", "team_alias", "user", "status_code", "user_email", "route", "model_id"`. Optionally includes `"stream"` — see [Emit Stream Label](#emit-stream-label). |
### Callback Logging Metrics
@@ -214,9 +214,31 @@ litellm_settings:
```
### Emit Stream Label
Add a `stream` label to `litellm_proxy_total_requests_metric` to split requests by streaming vs. non-streaming. Disabled by default.
```yaml title="config.yaml"
litellm_settings:
callbacks: ["prometheus"]
prometheus_emit_stream_label: true
```
When enabled, `litellm_proxy_total_requests_metric` gains a `stream` label with values `"True"`, `"False"`, or `"None"`.
```
litellm_proxy_total_requests_metric{..., stream="True"} 42
litellm_proxy_total_requests_metric{..., stream="False"} 100
```
:::note
This label is opt-in because adding a new label to an existing metric changes its cardinality and breaks existing Prometheus queries / Grafana dashboards that target this metric. Enable it only on fresh deployments or when you are ready to update your dashboards.
:::
## [BETA] Custom Metrics
Track custom metrics on prometheus on all events mentioned above.
Track custom metrics on prometheus on all events mentioned above.
### Custom Metadata Labels
docs/my-website/docs/proxy/public_routes.md
+1 -1
View File
@@ -5,7 +5,7 @@ import TabItem from '@theme/TabItem';
:::info
Requires a LiteLLM Enterprise License. [Get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat).
Requires a LiteLLM Enterprise License. [Get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions).
:::
docs/my-website/docs/proxy/tag_routing.md
+1 -1
View File
@@ -215,7 +215,7 @@ LiteLLM Proxy supports team-based tag routing, allowing you to associate specifi
:::info
This is an enterprise feature, [Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
This is an enterprise feature, [Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/team_logging.md
+2 -2
View File
@@ -26,7 +26,7 @@ Team 3 -> Disabled Logging (for GDPR compliance)
:::info
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
@@ -248,7 +248,7 @@ Use the `/key/generate` or `/key/update` endpoints to add logging callbacks to a
:::info
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/team_model_add.md
+1 -1
View File
@@ -5,7 +5,7 @@
This is an Enterprise feature.
[Enterprise Pricing](https://www.litellm.ai/#pricing)
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/token_auth.md
+1 -1
View File
@@ -11,7 +11,7 @@ Use JWT's to auth admins / users / projects into the proxy.
[Enterprise Pricing](https://www.litellm.ai/#pricing)
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/proxy/ui_store_model_db_setting.md
+92
View File
@@ -0,0 +1,92 @@
import Image from '@theme/IdealImage';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Store Model in DB Settings
Enable or disable storing model definitions in the database directly from the Admin UI—no config file edits or proxy restart required. This is especially useful for cloud deployments where updating the config is difficult or requires a long release process.
## Overview
Previously, the `store_model_in_db` setting had to be configured in `proxy_config.yaml` under `general_settings`. Changing it required editing the config and restarting the proxy, which was problematic for cloud users who don't have direct access to the config file or who want to avoid the downtime caused by restarts.
<Image img={require('../../img/ui_store_model_in_db.png')} />
**Store Model in DB Settings** lets you:
- **Enable or disable storing models in the database** – Control whether model definitions are cached in your database (useful for reducing config file size and improving scalability)
- **Apply changes immediately** – No proxy restart needed; settings take effect for new model operations as soon as you save
:::warning UI overrides config
Settings changed in the UI **override** the values in your config file. For example, if `store_model_in_db` is set to `false` in `general_settings`, enabling it in the UI will still persist model definitions to the database. Use the UI when you want runtime control without redeploying.
:::
## How Store Model in DB Works
When `store_model_in_db` is enabled, the LiteLLM proxy stores model definitions in the database instead of relying solely on your `proxy_config.yaml`. This provides several benefits:
- **Reduced config size** – Move model definitions out of YAML for easier maintenance
- **Scalability** – Database storage scales better than large YAML files
- **Dynamic updates** – Models can be added or updated without editing config files
- **Persistence** – Model definitions persist across proxy instances and restarts
The setting applies to all new model operations from the moment you save it.
## How to Configure Store Model in DB in the UI
### 1. Access Models + Endpoints Settings
Navigate to the Admin UI (e.g. `http://localhost:4000/ui` or your `PROXY_BASE_URL/ui`) and go to the **Models + Endpoints** page.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-22/55bc71f5-730f-4b2c-8539-8a4f46b8bd10/ascreenshot_0f7ba8f1c2694e94938996fd1b4adfcc_text_export.jpeg)
### 2. Open Settings
Click **Models + Endpoints** from the navigation menu.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-22/55bc71f5-730f-4b2c-8539-8a4f46b8bd10/ascreenshot_fc2b9e4812a9480087f4eb350fa0a792_text_export.jpeg)
### 3. Click the Settings Icon
Look for the settings (gear) icon on the Models + Endpoints page to open the configuration panel.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-22/7b394364-c281-4db8-8cad-ee322c76c935/ascreenshot_d7c8a6b234bc4e4d92aa7f09aefb13d3_text_export.jpeg)
### 4. Enable or Disable Store Model in DB
Toggle the **Store Model in DB** setting based on your preference:
- **Enabled**: Model definitions will be stored in the database
- **Disabled**: Models are read from the config file only
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-22/54a263ec-ad67-4b16-ba9f-2be57c3e4cb8/ascreenshot_501abda2a6c847f79d085efce814265d_text_export.jpeg)
### 5. Save Settings
Click **Save Settings** to apply the change. No proxy restart is required; the new setting takes effect immediately for subsequent model operations.
![](https://colony-recorder.s3.amazonaws.com/files/2026-02-22/7d13559a-d4e4-41f7-993b-cb20fbfa1f6e/ascreenshot_3245f3c5bd0d43cb96c5f5ff0ccb461d_text_export.jpeg)
## Use Cases
### Cloud and Managed Deployments
When the proxy runs in a managed or cloud environment, config may be in a separate repo, require a long release cycle, or be controlled by another team. Using the UI lets you change the `store_model_in_db` setting without going through a deployment process.
### Reducing Configuration Complexity
For large deployments with hundreds of models, storing model definitions in the database reduces the size and complexity of your `proxy_config.yaml`, making it easier to maintain and version control.
### Dynamic Model Management
Enable `store_model_in_db` to support dynamic model additions and updates without editing your config file. Teams can manage models through the UI or API without needing to redeploy the proxy.
### Zero-Downtime Updates
Change the setting from the UI and have it take effect immediately—perfect for production environments where downtime must be minimized.
## Related Documentation
- [Admin UI Overview](./ui_overview.md) – General guide to the LiteLLM Admin UI
- [Models and Endpoints](./models_and_endpoints.md) – Managing models and API endpoints
- [Config Settings](./config_settings.md) – `store_model_in_db` in `general_settings`
docs/my-website/docs/realtime.md
+1 -1
View File
@@ -85,7 +85,7 @@ const url = "ws://0.0.0.0:4000/v1/realtime?model=openai-gpt-4o-realtime-audio";
// const url = "wss://my-endpoint-sweden-berri992.openai.azure.com/openai/realtime?api-version=2024-10-01-preview&deployment=gpt-4o-realtime-preview";
const ws = new WebSocket(url, {
headers: {
"api-key": `f28ab7b695af4154bc53498e5bdccb07`,
"api-key": `sk-1234`,
"OpenAI-Beta": "realtime=v1",
},
});
docs/my-website/docs/reasoning_content.md
+19
View File
@@ -642,6 +642,25 @@ model_list:
model: openai/responses/gpt-5-mini
```
**Per-model configuration** (recommended when using Open WebUI or clients that cannot set `extra_body`):
```yaml
model_list:
- model_name: gpt-5.1
litellm_params:
model: openai/gpt-5.1
# String format - uses reasoning_auto_summary for summary when set
reasoning_effort: "high"
model_info:
mode: responses # if using Responses API bridge
- model_name: gpt-5.1-with-summary
litellm_params:
model: openai/gpt-5.1
# Dict format - explicit control over effort and summary
reasoning_effort: {"effort": "high", "summary": "detailed"}
```
</TabItem>
</Tabs>
docs/my-website/docs/response_api.md
+5 -1
View File
@@ -887,7 +887,8 @@ router = litellm.Router(
# `responses_api_deployment_check` ensures Requests with `previous_response_id`
# are routed to the same deployment. `deployment_affinity` adds sticky sessions
# for requests without `previous_response_id` (useful for implicit caching).
optional_pre_call_checks=["responses_api_deployment_check", "deployment_affinity"],
# `session_affinity` adds sticky sessions based on `session_id` metadata.
optional_pre_call_checks=["responses_api_deployment_check", "deployment_affinity", "session_affinity"],
# Optional (default is 3600 seconds / 1 hour)
deployment_affinity_ttl_seconds=3600,
)
@@ -919,10 +920,12 @@ follow_up = await router.aresponses(
To enable session continuity for Responses API in your LiteLLM proxy, set `optional_pre_call_checks` in your proxy config.yaml.
- `responses_api_deployment_check`: high priority routing when `previous_response_id` is provided
- `session_affinity`: sticky sessions based on session id (takes priority over `deployment_affinity`)
- `deployment_affinity`: sticky sessions based on user key (applies even without `previous_response_id`)
Notes:
- User-key affinity is keyed on `metadata.user_api_key_hash` (the API key hash). The OpenAI `user` request parameter is an end-user identifier and is intentionally not used for deployment affinity.
- Session-ID affinity is keyed on `metadata.session_id`. For proxy requests, this can be passed via the `x-litellm-session-id` HTTP header. For Python SDK requests, you can pass it via `litellm_metadata={"session_id": "value"}` in request args.
- `user_api_key_hash` is already SHA-256, and is used as-is (no double hashing).
- Affinity is scoped by a stable model identifier (the model-map key, e.g. `model_map_information.model_map_key`) so model aliases map to the same stickiness bucket.
- The mapping TTL is controlled by `deployment_affinity_ttl_seconds` (configured on Router init / proxy startup).
@@ -945,6 +948,7 @@ model_list:
router_settings:
optional_pre_call_checks:
- responses_api_deployment_check
- session_affinity
- deployment_affinity
# Optional (default is 3600 seconds / 1 hour)
deployment_affinity_ttl_seconds: 3600
docs/my-website/docs/secret.md
+1 -1
View File
@@ -6,7 +6,7 @@
[Enterprise Pricing](https://www.litellm.ai/#pricing)
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/secret_managers/aws_kms.md
+1 -1
View File
@@ -6,7 +6,7 @@
[Enterprise Pricing](https://www.litellm.ai/#pricing)
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/secret_managers/aws_secret_manager.md
+1 -1
View File
@@ -9,7 +9,7 @@ import TabItem from '@theme/TabItem';
[Enterprise Pricing](https://www.litellm.ai/#pricing)
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/secret_managers/azure_key_vault.md
+1 -1
View File
@@ -6,7 +6,7 @@
[Enterprise Pricing](https://www.litellm.ai/#pricing)
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/secret_managers/cyberark.md
+1 -1
View File
@@ -8,7 +8,7 @@ import Image from '@theme/IdealImage';
[Enterprise Pricing](https://www.litellm.ai/#pricing)
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/secret_managers/google_kms.md
+1 -1
View File
@@ -6,7 +6,7 @@
[Enterprise Pricing](https://www.litellm.ai/#pricing)
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/secret_managers/google_secret_manager.md
+1 -1
View File
@@ -6,7 +6,7 @@
[Enterprise Pricing](https://www.litellm.ai/#pricing)
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/secret_managers/hashicorp_vault.md
+1 -1
View File
@@ -8,7 +8,7 @@ import Image from '@theme/IdealImage';
[Enterprise Pricing](https://www.litellm.ai/#pricing)
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/secret_managers/overview.md
+1 -1
View File
@@ -8,7 +8,7 @@ import Image from '@theme/IdealImage';
[Enterprise Pricing](https://www.litellm.ai/#pricing)
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
:::
docs/my-website/docs/troubleshoot/prisma_migrations.md
+6 -2
View File
@@ -2,6 +2,8 @@
Common Prisma migration issues encountered when upgrading or downgrading LiteLLM proxy versions, and how to fix them.
For a full guide on safely reverting your LiteLLM version, see the **[Safe Rollback Guide](rollback)**.
## How Prisma Migrations Work in LiteLLM
- LiteLLM uses [Prisma](https://www.prisma.io/) to manage its PostgreSQL database schema.
@@ -46,6 +48,8 @@ After deleting the entry, restart LiteLLM — it will re-apply the migration on
If deleting the migration entry and restarting doesn't resolve the issue, sync the schema directly:
> **Warning:** `prisma db push` can cause **data loss** if the Prisma schema removes columns or tables that exist in your database. Only use this as a last resort and ensure you have a database backup first.
```bash
DATABASE_URL="<your_database_url>" prisma db push
```
@@ -76,7 +80,7 @@ DELETE FROM "_prisma_migrations"
WHERE migration_name = '<failed_migration_name>';
```
3. If that doesn't work, use `prisma db push`:
3. If that doesn't work, use `prisma db push` (see [warning above](#step-2--if-that-doesnt-work-use-prisma-db-push) — back up your database first):
```bash
DATABASE_URL="<your_database_url>" prisma db push
@@ -106,7 +110,7 @@ LIMIT 20;
3. Restart LiteLLM to re-run migrations.
4. If that doesn't work, use `prisma db push`:
4. If that doesn't work, use `prisma db push` (see [warning above](#step-2--if-that-doesnt-work-use-prisma-db-push) — back up your database first):
```bash
DATABASE_URL="<your_database_url>" prisma db push
docs/my-website/docs/troubleshoot/rollback.md
+115
View File
@@ -0,0 +1,115 @@
# Safe Rollback Guide
This guide outlines the process for safely rolling back a LiteLLM Proxy deployment to a previous version.
We recommend rolling back to the previous [stable release](https://github.com/BerriAI/litellm/releases). Stable releases come out every week and follow the `main-v<VERSION>-stable` tag convention (e.g., `main-v1.77.2-stable`).
## 1. Determine Rollback Scope
Before proceeding, identify why you are rolling back:
- **Application Logic Error**: Reverting code changes but keeping the database schema.
- **Database Migration Failure**: Reverting changes that included database schema updates.
- **Performance Regression**: Reverting to a known stable version.
## 2. Back Up the Database
> **Always back up before rolling back.** Before making any changes, take a database snapshot or dump. This is your safety net if something goes wrong during the rollback.
```bash
# PostgreSQL example
pg_dump -h <host> -U <user> -d <database> -F c -f litellm_backup_$(date +%Y%m%d_%H%M%S).dump
```
If you are on a managed database (e.g., AWS RDS, GCP Cloud SQL), create a snapshot through your cloud console instead.
## 3. Pre-Rollback Checks
Before reverting, review these items:
- **`LITELLM_SALT_KEY`**: Do **not** change this value during rollback. It is used to encrypt/decrypt your LLM API Key credentials stored in the database. Changing it will make existing credentials unreadable. See [Best Practices for Production](../proxy/prod#8-set-litellm-salt-key).
- **`config.yaml`**: If you added settings specific to the newer version, the older version may not recognize them. Review your config and remove or comment out any settings that were introduced in the version you are rolling back from.
- **`DISABLE_SCHEMA_UPDATE`**: If you use the [Helm PreSync hook for migrations](../proxy/prod#7-use-helm-presync-hook-for-database-migrations-beta) with `DISABLE_SCHEMA_UPDATE=true` on your pods, migrations will **not** auto-run on restart. You will need to handle migration cleanup manually (see Step 5) or re-run the PreSync hook against the older chart version.
## 4. Revert Application Version
Revert your deployment to the previous stable Docker image or Helm chart version.
### Docker
Update your deployment manifest (e.g., K8s Deployment, Docker Compose) to use the previous version:
```yaml
# Example: Reverting to the previous stable release
image: docker.litellm.ai/berriai/litellm:main-v<VERSION>-stable
```
See [all available images](https://github.com/orgs/BerriAI/packages).
### Helm
If you deployed via Helm, use `helm rollback`:
```bash
helm rollback <release-name> [revision-number]
```
## 5. Handle Database Migrations
If you are rolling back to a version that did not have a specific migration, you may need to resolve the migration state in the database.
> LiteLLM uses `prisma migrate deploy` for production (enabled via `USE_PRISMA_MIGRATE=True`). If a migration partially failed or you are reverting code that expects an older schema, you need to clean up the migration history in the `_prisma_migrations` table. See [Best Practices for Production](../proxy/prod#9-use-prisma-migrate-deploy).
### Option A — Delete stale migration entries (recommended)
Connect to your PostgreSQL database and remove migration entries that belong to the version you are rolling back from. This lets LiteLLM re-apply them cleanly if you upgrade again later.
```sql
-- View recent migrations
SELECT migration_name, finished_at, rolled_back_at, logs
FROM "_prisma_migrations"
ORDER BY started_at DESC
LIMIT 10;
-- Delete migration entries from the version you are rolling back from
DELETE FROM "_prisma_migrations"
WHERE migration_name = '<migration_name_from_newer_version>';
```
After deleting the entries, restart LiteLLM — it will re-apply the correct migrations for its version on startup.
> **Note:** If you have `DISABLE_SCHEMA_UPDATE=true` set on your pods, migrations will not auto-run. You need to either temporarily set it to `false`, or re-run the Helm PreSync migration job targeting the older version.
### Option B — Use `prisma migrate resolve` (if you have CLI access)
If you have access to the Prisma CLI (e.g., in a local development environment or a debug container with the `litellm-proxy-extras` package installed):
```bash
DATABASE_URL="<your_database_url>" prisma migrate resolve --rolled-back "<migration_name>"
```
> **Note:** This requires the Prisma CLI to be available in your environment (installed via `prisma-client-py`). If you don't have CLI access (e.g., no shell into the running container), use **Option A** (direct SQL) instead.
### Auto-Recovery Logic
LiteLLM's internal `ProxyExtrasDBManager` automatically attempts to handle idempotent migrations. In many cases, simply rolling back the version and restarting the proxy will be enough if the database changes are additive (e.g., new columns or tables).
## 6. Verification Checklist
After rolling back, verify the health of the system:
- [ ] **Health Endpoint**: Confirm the `/health` endpoint returns `200 OK`.
- [ ] **Check Logs**: Ensure no Prisma errors appear — look for `relation "..." does not exist`, `column "..." does not exist`, or `prisma migrate` failures in the logs.
- [ ] **Spend Tracking**: Run a test completion and confirm the spend is recorded in the `LiteLLM_SpendLogs` table.
- [ ] **Billing (Lago)**: If using Lago for billing (e.g., Lago → Stripe), check proxy logs for `Logged Lago Object` to confirm usage events are being sent.
- [ ] **State Consistency**: If using Redis for caching or rate limiting, consider clearing the cache if the newer version changed the cache key structure.
- [ ] **Admin UI**: Verify the Admin UI loads and shows correct data for keys and teams.
## 7. Troubleshooting
### "New migrations cannot be applied"
If you see this error after a rollback, it means the database has a migration in a "failed" state.
1. Identify the failed migration name (see the SQL query in Step 5).
2. Delete the failed entry from `_prisma_migrations`.
3. Restart the proxy.
### "relation X does not exist"
This typically means a migration entry exists in `_prisma_migrations` but the actual table/column was never created or was dropped.
1. Delete the stale migration entry.
2. Restart LiteLLM so it re-runs the migration.
For more details on Prisma errors, see [Prisma Migrations Troubleshoot](prisma_migrations).
docs/my-website/docs/tutorials/compare_llms.md
+1 -1
View File
@@ -82,7 +82,7 @@ Benchmark Results for 'When will BerriAI IPO?':
+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
```
## Support
**🤝 Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
**🤝 Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
<!--
docs/my-website/docs/tutorials/google_genai_sdk.md
+406
View File
@@ -0,0 +1,406 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Google GenAI SDK with LiteLLM
Use Google's official GenAI SDK (JavaScript/TypeScript and Python) with any LLM provider through LiteLLM Proxy.
The Google GenAI SDK (`@google/genai` for JS, `google-genai` for Python) provides a native interface for calling Gemini models. By pointing it to LiteLLM, you can use the same SDK with OpenAI, Anthropic, Bedrock, Azure, Vertex AI, or any other provider — while keeping the native Gemini request/response format.
## Why Use LiteLLM with Google GenAI SDK?
**Developer Benefits:**
- **Universal Model Access**: Use any LiteLLM-supported model (Anthropic, OpenAI, Vertex AI, Bedrock, etc.) through the Google GenAI SDK interface
- **Higher Rate Limits & Reliability**: Load balance across multiple models and providers to avoid hitting individual provider limits, with fallbacks to ensure you get responses even if one provider fails
**Proxy Admin Benefits:**
- **Centralized Management**: Control access to all models through a single LiteLLM proxy instance without giving developers API keys to each provider
- **Budget Controls**: Set spending limits and track costs across all SDK usage
- **Logging & Observability**: Track all requests with cost tracking, logging, and analytics
| Feature | Supported | Notes |
|---------|-----------|-------|
| Cost Tracking | ✅ | All models on `/generateContent` endpoint |
| Logging | ✅ | Works across all integrations |
| Streaming | ✅ | `streamGenerateContent` supported |
| Virtual Keys | ✅ | Use LiteLLM keys instead of Google keys |
| Load Balancing | ✅ | Via native router endpoints |
| Fallbacks | ✅ | Via native router endpoints |
## Quick Start
### 1. Install the SDK
<Tabs>
<TabItem value="js" label="JavaScript/TypeScript">
```bash
npm install @google/genai
```
</TabItem>
<TabItem value="python" label="Python">
```bash
pip install google-genai
```
</TabItem>
</Tabs>
### 2. Start LiteLLM Proxy
```yaml title="config.yaml" showLineNumbers
model_list:
- model_name: gemini-2.5-flash
litellm_params:
model: gemini/gemini-2.5-flash
api_key: os.environ/GEMINI_API_KEY
```
```bash
litellm --config config.yaml
```
### 3. Call the SDK through LiteLLM
<Tabs>
<TabItem value="js" label="JavaScript/TypeScript">
```javascript title="index.js" showLineNumbers
const { GoogleGenAI } = require("@google/genai");
const ai = new GoogleGenAI({
apiKey: "sk-1234", // LiteLLM virtual key (not a Google key)
httpOptions: {
baseUrl: "http://localhost:4000/gemini", // LiteLLM proxy URL
},
});
async function main() {
const response = await ai.models.generateContent({
model: "gemini-2.5-flash",
contents: "Explain how AI works",
});
console.log(response.text);
}
main();
```
</TabItem>
<TabItem value="python" label="Python">
```python title="main.py" showLineNumbers
from google import genai
client = genai.Client(
api_key="sk-1234", # LiteLLM virtual key (not a Google key)
http_options={"base_url": "http://localhost:4000/gemini"}, # LiteLLM proxy URL
)
response = client.models.generate_content(
model="gemini-2.5-flash",
contents="Explain how AI works",
)
print(response.text)
```
</TabItem>
<TabItem value="curl" label="curl">
```bash
curl "http://localhost:4000/gemini/v1beta/models/gemini-2.5-flash:generateContent?key=sk-1234" \
-H 'Content-Type: application/json' \
-X POST \
-d '{
"contents": [{
"parts": [{"text": "Explain how AI works"}]
}]
}'
```
</TabItem>
</Tabs>
## Streaming
<Tabs>
<TabItem value="js" label="JavaScript/TypeScript">
```javascript title="streaming.js" showLineNumbers
const { GoogleGenAI } = require("@google/genai");
const ai = new GoogleGenAI({
apiKey: "sk-1234",
httpOptions: {
baseUrl: "http://localhost:4000/gemini",
},
});
async function main() {
const response = await ai.models.generateContentStream({
model: "gemini-2.5-flash",
contents: "Write a short poem about the ocean",
});
for await (const chunk of response) {
process.stdout.write(chunk.text);
}
}
main();
```
</TabItem>
<TabItem value="python" label="Python">
```python title="streaming.py" showLineNumbers
from google import genai
client = genai.Client(
api_key="sk-1234",
http_options={"base_url": "http://localhost:4000/gemini"},
)
response = client.models.generate_content_stream(
model="gemini-2.5-flash",
contents="Write a short poem about the ocean",
)
for chunk in response:
print(chunk.text, end="")
```
</TabItem>
</Tabs>
## Multi-turn Chat
<Tabs>
<TabItem value="js" label="JavaScript/TypeScript">
```javascript title="chat.js" showLineNumbers
const { GoogleGenAI } = require("@google/genai");
const ai = new GoogleGenAI({
apiKey: "sk-1234",
httpOptions: {
baseUrl: "http://localhost:4000/gemini",
},
});
async function main() {
const chat = ai.chats.create({
model: "gemini-2.5-flash",
});
const response1 = await chat.sendMessage({ message: "I have 2 dogs and 3 cats." });
console.log(response1.text);
const response2 = await chat.sendMessage({ message: "How many pets is that in total?" });
console.log(response2.text);
}
main();
```
</TabItem>
<TabItem value="python" label="Python">
```python title="chat.py" showLineNumbers
from google import genai
client = genai.Client(
api_key="sk-1234",
http_options={"base_url": "http://localhost:4000/gemini"},
)
chat = client.chats.create(model="gemini-2.5-flash")
response1 = chat.send_message("I have 2 dogs and 3 cats.")
print(response1.text)
response2 = chat.send_message("How many pets is that in total?")
print(response2.text)
```
</TabItem>
</Tabs>
## Advanced: Use Any Model with the GenAI SDK
By default, the GenAI SDK talks to Gemini models. But with LiteLLM's router, you can route GenAI SDK requests to **any provider** — Anthropic, OpenAI, Bedrock, etc.
This works by using `model_group_alias` to map Gemini model names to your desired provider models. LiteLLM handles the format translation internally.
:::info
For this to work, point the SDK `baseUrl` to `http://localhost:4000` (without `/gemini`). This routes requests through LiteLLM's native Google endpoints, which go through the router and support model aliasing.
:::
<Tabs>
<TabItem value="anthropic" label="Anthropic">
Route `gemini-2.5-flash` requests to Claude Sonnet:
```yaml title="config.yaml" showLineNumbers
model_list:
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY
router_settings:
model_group_alias: {"gemini-2.5-flash": "claude-sonnet"}
```
</TabItem>
<TabItem value="openai" label="OpenAI">
Route `gemini-2.5-flash` requests to GPT-4o:
```yaml title="config.yaml" showLineNumbers
model_list:
- model_name: gpt-4o-model
litellm_params:
model: gpt-4o
api_key: os.environ/OPENAI_API_KEY
router_settings:
model_group_alias: {"gemini-2.5-flash": "gpt-4o-model"}
```
</TabItem>
<TabItem value="bedrock" label="Bedrock">
Route `gemini-2.5-flash` requests to Claude on Bedrock:
```yaml title="config.yaml" showLineNumbers
model_list:
- model_name: bedrock-claude
litellm_params:
model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
aws_region_name: us-east-1
router_settings:
model_group_alias: {"gemini-2.5-flash": "bedrock-claude"}
```
</TabItem>
<TabItem value="multi" label="Multi-Provider Load Balancing">
Load balance across Anthropic and OpenAI:
```yaml title="config.yaml" showLineNumbers
model_list:
- model_name: my-model
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: my-model
litellm_params:
model: gpt-4o
api_key: os.environ/OPENAI_API_KEY
router_settings:
model_group_alias: {"gemini-2.5-flash": "my-model"}
```
</TabItem>
</Tabs>
Then use the SDK with `baseUrl` pointing to LiteLLM (without `/gemini`):
<Tabs>
<TabItem value="js" label="JavaScript/TypeScript">
```javascript title="any_model.js" showLineNumbers
const { GoogleGenAI } = require("@google/genai");
const ai = new GoogleGenAI({
apiKey: "sk-1234",
httpOptions: {
baseUrl: "http://localhost:4000", // No /gemini — goes through the router
},
});
async function main() {
// This calls Claude/GPT-4o/Bedrock under the hood via model_group_alias
const response = await ai.models.generateContent({
model: "gemini-2.5-flash",
contents: "Hello from any model!",
});
console.log(response.text);
}
main();
```
</TabItem>
<TabItem value="python" label="Python">
```python title="any_model.py" showLineNumbers
from google import genai
client = genai.Client(
api_key="sk-1234",
http_options={"base_url": "http://localhost:4000"}, # No /gemini
)
# This calls Claude/GPT-4o/Bedrock under the hood via model_group_alias
response = client.models.generate_content(
model="gemini-2.5-flash",
contents="Hello from any model!",
)
print(response.text)
```
</TabItem>
</Tabs>
## Pass-through vs Native Router Endpoints
LiteLLM offers two ways to handle GenAI SDK requests:
| | Pass-through (`/gemini`) | Native Router (`/`) |
|---|---|---|
| **baseUrl** | `http://localhost:4000/gemini` | `http://localhost:4000` |
| **Models** | Gemini only | Any provider via `model_group_alias` |
| **Translation** | None — proxies directly to Google | Translates internally |
| **Cost Tracking** | ✅ | ✅ |
| **Virtual Keys** | ✅ | ✅ |
| **Load Balancing** | ❌ | ✅ |
| **Fallbacks** | ❌ | ✅ |
| **Best for** | Simple Gemini proxy | Multi-provider routing |
## Environment Variable Configuration
You can also configure the SDK via environment variables instead of code:
```bash
# For JavaScript SDK (@google/genai)
export GOOGLE_GEMINI_BASE_URL="http://localhost:4000/gemini"
export GEMINI_API_KEY="sk-1234"
# For Python SDK (google-genai)
# Note: The Python SDK does not support a base URL env var.
# Configure it in code with http_options={"base_url": "..."} instead.
export GEMINI_API_KEY="sk-1234"
```
This is especially useful for tools built on top of the GenAI SDK (like [Gemini CLI](./litellm_gemini_cli.md)).
## Related Resources
- [Gemini CLI with LiteLLM](./litellm_gemini_cli.md)
- [Google AI Studio Pass-Through](../pass_through/google_ai_studio)
- [Google ADK with LiteLLM](./google_adk.md)
- [LiteLLM Proxy Quick Start](../proxy/quick_start)
- [`@google/genai` npm package](https://www.npmjs.com/package/@google/genai)
- [`google-genai` PyPI package](https://pypi.org/project/google-genai/)
docs/my-website/docs/tutorials/openai_agents_sdk.md
+373
View File
@@ -0,0 +1,373 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# OpenAI Agents SDK with LiteLLM
Use OpenAI's Agents SDK with any LLM provider through LiteLLM Proxy.
This tutorial shows you how to build AI agents using the OpenAI Agents SDK with support for multiple LLM providers through LiteLLM.
## Overview
The OpenAI Agents SDK provides a high-level interface for building AI agents. By integrating with LiteLLM, you can:
- Use multiple LLM providers (Bedrock, Azure, Vertex AI, etc.) with the same agent code
- Switch easily between models from different providers
- Connect to a LiteLLM proxy for centralized model management
:::tip Built-in LiteLLM Extension
The OpenAI Agents SDK includes an official LiteLLM extension (`LitellmModel`) that works without a proxy. If you don't need centralized proxy features (cost tracking, rate limiting, load balancing), you can use it directly:
```python
from agents import Agent, Runner
from agents.extensions.models.litellm_model import LitellmModel
agent = Agent(
name="Assistant",
instructions="You are a helpful assistant.",
model=LitellmModel(model="anthropic/claude-sonnet-4-20250514"),
)
result = Runner.run_sync(agent, "Hello!")
print(result.final_output)
```
See the [Docs](https://openai.github.io/openai-agents-python/models/litellm/) for more details. The rest of this tutorial focuses on the **proxy-based approach** for teams that need centralized model management.
:::
## Prerequisites
- Python environment setup
- API keys for your LLM providers
- Basic understanding of LLMs and agent concepts
## Installation
```bash showLineNumbers title="Install dependencies"
pip install openai-agents litellm
```
## 1. Start LiteLLM Proxy
Configure and start the LiteLLM proxy with the models you want to use:
```yaml title="config.yaml" showLineNumbers
model_list:
- model_name: bedrock-claude-sonnet-4
litellm_params:
model: "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0"
aws_region_name: "us-east-1"
- model_name: gpt-4o
litellm_params:
model: "openai/gpt-4o"
- model_name: claude-sonnet-4
litellm_params:
model: "anthropic/claude-sonnet-4-20250514"
- model_name: bedrock-claude-haiku
litellm_params:
model: "bedrock/us.anthropic.claude-3-5-haiku-20241022-v1:0"
aws_region_name: "us-east-1"
- model_name: bedrock-nova-premier
litellm_params:
model: "bedrock/amazon.nova-premier-v1:0"
aws_region_name: "us-east-1"
```
```bash
litellm --config config.yaml
```
Required environment variables:
| Variable | Value | Description |
|----------|-------|-------------|
| `LITELLM_BASE_URL` | `http://localhost:4000` | LiteLLM proxy URL |
| `LITELLM_API_KEY` | `sk-1234` | Your LiteLLM API key (not your provider's key) |
## 2. Setting Up Environment
Import the necessary libraries and configure your LiteLLM proxy connection:
```python showLineNumbers title="Setup environment"
from __future__ import annotations
import asyncio
import os
from openai import AsyncOpenAI
from agents import (
Agent,
Model,
ModelProvider,
OpenAIChatCompletionsModel,
RunConfig,
Runner,
function_tool,
set_tracing_disabled,
)
# Point to LiteLLM proxy
BASE_URL = os.getenv("LITELLM_BASE_URL") or "http://localhost:4000"
API_KEY = os.getenv("LITELLM_API_KEY") or "sk-1234"
# Define model constants for cleaner code
MODEL_BEDROCK_SONNET = "bedrock-claude-sonnet-4"
MODEL_BEDROCK_HAIKU = "bedrock-claude-haiku"
MODEL_GPT_4O = "gpt-4o"
# Create the OpenAI client pointed at LiteLLM
client = AsyncOpenAI(base_url=BASE_URL, api_key=API_KEY)
# Disable tracing since we're not using OpenAI's platform directly
set_tracing_disabled(disabled=True)
```
## 3. Create a Custom Model Provider
The Agents SDK uses a `ModelProvider` to resolve model names. Create a custom provider that routes all requests through LiteLLM:
```python showLineNumbers title="Custom LiteLLM model provider"
class LiteLLMModelProvider(ModelProvider):
def get_model(self, model_name: str | None) -> Model:
return OpenAIChatCompletionsModel(
model=model_name or MODEL_BEDROCK_SONNET,
openai_client=client,
)
LITELLM_MODEL_PROVIDER = LiteLLMModelProvider()
```
## 4. Define a Simple Tool
Create a tool that your agent can use:
```python showLineNumbers title="Weather tool implementation"
@function_tool
def get_weather(city: str) -> str:
"""Retrieves the current weather report for a specified city.
Args:
city: The name of the city (e.g., "New York", "London", "Tokyo").
Returns:
A string containing the weather information for the city.
"""
print(f"[debug] getting weather for {city}")
mock_weather_db = {
"new york": "The weather in New York is sunny with a temperature of 25°C.",
"london": "It's cloudy in London with a temperature of 15°C.",
"tokyo": "Tokyo is experiencing light rain and a temperature of 18°C.",
}
city_normalized = city.lower()
if city_normalized in mock_weather_db:
return mock_weather_db[city_normalized]
else:
return f"Sorry, I don't have weather information for '{city}'."
```
## 5. Using Different Models with Agents
### 5.1 Using Bedrock Models
```python showLineNumbers title="Bedrock model via LiteLLM proxy"
async def test_bedrock_agent():
print("\n--- Testing Bedrock Claude Agent ---")
agent = Agent(
name="weather_agent_bedrock",
instructions="You are a helpful weather assistant powered by Claude. "
"Use the 'get_weather' tool for city weather requests. "
"Present information clearly.",
tools=[get_weather],
)
result = await Runner.run(
agent,
"What's the weather in Tokyo?",
run_config=RunConfig(
model_provider=LITELLM_MODEL_PROVIDER,
model="bedrock-claude-sonnet-4", # Uses the model name from your LiteLLM config
),
)
print(f"<<< Agent Response: {result.final_output}")
asyncio.run(test_bedrock_agent())
```
### 5.2 Using OpenAI Models
```python showLineNumbers title="OpenAI model via LiteLLM proxy"
async def test_openai_agent():
print("\n--- Testing OpenAI GPT Agent ---")
agent = Agent(
name="weather_agent_gpt",
instructions="You are a helpful weather assistant powered by GPT-4o. "
"Use the 'get_weather' tool for city weather requests. "
"Present information clearly.",
tools=[get_weather],
)
result = await Runner.run(
agent,
"What's the weather in London?",
run_config=RunConfig(
model_provider=LITELLM_MODEL_PROVIDER,
model="gpt-4o", # Uses the model name from your LiteLLM config
),
)
print(f"<<< Agent Response: {result.final_output}")
asyncio.run(test_openai_agent())
```
### 5.3 Using Anthropic Models
```python showLineNumbers title="Anthropic model via LiteLLM proxy"
async def test_anthropic_agent():
print("\n--- Testing Anthropic Claude Agent ---")
agent = Agent(
name="weather_agent_claude",
instructions="You are a helpful weather assistant powered by Claude. "
"Use the 'get_weather' tool for city weather requests. "
"Present information clearly.",
tools=[get_weather],
)
result = await Runner.run(
agent,
"What's the weather in New York?",
run_config=RunConfig(
model_provider=LITELLM_MODEL_PROVIDER,
model="claude-sonnet-4", # Uses the model name from your LiteLLM config
),
)
print(f"<<< Agent Response: {result.final_output}")
asyncio.run(test_anthropic_agent())
```
## 6. Complete Working Example
Here's a full end-to-end script you can copy and run:
```python showLineNumbers title="complete_agent.py"
from __future__ import annotations
import asyncio
import os
from openai import AsyncOpenAI
from agents import (
Agent,
Model,
ModelProvider,
OpenAIChatCompletionsModel,
RunConfig,
Runner,
function_tool,
set_tracing_disabled,
)
# Point to LiteLLM proxy
BASE_URL = os.getenv("LITELLM_BASE_URL") or "http://localhost:4000"
API_KEY = os.getenv("LITELLM_API_KEY") or "sk-1234"
MODEL_NAME = os.getenv("MODEL_NAME") or "bedrock-claude-sonnet-4"
client = AsyncOpenAI(base_url=BASE_URL, api_key=API_KEY)
set_tracing_disabled(disabled=True)
class LiteLLMModelProvider(ModelProvider):
def get_model(self, model_name: str | None) -> Model:
return OpenAIChatCompletionsModel(
model=model_name or MODEL_NAME,
openai_client=client,
)
LITELLM_MODEL_PROVIDER = LiteLLMModelProvider()
@function_tool
def get_weather(city: str) -> str:
"""Retrieves the current weather report for a specified city."""
print(f"[debug] getting weather for {city}")
mock_weather_db = {
"new york": "The weather in New York is sunny with a temperature of 25°C.",
"london": "It's cloudy in London with a temperature of 15°C.",
"tokyo": "Tokyo is experiencing light rain and a temperature of 18°C.",
}
city_normalized = city.lower()
if city_normalized in mock_weather_db:
return mock_weather_db[city_normalized]
else:
return f"Sorry, I don't have weather information for '{city}'."
async def main():
agent = Agent(
name="Assistant",
instructions="You are a helpful weather assistant. "
"Use the 'get_weather' tool for city weather requests. "
"Present information clearly and concisely.",
tools=[get_weather],
)
# Run with the default model (bedrock-claude-sonnet-4)
result = await Runner.run(
agent,
"What's the weather in Tokyo?",
run_config=RunConfig(model_provider=LITELLM_MODEL_PROVIDER),
)
print(result.final_output)
# Switch to a different model by passing model in RunConfig
result = await Runner.run(
agent,
"What's the weather in London?",
run_config=RunConfig(
model_provider=LITELLM_MODEL_PROVIDER,
model="gpt-4o",
),
)
print(result.final_output)
if __name__ == "__main__":
asyncio.run(main())
```
## Why Use LiteLLM with Agents SDK?
| Feature | Benefit |
|---------|---------|
| **Multi-Provider** | Use the same agent code with OpenAI, Bedrock, Azure, Vertex AI, etc. |
| **Cost Tracking** | Track spending across all agent conversations |
| **Rate Limiting** | Set budgets and limits on agent usage |
| **Load Balancing** | Distribute requests across multiple API keys or regions |
| **Fallbacks** | Automatically retry with different models if one fails |
## Related Resources
- [OpenAI Agents SDK Documentation](https://openai.github.io/openai-agents-python/)
- [LiteLLM Proxy Quick Start](../proxy/quick_start)
docs/my-website/docs/tutorials/openclaw_integration.md
+210
View File
@@ -0,0 +1,210 @@
---
sidebar_label: "OpenClaw"
---
# OpenClaw + LiteLLM Integration
[OpenClaw](https://openclaw.ai) is a self-hosted AI assistant that connects chat apps (WhatsApp, Telegram, Discord, and more) to LLM providers. By routing OpenClaw through LiteLLM Proxy, you get access to 100+ providers, cost tracking, spend limits, and automatic failover — all from a single gateway.
## What you'll set up
```
Chat apps → OpenClaw Gateway → LiteLLM Proxy → LLM Providers (OpenAI, Anthropic, etc.)
```
## Prerequisites
| Requirement | How to get it |
|---|---|
| **Node.js 22+** | `node --version` — install from [nodejs.org](https://nodejs.org) if needed |
| **Python 3.8+** | `python --version` |
| **At least one LLM API key** | OpenAI, Anthropic, Gemini, etc. |
## Step 1 — Install LiteLLM Proxy
```bash
pip install 'litellm[proxy]'
```
## Step 2 — Create a LiteLLM config file
Create a config file `litellm_config.yaml` with the models you want to use. Here's an example with OpenAI:
```yaml title="litellm_config.yaml"
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
general_settings:
master_key: sk-your-secret-key # pick any value — this is YOUR proxy password
```
:::tip Multi-provider example
You can add as many models as you want from different providers:
```yaml title="litellm_config.yaml"
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: gemini-flash
litellm_params:
model: gemini/gemini-2.0-flash
api_key: os.environ/GEMINI_API_KEY
general_settings:
master_key: sk-your-secret-key
```
See [LiteLLM proxy config docs](https://docs.litellm.ai/docs/proxy/configs) for all options.
:::
## Step 3 — Start the proxy
Make sure your API key(s) are available as environment variables (via `export`, `.env` file, or however you manage secrets), then start the proxy:
```bash
litellm --config litellm_config.yaml --port 4000
```
## Step 4 — Install OpenClaw
```bash
# macOS / Linux
curl -fsSL https://openclaw.ai/install.sh | bash
```
:::note Windows
On Windows, use PowerShell: `iwr -useb https://openclaw.ai/install.ps1 | iex`
WSL2 is recommended over native Windows.
:::
## Step 5 — Connect OpenClaw to LiteLLM
Run the onboarding wizard:
```bash
openclaw onboard --install-daemon
```
When prompted:
1. Choose **QuickStart** or **Manual** as the onboarding mode (both work — Manual gives you more options for gateway settings)
2. Select **LiteLLM** as the model/auth provider
3. Enter your LiteLLM `master_key` from Step 2 and set the base URL to your proxy address (e.g., `http://localhost:4000`)
4. When asked for the default model, choose **Enter model manually** and type the model name from your `litellm_config.yaml` (e.g., `litellm/gpt-4o`)
You can also set or change the model after onboarding:
```bash
openclaw models set litellm/gpt-4o
```
For scripted / CI environments, you can skip the prompts entirely:
```bash
openclaw onboard --non-interactive --accept-risk \
--auth-choice litellm-api-key \
--litellm-api-key "sk-your-secret-key" \
--custom-base-url "http://localhost:4000" \
--install-daemon --skip-channels --skip-skills
```
## Step 6 — Verify
Check the gateway is healthy:
```bash
openclaw health
```
Then send a test message:
```bash
openclaw dashboard # web UI
openclaw tui # terminal UI
openclaw agent --agent main -m "Hello, what model are you?" # one-shot CLI
```
If you get a response from your model, the integration is working.
Check which model is active:
```bash
openclaw models status
```
## Config reference
After onboarding, OpenClaw stores the LiteLLM provider config in `~/.openclaw/openclaw.json`. The relevant sections are something like this:
```json5 title="~/.openclaw/openclaw.json (excerpt)"
{
"models": {
"providers": {
"litellm": {
"baseUrl": "http://localhost:4000",
"apiKey": "sk-your-secret-key",
"api": "openai-completions",
"models": [
{
"id": "gpt-4o",
"name": "GPT-4o via LiteLLM"
}
]
}
}
},
"agents": {
"defaults": {
"model": { "primary": "litellm/gpt-4o" }
}
}
}
```
You can edit this file directly to add more models or change the `baseUrl`. OpenClaw hot-reloads changes automatically.
## Troubleshooting
**Connection refused / proxy not reachable**
Make sure the LiteLLM proxy is running and that the `baseUrl` in your OpenClaw config matches:
```bash
curl http://localhost:4000/health -H "Authorization: Bearer sk-your-secret-key"
```
**Wrong model or "Invalid model name"**
The model name in OpenClaw must match a `model_name` from your `litellm_config.yaml`. Switch the active model with:
```bash
openclaw models set litellm/gpt-4o
```
**Gateway pairing issues after reinstall**
If the CLI can't connect to the gateway after a reinstall, stop the service and reinstall it:
```bash
openclaw gateway stop
openclaw gateway install
```
## References
- [OpenClaw docs](https://docs.openclaw.ai)
- [OpenClaw LiteLLM provider docs](https://docs.openclaw.ai/providers/litellm)
- [OpenClaw model providers](https://docs.openclaw.ai/concepts/model-providers)
- [LiteLLM proxy configuration](https://docs.litellm.ai/docs/proxy/configs)
docs/my-website/docs/tutorials/presidio_pii_masking.md
+25
View File
@@ -685,3 +685,28 @@ Congratulations! 🎉 You've successfully set up PII masking with Presidio and L
---
**Need help?** Join our [Discord community](https://discord.com/invite/wuPM9dRgDw) or open an issue on GitHub!
### Suppressing False Positives
Presidio can sometimes trigger false positive detections. For example, short alphanumeric strings might be incorrectly flagged as `US_DRIVER_LICENSE`.
You can suppress these false positives using `presidio_score_thresholds` or `presidio_entities_deny_list`.
```yaml
guardrails:
- guardrail_name: presidio-pii
litellm_params:
guardrail: presidio
mode: "pre_call"
presidio_analyzer_api_base: "http://localhost:5002/"
presidio_anonymizer_api_base: "http://localhost:5001/"
# Use high score thresholds to reduce false positives
presidio_score_thresholds:
US_DRIVER_LICENSE: 0.85
ALL: 0.5
# Or exclude certain entity types entirely from detection
presidio_entities_deny_list:
- US_DRIVER_LICENSE
```
docs/my-website/img/litellm_proxy_setup.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 538 KiB

docs/my-website/img/release_notes/compliance_playground.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 780 KiB

docs/my-website/img/release_notes/guardrail_garden.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.2 MiB

docs/my-website/img/release_notes/v1_81_14_perf.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 625 KiB

docs/my-website/img/ui_store_model_in_db.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 234 KiB

docs/my-website/package-lock.json
Generated
+82 -77
View File
@@ -8339,9 +8339,9 @@
}
},
"node_modules/ajv": {
"version": "8.17.1",
"resolved": "https://registry.npmjs.org/ajv/-/ajv-8.17.1.tgz",
"integrity": "sha512-B/gBuNg5SiMTrPkC+A2+cW0RszwxYmn6VYxB/inlBStS5nx6xHIt/ehKRhIMhqusl7a8LjQoZnjCs5vhwxOQ1g==",
"version": "8.18.0",
"resolved": "https://registry.npmjs.org/ajv/-/ajv-8.18.0.tgz",
"integrity": "sha512-PlXPeEWMXMZ7sPYOHqmDyCJzcfNrUr3fGNKtezX14ykXOEIvyK81d+qydx89KY5O71FKMPaQ2vBfBFI5NHR63A==",
"license": "MIT",
"dependencies": {
"fast-deep-equal": "^3.1.3",
@@ -8721,10 +8721,13 @@
}
},
"node_modules/balanced-match": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
"integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==",
"license": "MIT"
"version": "4.0.3",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.3.tgz",
"integrity": "sha512-1pHv8LX9CpKut1Zp4EXey7Z8OfH11ONNH6Dhi2WDUt31VVZFXZzKwXcysBgqSumFCmR+0dqjMK5v5JiFHzi0+g==",
"license": "MIT",
"engines": {
"node": "20 || >=22"
}
},
"node_modules/bare-events": {
"version": "2.8.2",
@@ -8838,12 +8841,15 @@
"license": "MIT"
},
"node_modules/baseline-browser-mapping": {
"version": "2.8.30",
"resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.8.30.tgz",
"integrity": "sha512-aTUKW4ptQhS64+v2d6IkPzymEzzhw+G0bA1g3uBRV3+ntkH+svttKseW5IOR4Ed6NUVKqnY7qT3dKvzQ7io4AA==",
"version": "2.10.0",
"resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.10.0.tgz",
"integrity": "sha512-lIyg0szRfYbiy67j9KN8IyeD7q7hcmqnJ1ddWmNt19ItGpNN64mnllmxUNFIOdOm6by97jlL6wfpTTJrmnjWAA==",
"license": "Apache-2.0",
"bin": {
"baseline-browser-mapping": "dist/cli.js"
"baseline-browser-mapping": "dist/cli.cjs"
},
"engines": {
"node": ">=6.0.0"
}
},
"node_modules/batch": {
@@ -9024,13 +9030,15 @@
}
},
"node_modules/brace-expansion": {
"version": "1.1.12",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.12.tgz",
"integrity": "sha512-9T9UjW3r0UW5c1Q7GTwllptXwhvYmEzFhzMfZ9H7FQWt+uZePjZPjBP/W1ZEyZ1twGWom5/56TF4lPcqjnDHcg==",
"version": "5.0.2",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.2.tgz",
"integrity": "sha512-Pdk8c9poy+YhOgVWw1JNN22/HcivgKWwpxKq04M/jTmHyCZn12WPJebZxdjSa5TmBqISrUSgNYU3eRORljfCCw==",
"license": "MIT",
"dependencies": {
"balanced-match": "^1.0.0",
"concat-map": "0.0.1"
"balanced-match": "^4.0.2"
},
"engines": {
"node": "20 || >=22"
}
},
"node_modules/braces": {
@@ -9046,9 +9054,9 @@
}
},
"node_modules/browserslist": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.0.tgz",
"integrity": "sha512-tbydkR/CxfMwelN0vwdP/pLkDwyAASZ+VfWm4EOwlB6SWhx1sYnWLqo8N5j0rAzPfzfRaxt0mM/4wPU/Su84RQ==",
"version": "4.28.1",
"resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.1.tgz",
"integrity": "sha512-ZC5Bd0LgJXgwGqUknZY/vkUQ04r8NXnJZ3yYi4vDmSiZmC/pdSN0NbNRPxZpbtO4uAfDUAFffO8IZoM3Gj8IkA==",
"funding": [
{
"type": "opencollective",
@@ -9065,11 +9073,11 @@
],
"license": "MIT",
"dependencies": {
"baseline-browser-mapping": "^2.8.25",
"caniuse-lite": "^1.0.30001754",
"electron-to-chromium": "^1.5.249",
"baseline-browser-mapping": "^2.9.0",
"caniuse-lite": "^1.0.30001759",
"electron-to-chromium": "^1.5.263",
"node-releases": "^2.0.27",
"update-browserslist-db": "^1.1.4"
"update-browserslist-db": "^1.2.0"
},
"bin": {
"browserslist": "cli.js"
@@ -9262,9 +9270,9 @@
}
},
"node_modules/caniuse-lite": {
"version": "1.0.30001756",
"resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001756.tgz",
"integrity": "sha512-4HnCNKbMLkLdhJz3TToeVWHSnfJvPaq6vu/eRP0Ahub/07n484XHhBF5AJoSGHdVrS8tKFauUQz8Bp9P7LVx7A==",
"version": "1.0.30001770",
"resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001770.tgz",
"integrity": "sha512-x/2CLQ1jHENRbHg5PSId2sXq1CIO1CISvwWAj027ltMVG2UNgW+w9oH2+HzgEIRFembL8bUlXtfbBHR1fCg2xw==",
"funding": [
{
"type": "opencollective",
@@ -9764,12 +9772,6 @@
"integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==",
"license": "MIT"
},
"node_modules/concat-map": {
"version": "0.0.1",
"resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
"integrity": "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg==",
"license": "MIT"
},
"node_modules/confbox": {
"version": "0.2.2",
"resolved": "https://registry.npmjs.org/confbox/-/confbox-0.2.2.tgz",
@@ -11413,9 +11415,9 @@
"license": "MIT"
},
"node_modules/electron-to-chromium": {
"version": "1.5.259",
"resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.259.tgz",
"integrity": "sha512-I+oLXgpEJzD6Cwuwt1gYjxsDmu/S/Kd41mmLA3O+/uH2pFRO/DvOjUyGozL8j3KeLV6WyZ7ssPwELMsXCcsJAQ==",
"version": "1.5.302",
"resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.302.tgz",
"integrity": "sha512-sM6HAN2LyK82IyPBpznDRqlTQAtuSaO+ShzFiWTvoMJLHyZ+Y39r8VMfHzwbU8MVBzQ4Wdn85+wlZl2TLGIlwg==",
"license": "ISC"
},
"node_modules/emoji-regex": {
@@ -11468,13 +11470,13 @@
}
},
"node_modules/enhanced-resolve": {
"version": "5.18.3",
"resolved": "https://registry.npmjs.org/enhanced-resolve/-/enhanced-resolve-5.18.3.tgz",
"integrity": "sha512-d4lC8xfavMeBjzGr2vECC3fsGXziXZQyJxD868h2M/mBI3PwAuODxAkLkq5HYuvrPYcUtiLzsTo8U3PgX3Ocww==",
"version": "5.19.0",
"resolved": "https://registry.npmjs.org/enhanced-resolve/-/enhanced-resolve-5.19.0.tgz",
"integrity": "sha512-phv3E1Xl4tQOShqSte26C7Fl84EwUdZsyOuSSk9qtAGyyQs2s3jJzComh+Abf4g187lUUAvH+H26omrqia2aGg==",
"license": "MIT",
"dependencies": {
"graceful-fs": "^4.2.4",
"tapable": "^2.2.0"
"tapable": "^2.3.0"
},
"engines": {
"node": ">=10.13.0"
@@ -11520,9 +11522,9 @@
}
},
"node_modules/es-module-lexer": {
"version": "1.7.0",
"resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-1.7.0.tgz",
"integrity": "sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==",
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-2.0.0.tgz",
"integrity": "sha512-5POEcUuZybH7IdmGsD8wlf0AI55wMecM9rVBTI/qEAy2c1kTOm3DjFYjrBdI2K3BaJjJYfYFeRtM0t9ssnRuxw==",
"license": "MIT"
},
"node_modules/es-object-atoms": {
@@ -12142,9 +12144,9 @@
}
},
"node_modules/file-loader/node_modules/ajv": {
"version": "6.12.6",
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.12.6.tgz",
"integrity": "sha512-j3fVLgvTo527anyYyJOGTYJbG+vnnQYvE0m5mmkc1TK+nxAppkCLMIL0aZ4dblVCNoGShhm+kzE4ZUykBoMg4g==",
"version": "6.14.0",
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.14.0.tgz",
"integrity": "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw==",
"license": "MIT",
"dependencies": {
"fast-deep-equal": "^3.1.1",
@@ -16720,15 +16722,18 @@
"license": "ISC"
},
"node_modules/minimatch": {
"version": "3.1.2",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.2.tgz",
"integrity": "sha512-J7p63hRiAjw1NDEww1W7i37+ByIrOWO5XQQAzZ3VOcL0PNybwpfmV/N05zFAzwQ9USyEcX6t3UO+K5aqBQOIHw==",
"license": "ISC",
"version": "10.2.1",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.1.tgz",
"integrity": "sha512-MClCe8IL5nRRmawL6ib/eT4oLyeKMGCghibcDWK+J0hh0Q8kqSdia6BvbRMVk6mPa6WqUa5uR2oxt6C5jd533A==",
"license": "BlueOak-1.0.0",
"dependencies": {
"brace-expansion": "^1.1.7"
"brace-expansion": "^5.0.2"
},
"engines": {
"node": "*"
"node": "20 || >=22"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/minimist": {
@@ -17021,9 +17026,9 @@
}
},
"node_modules/null-loader/node_modules/ajv": {
"version": "6.12.6",
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.12.6.tgz",
"integrity": "sha512-j3fVLgvTo527anyYyJOGTYJbG+vnnQYvE0m5mmkc1TK+nxAppkCLMIL0aZ4dblVCNoGShhm+kzE4ZUykBoMg4g==",
"version": "6.14.0",
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.14.0.tgz",
"integrity": "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw==",
"license": "MIT",
"dependencies": {
"fast-deep-equal": "^3.1.1",
@@ -19294,9 +19299,9 @@
}
},
"node_modules/qs": {
"version": "6.14.1",
"resolved": "https://registry.npmjs.org/qs/-/qs-6.14.1.tgz",
"integrity": "sha512-4EK3+xJl8Ts67nLYNwqw/dsFVnCf+qR7RgXSK9jEEm9unao3njwMDdmsdvoKBKHzxd7tCYz5e5M+SnMjdtXGQQ==",
"version": "6.14.2",
"resolved": "https://registry.npmjs.org/qs/-/qs-6.14.2.tgz",
"integrity": "sha512-V/yCWTTF7VJ9hIh18Ugr2zhJMP01MY7c5kh4J870L7imm6/DIzBsNLTXzMwUA3yZ5b/KBqLx8Kp3uRvd7xSe3Q==",
"license": "BSD-3-Clause",
"dependencies": {
"side-channel": "^1.1.0"
@@ -21471,9 +21476,9 @@
}
},
"node_modules/terser-webpack-plugin": {
"version": "5.3.14",
"resolved": "https://registry.npmjs.org/terser-webpack-plugin/-/terser-webpack-plugin-5.3.14.tgz",
"integrity": "sha512-vkZjpUjb6OMS7dhV+tILUW6BhpDR7P2L/aQSAv+Uwk+m8KATX9EccViHTJR2qDtACKPIYndLGCyl3FMo+r2LMw==",
"version": "5.3.16",
"resolved": "https://registry.npmjs.org/terser-webpack-plugin/-/terser-webpack-plugin-5.3.16.tgz",
"integrity": "sha512-h9oBFCWrq78NyWWVcSwZarJkZ01c2AyGrzs1crmHZO3QUg9D61Wu4NPjBy69n7JqylFF5y+CsUZYmYEIZ3mR+Q==",
"license": "MIT",
"dependencies": {
"@jridgewell/trace-mapping": "^0.3.25",
@@ -21928,9 +21933,9 @@
}
},
"node_modules/update-browserslist-db": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.1.4.tgz",
"integrity": "sha512-q0SPT4xyU84saUX+tomz1WLkxUbuaJnR1xWt17M7fJtEJigJeWUNGUqrauFXsHnqev9y9JTRGwk13tFBuKby4A==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.2.3.tgz",
"integrity": "sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w==",
"funding": [
{
"type": "opencollective",
@@ -22068,9 +22073,9 @@
}
},
"node_modules/url-loader/node_modules/ajv": {
"version": "6.12.6",
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.12.6.tgz",
"integrity": "sha512-j3fVLgvTo527anyYyJOGTYJbG+vnnQYvE0m5mmkc1TK+nxAppkCLMIL0aZ4dblVCNoGShhm+kzE4ZUykBoMg4g==",
"version": "6.14.0",
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.14.0.tgz",
"integrity": "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw==",
"license": "MIT",
"dependencies": {
"fast-deep-equal": "^3.1.1",
@@ -22372,9 +22377,9 @@
"license": "MIT"
},
"node_modules/watchpack": {
"version": "2.4.4",
"resolved": "https://registry.npmjs.org/watchpack/-/watchpack-2.4.4.tgz",
"integrity": "sha512-c5EGNOiyxxV5qmTtAB7rbiXxi1ooX1pQKMLX/MIabJjRA0SJBQOjKF+KSVfHkr9U1cADPon0mRiVe/riyaiDUA==",
"version": "2.5.1",
"resolved": "https://registry.npmjs.org/watchpack/-/watchpack-2.5.1.tgz",
"integrity": "sha512-Zn5uXdcFNIA1+1Ei5McRd+iRzfhENPCe7LeABkJtNulSxjma+l7ltNx55BWZkRlwRnpOgHqxnjyaDgJnNXnqzg==",
"license": "MIT",
"dependencies": {
"glob-to-regexp": "^0.4.1",
@@ -22419,9 +22424,9 @@
"license": "BSD-2-Clause"
},
"node_modules/webpack": {
"version": "5.103.0",
"resolved": "https://registry.npmjs.org/webpack/-/webpack-5.103.0.tgz",
"integrity": "sha512-HU1JOuV1OavsZ+mfigY0j8d1TgQgbZ6M+J75zDkpEAwYeXjWSqrGJtgnPblJjd/mAyTNQ7ygw0MiKOn6etz8yw==",
"version": "5.105.2",
"resolved": "https://registry.npmjs.org/webpack/-/webpack-5.105.2.tgz",
"integrity": "sha512-dRXm0a2qcHPUBEzVk8uph0xWSjV/xZxenQQbLwnwP7caQCYpqG1qddwlyEkIDkYn0K8tvmcrZ+bOrzoQ3HxCDw==",
"license": "MIT",
"dependencies": {
"@types/eslint-scope": "^3.7.7",
@@ -22432,10 +22437,10 @@
"@webassemblyjs/wasm-parser": "^1.14.1",
"acorn": "^8.15.0",
"acorn-import-phases": "^1.0.3",
"browserslist": "^4.26.3",
"browserslist": "^4.28.1",
"chrome-trace-event": "^1.0.2",
"enhanced-resolve": "^5.17.3",
"es-module-lexer": "^1.2.1",
"enhanced-resolve": "^5.19.0",
"es-module-lexer": "^2.0.0",
"eslint-scope": "5.1.1",
"events": "^3.2.0",
"glob-to-regexp": "^0.4.1",
@@ -22446,8 +22451,8 @@
"neo-async": "^2.6.2",
"schema-utils": "^4.3.3",
"tapable": "^2.3.0",
"terser-webpack-plugin": "^5.3.11",
"watchpack": "^2.4.4",
"terser-webpack-plugin": "^5.3.16",
"watchpack": "^2.5.1",
"webpack-sources": "^3.3.3"
},
"bin": {
docs/my-website/package.json
+29 -2
View File
@@ -61,10 +61,37 @@
"mermaid": ">=11.10.0",
"gray-matter": "4.0.3",
"glob": ">=11.1.0",
"tar": ">=7.5.7",
"tar": ">=7.5.8",
"minimatch": ">=10.2.1",
"diff": ">=8.0.3",
"@isaacs/brace-expansion": ">=5.0.1",
"node-forge": ">=1.3.2",
"mdast-util-to-hast": ">=13.2.1",
"lodash-es": ">=4.17.23"
"lodash-es": ">=4.17.23",
"schema-utils@3": {
"ajv": "6.14.0"
},
"schema-utils@4": {
"ajv": "8.18.0"
},
"file-loader": {
"ajv": "6.14.0"
},
"null-loader": {
"ajv": "6.14.0"
},
"url-loader": {
"ajv": "6.14.0"
},
"@babel/traverse": ">=7.23.2",
"ws": ">=7.5.10",
"http-proxy-middleware": ">=2.0.9",
"tar-fs": ">=2.1.4",
"webpack-dev-middleware": ">=5.3.4",
"braces": ">=3.0.3",
"axios": ">=0.30.2",
"webpack": ">=5.94.0",
"serve-static": ">=1.16.0",
"path-to-regexp": ">=0.1.12"
}
}
docs/my-website/release_notes/v1.81.12.md
+3 -3
View File
@@ -1,5 +1,5 @@
---
title: "[Preview] v1.81.12 - Guardrail Policy Templates & Action Builder"
title: "v1.81.12-stable - Guardrail Policy Templates & Action Builder"
slug: "v1-81-12"
date: 2026-02-14T00:00:00
authors:
@@ -27,14 +27,14 @@ import Image from '@theme/IdealImage';
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.81.12.rc.1
ghcr.io/berriai/litellm:main-v1.81.12-stable
```
</TabItem>
<TabItem value="pip" label="Pip">
``` showLineNumbers title="pip install litellm"
pip install litellm==1.81.12.rc1
pip install litellm==1.81.12
```
</TabItem>
docs/my-website/release_notes/v1.81.14.md
+526
View File
@@ -0,0 +1,526 @@
---
title: "[Preview] v1.81.14 - New Gateway Level Guardrails & Compliance Playground"
slug: "v1-81-14"
date: 2026-02-21T00:00:00
authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
hide_table_of_contents: false
---
## Deploy this version
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
<Tabs>
<TabItem value="docker" label="Docker">
``` showLineNumbers title="docker run litellm"
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.81.14.rc.1
```
</TabItem>
<TabItem value="pip" label="Pip">
``` showLineNumbers title="pip install litellm"
pip install litellm==1.81.14
```
</TabItem>
</Tabs>
## Key Highlights
- **Guardrail Garden** — [Browse built-in and partner guardrails by use case — competitor blocking, topic filtering, GDPR, prompt injection, and more. Pick a template, customize it, attach it to a team or key.](../../docs/proxy/guardrails/policy_templates)
- **Compliance Playground** — [Test any guardrail policy against your own traffic before it goes live. See precision, recall, and false positive rate — so you know how it'll behave in production.](../../docs/proxy/guardrails/policy_templates)
- **3 new zero-cost built-in guardrails** — [Competitor name blocker, topic blocker, and insults filter — all gateway-level, &lt;0.1ms latency, no external API, configurable per-team or key](../../docs/proxy/guardrails)
- **Store Model in DB Settings via UI** - [Configure model storage directly in the Admin UI without editing config files or restarting the proxy—perfect for cloud deployments](../../docs/proxy/ui_store_model_db_setting)
- **Claude Sonnet 4.6 — day 0** — [Full support across Anthropic and Vertex AI: reasoning, computer use, prompt caching, 200K context](../../docs/providers/anthropic)
- **20+ performance optimizations** — Faster routing, lower logging overhead, reduced cost-calculator latency, and connection pool fixes — meaningfully less CPU and latency on every request
---
### Guardrail Garden
AI Platform Admins can now browse built-in and partner guardrails from the Guardrail Garden. Guardrails are organized by use case — blocking financial advice, filtering insults, detecting competitor mentions, and more — so you can find the right one and deploy it in a few clicks.
![Guardrail Garden](../img/release_notes/guardrail_garden.png)
### 3 New Built-in Guardrails
This release brings 3 new built-in guardrails that run directly on the gateway. This is great for AI Gateway Admins who need low latency, zero cost guardrails for their scenarios.
- **Denied Financial Advice** — detects requests for personalized financial advice, investment recommendations, or financial planning
- **Denied Insults** — detects insults, name-calling, and personal attacks directed at the chatbot, staff, or other people
- **Competitor Name Blocker** — detects mentions of competitor brands in responses
These guardrails are built for production and on our benchmarks had a 100% Recall and Precision.
### Store Model in DB Settings via UI
Previously, the `store_model_in_db` setting could only be configured in `proxy_config.yaml` under `general_settings`, requiring a proxy restart to take effect. Now you can enable or disable this setting directly from the Admin UI without any restarts. This is especially useful for cloud deployments where you don't have direct access to config files or want to avoid downtime. Enable `store_model_in_db` to move model definitions from your YAML into the database—reducing config complexity, improving scalability, and enabling dynamic model management across multiple proxy instances.
![Store model in DB Setting](../img/ui_store_model_in_db.png)
#### Eval results
We benchmarked our new built-in guardrails against labeled datasets before shipping. You can see the results for Denied Financial Advice (207 cases) and Denied Insults (299 cases):
| Guardrail | Precision | Recall | F1 | Latency p50 | Cost/req |
|-----------|-----------|--------|----|-------------|----------|
| Denied Financial Advice | 100% | 100% | 100% | &lt;0.1ms | $0 |
| Denied Insults | 100% | 100% | 100% | &lt;0.1ms | $0 |
100% precision means zero false positives — no legitimate messages were incorrectly blocked. 100% recall means zero false negatives — every message that should have been blocked was caught.
### Compliance Playground
The Compliance Playground lets you test any guardrail against our pre-built eval datasets or your own custom datasets, so you can see precision, recall, and false positive rate before rolling it out to production.
![Compliance Playground](../img/release_notes/compliance_playground.png)
---
## Performance & Reliability — Up to 13% Lower Latency
<Image img={require('../img/release_notes/v1_81_14_perf.png')} />
This release cuts latency across all percentiles through 20+ micro-optimizations across logging, cost calculation, routing, and connection management. See [benchmarking](../../docs/benchmarks) for more info about how to benchmark yourself.
- **Mean latency:** 78.4 ms → **70.3 ms** (−10.3%)
- **p50 latency:** 64.8 ms → **57.3 ms** (−11.7%)
- **p99 latency:** 288.9 ms → **250.0 ms** (−13.4%)
**Streaming Connection Pool Fix**
Fixed a 3-fold connection leak that caused TCP connection starvation under streaming workloads: the aiohttp transport wasn't closing connections, no `finally` blocks were calling close on disconnect, and a Uvicorn bug prevented disconnect signaling. [PR #21213](https://github.com/BerriAI/litellm/pull/21213)
```mermaid
graph LR
A[Client Disconnects] --> B[Stream Abandoned]
B --> C{Connection cleaned up?}
C -->|Before| D["❌ No — connection leaked"]
C -->|After| E["✅ Yes — connection returned to pool"]
```
**Redis Connection Pool Reliability**
Fixed 4 separate connection pool bugs to make how we use Redis more reliable. The most important change was on pools being leaked on cache expiry and the other fixes are detailed here in [PR #21717](https://github.com/BerriAI/litellm/pull/21717).
```mermaid
graph LR
A[Cache Entry Expires] --> B{Pool cleanup?}
B -->|Before| C["❌ New untracked pool created — leaked"]
B -->|After| D["✅ Pool closed on eviction"]
```
---
## New Providers and Endpoints
### New Providers (1 new provider)
| Provider | Supported LiteLLM Endpoints | Description |
| -------- | --------------------------- | ----------- |
| [IBM watsonx.ai](../../docs/providers/watsonx) | `/rerank` | Rerank support for IBM watsonx.ai models |
### New LLM API Endpoints (1 new endpoint)
| Endpoint | Method | Description | Documentation |
| -------- | ------ | ----------- | ------------- |
| `/v1/evals` | POST/GET | OpenAI-compatible Evals API for model evaluation | [Docs](../../docs/evals_api) |
---
## New Models / Updated Models
#### New Model Support (13 new models)
| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features |
| -------- | ----- | -------------- | ------------------- | -------------------- | -------- |
| Anthropic | `claude-sonnet-4-6` | 200K | $3.00 | $15.00 | Reasoning, computer use, prompt caching, vision, PDF |
| Vertex AI | `vertex_ai/claude-opus-4-6@default` | 1M | $5.00 | $25.00 | Reasoning, computer use, prompt caching |
| Google Gemini | `gemini/gemini-3.1-pro-preview` | 1M | $2.00 | $12.00 | Audio, video, images, PDF |
| Google Gemini | `gemini/gemini-3.1-pro-preview-customtools` | 1M | $2.00 | $12.00 | Custom tools |
| GitHub Copilot | `github_copilot/gpt-5.3-codex` | 128K | - | - | Responses API, function calling, vision |
| GitHub Copilot | `github_copilot/claude-opus-4.6-fast` | 128K | - | - | Chat completions, function calling, vision |
| Mistral | `mistral/devstral-small-latest` | 256K | $0.10 | $0.30 | Function calling, response schema |
| Mistral | `mistral/devstral-latest` | 256K | $0.40 | $2.00 | Function calling, response schema |
| Mistral | `mistral/devstral-medium-latest` | 256K | $0.40 | $2.00 | Function calling, response schema |
| OpenRouter | `openrouter/minimax/minimax-m2.5` | 196K | $0.30 | $1.10 | Function calling, reasoning, prompt caching |
| Fireworks AI | `fireworks_ai/accounts/fireworks/models/glm-4p7` | - | - | - | Chat completions |
| Fireworks AI | `fireworks_ai/accounts/fireworks/models/minimax-m2p1` | - | - | - | Chat completions |
| Fireworks AI | `fireworks_ai/accounts/fireworks/models/kimi-k2p5` | - | - | - | Chat completions |
#### Features
- **[Anthropic](../../docs/providers/anthropic)**
- Day 0 support for Claude Sonnet 4.6 with reasoning, computer use, and 200K context - [PR #21401](https://github.com/BerriAI/litellm/pull/21401)
- Add Claude Sonnet 4.6 pricing - [PR #21395](https://github.com/BerriAI/litellm/pull/21395)
- Add day 0 feature support for Claude Sonnet 4.6 (streaming, function calling, vision) - [PR #21448](https://github.com/BerriAI/litellm/pull/21448)
- Add `reasoning` effort and extended thinking support for Sonnet 4.6 - [PR #21598](https://github.com/BerriAI/litellm/pull/21598)
- Fix empty system messages in `translate_system_message` - [PR #21630](https://github.com/BerriAI/litellm/pull/21630)
- Sanitize Anthropic messages for multi-turn compatibility - [PR #21464](https://github.com/BerriAI/litellm/pull/21464)
- Map `websearch` tool from `/v1/messages` to `/chat/completions` - [PR #21465](https://github.com/BerriAI/litellm/pull/21465)
- Forward `reasoning` field as `reasoning_content` in delta streaming - [PR #21468](https://github.com/BerriAI/litellm/pull/21468)
- Add server-side compaction translation from OpenAI to Anthropic format - [PR #21555](https://github.com/BerriAI/litellm/pull/21555)
- **[AWS Bedrock](../../docs/providers/bedrock)**
- Native structured outputs API support (`outputConfig.textFormat`) - [PR #21222](https://github.com/BerriAI/litellm/pull/21222)
- Support `nova/` and `nova-2/` spec prefixes for custom imported models - [PR #21359](https://github.com/BerriAI/litellm/pull/21359)
- Broaden Nova 2 model detection to support all `nova-2-*` variants - [PR #21358](https://github.com/BerriAI/litellm/pull/21358)
- Clamp `thinking.budget_tokens` to minimum 1024 - [PR #21306](https://github.com/BerriAI/litellm/pull/21306)
- Fix `parallel_tool_calls` mapping for Bedrock Converse - [PR #21659](https://github.com/BerriAI/litellm/pull/21659)
- **[Google Gemini / Vertex AI](../../docs/providers/gemini)**
- Day 0 support for `gemini-3.1-pro-preview` - [PR #21568](https://github.com/BerriAI/litellm/pull/21568)
- Fix `_map_reasoning_effort_to_thinking_level` for all Gemini 3 family models - [PR #21654](https://github.com/BerriAI/litellm/pull/21654)
- Add reasoning support via config for Gemini models - [PR #21663](https://github.com/BerriAI/litellm/pull/21663)
- **[Databricks](../../docs/providers/databricks)**
- Add Databricks to supported providers for response schema - [PR #21368](https://github.com/BerriAI/litellm/pull/21368)
- Native Responses API support for Databricks GPT models - [PR #21460](https://github.com/BerriAI/litellm/pull/21460)
- **[GitHub Copilot](../../docs/providers/github_copilot)**
- Add `github_copilot/gpt-5.3-codex` and `github_copilot/claude-opus-4.6-fast` models - [PR #21316](https://github.com/BerriAI/litellm/pull/21316)
- Fix unsupported params for ChatGPT Codex - [PR #21209](https://github.com/BerriAI/litellm/pull/21209)
- Allow GitHub model aliases to reuse upstream model metadata - [PR #21497](https://github.com/BerriAI/litellm/pull/21497)
- **[Mistral](../../docs/providers/mistral)**
- Add `devstral-2512` model aliases (`devstral-small-latest`, `devstral-latest`, `devstral-medium-latest`) - [PR #21372](https://github.com/BerriAI/litellm/pull/21372)
- **[IBM watsonx.ai](../../docs/providers/watsonx)**
- Add native rerank support - [PR #21303](https://github.com/BerriAI/litellm/pull/21303)
- **[xAI](../../docs/providers/xai)**
- Fix usage object in xAI responses - [PR #21559](https://github.com/BerriAI/litellm/pull/21559)
- **[Dashscope](../../docs/providers/dashscope)**
- Remove list-to-str transformation that caused incorrect request formatting - [PR #21547](https://github.com/BerriAI/litellm/pull/21547)
- **[hosted_vllm](../../docs/providers/vllm)**
- Convert thinking blocks to content blocks for multi-turn conversations - [PR #21557](https://github.com/BerriAI/litellm/pull/21557)
- **[OCI / Oracle](../../docs/providers/oci_cohere)**
- Fix Grok output pricing - [PR #21329](https://github.com/BerriAI/litellm/pull/21329)
- **[AU Anthropic](../../docs/providers/anthropic)**
- Fix `au.anthropic.claude-opus-4-6-v1` model ID - [PR #20731](https://github.com/BerriAI/litellm/pull/20731)
- **General**
- Add routing based on reasoning support — skip deployments that don't support reasoning when `thinking` params are present - [PR #21302](https://github.com/BerriAI/litellm/pull/21302)
- Add `stop` as supported param for OpenAI and Azure - [PR #21539](https://github.com/BerriAI/litellm/pull/21539)
- Add `store` and other missing params to `OPENAI_CHAT_COMPLETION_PARAMS` - [PR #21195](https://github.com/BerriAI/litellm/pull/21195), [PR #21360](https://github.com/BerriAI/litellm/pull/21360)
- Preserve `provider_specific_fields` from proxy responses - [PR #21220](https://github.com/BerriAI/litellm/pull/21220)
- Add default usage data configuration - [PR #21550](https://github.com/BerriAI/litellm/pull/21550)
### Bug Fixes
- **[AWS Bedrock](../../docs/providers/bedrock)**
- Fix service_tier cost propagation - [PR #21172](https://github.com/BerriAI/litellm/pull/21172)
- Fix per-image pricing for multimodal embeddings - [PR #21646](https://github.com/BerriAI/litellm/pull/21646)
- Use `batch_` prefix for Vertex AI batch IDs in `encode_file_id_with_model` - [PR #21624](https://github.com/BerriAI/litellm/pull/21624)
- **[Bedrock Converse](../../docs/providers/bedrock)**
- Fix Anthropic usage object to match v1/messages spec - [PR #21295](https://github.com/BerriAI/litellm/pull/21295)
- **[Fireworks AI](../../docs/providers/fireworks_ai)**
- Add missing model pricing for `glm-4p7`, `minimax-m2p1`, `kimi-k2p5` - [PR #21642](https://github.com/BerriAI/litellm/pull/21642)
- **[Responses API](../../docs/response_api)**
- Fix `use None` instead of `Reasoning()` for reasoning parameter - [PR #21103](https://github.com/BerriAI/litellm/pull/21103)
- Preserve metadata for custom callbacks on codex/responses path - [PR #21243](https://github.com/BerriAI/litellm/pull/21243)
---
## LLM API Endpoints
#### Features
- **[Responses API](../../docs/response_api)**
- Return `finish_reason='tool_calls'` when response contains function_call items - [PR #19745](https://github.com/BerriAI/litellm/pull/19745)
- Eliminate per-chunk thread spawning in async streaming path for significantly better throughput - [PR #21709](https://github.com/BerriAI/litellm/pull/21709)
- **[Evals API](../../docs/evals_api)**
- Add support for OpenAI Evals API - [PR #21375](https://github.com/BerriAI/litellm/pull/21375)
- **[Batch API](../../docs/batches)**
- Add file deletion criteria with batch references - [PR #21456](https://github.com/BerriAI/litellm/pull/21456)
- Misc bug fixes for managed batches - [PR #21157](https://github.com/BerriAI/litellm/pull/21157)
- **[Pass-Through Endpoints](../../docs/pass_through/bedrock)**
- Add method-based routing for passthrough endpoints - [PR #21543](https://github.com/BerriAI/litellm/pull/21543)
- Preserve and forward OAuth Authorization headers through proxy layer - [PR #19912](https://github.com/BerriAI/litellm/pull/19912)
- **[Websearch / Tool Calling](../../docs/completion/input)**
- Add DuckDuckGo as a search tool - [PR #21467](https://github.com/BerriAI/litellm/pull/21467)
- Fix `pre_call_deployment_hook` not triggering via proxy router for websearch - [PR #21433](https://github.com/BerriAI/litellm/pull/21433)
- **General**
- Exclude tool params for models without function calling support - [PR #21244](https://github.com/BerriAI/litellm/pull/21244)
- Add `store` param to OpenAI chat completion params - [PR #21195](https://github.com/BerriAI/litellm/pull/21195)
- Add reasoning support via config for per-model reasoning configuration - [PR #21663](https://github.com/BerriAI/litellm/pull/21663)
#### Bugs
- **General**
- Fix `api_base` resolution error for models with multiple potential endpoints - [PR #21658](https://github.com/BerriAI/litellm/pull/21658)
- Fix session grouping broken for dict rows from `query_raw` - [PR #21435](https://github.com/BerriAI/litellm/pull/21435)
---
## Management Endpoints / UI
#### Features
- **Access Groups**
- Add Access Group Selector to Create and Edit flow for Keys/Teams - [PR #21234](https://github.com/BerriAI/litellm/pull/21234)
- **Virtual Keys**
- Fix virtual key grace period from env/UI - [PR #20321](https://github.com/BerriAI/litellm/pull/20321)
- Fix key expiry default duration - [PR #21362](https://github.com/BerriAI/litellm/pull/21362)
- Key Last Active Tracking — see when a key was last used - [PR #21545](https://github.com/BerriAI/litellm/pull/21545)
- Fix `/v1/models` returning wildcard instead of expanded models for BYOK team keys - [PR #21408](https://github.com/BerriAI/litellm/pull/21408)
- Return `failed_tokens` in delete_verification_tokens response - [PR #21609](https://github.com/BerriAI/litellm/pull/21609)
- **Models + Endpoints**
- Add Model Settings Modal to Models & Endpoints page - [PR #21516](https://github.com/BerriAI/litellm/pull/21516)
- Allow `store_model_in_db` to be set via database (not just config) - [PR #21511](https://github.com/BerriAI/litellm/pull/21511)
- Fix `input_cost_per_token` masked/hidden in Model Info UI - [PR #21723](https://github.com/BerriAI/litellm/pull/21723)
- Fix credentials for UI-created models in batch file uploads - [PR #21502](https://github.com/BerriAI/litellm/pull/21502)
- Resolve credentials for UI-created models - [PR #21502](https://github.com/BerriAI/litellm/pull/21502)
- **Teams**
- Allow team members to view entire team usage - [PR #21537](https://github.com/BerriAI/litellm/pull/21537)
- Fix service account visibility for team members - [PR #21627](https://github.com/BerriAI/litellm/pull/21627)
- Organization Info page: show member email, AntD tabs, reusable MemberTable - [PR #21745](https://github.com/BerriAI/litellm/pull/21745)
- **Usage / Spend Logs**
- Allow filtering Usage by User - [PR #21351](https://github.com/BerriAI/litellm/pull/21351)
- Inject Credential Name as Tag for Usage Page filtering - [PR #21715](https://github.com/BerriAI/litellm/pull/21715)
- Prefix credential tags and update Tag usage banner - [PR #21739](https://github.com/BerriAI/litellm/pull/21739)
- Show retry count for requests in Logs view - [PR #21704](https://github.com/BerriAI/litellm/pull/21704)
- Fix Aggregated Daily Activity Endpoint performance - [PR #21613](https://github.com/BerriAI/litellm/pull/21613)
- **SSO / Auth**
- Fix SSO PKCE support in multi-pod Kubernetes deployments - [PR #20314](https://github.com/BerriAI/litellm/pull/20314)
- Preserve SSO role regardless of `role_mappings` config - [PR #21503](https://github.com/BerriAI/litellm/pull/21503)
- **Proxy CLI / Master Key**
- Fix master key rotation Prisma validation errors - [PR #21330](https://github.com/BerriAI/litellm/pull/21330)
- Handle missing `DATABASE_URL` in `append_query_params` - [PR #21239](https://github.com/BerriAI/litellm/pull/21239)
- **Project Management**
- Add Project Management APIs for organizing resources - [PR #21078](https://github.com/BerriAI/litellm/pull/21078)
- **UI Improvements**
- Content Filters: help edit/view categories and 1-click add with pagination - [PR #21223](https://github.com/BerriAI/litellm/pull/21223)
- Playground: test fallbacks with UI - [PR #21007](https://github.com/BerriAI/litellm/pull/21007)
- Add `forward_client_headers_to_llm_api` toggle to general settings - [PR #21776](https://github.com/BerriAI/litellm/pull/21776)
- Fix `is_premium()` debug log spam on every request - [PR #20841](https://github.com/BerriAI/litellm/pull/20841)
#### Bugs
- Spend Logs: Fix cost calculation - [PR #21152](https://github.com/BerriAI/litellm/pull/21152)
- Logs: Fix table not updating and pagination issues - [PR #21708](https://github.com/BerriAI/litellm/pull/21708)
- Fix `/get_image` ignoring `UI_LOGO_PATH` when `cached_logo.jpg` exists - [PR #21637](https://github.com/BerriAI/litellm/pull/21637)
- Fix duplicate URL in `tagsSpendLogsCall` query string - [PR #20909](https://github.com/BerriAI/litellm/pull/20909)
- Preserve `key_alias` and `team_id` metadata in `/user/daily/activity/aggregated` after key deletion or regeneration - [PR #20684](https://github.com/BerriAI/litellm/pull/20684)
- Uncomment `response_model` in `user_info` endpoint - [PR #17430](https://github.com/BerriAI/litellm/pull/17430)
- Allow `internal_user_viewer` to access RAG endpoints; restrict ingest to existing vector stores - [PR #21508](https://github.com/BerriAI/litellm/pull/21508)
- Suppress warning for `litellm-dashboard` team in agent permission handler - [PR #21721](https://github.com/BerriAI/litellm/pull/21721)
---
## AI Integrations
### Logging
- **[DataDog](../../docs/proxy/logging#datadog)**
- Add `team` tag to logs, metrics, and cost management - [PR #21449](https://github.com/BerriAI/litellm/pull/21449)
- **[Prometheus](../../docs/proxy/logging#prometheus)**
- Fix double-counting of `litellm_proxy_total_requests_metric` - [PR #21159](https://github.com/BerriAI/litellm/pull/21159)
- Guard against None metadata in Prometheus metrics - [PR #21489](https://github.com/BerriAI/litellm/pull/21489)
- Add ASGI middleware for improved Prometheus metrics collection - [PR #20434](https://github.com/BerriAI/litellm/pull/20434)
- **[Langfuse](../../docs/proxy/logging#langfuse)**
- Improve Langfuse test isolation (multiple stability fixes) - [PR #21214](https://github.com/BerriAI/litellm/pull/21214)
- **General**
- Fix cost to 0 for cached responses in logging - [PR #21816](https://github.com/BerriAI/litellm/pull/21816)
- Improve streaming proxy throughput by fixing middleware and logging bottlenecks - [PR #21501](https://github.com/BerriAI/litellm/pull/21501)
- Reduce proxy overhead for large base64 payloads - [PR #21594](https://github.com/BerriAI/litellm/pull/21594)
- Close streaming connections to prevent connection pool exhaustion - [PR #21213](https://github.com/BerriAI/litellm/pull/21213)
### Guardrails
- **Guardrail Garden**
- Launch Guardrail Garden — a marketplace for pre-built guardrails deployable in one click - [PR #21732](https://github.com/BerriAI/litellm/pull/21732)
- Redesign guardrail creation form with vertical stepper UI - [PR #21727](https://github.com/BerriAI/litellm/pull/21727)
- Add guardrail jump link in log detail view - [PR #21437](https://github.com/BerriAI/litellm/pull/21437)
- Guardrail tracing UI: show policy, detection method, and match details - [PR #21349](https://github.com/BerriAI/litellm/pull/21349)
- **AI Policy Templates**
- Seven new ready-to-deploy policy templates ship in this release:
- GDPR Art. 32 EU PII Protection - [PR #21340](https://github.com/BerriAI/litellm/pull/21340)
- EU AI Act Article 5 (5 sub-guardrails, with French language support) - [PR #21342](https://github.com/BerriAI/litellm/pull/21342), [PR #21453](https://github.com/BerriAI/litellm/pull/21453), [PR #21427](https://github.com/BerriAI/litellm/pull/21427)
- Prompt injection detection - [PR #21520](https://github.com/BerriAI/litellm/pull/21520)
- Aviation and UAE topic filters with tag-based routing - [PR #21518](https://github.com/BerriAI/litellm/pull/21518)
- Airline off-topic restriction - [PR #21607](https://github.com/BerriAI/litellm/pull/21607)
- SQL injection - [PR #21806](https://github.com/BerriAI/litellm/pull/21806)
- AI-powered policy template suggestions with latency overhead estimates - [PR #21589](https://github.com/BerriAI/litellm/pull/21589), [PR #21608](https://github.com/BerriAI/litellm/pull/21608), [PR #21620](https://github.com/BerriAI/litellm/pull/21620)
- **Compliance Checker**
- Add compliance checker endpoints + UI panel - [PR #21432](https://github.com/BerriAI/litellm/pull/21432)
- CSV dataset upload to compliance playground for batch testing - [PR #21526](https://github.com/BerriAI/litellm/pull/21526)
- **Built-in Guardrails**
- Competitor name blocker: blocks by name, handles streaming, supports name variations, and splits pre/post call - [PR #21719](https://github.com/BerriAI/litellm/pull/21719), [PR #21533](https://github.com/BerriAI/litellm/pull/21533)
- Topic blocker with both keyword and embedding-based implementations - [PR #21713](https://github.com/BerriAI/litellm/pull/21713)
- Insults content filter - [PR #21729](https://github.com/BerriAI/litellm/pull/21729)
- MCP Security guardrail to block unregistered MCP servers - [PR #21429](https://github.com/BerriAI/litellm/pull/21429)
- **[Generic Guardrails](../../docs/proxy/guardrails)**
- Add configurable fallback to handle generic guardrail endpoint connection failures - [PR #21245](https://github.com/BerriAI/litellm/pull/21245)
- **[Presidio](../../docs/proxy/guardrails)**
- Fix Presidio controls configuration - [PR #21798](https://github.com/BerriAI/litellm/pull/21798)
- **[LakeraAI](../../docs/proxy/guardrails)**
- Avoid `KeyError` on missing `LAKERA_API_KEY` during initialization - [PR #21422](https://github.com/BerriAI/litellm/pull/21422)
### Auto Routing
- **Complexity-based auto routing** — new router strategy that scores requests across 7 dimensions (token count, code presence, reasoning markers, technical terms, etc.) and routes to the appropriate model tier — no embeddings or API calls required - [PR #21789](https://github.com/BerriAI/litellm/pull/21789), [Docs](../../docs/proxy/auto_routing)
### Prompt Management
- **Prompt Management API**
- New API to interact with prompt management integrations without requiring a PR - [PR #17800](https://github.com/BerriAI/litellm/pull/17800), [PR #17946](https://github.com/BerriAI/litellm/pull/17946)
- Fix prompt registry configuration issues - [PR #21402](https://github.com/BerriAI/litellm/pull/21402)
---
## Spend Tracking, Budgets and Rate Limiting
- **Fix Bedrock service_tier cost propagation** — costs from service-tier responses now correctly flow through to spend tracking - [PR #21172](https://github.com/BerriAI/litellm/pull/21172)
- **Fix cost for cached responses** — cached responses now correctly log $0 cost instead of re-billing - [PR #21816](https://github.com/BerriAI/litellm/pull/21816)
- **Aggregate daily activity endpoint performance** — faster queries for `/user/daily/activity/aggregated` - [PR #21613](https://github.com/BerriAI/litellm/pull/21613)
- **Preserve key_alias and team_id metadata** in `/user/daily/activity/aggregated` after key deletion or regeneration - [PR #20684](https://github.com/BerriAI/litellm/pull/20684)
- **Inject Credential Name as Tag** for granular usage page filtering by credential - [PR #21715](https://github.com/BerriAI/litellm/pull/21715)
---
## MCP Gateway
- **OpenAPI-to-MCP** — Convert any OpenAPI spec to an MCP server via API or UI - [PR #21575](https://github.com/BerriAI/litellm/pull/21575), [PR #21662](https://github.com/BerriAI/litellm/pull/21662)
- **MCP User Permissions** — Fine-grained permissions for end users on MCP servers - [PR #21462](https://github.com/BerriAI/litellm/pull/21462)
- **MCP Security Guardrail** — Block calls to unregistered MCP servers - [PR #21429](https://github.com/BerriAI/litellm/pull/21429)
- **Fix StreamableHTTPSessionManager** — Revert to stateless mode to prevent session state issues - [PR #21323](https://github.com/BerriAI/litellm/pull/21323)
- **Fix Bedrock AgentCore Accept header** — Add required Accept header for AgentCore MCP server requests - [PR #21551](https://github.com/BerriAI/litellm/pull/21551)
---
## Performance / Loadbalancing / Reliability improvements
**Logging & callback overhead**
- Move async/sync callback separation from per-request to callback registration time — ~30% speedup for callback-heavy deployments - [PR #20354](https://github.com/BerriAI/litellm/pull/20354)
- Skip Pydantic Usage round-trip in logging payload — reduces serialization overhead per request - [PR #21003](https://github.com/BerriAI/litellm/pull/21003)
- Skip duplicate `get_standard_logging_object_payload` calls for non-streaming requests - [PR #20440](https://github.com/BerriAI/litellm/pull/20440)
- Reuse `LiteLLM_Params` object across the request lifecycle - [PR #20593](https://github.com/BerriAI/litellm/pull/20593)
- Optimize `add_litellm_data_to_request` hot path - [PR #20526](https://github.com/BerriAI/litellm/pull/20526)
- Optimize `model_dump_with_preserved_fields` - [PR #20882](https://github.com/BerriAI/litellm/pull/20882)
- Pre-compute OpenAI client init params at module load instead of per-request - [PR #20789](https://github.com/BerriAI/litellm/pull/20789)
- Reduce proxy overhead for large base64 payloads - [PR #21594](https://github.com/BerriAI/litellm/pull/21594)
- Improve streaming proxy throughput by fixing middleware and logging bottlenecks - [PR #21501](https://github.com/BerriAI/litellm/pull/21501)
- Eliminate per-chunk thread spawning in Responses API async streaming - [PR #21709](https://github.com/BerriAI/litellm/pull/21709)
**Cost calculation**
- Optimize `completion_cost()` with early-exit and caching - [PR #20448](https://github.com/BerriAI/litellm/pull/20448)
- Cost calculator: reduce repeated lookups and dict copies - [PR #20541](https://github.com/BerriAI/litellm/pull/20541)
**Router & load balancing**
- Remove quadratic deployment scan in usage-based routing v2 - [PR #21211](https://github.com/BerriAI/litellm/pull/21211)
- Avoid O(n²) membership scans in team deployment filter - [PR #21210](https://github.com/BerriAI/litellm/pull/21210)
- Avoid O(n) alias scan for non-alias `get_model_list` lookups - [PR #21136](https://github.com/BerriAI/litellm/pull/21136)
- Increase default LRU cache size to reduce multi-model cache thrash - [PR #21139](https://github.com/BerriAI/litellm/pull/21139)
- Cache `get_model_access_groups()` no-args result on Router - [PR #20374](https://github.com/BerriAI/litellm/pull/20374)
- Deployment affinity routing callback — route to the same deployment for a session - [PR #19143](https://github.com/BerriAI/litellm/pull/19143)
- Session-ID-based routing — use `session_id` for consistent routing within a session - [PR #21763](https://github.com/BerriAI/litellm/pull/21763)
**Connection management & reliability**
- Fix Redis connection pool reliability — prevent connection exhaustion under load - [PR #21717](https://github.com/BerriAI/litellm/pull/21717)
- Fix Prisma connection self-heal for auth and runtime reconnection (reverted, will be re-introduced with fixes) - [PR #21706](https://github.com/BerriAI/litellm/pull/21706)
- Close streaming connections to prevent connection pool exhaustion - [PR #21213](https://github.com/BerriAI/litellm/pull/21213)
- Make `PodLockManager.release_lock` atomic compare-and-delete - [PR #21226](https://github.com/BerriAI/litellm/pull/21226)
---
## Database Changes
### Schema Updates
| Table | Change Type | Description | PR |
| ----- | ----------- | ----------- | -- |
| `LiteLLM_DeletedVerificationToken` | New Column | Added `project_id` column | [PR #21587](https://github.com/BerriAI/litellm/pull/21587) |
| `LiteLLM_ProjectTable` | New Table | Project management for organizing resources | [PR #21078](https://github.com/BerriAI/litellm/pull/21078) |
| `LiteLLM_VerificationToken` | New Column | Added `last_active` timestamp for key activity tracking | [PR #21545](https://github.com/BerriAI/litellm/pull/21545) |
| `LiteLLM_ManagedVectorStoreTable` | Migration | Make vector store migration idempotent | [PR #21325](https://github.com/BerriAI/litellm/pull/21325) |
---
## Documentation Updates
- Add OpenAI Agents SDK with LiteLLM guide - [PR #21311](https://github.com/BerriAI/litellm/pull/21311)
- Access Groups documentation - [PR #21236](https://github.com/BerriAI/litellm/pull/21236)
- Anthropic beta headers documentation - [PR #21320](https://github.com/BerriAI/litellm/pull/21320)
- Latency overhead troubleshooting guide - [PR #21600](https://github.com/BerriAI/litellm/pull/21600), [PR #21603](https://github.com/BerriAI/litellm/pull/21603)
- Add rollback safety check guide - [PR #21743](https://github.com/BerriAI/litellm/pull/21743)
- Incident report: vLLM Embeddings broken by encoding_format parameter - [PR #21474](https://github.com/BerriAI/litellm/pull/21474)
- Incident report: Claude Code beta headers - [PR #21485](https://github.com/BerriAI/litellm/pull/21485)
- Mark v1.81.12 as stable - [PR #21809](https://github.com/BerriAI/litellm/pull/21809)
---
## New Contributors
* @mjkam made their first contribution in [PR #21306](https://github.com/BerriAI/litellm/pull/21306)
* @saneroen made their first contribution in [PR #21243](https://github.com/BerriAI/litellm/pull/21243)
* @vincentkoc made their first contribution in [PR #21239](https://github.com/BerriAI/litellm/pull/21239)
* @felixti made their first contribution in [PR #19745](https://github.com/BerriAI/litellm/pull/19745)
* @anttttti made their first contribution in [PR #20731](https://github.com/BerriAI/litellm/pull/20731)
* @ndgigliotti made their first contribution in [PR #21222](https://github.com/BerriAI/litellm/pull/21222)
* @iamadamreed made their first contribution in [PR #19912](https://github.com/BerriAI/litellm/pull/19912)
* @sahukanishka made their first contribution in [PR #21220](https://github.com/BerriAI/litellm/pull/21220)
* @namabile made their first contribution in [PR #21195](https://github.com/BerriAI/litellm/pull/21195)
* @stronk7 made their first contribution in [PR #21372](https://github.com/BerriAI/litellm/pull/21372)
* @ZeroAurora made their first contribution in [PR #21547](https://github.com/BerriAI/litellm/pull/21547)
* @SolitudePy made their first contribution in [PR #21497](https://github.com/BerriAI/litellm/pull/21497)
* @SherifWaly made their first contribution in [PR #21557](https://github.com/BerriAI/litellm/pull/21557)
* @dkindlund made their first contribution in [PR #21633](https://github.com/BerriAI/litellm/pull/21633)
* @cagojeiger made their first contribution in [PR #21664](https://github.com/BerriAI/litellm/pull/21664)
---
## Full Changelog
[v1.81.12.rc.1...v1.81.14.rc.1](https://github.com/BerriAI/litellm/compare/v1.81.12.rc.1...v1.81.14.rc.1)
docs/my-website/sidebars.js
+7
View File
@@ -45,6 +45,7 @@ const sidebars = {
"proxy/guardrails/guardrail_load_balancing",
"proxy/guardrails/test_playground",
"proxy/guardrails/litellm_content_filter",
"proxy/guardrails/realtime_guardrails",
{
type: "category",
label: "Providers",
@@ -161,10 +162,12 @@ const sidebars = {
]
},
"tutorials/opencode_integration",
"tutorials/openclaw_integration",
"tutorials/cost_tracking_coding",
"tutorials/cursor_integration",
"tutorials/github_copilot_integration",
"tutorials/litellm_gemini_cli",
"tutorials/google_genai_sdk",
"tutorials/litellm_qwen_code_cli",
"tutorials/openai_codex"
]
@@ -179,6 +182,7 @@ const sidebars = {
slug: "/agent_sdks"
},
items: [
"tutorials/openai_agents_sdk",
"tutorials/claude_agent_sdk",
"tutorials/copilotkit_sdk",
"tutorials/google_adk",
@@ -334,6 +338,7 @@ const sidebars = {
"proxy/ui_credentials",
"proxy/ai_hub",
"proxy/model_compare_ui",
"proxy/ui_store_model_db_setting",
]
},
{
@@ -417,6 +422,7 @@ const sidebars = {
"proxy/dynamic_rate_limit",
"proxy/rate_limit_tiers",
"proxy/temporary_budget_increase",
"proxy/budget_reset_and_tz",
],
},
"proxy/caching",
@@ -1130,6 +1136,7 @@ const sidebars = {
"troubleshoot/prisma_migrations",
],
},
"troubleshoot/rollback",
"troubleshoot",
],
},
enterprise/LICENSE.md
+1 -1
View File
@@ -7,7 +7,7 @@ With regard to the BerriAI Software:
This software and associated documentation files (the "Software") may only be
used in production, if you (and any entity that you represent) have agreed to,
and are in compliance with, the BerriAI Subscription Terms of Service, available
via [call](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) or email (info@berri.ai) (the "Enterprise Terms"), or other
via [call](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions) or email (info@berri.ai) (the "Enterprise Terms"), or other
agreement governing the use of the Software, as agreed by you and BerriAI,
and otherwise have a valid BerriAI Enterprise license for the
correct number of user seats. Subject to the foregoing sentence, you are free to
enterprise/README.md
+1 -1
View File
@@ -4,6 +4,6 @@ Code in this folder is licensed under a commercial license. Please review the [L
**These features are covered under the LiteLLM Enterprise contract**
👉 **Using in an Enterprise / Need specific features ?** Meet with us [here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat?month=2024-02)
👉 **Using in an Enterprise / Need specific features ?** Meet with us [here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions?month=2024-02)
See all Enterprise Features here 👉 [Docs](https://docs.litellm.ai/docs/proxy/enterprise)
enterprise/litellm_enterprise/enterprise_callbacks/send_emails/base_email.py
+128 -68
View File
@@ -16,6 +16,10 @@ from litellm_enterprise.types.enterprise_callbacks.send_emails import (
from litellm._logging import verbose_proxy_logger
from litellm.caching.caching import DualCache
from litellm.constants import (
EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE,
EMAIL_BUDGET_ALERT_TTL,
)
from litellm.integrations.custom_logger import CustomLogger
from litellm.integrations.email_templates.email_footer import EMAIL_FOOTER
from litellm.integrations.email_templates.key_created_email import (
@@ -24,14 +28,14 @@ from litellm.integrations.email_templates.key_created_email import (
from litellm.integrations.email_templates.key_rotated_email import (
KEY_ROTATED_EMAIL_TEMPLATE,
)
from litellm.integrations.email_templates.user_invitation_email import (
USER_INVITATION_EMAIL_TEMPLATE,
)
from litellm.integrations.email_templates.templates import (
MAX_BUDGET_ALERT_EMAIL_TEMPLATE,
SOFT_BUDGET_ALERT_EMAIL_TEMPLATE,
TEAM_SOFT_BUDGET_ALERT_EMAIL_TEMPLATE,
)
from litellm.integrations.email_templates.user_invitation_email import (
USER_INVITATION_EMAIL_TEMPLATE,
)
from litellm.proxy._types import (
CallInfo,
InvitationNew,
@@ -41,10 +45,6 @@ from litellm.proxy._types import (
)
from litellm.secret_managers.main import get_secret_bool
from litellm.types.integrations.slack_alerting import LITELLM_LOGO_URL
from litellm.constants import (
EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE,
EMAIL_BUDGET_ALERT_TTL,
)
class BaseEmailLogger(CustomLogger):
@@ -121,10 +121,16 @@ class BaseEmailLogger(CustomLogger):
)
# Check if API key should be included in email
include_api_key = get_secret_bool(secret_name="EMAIL_INCLUDE_API_KEY", default_value=True)
include_api_key = get_secret_bool(
secret_name="EMAIL_INCLUDE_API_KEY", default_value=True
)
if include_api_key is None:
include_api_key = True # Default to True if not set
key_token_display = send_key_created_email_event.virtual_key if include_api_key else "[Key hidden for security - retrieve from dashboard]"
key_token_display = (
send_key_created_email_event.virtual_key
if include_api_key
else "[Key hidden for security - retrieve from dashboard]"
)
email_html_content = KEY_CREATED_EMAIL_TEMPLATE.format(
email_logo_url=email_params.logo_url,
@@ -162,10 +168,16 @@ class BaseEmailLogger(CustomLogger):
)
# Check if API key should be included in email
include_api_key = get_secret_bool(secret_name="EMAIL_INCLUDE_API_KEY", default_value=True)
include_api_key = get_secret_bool(
secret_name="EMAIL_INCLUDE_API_KEY", default_value=True
)
if include_api_key is None:
include_api_key = True # Default to True if not set
key_token_display = send_key_rotated_email_event.virtual_key if include_api_key else "[Key hidden for security - retrieve from dashboard]"
key_token_display = (
send_key_rotated_email_event.virtual_key
if include_api_key
else "[Key hidden for security - retrieve from dashboard]"
)
email_html_content = KEY_ROTATED_EMAIL_TEMPLATE.format(
email_logo_url=email_params.logo_url,
@@ -201,7 +213,9 @@ class BaseEmailLogger(CustomLogger):
)
# Format budget values
soft_budget_str = f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
soft_budget_str = (
f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
)
spend_str = f"${event.spend}" if event.spend is not None else "$0.00"
max_budget_info = ""
if event.max_budget is not None:
@@ -231,13 +245,13 @@ class BaseEmailLogger(CustomLogger):
"""
# Collect all recipient emails
recipient_emails: List[str] = []
# Add additional alert emails from team metadata.soft_budget_alert_emails
if hasattr(event, "alert_emails") and event.alert_emails:
for email in event.alert_emails:
if email and email not in recipient_emails: # Avoid duplicates
recipient_emails.append(email)
# If no recipients found, skip sending
if not recipient_emails:
verbose_proxy_logger.warning(
@@ -268,7 +282,9 @@ class BaseEmailLogger(CustomLogger):
)
# Format budget values
soft_budget_str = f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
soft_budget_str = (
f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
)
spend_str = f"${event.spend}" if event.spend is not None else "$0.00"
max_budget_info = ""
if event.max_budget is not None:
@@ -286,7 +302,7 @@ class BaseEmailLogger(CustomLogger):
base_url=email_params.base_url,
email_support_contact=email_params.support_contact,
)
# Send email to all recipients
await self.send_email(
from_email=self.DEFAULT_LITELLM_EMAIL,
@@ -313,11 +329,17 @@ class BaseEmailLogger(CustomLogger):
# Format budget values
spend_str = f"${event.spend}" if event.spend is not None else "$0.00"
max_budget_str = f"${event.max_budget}" if event.max_budget is not None else "N/A"
max_budget_str = (
f"${event.max_budget}" if event.max_budget is not None else "N/A"
)
# Calculate percentage and alert threshold
percentage = int(EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE * 100)
alert_threshold_str = f"${event.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE:.2f}" if event.max_budget is not None else "N/A"
alert_threshold_str = (
f"${event.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE:.2f}"
if event.max_budget is not None
else "N/A"
)
email_html_content = MAX_BUDGET_ALERT_EMAIL_TEMPLATE.format(
email_logo_url=email_params.logo_url,
@@ -382,7 +404,10 @@ class BaseEmailLogger(CustomLogger):
# For non-team alerts, require either max_budget or soft_budget
if user_info.max_budget is None and user_info.soft_budget is None:
return
if user_info.soft_budget is not None and user_info.spend >= user_info.soft_budget:
if (
user_info.soft_budget is not None
and user_info.spend >= user_info.soft_budget
):
# Generate cache key based on event type and identifier
# Use appropriate ID based on event_group to ensure unique cache keys per entity type
if user_info.event_group == Litellm_EntityType.TEAM:
@@ -395,7 +420,7 @@ class BaseEmailLogger(CustomLogger):
# For KEY and other types, use token or user_id
_id = user_info.token or user_info.user_id or "default_id"
_cache_key = f"email_budget_alerts:soft_budget_crossed:{_id}"
# Check if we've already sent this alert
result = await _cache.async_get_cache(key=_cache_key)
if result is None:
@@ -420,14 +445,14 @@ class BaseEmailLogger(CustomLogger):
event_group=user_info.event_group,
alert_emails=user_info.alert_emails,
)
try:
# Use team-specific function for team alerts, otherwise use standard function
if user_info.event_group == Litellm_EntityType.TEAM:
await self.send_team_soft_budget_alert_email(webhook_event)
else:
await self.send_soft_budget_alert_email(webhook_event)
# Cache the alert to prevent duplicate sends
await _cache.async_set_cache(
key=_cache_key,
@@ -444,20 +469,27 @@ class BaseEmailLogger(CustomLogger):
# For max_budget_alert, check if we've already sent an alert
if type == "max_budget_alert":
if user_info.max_budget is not None and user_info.spend is not None:
alert_threshold = user_info.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE
alert_threshold = (
user_info.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE
)
# Only alert if we've crossed the threshold but haven't exceeded max_budget yet
if user_info.spend >= alert_threshold and user_info.spend < user_info.max_budget:
if (
user_info.spend >= alert_threshold
and user_info.spend < user_info.max_budget
):
# Generate cache key based on event type and identifier
_id = user_info.token or user_info.user_id or "default_id"
_cache_key = f"email_budget_alerts:max_budget_alert:{_id}"
# Check if we've already sent this alert
result = await _cache.async_get_cache(key=_cache_key)
if result is None:
# Calculate percentage
percentage = int(EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE * 100)
percentage = int(
EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE * 100
)
# Create WebhookEvent for max budget alert
event_message = f"Max Budget Alert - {percentage}% of Maximum Budget Reached"
webhook_event = WebhookEvent(
@@ -478,10 +510,10 @@ class BaseEmailLogger(CustomLogger):
projected_spend=user_info.projected_spend,
event_group=user_info.event_group,
)
try:
await self.send_max_budget_alert_email(webhook_event)
# Cache the alert to prevent duplicate sends
await _cache.async_set_cache(
key=_cache_key,
@@ -525,9 +557,14 @@ class BaseEmailLogger(CustomLogger):
unused_custom_fields = []
# Function to safely get custom value or default
def get_custom_or_default(custom_value: Optional[str], default_value: str, field_name: str) -> str:
if custom_value is not None: # Only check premium if trying to use custom value
def get_custom_or_default(
custom_value: Optional[str], default_value: str, field_name: str
) -> str:
if (
custom_value is not None
): # Only check premium if trying to use custom value
from litellm.proxy.proxy_server import premium_user
if premium_user is not True:
unused_custom_fields.append(field_name)
return default_value
@@ -536,38 +573,48 @@ class BaseEmailLogger(CustomLogger):
# Get parameters, falling back to defaults if custom values aren't allowed
logo_url = get_custom_or_default(custom_logo, LITELLM_LOGO_URL, "logo URL")
support_contact = get_custom_or_default(custom_support, self.DEFAULT_SUPPORT_EMAIL, "support contact")
base_url = os.getenv("PROXY_BASE_URL", "http://0.0.0.0:4000") # Not a premium feature
signature = get_custom_or_default(custom_signature, EMAIL_FOOTER, "email signature")
support_contact = get_custom_or_default(
custom_support, self.DEFAULT_SUPPORT_EMAIL, "support contact"
)
base_url = os.getenv(
"PROXY_BASE_URL", "http://0.0.0.0:4000"
) # Not a premium feature
signature = get_custom_or_default(
custom_signature, EMAIL_FOOTER, "email signature"
)
# Get custom subject template based on email event type
if email_event == EmailEvent.new_user_invitation:
subject_template = get_custom_or_default(
custom_subject_invitation,
self.DEFAULT_SUBJECT_TEMPLATES[EmailEvent.new_user_invitation],
"invitation subject template"
"invitation subject template",
)
elif email_event == EmailEvent.virtual_key_created:
subject_template = get_custom_or_default(
custom_subject_key_created,
self.DEFAULT_SUBJECT_TEMPLATES[EmailEvent.virtual_key_created],
"key created subject template"
"key created subject template",
)
elif email_event == EmailEvent.virtual_key_rotated:
custom_subject_key_rotated = os.getenv("EMAIL_SUBJECT_KEY_ROTATED", None)
subject_template = get_custom_or_default(
custom_subject_key_rotated,
self.DEFAULT_SUBJECT_TEMPLATES[EmailEvent.virtual_key_rotated],
"key rotated subject template"
"key rotated subject template",
)
else:
subject_template = "LiteLLM: {event_message}"
subject = subject_template.format(event_message=event_message) if event_message else "LiteLLM Notification"
subject = (
subject_template.format(event_message=event_message)
if event_message
else "LiteLLM Notification"
)
recipient_email: Optional[
str
] = user_email or await self._lookup_user_email_from_db(user_id=user_id)
recipient_email: Optional[str] = (
user_email or await self._lookup_user_email_from_db(user_id=user_id)
)
if recipient_email is None:
raise ValueError(
f"User email not found for user_id: {user_id}. User email is required to send email."
@@ -585,11 +632,9 @@ class BaseEmailLogger(CustomLogger):
warning_msg = (
f"Email sent with default values instead of custom values for: {fields_str}. "
"This is an Enterprise feature. To use custom email fields, please upgrade to LiteLLM Enterprise. "
"Schedule a meeting here: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat"
)
verbose_proxy_logger.warning(
f"{warning_msg}"
"Schedule a meeting here: https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions"
)
verbose_proxy_logger.warning(f"{warning_msg}")
return EmailParams(
logo_url=logo_url,
@@ -636,44 +681,49 @@ class BaseEmailLogger(CustomLogger):
if not user_id:
verbose_proxy_logger.debug("No user_id provided for invitation link")
return base_url
if not await self._is_prisma_client_available():
return base_url
# Wait for any concurrent invitation creation to complete
await self._wait_for_invitation_creation()
# Get or create invitation
invitation = await self._get_or_create_invitation(user_id)
if not invitation:
verbose_proxy_logger.warning(f"Failed to get/create invitation for user_id: {user_id}")
verbose_proxy_logger.warning(
f"Failed to get/create invitation for user_id: {user_id}"
)
return base_url
return self._construct_invitation_link(invitation.id, base_url)
async def _is_prisma_client_available(self) -> bool:
"""Check if Prisma client is available"""
from litellm.proxy.proxy_server import prisma_client
if prisma_client is None:
verbose_proxy_logger.debug("Prisma client not found. Unable to lookup invitation")
verbose_proxy_logger.debug(
"Prisma client not found. Unable to lookup invitation"
)
return False
return True
async def _wait_for_invitation_creation(self) -> None:
"""
Wait for any concurrent invitation creation to complete.
The UI calls /invitation/new to generate the invitation link.
We wait to ensure any pending invitation creation is completed.
"""
import asyncio
await asyncio.sleep(10)
async def _get_or_create_invitation(self, user_id: str):
"""
Get existing invitation or create a new one for the user
Returns:
Invitation object with id attribute, or None if failed
"""
@@ -681,31 +731,41 @@ class BaseEmailLogger(CustomLogger):
create_invitation_for_user,
)
from litellm.proxy.proxy_server import prisma_client
if prisma_client is None:
verbose_proxy_logger.error("Prisma client is None in _get_or_create_invitation")
verbose_proxy_logger.error(
"Prisma client is None in _get_or_create_invitation"
)
return None
try:
# Try to get existing invitation
existing_invitations = await prisma_client.db.litellm_invitationlink.find_many(
where={"user_id": user_id},
order={"created_at": "desc"},
existing_invitations = (
await prisma_client.db.litellm_invitationlink.find_many(
where={"user_id": user_id},
order={"created_at": "desc"},
)
)
if existing_invitations and len(existing_invitations) > 0:
verbose_proxy_logger.debug(f"Found existing invitation for user_id: {user_id}")
verbose_proxy_logger.debug(
f"Found existing invitation for user_id: {user_id}"
)
return existing_invitations[0]
# Create new invitation if none exists
verbose_proxy_logger.debug(f"Creating new invitation for user_id: {user_id}")
verbose_proxy_logger.debug(
f"Creating new invitation for user_id: {user_id}"
)
return await create_invitation_for_user(
data=InvitationNew(user_id=user_id),
user_api_key_dict=UserAPIKeyAuth(user_id=user_id),
)
except Exception as e:
verbose_proxy_logger.error(f"Error getting/creating invitation for user_id {user_id}: {e}")
verbose_proxy_logger.error(
f"Error getting/creating invitation for user_id {user_id}: {e}"
)
return None
def _construct_invitation_link(self, invitation_id: str, base_url: str) -> str:
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
+53 -5
View File
@@ -13,6 +13,9 @@ if TYPE_CHECKING:
from litellm.router import Router
CHECK_BATCH_COST_USER_AGENT = "LiteLLM Proxy/CheckBatchCost"
class CheckBatchCost:
def __init__(
self,
@@ -27,6 +30,25 @@ class CheckBatchCost:
self.prisma_client: PrismaClient = prisma_client
self.llm_router: Router = llm_router
async def _get_user_info(self, batch_id, user_id) -> dict:
"""
Look up user email and key alias by user_id for enriching the S3 callback metadata.
Returns a dict with user_api_key_user_email and user_api_key_alias (both may be None).
"""
try:
user_row = await self.prisma_client.db.litellm_usertable.find_unique(
where={"user_id": user_id}
)
if user_row is None:
return {}
return {
"user_api_key_user_email": getattr(user_row, "user_email", None),
"user_api_key_alias": getattr(user_row, "user_alias", None),
}
except Exception as e:
verbose_proxy_logger.error(f"CheckBatchCost: could not look up user {user_id} for batch {batch_id}: {e}")
return {}
async def check_batch_cost(self):
"""
Check if the batch JOB has been tracked.
@@ -48,10 +70,12 @@ class CheckBatchCost:
get_model_id_from_unified_batch_id,
)
# Look for all batches that have not yet been processed by CheckBatchCost
jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
where={
"status": {"in": ["validating", "in_progress", "finalizing"]},
"file_purpose": "batch",
"batch_processed" : False,
"status": {"not_in": ["failed", "expired", "cancelled"]}
}
)
completed_jobs = []
@@ -107,6 +131,21 @@ class CheckBatchCost:
f"Batch ID: {batch_id} is complete, tracking cost and usage"
)
# aretrieve_batch is called with the raw provider batch ID, so response.id
# is the raw provider value (e.g. "batch_20260223-0518.234"). We need the
# unified base64 ID in the S3 log so downstream consumers can correlate it
# back to the batch they submitted via the proxy.
#
# CheckBatchCost builds its own LiteLLMLogging object (logging_obj below) and
# calls async_success_handler(result=response) directly. That handler calls
# _build_standard_logging_payload(response, ...) which reads response.id at
# that point — so setting response.id here is sufficient.
#
# The HTTP endpoint does this substitution via the managed files hook
# (async_post_call_success_hook). CheckBatchCost bypasses that hook entirely,
# so we do it explicitly here.
response.id = job.unified_object_id
# This background job runs as default_user_id, so going through the HTTP endpoint
# would trigger check_managed_file_id_access and get 403. Instead, extract the raw
# provider file ID and call afile_content directly with deployment credentials.
@@ -171,11 +210,21 @@ class CheckBatchCost:
function_id=str(uuid.uuid4()),
)
creator_user_id = job.created_by
user_info = await self._get_user_info(batch_id, job.created_by)
logging_obj.update_environment_variables(
litellm_params={
# set the user-agent header so that S3 callback consumers can easily identify CheckBatchCost callbacks
"proxy_server_request": {
"headers": {
"user-agent": CHECK_BATCH_COST_USER_AGENT,
}
},
"metadata": {
"user_api_key_user_id": job.created_by or "default-user-id",
}
"user_api_key_user_id": creator_user_id,
**user_info,
},
},
optional_params={},
)
@@ -191,8 +240,7 @@ class CheckBatchCost:
completed_jobs.append(job)
if len(completed_jobs) > 0:
# mark the jobs as complete
await self.prisma_client.db.litellm_managedobjecttable.update_many(
where={"id": {"in": [job.id for job in completed_jobs]}},
data={"status": "complete"},
data={"batch_processed": True, "status": "complete"},
)
enterprise/litellm_enterprise/proxy/hooks/managed_files.py
+5 -7
View File
@@ -1086,11 +1086,8 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
self, file_id: str
) -> List[Dict[str, Any]]:
"""
Find batches in non-terminal states that reference this file.
Non-terminal states: validating, in_progress, finalizing
Terminal states: completed, complete, failed, expired, cancelled
Find batches that reference this file and still need cost tracking.
Find batches that are in non-terminal state and have not yet been processed by CheckBatchCost.
Args:
file_id: The unified file ID to check
@@ -1121,7 +1118,8 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
batches = await self.prisma_client.db.litellm_managedobjecttable.find_many(
where={
"file_purpose": "batch",
"status": {"in": ["validating", "in_progress", "finalizing"]},
"batch_processed": False,
"status": {"not_in": ["failed", "expired", "cancelled"]}
},
take=MAX_MATCHES_TO_RETURN,
order={"created_at": "desc"},
@@ -1205,7 +1203,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
error_message += (
f"To delete this file before complete cost tracking, please delete or cancel the referencing batch(es) first. "
f"Alternatively, wait for all batches to complete processing."
f"Alternatively, wait for all batches to complete and for cost to be computed (batch_processed=true)."
)
raise HTTPException(
license_cache.json
+9
View File
@@ -0,0 +1,9 @@
{
"tornado:6.5.3": "Apache-2.0",
"redisvl:0.4.1": "MIT",
"google-cloud-iam:2.19.1": "Apache 2.0",
"google-genai:1.37.0": "Apache-2.0",
"azure-keyvault:4.2.0": "MIT License",
"soundfile:0.12.1": "BSD 3-Clause License",
"openapi-core:0.21.0": "BSD-3-Clause"
}
litellm-js/spend-logs/package.json
+15 -3
View File
@@ -12,7 +12,19 @@
},
"overrides": {
"glob": ">=11.1.0",
"tar": ">=7.5.7",
"@isaacs/brace-expansion": ">=5.0.1"
"tar": ">=7.5.8",
"minimatch": ">=10.2.1",
"diff": ">=8.0.3",
"@isaacs/brace-expansion": ">=5.0.1",
"@babel/traverse": ">=7.23.2",
"ws": ">=7.5.10",
"http-proxy-middleware": ">=2.0.9",
"tar-fs": ">=2.1.4",
"webpack-dev-middleware": ">=5.3.4",
"braces": ">=3.0.3",
"axios": ">=0.30.2",
"webpack": ">=5.94.0",
"serve-static": ">=1.16.0",
"path-to-regexp": ">=0.1.12"
}
}
}

Some files were not shown because too many files have changed in this diff Show More