diff --git a/docs/my-website/docs/proxy/config_settings.md b/docs/my-website/docs/proxy/config_settings.md index 5c2d6a79a3..809478bb6d 100644 --- a/docs/my-website/docs/proxy/config_settings.md +++ b/docs/my-website/docs/proxy/config_settings.md @@ -758,7 +758,7 @@ router_settings: | LITELLM_CLI_JWT_EXPIRATION_HOURS | Expiration time in hours for CLI-generated JWT tokens. Default is 24 hours | LITELLM_DD_AGENT_HOST | Hostname or IP of DataDog agent for LiteLLM-specific logging. When set, logs are sent to agent instead of direct API | LITELLM_DEPLOYMENT_ENVIRONMENT | Environment name for the deployment (e.g., "production", "staging"). Used as a fallback when OTEL_ENVIRONMENT_NAME is not set. Sets the `environment` tag in telemetry data -| LITELLM_DETAILED_TIMING | When true, adds detailed per-phase timing headers to responses (x-litellm-timing-{pre-processing,llm-api,post-processing,message-copy}-ms). Default is false. See [latency overhead docs](../troubleshoot/latency_overhead.md) +| LITELLM_DETAILED_TIMING | When true, adds detailed per-phase timing headers to responses (`x-litellm-timing-{pre-processing,llm-api,post-processing,message-copy}-ms`). Default is false. See [latency overhead docs](../troubleshoot/latency_overhead.md) | LITELLM_DD_AGENT_PORT | Port of DataDog agent for LiteLLM-specific log intake. Default is 10518 | LITELLM_DD_LLM_OBS_PORT | Port for Datadog LLM Observability agent. Default is 8126 | LITELLM_DONT_SHOW_FEEDBACK_BOX | Flag to hide feedback box in LiteLLM UI diff --git a/docs/my-website/img/release_notes/compliance_playground.png b/docs/my-website/img/release_notes/compliance_playground.png new file mode 100644 index 0000000000..1b5c5dfd88 Binary files /dev/null and b/docs/my-website/img/release_notes/compliance_playground.png differ diff --git a/docs/my-website/img/release_notes/guardrail_garden.png b/docs/my-website/img/release_notes/guardrail_garden.png new file mode 100644 index 0000000000..072a15bcb3 Binary files /dev/null and b/docs/my-website/img/release_notes/guardrail_garden.png differ diff --git a/docs/my-website/release_notes/v1.81.14.md b/docs/my-website/release_notes/v1.81.14.md index e7c5cfad6b..73a9b5e28f 100644 --- a/docs/my-website/release_notes/v1.81.14.md +++ b/docs/my-website/release_notes/v1.81.14.md @@ -1,5 +1,5 @@ --- -title: "v1.81.14-stable - Claude Sonnet 4.6, Guardrail Garden & Major Performance Improvements" +title: "[Preview] v1.81.14 - New Gateway Level Guardrails & Compliance Playground" slug: "v1-81-14" date: 2026-02-21T00:00:00 authors: @@ -27,7 +27,7 @@ import Image from '@theme/IdealImage'; docker run \ -e STORE_MODEL_IN_DB=True \ -p 4000:4000 \ -ghcr.io/berriai/litellm:main-v1.81.14-stable +ghcr.io/berriai/litellm:main-v1.81.14.rc.1 ``` @@ -42,41 +42,54 @@ pip install litellm==1.81.14 ## Key Highlights -- **Guardrail Model Garden** - [Easily browse built-in and partner guardrails by use case — competitor blockers, topic filters, keyword lists, GDPR PII, EU AI Act, prompt injection, and more](../../docs/proxy/guardrails/policy_templates) -- **3 new built-in guardrails** - [Competitor name blocker, topic blocker, and insults/keyword filter — all run at the gateway level, no external API, configurable per-team or key](../../docs/proxy/guardrails) -- **Compliance Playground** - [Run your guardrails against your own datasets or pre-built ones to measure performance before going live](../../docs/proxy/guardrails/policy_templates) +- **Guardrail Garden** — [Browse built-in and partner guardrails by use case — competitor blocking, topic filtering, GDPR, prompt injection, and more. Pick a template, customize it, attach it to a team or key.](../../docs/proxy/guardrails/policy_templates) +- **Compliance Playground** — [Test any guardrail policy against your own traffic before it goes live. See precision, recall, and false positive rate — so you know how it'll behave in production.](../../docs/proxy/guardrails/policy_templates) +- **3 new zero-cost built-in guardrails** — [Competitor name blocker, topic blocker, and insults filter — all gateway-level, <0.1ms latency, no external API, configurable per-team or key](../../docs/proxy/guardrails) - **Store Model in DB Settings via UI** - [Configure model storage directly in the Admin UI without editing config files or restarting the proxy—perfect for cloud deployments](../../docs/proxy/ui_store_model_db_setting) -- **Claude Sonnet 4.6 — day 0** - [Full support across Anthropic and Vertex AI: reasoning, computer use, prompt caching, 200K context](../../docs/providers/anthropic) -- **Major performance batch** - 20+ targeted optimizations across router algorithms, logging overhead, cost calculator, and connection management — meaningfully lower latency and CPU overhead on every request +- **Claude Sonnet 4.6 — day 0** — [Full support across Anthropic and Vertex AI: reasoning, computer use, prompt caching, 200K context](../../docs/providers/anthropic) +- **20+ performance optimizations** — Faster routing, lower logging overhead, reduced cost-calculator latency, and connection pool fixes — meaningfully less CPU and latency on every request --- -## Guardrail Model Garden -Browse built-in and partner guardrails organized by use case — competitor blocking, topic filtering, keyword lists, GDPR/EU AI Act compliance, prompt injection detection, and more. Pick a template, customize the parameters (keyword lists, blocked topics, score thresholds), and attach it to a team or key. +### Guardrail Garden -## 3 New Built-in Guardrails +AI Platform Admins can now browse built-in and partner guardrails from the Guardrail Garden. Guardrails are organized by use case — blocking financial advice, filtering insults, detecting competitor mentions, and more — so you can find the right one and deploy it in a few clicks. -A competitor name blocker, a topic blocker (keyword and embedding-based), and an insults/keyword filter. All run at the gateway level with no external API call. They're configurable per-team or per-key, and you can swap in AWS Bedrock Guardrails or Azure Content Safety on the same endpoint without changing your application code. + -## Compliance Playground +### 3 New Built-in Guardrails -Upload your own dataset or use a pre-built one to measure how a guardrail policy performs before it goes live. See precision, recall, and false positive rate on your actual traffic patterns — so you know how the policy will behave in production before you deploy it. +This release brings 3 new built-in guardrails that run directly on the gateway. This is great for AI Gateway Admins who need low latency, zero cost guardrails for their scenarios. -## Store Model in DB Settings via UI +- **Denied Financial Advice** — detects requests for personalized financial advice, investment recommendations, or financial planning +- **Denied Insults** — detects insults, name-calling, and personal attacks directed at the chatbot, staff, or other people +- **Competitor Name Blocker** — detects mentions of competitor brands in responses + +These guardrails are built for production and on our benchmarks had a 100% Recall and Precision. + +### Store Model in DB Settings via UI Previously, the `store_model_in_db` setting could only be configured in `proxy_config.yaml` under `general_settings`, requiring a proxy restart to take effect. Now you can enable or disable this setting directly from the Admin UI without any restarts. This is especially useful for cloud deployments where you don't have direct access to config files or want to avoid downtime. Enable `store_model_in_db` to move model definitions from your YAML into the database—reducing config complexity, improving scalability, and enabling dynamic model management across multiple proxy instances. -### Eval results +#### Eval results -We benchmark every built-in guardrail against labeled datasets before shipping. Results for the two policies most relevant to topic and keyword blocking (207 investment-question cases, 299 insult cases): +We benchmarked our new built-in guardrails against labeled datasets before shipping. You can see the results for Denied Financial Advice (207 cases) and Denied Insults (299 cases): | Guardrail | Precision | Recall | F1 | Latency p50 | Cost/req | |-----------|-----------|--------|----|-------------|----------| -| Block investment questions | 100% | 100% | 100% | <0.1ms | $0 | -| Block insults / keywords | 100% | 100% | 100% | <0.1ms | $0 | +| Denied Financial Advice | 100% | 100% | 100% | <0.1ms | $0 | +| Denied Insults | 100% | 100% | 100% | <0.1ms | $0 | + +100% precision means zero false positives — no legitimate messages were incorrectly blocked. 100% recall means zero false negatives — every message that should have been blocked was caught. + + +### Compliance Playground + +The Compliance Playground lets you test any guardrail against our pre-built eval datasets or your own custom datasets, so you can see precision, recall, and false positive rate before rolling it out to production. + + -For reference, ONNX embedding approaches on the same eval set hit 95–98% precision at 2–20ms latency and require additional dependencies. The built-in content filter uses no ML model — just structured YAML rules with layered matching — so there's nothing to download, no API key needed, and latency is effectively zero. --- diff --git a/litellm/proxy/_experimental/out/404.html b/litellm/proxy/_experimental/out/404.html index d151ca6146..749b925129 100644 --- a/litellm/proxy/_experimental/out/404.html +++ b/litellm/proxy/_experimental/out/404.html @@ -1 +1 @@ -