Merge branch 'main' into litellm_oss_staging_03_11_2026

2026-08-08 22:25:25 +00:00 · 2026-03-12 10:43:08 -03:00
parent fa68d69bcf 291e6e1841
commit 6bd7cd7573
125 changed files with 7141 additions and 857 deletions
@@ -944,7 +944,7 @@ router_settings:
 | QDRANT_URL | Connection URL for Qdrant database
 | QDRANT_VECTOR_SIZE | Vector size for Qdrant operations. Default is 1536
 | REDIS_CONNECTION_POOL_TIMEOUT | Timeout in seconds for Redis connection pool. Default is 5
-| REDIS_CLUSTER_NODES | JSON-formatted list of Redis cluster startup nodes for Redis Cluster mode. Example: '[{"host": "node1", "port": 6379}]'
+| REDIS_CLUSTER_NODES | JSON-formatted list of Redis cluster startup nodes for Redis Cluster mode. Example: `[{"host": "node1", "port": 6379}]`
 | REDIS_HOST | Hostname for Redis server
 | REDIS_PASSWORD | Password for Redis service
 | REDIS_PORT | Port number for Redis server
@@ -309,6 +309,10 @@ Response:
 </TabItem>
 </Tabs>

+## Policy Flow Builder
+
+For conditional execution (e.g., run a second guardrail only if the first fails), use the [Policy Flow Builder](./policy_flow_builder) to define pipelines with per-step pass/fail actions.
+
 ## Config Reference

 ### `policies`
@@ -323,6 +327,7 @@ policies:
      remove: [...]
    condition:
      model: ...
+    pipeline: ...  # optional; see Policy Flow Builder
 ```

 | Field | Type | Description |
@@ -332,6 +337,7 @@ policies:
 | `guardrails.add` | `list[string]` | Guardrails to enable. |
 | `guardrails.remove` | `list[string]` | Guardrails to disable (useful with inheritance). |
 | `condition.model` | `string` or `list[string]` | Optional. Only apply when model matches. Supports regex. |
+| `pipeline` | `object` | Optional. Ordered guardrail execution with per-step actions. See [Policy Flow Builder](./policy_flow_builder). |

 ### `policy_attachments`

@@ -0,0 +1,219 @@
+# Policy Flow Builder
+
+The Policy Flow Builder lets you design guardrail pipelines with **conditional execution**. Instead of running guardrails independently, you chain them into ordered steps and control what happens when each guardrail passes or fails.
+
+Two powerful patterns it enables: **guardrail fallbacks** (try a different guardrail when one fails) and **retrying the same guardrail** (run the same guardrail again if it fails, e.g. to handle transient errors).
+
+## When to use the Flow Builder
+
+| Approach | Use case |
+|----------|----------|
+| **Simple policy** (`guardrails.add`) | All guardrails run in parallel; any failure blocks the request. |
+| **Flow Builder** (pipeline) | Guardrails run in sequence; you choose actions per step (next, block, allow, custom response). |
+
+Use the Flow Builder when you need:
+
+- **Guardrail fallbacks** — use `on_fail: next` to try a different guardrail when one fails (e.g., fast filter → stricter filter)
+- **Retrying the same guardrail** — add the same guardrail as multiple steps; if it fails, `on_fail: next` moves to the next step, which can be the same guardrail again (useful for transient API errors or rate limits)
+- **Conditional routing** — e.g., if a fast guardrail fails, run a more advanced one instead of blocking immediately
+- **Custom responses** — return a specific message when a guardrail fails instead of a generic block
+- **Data chaining** — pass modified data (e.g., PII-masked content) from one step to the next
+- **Fine-grained control** — different actions on pass vs. fail per step
+
+## Concepts
+
+### Pipeline
+
+A pipeline has:
+
+- **Mode**: `pre_call` (before the LLM) or `post_call` (after the LLM)
+- **Steps**: Ordered list of guardrail steps
+
+### Step actions
+
+Each step defines what happens when the guardrail **passes** and when it **fails**:
+
+| Action | Description |
+|--------|-------------|
+| **Next Step** | Continue to the next guardrail in the pipeline |
+| **Allow** | Stop the pipeline and allow the request to proceed |
+| **Block** | Stop the pipeline and block the request |
+| **Custom Response** | Return a custom message instead of the default block |
+
+### Step options
+
+| Field | Type | Description |
+|-------|------|--------------|
+| `guardrail` | `string` | Name of the guardrail to run |
+| `on_pass` | `string` | Action when guardrail passes: `next`, `allow`, `block`, `modify_response` |
+| `on_fail` | `string` | Action when guardrail fails: `next`, `allow`, `block`, `modify_response` |
+| `pass_data` | `boolean` | Forward modified request data (e.g., PII-masked) to the next step |
+| `modify_response_message` | `string` | Custom message when using `modify_response` action |
+
+## Using the Flow Builder (UI)
+
+1. Go to **Policies** in the LiteLLM Admin UI
+2. Click **+ Create New Policy** or **Edit** on an existing policy
+3. Select **Flow Builder** (instead of the simple form)
+4. Design your flow:
+   - **Trigger** — Incoming LLM request (runs when the policy matches)
+   - **Steps** — Add guardrails, set ON PASS and ON FAIL actions per step
+   - **End** — Request proceeds to the LLM
+5. Use the **+** between steps to insert new steps
+6. Use the **Test** panel to run sample messages through the pipeline before saving
+7. Click **Save** to create or update the policy
+
+## Config (YAML)
+
+Define a pipeline in your policy config:
+
+```yaml showLineNumbers title="config.yaml"
+guardrails:
+  - guardrail_name: pii_masking
+    litellm_params:
+      guardrail: presidio
+      mode: pre_call
+
+  - guardrail_name: prompt_injection
+    litellm_params:
+      guardrail: lakera
+      mode: pre_call
+
+policies:
+  my-pipeline-policy:
+    description: "PII mask first, then check for prompt injection"
+    guardrails:
+      add:
+        - pii_masking
+        - prompt_injection
+    pipeline:
+      mode: pre_call
+      steps:
+        - guardrail: pii_masking
+          on_pass: next
+          on_fail: block
+          pass_data: true
+        - guardrail: prompt_injection
+          on_pass: allow
+          on_fail: block
+
+policy_attachments:
+  - policy: my-pipeline-policy
+    scope: "*"
+```
+
+## Fallbacks and retries
+
+### Guardrail fallbacks
+
+Use `on_fail: next` to fall back to another guardrail when one fails. Run a lightweight guardrail first; if it fails, escalate to a stricter or different provider:
+
+```yaml
+policies:
+  fallback-policy:
+    guardrails:
+      add:
+        - fast_content_filter
+        - strict_content_filter
+    pipeline:
+      mode: pre_call
+      steps:
+        - guardrail: fast_content_filter
+          on_pass: allow
+          on_fail: next
+        - guardrail: strict_content_filter
+          on_pass: allow
+          on_fail: block
+```
+
+If `fast_content_filter` passes → allow. If it fails → run `strict_content_filter`; pass → allow, fail → block.
+
+### Retrying the same guardrail
+
+Add the same guardrail as multiple steps to retry on failure. Useful for transient errors (API timeouts, rate limits):
+
+```yaml
+policies:
+  retry-policy:
+    guardrails:
+      add:
+        - lakera_prompt_injection
+    pipeline:
+      mode: pre_call
+      steps:
+        - guardrail: lakera_prompt_injection
+          on_pass: allow
+          on_fail: next
+        - guardrail: lakera_prompt_injection
+          on_pass: allow
+          on_fail: block
+```
+
+First attempt passes → allow. First attempt fails → retry the same guardrail; second pass → allow, second fail → block.
+
+## Example: Custom response on fail
+
+Return a branded message instead of a generic block:
+
+```yaml
+policies:
+  branded-block-policy:
+    guardrails:
+      add:
+        - pii_detector
+    pipeline:
+      mode: pre_call
+      steps:
+        - guardrail: pii_detector
+          on_pass: allow
+          on_fail: modify_response
+          modify_response_message: "Your message contains sensitive information. Please remove PII and try again."
+```
+
+## Test a pipeline (API)
+
+Test a pipeline with sample messages before attaching it:
+
+```bash
+curl -X POST "http://localhost:4000/policies/test-pipeline" \
+  -H "Authorization: Bearer <your_api_key>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "pipeline": {
+      "mode": "pre_call",
+      "steps": [
+        {
+          "guardrail": "pii_masking",
+          "on_pass": "next",
+          "on_fail": "block",
+          "pass_data": true
+        },
+        {
+          "guardrail": "prompt_injection",
+          "on_pass": "allow",
+          "on_fail": "block"
+        }
+      ]
+    },
+    "test_messages": [
+      {"role": "user", "content": "What is 2+2?"},
+      {"role": "user", "content": "My SSN is 123-45-6789"}
+    ]
+  }'
+```
+
+Response includes per-step outcomes (pass/fail/error), actions taken, and timing.
+
+## Pipeline vs simple policy
+
+When a policy has a `pipeline`, the pipeline defines execution order and actions. The `guardrails.add` list must include all guardrails used in the pipeline steps.
+
+| Policy type | Execution |
+|-------------|-----------|
+| Simple (`guardrails.add` only) | All guardrails run; any failure blocks |
+| Pipeline (`pipeline` present) | Steps run in order; actions control flow |
+
+## Related docs
+
+- [Guardrail Policies](./guardrail_policies) — Policy basics, attachments, inheritance
+- [Policy Templates](./policy_templates) — Pre-built policy templates
@@ -0,0 +1,143 @@
+import Image from '@theme/IdealImage';
+
+# Retool Assist
+
+This guide walks you through connecting [Retool Assist](https://docs.retool.com/apps/guides/assist/) to LiteLLM Proxy. Retool Assist uses AI to generate and edit apps from within the Retool app IDE. Using LiteLLM with Retool Assist allows you to:
+
+- Access 100+ LLMs through Retool Assist
+- Track spend and usage, set budget limits per virtual key
+- Control which models Retool Assist can access
+- Use your own LLM providers via a unified OpenAI-compatible API
+
+<div style={{ maxWidth: '100%', overflow: 'hidden', paddingBottom: '59.52%', position: 'relative', height: 0 }}>
+  <iframe 
+    style={{ position: 'absolute', top: 0, left: 0, width: '100%', height: '100%', maxWidth: '840px' }}
+    src="https://www.youtube.com/embed/aN-Iua5dHGg" 
+    frameborder="0" 
+    webkitallowfullscreen 
+    mozallowfullscreen 
+    allowfullscreen
+  ></iframe>
+</div>
+
+---
+
+:::info
+**Hosted Retool requires a public URL.** Retool Cloud runs on Retool's servers, so `localhost` will not work. You must expose your LiteLLM proxy via ngrok, Cloudflare Tunnel, or by deploying to a cloud provider.
+:::
+
+## Quick Reference
+
+| Setting | Value |
+|---------|-------|
+| Provider Schema | OpenAI |
+| Base URL | Your ngrok URL (e.g. `https://abc123.ngrok-free.app`) or deployed proxy URL |
+| API Key | Your LiteLLM Virtual Key |
+| Model | Public model name from LiteLLM (e.g. `openai/gpt-4o-mini`, `openai/gpt-5.2-2025-12-11`) |
+
+---
+
+## Prerequisites
+
+- LiteLLM Proxy running locally or deployed
+- [ngrok](https://ngrok.com/download) (or similar tunnel) for local development with hosted Retool
+- A [Retool](https://retool.com) account (Cloud or self-hosted)
+
+## 1. Start LiteLLM Proxy
+
+Set up LiteLLM Proxy following the [Getting Started Guide](https://docs.litellm.ai/docs/proxy/docker_quick_start). Ensure your proxy is running on port 4000.
+
+## 2. Expose LiteLLM with a Public URL
+
+<Image img={require('../../img/ngrok_public_url.gif')} />
+
+Retool Cloud runs on Retool's servers. You must expose your local LiteLLM proxy with a public URL.
+
+### Using ngrok
+
+- Install [ngrok](https://ngrok.com/download)
+- In a separate terminal, run:
+  
+```bash
+ngrok http 4000
+```
+- Copy the generated HTTPS URL (e.g. `https://abc123.ngrok-free.app`). This is your **Base URL** for Retool.
+
+
+### Alternative
+
+If you deploy LiteLLM to Railway, Render, Fly.io, or another cloud provider, use that public URL as your Base URL. See the [Deploy guide](https://docs.litellm.ai/docs/proxy/deploy) for details.
+
+## 3. Generate a Virtual Key
+
+<Image img={require('../../img/litellm_virtual_key.gif')} />
+
+Create a virtual key that Retool Assist will use to authenticate with LiteLLM. The key must have access to the models you want to use (e.g. `openai/*` for all OpenAI models).
+
+### Via LiteLLM UI
+
+- Navigate to [http://localhost:4000/ui](http://localhost:4000/ui)
+- Go to **Virtual Keys** → **+ Create New Key**
+- Select the models you need (or `openai/*` for all OpenAI models)
+- Copy the key
+
+## 4. Add LiteLLM as a Custom Provider in Retool
+
+Inside your Retool dashboard, configure LiteLLM as a custom AI resource:
+
+<Image img={require('../../img/retool_resource_setup.gif')} />
+
+1. Go to **Resources**
+
+2. Under the **AI** category, select **Custom Provider**
+
+3. Fill in the form:
+   - **Name:** `LiteLLM`
+   - **Description:** (optional) e.g. `LiteLLM Proxy - 100+ LLMs`
+   - **Provider Schema:** `OpenAI`
+   - **Base URL:** Your ngrok-generated URL (e.g. `https://abc123.ngrok-free.app`) or deployed proxy URL—do not add `/v1` unless Retool requires it
+   - **API Key:** Your LiteLLM virtual key from Step 3
+4. **Add model names** from your LiteLLM proxy (e.g. `openai/gpt-4o-mini`, `openai/gpt-5.2-2025-12-11`).
+5. Click **Create Resource**
+
+<Image img={require('../../img/retool_llm_setup.gif')} />
+
+## 5. Test the Connection
+
+<Image img={require('../../img/retool_litellm_connection.gif')} />
+
+- Open an app in Retool and enable **Assist** (if not already enabled in your organization)
+- Use Assist to generate or edit app elements, it will route requests through LiteLLM
+- Use the code option from the Sidebar to add a resource query, select the LiteLLM resource, and run it to test the setup.
+- Check the LiteLLM **Logs** section to verify requests and track usage
+
+<Image img={require('../../img/retool_litellm_logs.gif')} />
+
+---
+
+## Troubleshooting
+
+### 401 Unauthorized
+
+- Ensure the **API Key** in Retool matches your LiteLLM virtual key exactly
+- Verify the key is not expired or blocked in LiteLLM
+
+### 401 "key not allowed to access model"
+
+Your virtual key is restricted to specific models. Generate a new key with `openai/*` or include the model you need (e.g. `openai/gpt-5.2-2025-12-11`) in the key's allowed models list.
+
+### 500 "api_key client option must be set"
+
+LiteLLM could not use your OpenAI API key to call the provider. Ensure `OPENAI_API_KEY` is set in your LiteLLM environment (e.g. in `.env` or `docker-compose.yml`) when using `openai/*` models.
+
+### localhost does not work
+
+Retool Cloud cannot reach `localhost` it points to Retool's servers. Use ngrok or deploy LiteLLM to a public URL.
+
+---
+
+## Additional Resources
+
+- [Virtual Keys](https://docs.litellm.ai/docs/proxy/virtual_keys) – Create and manage API keys
+- [Deploy LiteLLM](https://docs.litellm.ai/docs/proxy/deploy) – Production deployment options
+- [Retool Assist Documentation](https://docs.retool.com/apps/guides/assist/) – Configure Assist and prompting guides
@@ -1,5 +1,5 @@
 ---
-title: "[Preview] v1.82.0 - Realtime Guardrails, Projects Management, and 10+ Performance Optimizations"
+title: "v1.82.0 - Realtime Guardrails, Projects Management, and 10+ Performance Optimizations"
 slug: "v1-82-0"
 date: 2026-02-28T00:00:00
 authors:
@@ -26,7 +26,7 @@ import TabItem from '@theme/TabItem';
 docker run \
 -e STORE_MODEL_IN_DB=True \
 -p 4000:4000 \
-ghcr.io/berriai/litellm:main-1.82.0
+ghcr.io/berriai/litellm:main-1.82.0-stable
 ```

 </TabItem>
@@ -100,6 +100,7 @@ const sidebars = {
      label: "Policies",
      items: [
        "proxy/guardrails/guardrail_policies",
+        "proxy/guardrails/policy_flow_builder",
        "proxy/guardrails/policy_templates",
        "proxy/guardrails/policy_tags",
      ],
@@ -172,7 +173,8 @@ const sidebars = {
        "tutorials/litellm_gemini_cli",
        "tutorials/google_genai_sdk",
        "tutorials/litellm_qwen_code_cli",
-        "tutorials/openai_codex"
+        "tutorials/openai_codex",
+        "tutorials/retool_assist"
      ]
    },
    {