mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-02 04:21:34 +00:00
[Docs] Pii masking with LiteLLM (#10924)
* doc pii masking * remove old doc * docs add deployment options * docs guardrail pii masking * docs pii masking / tracing * add images of tracing guards litellm ui * docs presidio pii
This commit is contained in:
Ishaan Jaff
@@ -2,15 +2,56 @@ import Image from '@theme/IdealImage';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# PII Masking - Presidio
|
||||
# PII, PHI Masking - Presidio
|
||||
|
||||
## Overview
|
||||
|
||||
| Property | Details |
|
||||
|-------|-------|
|
||||
| Description | Use this guardrail to mask PII (Personally Identifiable Information), PHI (Protected Health Information), and other sensitive data. |
|
||||
| Provider | [Microsoft Presidio](https://github.com/microsoft/presidio/) |
|
||||
| Supported Entity Types | All Presidio Entity Types |
|
||||
| Supported Actions | `MASK`, `BLOCK` |
|
||||
| Supported Modes | `pre_call`, `during_call`, `post_call`, `logging_only` |
|
||||
|
||||
## Deployment options
|
||||
|
||||
For this guardrail you need a deployed Presidio Analyzer and Presido Anonymizer containers.
|
||||
|
||||
| Deployment Option | Details |
|
||||
|------------------|----------|
|
||||
| Deploy Presidio Docker Containers | - [Presidio Analyzer Docker Container](https://hub.docker.com/r/microsoft/presidio-analyzer)<br/>- [Presidio Anonymizer Docker Container](https://hub.docker.com/r/microsoft/presidio-anonymizer) |
|
||||
|
||||
## Quick Start
|
||||
|
||||
LiteLLM supports [Microsoft Presidio](https://github.com/microsoft/presidio/) for PII masking.
|
||||
<Tabs>
|
||||
<TabItem value="ui" label="LiteLLM UI">
|
||||
|
||||
### 1. Define Guardrails on your LiteLLM config.yaml
|
||||
### 1. Create a PII, PHI Masking Guardrail
|
||||
|
||||
On the LiteLLM UI, navigate to Guardrails. Click "Add Guardrail". On this dropdown select "Presidio PII" and enter your presidio analyzer and anonymizer endpoints.
|
||||
|
||||
<Image
|
||||
img={require('../../../img/presidio_1.png')}
|
||||
style={{width: '80%', display: 'block', margin: '0'}}
|
||||
/>
|
||||
|
||||
#### 1.2 Configure Entity Types
|
||||
|
||||
Now select the entity types you want to mask. See the [supported actions here](#supported-actions)
|
||||
|
||||
<Image
|
||||
img={require('../../../img/presidio_2.png')}
|
||||
style={{width: '50%', display: 'block', margin: '0'}}
|
||||
/>
|
||||
|
||||
</TabItem>
|
||||
|
||||
|
||||
<TabItem value="config" label="Config.yaml">
|
||||
|
||||
Define your guardrails under the `guardrails` section
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
model_list:
|
||||
- model_name: gpt-3.5-turbo
|
||||
@@ -19,7 +60,7 @@ model_list:
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
|
||||
guardrails:
|
||||
- guardrail_name: "presidio-pre-guard"
|
||||
- guardrail_name: "presidio-pii"
|
||||
litellm_params:
|
||||
guardrail: presidio # supported values: "aporia", "bedrock", "lakera", "presidio"
|
||||
mode: "pre_call"
|
||||
@@ -38,15 +79,35 @@ export PRESIDIO_ANONYMIZER_API_BASE="http://localhost:5001"
|
||||
- `post_call` Run **after** LLM call, on **input & output**
|
||||
- `logging_only` Run **after** LLM call, only apply PII Masking before logging to Langfuse, etc. Not on the actual llm api request / response.
|
||||
|
||||
|
||||
### 2. Start LiteLLM Gateway
|
||||
|
||||
|
||||
```shell title="Start Gateway" showLineNumbers
|
||||
litellm --config config.yaml --detailed_debug
|
||||
```
|
||||
|
||||
### 3. Test request
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
### 3. Test it!
|
||||
|
||||
#### 3.1 LiteLLM UI
|
||||
|
||||
On the litellm UI, navigate to the 'Test Keys' page, select the guardrail you created and send the following messaged filled with PII data.
|
||||
|
||||
```text title="PII Request" showLineNumbers
|
||||
My credit card is 4111-1111-1111-1111 and my email is test@example.com.
|
||||
```
|
||||
|
||||
<Image
|
||||
img={require('../../../img/presidio_3.png')}
|
||||
style={{width: '100%', display: 'block', margin: '0'}}
|
||||
/>
|
||||
|
||||
|
||||
#### 3.2 Test in code
|
||||
|
||||
In order to apply a guardrail for a request send `guardrails=["presidio-pii"]` in the request body.
|
||||
|
||||
**[Langchain, OpenAI SDK Usage Examples](../proxy/user_keys#request-format)**
|
||||
|
||||
@@ -64,7 +125,7 @@ curl http://localhost:4000/chat/completions \
|
||||
"messages": [
|
||||
{"role": "user", "content": "Hello my name is Jane Doe"}
|
||||
],
|
||||
"guardrails": ["presidio-pre-guard"],
|
||||
"guardrails": ["presidio-pii"],
|
||||
}'
|
||||
```
|
||||
|
||||
@@ -111,15 +172,36 @@ curl http://localhost:4000/chat/completions \
|
||||
"messages": [
|
||||
{"role": "user", "content": "Hello good morning"}
|
||||
],
|
||||
"guardrails": ["presidio-pre-guard"],
|
||||
"guardrails": ["presidio-pii"],
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
|
||||
</Tabs>
|
||||
|
||||
|
||||
## Tracing Guardrail requests
|
||||
|
||||
Once your guardrail is live in production, you will also be able to trace your guardrail on LiteLLM Logs, Langfuse, Arize Phoenix, etc, all LiteLLM logging integrations.
|
||||
|
||||
### LiteLLM UI
|
||||
|
||||
On the LiteLLM logs page you can see that the PII content was masked for this specific request. And you can see detailed tracing for the guardrail. This allows you to monitor entity types masked with their corresponding confidence score and the duration of the guardrail execution.
|
||||
|
||||
<Image
|
||||
img={require('../../../img/presidio_4.png')}
|
||||
style={{width: '60%', display: 'block', margin: '0'}}
|
||||
/>
|
||||
|
||||
### Langfuse
|
||||
|
||||
When connecting Litellm to Langfuse, you can see the guardrail information on the Langfuse Trace.
|
||||
|
||||
<Image
|
||||
img={require('../../../img/presidio_5.png')}
|
||||
style={{width: '60%', display: 'block', margin: '0'}}
|
||||
/>
|
||||
|
||||
## Entity Type Configuration
|
||||
|
||||
You can configure specific entity types for PII detection and decide how to handle each entity type (mask or block).
|
||||
|
||||
@@ -1,246 +0,0 @@
|
||||
import Image from '@theme/IdealImage';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# PII Masking - LiteLLM Gateway (Deprecated Version)
|
||||
|
||||
:::warning
|
||||
|
||||
This is deprecated, please use [our new Presidio pii masking integration](./guardrails/pii_masking_v2)
|
||||
|
||||
:::
|
||||
|
||||
LiteLLM supports [Microsoft Presidio](https://github.com/microsoft/presidio/) for PII masking.
|
||||
|
||||
|
||||
## Quick Start
|
||||
### Step 1. Add env
|
||||
|
||||
```bash
|
||||
export PRESIDIO_ANALYZER_API_BASE="http://localhost:5002"
|
||||
export PRESIDIO_ANONYMIZER_API_BASE="http://localhost:5001"
|
||||
```
|
||||
|
||||
### Step 2. Set it as a callback in config.yaml
|
||||
|
||||
```yaml
|
||||
litellm_settings:
|
||||
callbacks = ["presidio", ...] # e.g. ["presidio", custom_callbacks.proxy_handler_instance]
|
||||
```
|
||||
|
||||
### Step 3. Start proxy
|
||||
|
||||
|
||||
```
|
||||
litellm --config /path/to/config.yaml
|
||||
```
|
||||
|
||||
|
||||
This will mask the input going to the llm provider
|
||||
|
||||
<Image img={require('../../img/presidio_screenshot.png')} />
|
||||
|
||||
## Output parsing
|
||||
|
||||
LLM responses can sometimes contain the masked tokens.
|
||||
|
||||
For presidio 'replace' operations, LiteLLM can check the LLM response and replace the masked token with the user-submitted values.
|
||||
|
||||
Just set `litellm.output_parse_pii = True`, to enable this.
|
||||
|
||||
|
||||
```yaml
|
||||
litellm_settings:
|
||||
output_parse_pii: true
|
||||
```
|
||||
|
||||
**Expected Flow: **
|
||||
|
||||
1. User Input: "hello world, my name is Jane Doe. My number is: 034453334"
|
||||
|
||||
2. LLM Input: "hello world, my name is [PERSON]. My number is: [PHONE_NUMBER]"
|
||||
|
||||
3. LLM Response: "Hey [PERSON], nice to meet you!"
|
||||
|
||||
4. User Response: "Hey Jane Doe, nice to meet you!"
|
||||
|
||||
## Ad-hoc recognizers
|
||||
|
||||
Send ad-hoc recognizers to presidio `/analyze` by passing a json file to the proxy
|
||||
|
||||
[**Example** ad-hoc recognizer](../../../../litellm/proxy/hooks/example_presidio_ad_hoc_recognizer.json)
|
||||
|
||||
```yaml
|
||||
litellm_settings:
|
||||
callbacks: ["presidio"]
|
||||
presidio_ad_hoc_recognizers: "./hooks/example_presidio_ad_hoc_recognizer.json"
|
||||
```
|
||||
|
||||
You can see this working, when you run the proxy:
|
||||
|
||||
```bash
|
||||
litellm --config /path/to/config.yaml --debug
|
||||
```
|
||||
|
||||
Make a chat completions request, example:
|
||||
|
||||
```
|
||||
{
|
||||
"model": "azure-gpt-3.5",
|
||||
"messages": [{"role": "user", "content": "John Smith AHV number is 756.3026.0705.92. Zip code: 1334023"}]
|
||||
}
|
||||
```
|
||||
|
||||
And search for any log starting with `Presidio PII Masking`, example:
|
||||
```
|
||||
Presidio PII Masking: Redacted pii message: <PERSON> AHV number is <AHV_NUMBER>. Zip code: <US_DRIVER_LICENSE>
|
||||
```
|
||||
|
||||
|
||||
## Turn on/off per key
|
||||
|
||||
Turn off PII masking for a given key.
|
||||
|
||||
Do this by setting `permissions: {"pii": false}`, when generating a key.
|
||||
|
||||
```shell
|
||||
curl --location 'http://0.0.0.0:4000/key/generate' \
|
||||
--header 'Authorization: Bearer sk-1234' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"permissions": {"pii": false}
|
||||
}'
|
||||
```
|
||||
|
||||
|
||||
## Turn on/off per request
|
||||
|
||||
The proxy support 2 request-level PII controls:
|
||||
|
||||
- *no-pii*: Optional(bool) - Allow user to turn off pii masking per request.
|
||||
- *output_parse_pii*: Optional(bool) - Allow user to turn off pii output parsing per request.
|
||||
|
||||
### Usage
|
||||
|
||||
**Step 1. Create key with pii permissions**
|
||||
|
||||
Set `allow_pii_controls` to true for a given key. This will allow the user to set request-level PII controls.
|
||||
|
||||
```bash
|
||||
curl --location 'http://0.0.0.0:4000/key/generate' \
|
||||
--header 'Authorization: Bearer my-master-key' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"permissions": {"allow_pii_controls": true}
|
||||
}'
|
||||
```
|
||||
|
||||
**Step 2. Turn off pii output parsing**
|
||||
|
||||
```python
|
||||
import os
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
# This is the default and can be omitted
|
||||
api_key=os.environ.get("OPENAI_API_KEY"),
|
||||
base_url="http://0.0.0.0:4000"
|
||||
)
|
||||
|
||||
chat_completion = client.chat.completions.create(
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "My name is Jane Doe, my number is 8382043839",
|
||||
}
|
||||
],
|
||||
model="gpt-3.5-turbo",
|
||||
extra_body={
|
||||
"content_safety": {"output_parse_pii": False}
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
**Step 3: See response**
|
||||
|
||||
```
|
||||
{
|
||||
"id": "chatcmpl-8c5qbGTILZa1S4CK3b31yj5N40hFN",
|
||||
"choices": [
|
||||
{
|
||||
"finish_reason": "stop",
|
||||
"index": 0,
|
||||
"message": {
|
||||
"content": "Hi [PERSON], what can I help you with?",
|
||||
"role": "assistant"
|
||||
}
|
||||
}
|
||||
],
|
||||
"created": 1704089632,
|
||||
"model": "gpt-35-turbo",
|
||||
"object": "chat.completion",
|
||||
"system_fingerprint": null,
|
||||
"usage": {
|
||||
"completion_tokens": 47,
|
||||
"prompt_tokens": 12,
|
||||
"total_tokens": 59
|
||||
},
|
||||
"_response_ms": 1753.426
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## Turn on for logging only
|
||||
|
||||
Only apply PII Masking before logging to Langfuse, etc.
|
||||
|
||||
Not on the actual llm api request / response.
|
||||
|
||||
:::note
|
||||
This is currently only applied for
|
||||
- `/chat/completion` requests
|
||||
- on 'success' logging
|
||||
|
||||
:::
|
||||
|
||||
1. Setup config.yaml
|
||||
```yaml
|
||||
litellm_settings:
|
||||
presidio_logging_only: true
|
||||
|
||||
model_list:
|
||||
- model_name: gpt-3.5-turbo
|
||||
litellm_params:
|
||||
model: gpt-3.5-turbo
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
```
|
||||
|
||||
2. Start proxy
|
||||
|
||||
```bash
|
||||
litellm --config /path/to/config.yaml
|
||||
```
|
||||
|
||||
3. Test it!
|
||||
|
||||
```bash
|
||||
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-D '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Hi, my name is Jane!"
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
|
||||
**Expected Logged Response**
|
||||
|
||||
```
|
||||
Hi, my name is <PERSON>!
|
||||
```
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 198 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 141 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 178 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 159 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 203 KiB |
@@ -574,7 +574,6 @@ const sidebars = {
|
||||
"projects/GPTLocalhost",
|
||||
],
|
||||
},
|
||||
"proxy/pii_masking",
|
||||
"extras/code_quality",
|
||||
"rules",
|
||||
"proxy/team_based_routing",
|
||||
|
||||
Reference in New Issue
Block a user