diff --git a/docs/my-website/docs/proxy/guardrails/pii_masking_v2.md b/docs/my-website/docs/proxy/guardrails/pii_masking_v2.md index 1d1efd4e46..abea90f617 100644 --- a/docs/my-website/docs/proxy/guardrails/pii_masking_v2.md +++ b/docs/my-website/docs/proxy/guardrails/pii_masking_v2.md @@ -2,15 +2,56 @@ import Image from '@theme/IdealImage'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# PII Masking - Presidio +# PII, PHI Masking - Presidio + +## Overview + +| Property | Details | +|-------|-------| +| Description | Use this guardrail to mask PII (Personally Identifiable Information), PHI (Protected Health Information), and other sensitive data. | +| Provider | [Microsoft Presidio](https://github.com/microsoft/presidio/) | +| Supported Entity Types | All Presidio Entity Types | +| Supported Actions | `MASK`, `BLOCK` | +| Supported Modes | `pre_call`, `during_call`, `post_call`, `logging_only` | + +## Deployment options + +For this guardrail you need a deployed Presidio Analyzer and Presido Anonymizer containers. + +| Deployment Option | Details | +|------------------|----------| +| Deploy Presidio Docker Containers | - [Presidio Analyzer Docker Container](https://hub.docker.com/r/microsoft/presidio-analyzer)
- [Presidio Anonymizer Docker Container](https://hub.docker.com/r/microsoft/presidio-anonymizer) | ## Quick Start -LiteLLM supports [Microsoft Presidio](https://github.com/microsoft/presidio/) for PII masking. + + -### 1. Define Guardrails on your LiteLLM config.yaml +### 1. Create a PII, PHI Masking Guardrail + +On the LiteLLM UI, navigate to Guardrails. Click "Add Guardrail". On this dropdown select "Presidio PII" and enter your presidio analyzer and anonymizer endpoints. + + + +#### 1.2 Configure Entity Types + +Now select the entity types you want to mask. See the [supported actions here](#supported-actions) + + + + + + + Define your guardrails under the `guardrails` section + ```yaml title="config.yaml" showLineNumbers model_list: - model_name: gpt-3.5-turbo @@ -19,7 +60,7 @@ model_list: api_key: os.environ/OPENAI_API_KEY guardrails: - - guardrail_name: "presidio-pre-guard" + - guardrail_name: "presidio-pii" litellm_params: guardrail: presidio # supported values: "aporia", "bedrock", "lakera", "presidio" mode: "pre_call" @@ -38,15 +79,35 @@ export PRESIDIO_ANONYMIZER_API_BASE="http://localhost:5001" - `post_call` Run **after** LLM call, on **input & output** - `logging_only` Run **after** LLM call, only apply PII Masking before logging to Langfuse, etc. Not on the actual llm api request / response. - ### 2. Start LiteLLM Gateway - ```shell title="Start Gateway" showLineNumbers litellm --config config.yaml --detailed_debug ``` -### 3. Test request + + + + +### 3. Test it! + +#### 3.1 LiteLLM UI + +On the litellm UI, navigate to the 'Test Keys' page, select the guardrail you created and send the following messaged filled with PII data. + +```text title="PII Request" showLineNumbers +My credit card is 4111-1111-1111-1111 and my email is test@example.com. +``` + + + + +#### 3.2 Test in code + +In order to apply a guardrail for a request send `guardrails=["presidio-pii"]` in the request body. **[Langchain, OpenAI SDK Usage Examples](../proxy/user_keys#request-format)** @@ -64,7 +125,7 @@ curl http://localhost:4000/chat/completions \ "messages": [ {"role": "user", "content": "Hello my name is Jane Doe"} ], - "guardrails": ["presidio-pre-guard"], + "guardrails": ["presidio-pii"], }' ``` @@ -111,15 +172,36 @@ curl http://localhost:4000/chat/completions \ "messages": [ {"role": "user", "content": "Hello good morning"} ], - "guardrails": ["presidio-pre-guard"], + "guardrails": ["presidio-pii"], }' ``` - - + +## Tracing Guardrail requests + +Once your guardrail is live in production, you will also be able to trace your guardrail on LiteLLM Logs, Langfuse, Arize Phoenix, etc, all LiteLLM logging integrations. + +### LiteLLM UI + +On the LiteLLM logs page you can see that the PII content was masked for this specific request. And you can see detailed tracing for the guardrail. This allows you to monitor entity types masked with their corresponding confidence score and the duration of the guardrail execution. + + + +### Langfuse + +When connecting Litellm to Langfuse, you can see the guardrail information on the Langfuse Trace. + + + ## Entity Type Configuration You can configure specific entity types for PII detection and decide how to handle each entity type (mask or block). diff --git a/docs/my-website/docs/proxy/pii_masking.md b/docs/my-website/docs/proxy/pii_masking.md deleted file mode 100644 index 83e4965a49..0000000000 --- a/docs/my-website/docs/proxy/pii_masking.md +++ /dev/null @@ -1,246 +0,0 @@ -import Image from '@theme/IdealImage'; -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - -# PII Masking - LiteLLM Gateway (Deprecated Version) - -:::warning - -This is deprecated, please use [our new Presidio pii masking integration](./guardrails/pii_masking_v2) - -::: - -LiteLLM supports [Microsoft Presidio](https://github.com/microsoft/presidio/) for PII masking. - - -## Quick Start -### Step 1. Add env - -```bash -export PRESIDIO_ANALYZER_API_BASE="http://localhost:5002" -export PRESIDIO_ANONYMIZER_API_BASE="http://localhost:5001" -``` - -### Step 2. Set it as a callback in config.yaml - -```yaml -litellm_settings: - callbacks = ["presidio", ...] # e.g. ["presidio", custom_callbacks.proxy_handler_instance] -``` - -### Step 3. Start proxy - - -``` -litellm --config /path/to/config.yaml -``` - - -This will mask the input going to the llm provider - - - -## Output parsing - -LLM responses can sometimes contain the masked tokens. - -For presidio 'replace' operations, LiteLLM can check the LLM response and replace the masked token with the user-submitted values. - -Just set `litellm.output_parse_pii = True`, to enable this. - - -```yaml -litellm_settings: - output_parse_pii: true -``` - -**Expected Flow: ** - -1. User Input: "hello world, my name is Jane Doe. My number is: 034453334" - -2. LLM Input: "hello world, my name is [PERSON]. My number is: [PHONE_NUMBER]" - -3. LLM Response: "Hey [PERSON], nice to meet you!" - -4. User Response: "Hey Jane Doe, nice to meet you!" - -## Ad-hoc recognizers - -Send ad-hoc recognizers to presidio `/analyze` by passing a json file to the proxy - -[**Example** ad-hoc recognizer](../../../../litellm/proxy/hooks/example_presidio_ad_hoc_recognizer.json) - -```yaml -litellm_settings: - callbacks: ["presidio"] - presidio_ad_hoc_recognizers: "./hooks/example_presidio_ad_hoc_recognizer.json" -``` - -You can see this working, when you run the proxy: - -```bash -litellm --config /path/to/config.yaml --debug -``` - -Make a chat completions request, example: - -``` -{ - "model": "azure-gpt-3.5", - "messages": [{"role": "user", "content": "John Smith AHV number is 756.3026.0705.92. Zip code: 1334023"}] -} -``` - -And search for any log starting with `Presidio PII Masking`, example: -``` -Presidio PII Masking: Redacted pii message: AHV number is . Zip code: -``` - - -## Turn on/off per key - -Turn off PII masking for a given key. - -Do this by setting `permissions: {"pii": false}`, when generating a key. - -```shell -curl --location 'http://0.0.0.0:4000/key/generate' \ ---header 'Authorization: Bearer sk-1234' \ ---header 'Content-Type: application/json' \ ---data '{ - "permissions": {"pii": false} -}' -``` - - -## Turn on/off per request - -The proxy support 2 request-level PII controls: - -- *no-pii*: Optional(bool) - Allow user to turn off pii masking per request. -- *output_parse_pii*: Optional(bool) - Allow user to turn off pii output parsing per request. - -### Usage - -**Step 1. Create key with pii permissions** - -Set `allow_pii_controls` to true for a given key. This will allow the user to set request-level PII controls. - -```bash -curl --location 'http://0.0.0.0:4000/key/generate' \ ---header 'Authorization: Bearer my-master-key' \ ---header 'Content-Type: application/json' \ ---data '{ - "permissions": {"allow_pii_controls": true} -}' -``` - -**Step 2. Turn off pii output parsing** - -```python -import os -from openai import OpenAI - -client = OpenAI( - # This is the default and can be omitted - api_key=os.environ.get("OPENAI_API_KEY"), - base_url="http://0.0.0.0:4000" -) - -chat_completion = client.chat.completions.create( - messages=[ - { - "role": "user", - "content": "My name is Jane Doe, my number is 8382043839", - } - ], - model="gpt-3.5-turbo", - extra_body={ - "content_safety": {"output_parse_pii": False} - } -) -``` - -**Step 3: See response** - -``` -{ - "id": "chatcmpl-8c5qbGTILZa1S4CK3b31yj5N40hFN", - "choices": [ - { - "finish_reason": "stop", - "index": 0, - "message": { - "content": "Hi [PERSON], what can I help you with?", - "role": "assistant" - } - } - ], - "created": 1704089632, - "model": "gpt-35-turbo", - "object": "chat.completion", - "system_fingerprint": null, - "usage": { - "completion_tokens": 47, - "prompt_tokens": 12, - "total_tokens": 59 - }, - "_response_ms": 1753.426 -} -``` - - -## Turn on for logging only - -Only apply PII Masking before logging to Langfuse, etc. - -Not on the actual llm api request / response. - -:::note -This is currently only applied for -- `/chat/completion` requests -- on 'success' logging - -::: - -1. Setup config.yaml -```yaml -litellm_settings: - presidio_logging_only: true - -model_list: - - model_name: gpt-3.5-turbo - litellm_params: - model: gpt-3.5-turbo - api_key: os.environ/OPENAI_API_KEY -``` - -2. Start proxy - -```bash -litellm --config /path/to/config.yaml -``` - -3. Test it! - -```bash -curl -X POST 'http://0.0.0.0:4000/chat/completions' \ --H 'Content-Type: application/json' \ --H 'Authorization: Bearer sk-1234' \ --D '{ - "model": "gpt-3.5-turbo", - "messages": [ - { - "role": "user", - "content": "Hi, my name is Jane!" - } - ] - }' -``` - - -**Expected Logged Response** - -``` -Hi, my name is ! -``` \ No newline at end of file diff --git a/docs/my-website/img/presidio_1.png b/docs/my-website/img/presidio_1.png new file mode 100644 index 0000000000..6cc13cfacf Binary files /dev/null and b/docs/my-website/img/presidio_1.png differ diff --git a/docs/my-website/img/presidio_2.png b/docs/my-website/img/presidio_2.png new file mode 100644 index 0000000000..2bdab8821b Binary files /dev/null and b/docs/my-website/img/presidio_2.png differ diff --git a/docs/my-website/img/presidio_3.png b/docs/my-website/img/presidio_3.png new file mode 100644 index 0000000000..7e6e0039d3 Binary files /dev/null and b/docs/my-website/img/presidio_3.png differ diff --git a/docs/my-website/img/presidio_4.png b/docs/my-website/img/presidio_4.png new file mode 100644 index 0000000000..b7732ba0fe Binary files /dev/null and b/docs/my-website/img/presidio_4.png differ diff --git a/docs/my-website/img/presidio_5.png b/docs/my-website/img/presidio_5.png new file mode 100644 index 0000000000..a0d903f8ed Binary files /dev/null and b/docs/my-website/img/presidio_5.png differ diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index 87dfa77020..edda8a5acb 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -574,7 +574,6 @@ const sidebars = { "projects/GPTLocalhost", ], }, - "proxy/pii_masking", "extras/code_quality", "rules", "proxy/team_based_routing",