From 43fc152a91bb57950a0ba442160352ba2bcc9433 Mon Sep 17 00:00:00 2001 From: ishaan-jaff Date: Thu, 14 Dec 2023 16:33:56 +0530 Subject: [PATCH] (docs) caching - proxy --- docs/my-website/docs/proxy/caching.md | 60 +++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 7 deletions(-) diff --git a/docs/my-website/docs/proxy/caching.md b/docs/my-website/docs/proxy/caching.md index 37d3658e76..395250f6c9 100644 --- a/docs/my-website/docs/proxy/caching.md +++ b/docs/my-website/docs/proxy/caching.md @@ -1,20 +1,24 @@ # Caching Cache LLM Responses +## Quick Start Caching can be enabled by adding the `cache` key in the `config.yaml` -#### Step 1: Add `cache` to the config.yaml +### Step 1: Add `cache` to the config.yaml ```yaml model_list: - model_name: gpt-3.5-turbo litellm_params: model: gpt-3.5-turbo + - model_name: text-embedding-ada-002 + litellm_params: + model: text-embedding-ada-002 litellm_settings: set_verbose: True cache: True # set cache responses to True, litellm defaults to using a redis cache ``` -#### Step 2: Add Redis Credentials to .env +### Step 2: Add Redis Credentials to .env Set either `REDIS_URL` or the `REDIS_HOST` in your os environment, to enable caching. ```shell @@ -32,12 +36,12 @@ REDIS_ = "" ``` [**See how it's read from the environment**](https://github.com/BerriAI/litellm/blob/4d7ff1b33b9991dcf38d821266290631d9bcd2dd/litellm/_redis.py#L40) -#### Step 3: Run proxy with config +### Step 3: Run proxy with config ```shell $ litellm --config /path/to/config.yaml ``` -#### Using Caching +## Using Caching - /chat/completions Send the same request twice: ```shell curl http://0.0.0.0:8000/v1/chat/completions \ @@ -57,9 +61,27 @@ curl http://0.0.0.0:8000/v1/chat/completions \ }' ``` -#### Control caching per completion request +## Using Caching - /embeddings +Send the same request twice: +```shell +curl --location 'http://0.0.0.0:8000/embeddings' \ + --header 'Content-Type: application/json' \ + --data ' { + "model": "text-embedding-ada-002", + "input": ["write a litellm poem"] + }' + +curl --location 'http://0.0.0.0:8000/embeddings' \ + --header 'Content-Type: application/json' \ + --data ' { + "model": "text-embedding-ada-002", + "input": ["write a litellm poem"] + }' +``` + +## Override caching per `chat/completions` request Caching can be switched on/off per `/chat/completions` request -- Caching **on** for completion - pass `caching=True`: +- Caching **on** for individual completion - pass `caching=True`: ```shell curl http://0.0.0.0:8000/v1/chat/completions \ -H "Content-Type: application/json" \ @@ -70,7 +92,7 @@ Caching can be switched on/off per `/chat/completions` request "caching": true }' ``` -- Caching **off** for completion - pass `caching=False`: +- Caching **off** for individual completion - pass `caching=False`: ```shell curl http://0.0.0.0:8000/v1/chat/completions \ -H "Content-Type: application/json" \ @@ -80,4 +102,28 @@ Caching can be switched on/off per `/chat/completions` request "temperature": 0.7, "caching": false }' + ``` + + +## Override caching per `/embeddings` request +Caching can be switched on/off per `/embeddings` request +- Caching **on** for embedding - pass `caching=True`: + ```shell + curl --location 'http://0.0.0.0:8000/embeddings' \ + --header 'Content-Type: application/json' \ + --data ' { + "model": "text-embedding-ada-002", + "input": ["write a litellm poem"], + "caching": true + }' + ``` +- Caching **off** for completion - pass `caching=False`: + ```shell + curl --location 'http://0.0.0.0:8000/embeddings' \ + --header 'Content-Type: application/json' \ + --data ' { + "model": "text-embedding-ada-002", + "input": ["write a litellm poem"], + "caching": false + }' ``` \ No newline at end of file