diff --git a/docs/my-website/docs/completion/prompt_caching.md b/docs/my-website/docs/completion/prompt_caching.md index dca5f5c0cf..402c7b9f4c 100644 --- a/docs/my-website/docs/completion/prompt_caching.md +++ b/docs/my-website/docs/completion/prompt_caching.md @@ -6,6 +6,8 @@ import TabItem from '@theme/TabItem'; Supported Providers: - OpenAI (`openai/`) - Anthropic API (`anthropic/`) +- Google AI Studio (`gemini/`) +- Vertex AI (`vertex_ai/`, `vertex_ai_beta/`) - Bedrock (`bedrock/`, `bedrock/invoke/`, `bedrock/converse`) ([All models bedrock supports prompt caching on](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html)) - Deepseek API (`deepseek/`) @@ -257,7 +259,7 @@ Anthropic charges for cache writes. Specify the content to cache with `"cache_control": {"type": "ephemeral"}`. -If you pass that in for any other llm provider, it will be ignored. +This same format also works for [Gemini / Vertex AI](#google-ai-studio--vertex-ai-gemini-example). For other providers, it will be ignored. @@ -356,6 +358,208 @@ print(response.usage) +### Google AI Studio / Vertex AI (Gemini) Example + +Use the same Anthropic-style `cache_control` format — LiteLLM automatically translates it to Google's [context caching API](https://ai.google.dev/api/caching). + +**How it works under the hood:** +1. Messages with `cache_control` are separated and sent to Google's `cachedContents` API +2. The cached content ID is then passed as `cachedContent` in the Gemini request body +3. Works across all three providers: `gemini/` (Google AI Studio), `vertex_ai/`, and `vertex_ai_beta/` +4. Requires a minimum of **1024 tokens** in the cached content — below that, caching is silently skipped + + + + +```python +from litellm import completion +import os + +os.environ["GEMINI_API_KEY"] = "" + +response = completion( + model="gemini/gemini-2.5-flash", + messages=[ + { + "role": "system", + "content": [ + { + "type": "text", + "text": "You are an AI assistant tasked with analyzing legal documents.", + }, + { + "type": "text", + "text": "Here is the full text of a complex legal agreement" * 400, + "cache_control": {"type": "ephemeral"}, + }, + ], + }, + { + "role": "user", + "content": "what are the key terms and conditions in this agreement?", + }, + ], +) + +print(response.usage) +``` + + + +1. Setup config.yaml + +```yaml +model_list: + - model_name: gemini-2.5-flash + litellm_params: + model: gemini/gemini-2.5-flash + api_key: os.environ/GEMINI_API_KEY +``` + +2. Start proxy + +```bash +litellm --config /path/to/config.yaml +``` + +3. Test it! + +```python +from openai import OpenAI + +client = OpenAI( + api_key="LITELLM_PROXY_KEY", # sk-1234 + base_url="LITELLM_PROXY_BASE", # http://0.0.0.0:4000 +) + +response = client.chat.completions.create( + model="gemini-2.5-flash", + messages=[ + { + "role": "system", + "content": [ + { + "type": "text", + "text": "You are an AI assistant tasked with analyzing legal documents.", + }, + { + "type": "text", + "text": "Here is the full text of a complex legal agreement" * 400, + "cache_control": {"type": "ephemeral"}, + }, + ], + }, + { + "role": "user", + "content": "what are the key terms and conditions in this agreement?", + }, + ], +) + +print(response.usage) +``` + + + + +#### Vertex AI + +For Vertex AI, use `vertex_ai/` prefix: + + + + +```python +from litellm import completion + +response = completion( + model="vertex_ai/gemini-2.5-flash", + vertex_project="my-gcp-project", + vertex_location="us-central1", + messages=[ + { + "role": "system", + "content": [ + { + "type": "text", + "text": "You are an AI assistant tasked with analyzing legal documents.", + }, + { + "type": "text", + "text": "Here is the full text of a complex legal agreement" * 400, + "cache_control": {"type": "ephemeral"}, + }, + ], + }, + { + "role": "user", + "content": "what are the key terms and conditions in this agreement?", + }, + ], +) + +print(response.usage) +``` + + + +1. Setup config.yaml + +```yaml +model_list: + - model_name: gemini-2.5-flash + litellm_params: + model: vertex_ai/gemini-2.5-flash + vertex_project: my-gcp-project + vertex_location: us-central1 +``` + +2. Start proxy + +```bash +litellm --config /path/to/config.yaml +``` + +3. Test it! + +```python +from openai import OpenAI + +client = OpenAI( + api_key="LITELLM_PROXY_KEY", # sk-1234 + base_url="LITELLM_PROXY_BASE", # http://0.0.0.0:4000 +) + +response = client.chat.completions.create( + model="gemini-2.5-flash", + messages=[ + { + "role": "system", + "content": [ + { + "type": "text", + "text": "You are an AI assistant tasked with analyzing legal documents.", + }, + { + "type": "text", + "text": "Here is the full text of a complex legal agreement" * 400, + "cache_control": {"type": "ephemeral"}, + }, + ], + }, + { + "role": "user", + "content": "what are the key terms and conditions in this agreement?", + }, + ], +) + +print(response.usage) +``` + + + + ### Deepeek Example Works the same as OpenAI.