From 17efd96e6173bd2eee3ee5cc8ee1640dc0799cb4 Mon Sep 17 00:00:00 2001 From: Sameer Kankute Date: Wed, 18 Mar 2026 17:49:03 +0530 Subject: [PATCH 1/4] docs(vertex): add concise PayGo/Priority guide with cost-tracking flow Document how to send Vertex Priority PayGo headers and explain how trafficType maps to service-tier pricing in LiteLLM, including an embedded flow diagram for quick understanding. Made-with: Cursor --- .../docs/tutorials/vertex_ai_pay_go.md | 151 ++++++++++++++++++ .../static/img/vertex_cost_tracking_flow.svg | 62 +++++++ 2 files changed, 213 insertions(+) create mode 100644 docs/my-website/docs/tutorials/vertex_ai_pay_go.md create mode 100644 docs/my-website/static/img/vertex_cost_tracking_flow.svg diff --git a/docs/my-website/docs/tutorials/vertex_ai_pay_go.md b/docs/my-website/docs/tutorials/vertex_ai_pay_go.md new file mode 100644 index 0000000000..625aff35e1 --- /dev/null +++ b/docs/my-website/docs/tutorials/vertex_ai_pay_go.md @@ -0,0 +1,151 @@ +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Vertex AI PayGo and Priority + +## Priority PayGo + +LiteLLM supports Priority PayGo. +Send a priority header, get priority queueing, and pay priority token rates. + +:::info Which models support Priority PayGo? +As of this writing: `gemini/gemini-2.5-pro`, `vertex_ai/gemini-3-pro-preview`, `vertex_ai/gemini-3.1-pro-preview`, `vertex_ai/gemini-3-flash-preview`, and their variants. +Check `supports_service_tier: true` in LiteLLM's [model pricing JSON](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json). +::: + +### Send a priority request + +Use this header: + +`X-Vertex-AI-LLM-Shared-Request-Type: priority` + + + + +```python +import litellm + +response = litellm.completion( + model="vertex_ai/gemini-3-pro-preview", + messages=[{"role": "user", "content": "Summarize the Gettysburg Address."}], + vertex_project="YOUR_PROJECT_ID", + vertex_location="us-central1", + extra_headers={"X-Vertex-AI-LLM-Shared-Request-Type": "priority"}, +) + +print(response.choices[0].message.content) +``` + + + + +```yaml title="config.yaml" +model_list: + - model_name: gemini-priority + litellm_params: + model: vertex_ai/gemini-3-pro-preview + vertex_project: "YOUR_PROJECT_ID" + vertex_location: "us-central1" + vertex_credentials: os.environ/GOOGLE_APPLICATION_CREDENTIALS + extra_headers: + X-Vertex-AI-LLM-Shared-Request-Type: priority +``` + +```bash +curl http://localhost:4000/v1/chat/completions \ + -H "Authorization: Bearer sk-your-key" \ + -H "Content-Type: application/json" \ + -d '{"model": "gemini-priority", "messages": [{"role": "user", "content": "Hello"}]}' +``` + + + + +Use `x-pass-` so LiteLLM forwards provider-specific headers. + +```bash +MODEL_ID="gemini-3-pro-preview-0325" +PROJECT_ID="YOUR_PROJECT_ID" + +curl -X POST \ + "${LITELLM_PROXY_BASE_URL}/vertex_ai/v1/projects/${PROJECT_ID}/locations/global/publishers/google/models/${MODEL_ID}:generateContent" \ + -H "Authorization: Bearer sk-your-litellm-key" \ + -H "Content-Type: application/json" \ + -H "x-pass-X-Vertex-AI-LLM-Shared-Request-Type: priority" \ + -d '{"contents": [{"role": "user", "parts": [{"text": "Hello!"}]}]}' +``` + + + + +### How cost tracking works + +![Vertex AI Priority PayGo Cost Tracking Flow](/img/vertex_cost_tracking_flow.svg) + +**`trafficType` → `service_tier` mapping** + +| `usageMetadata.trafficType` | `service_tier` | Pricing keys used | +|---|---|---| +| `ON_DEMAND` | `None` | `input_cost_per_token` | +| `ON_DEMAND_PRIORITY` | `"priority"` | `input_cost_per_token_priority` | +| `FLEX` / `BATCH` | `"flex"` | `input_cost_per_token_flex` | + +If a tier-specific key is missing, LiteLLM falls back to standard pricing keys. + +--- + +## Standard PayGo vs Provisioned Throughput + +This is a different header from priority routing: + +| Header value | Behavior | +|---|---| +| `X-Vertex-AI-LLM-Request-Type: shared` | Force standard PayGo (bypass PT) | +| `X-Vertex-AI-LLM-Request-Type: dedicated` | Force Provisioned Throughput only (`429` if exhausted) | + +### Native route example + +```python +import litellm + +response = litellm.completion( + model="vertex_ai/gemini-2.0-flash", + messages=[{"role": "user", "content": "Hello!"}], + vertex_project="YOUR_PROJECT_ID", + vertex_location="us-central1", + extra_headers={"X-Vertex-AI-LLM-Request-Type": "shared"}, +) +``` + +### Pass-through example + +```bash +MODEL_ID="gemini-2.0-flash-001" +PROJECT_ID="YOUR_PROJECT_ID" + +curl -X POST \ + "${LITELLM_PROXY_BASE_URL}/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:generateContent" \ + -H "Authorization: Bearer sk-your-litellm-key" \ + -H "Content-Type: application/json" \ + -H "x-pass-X-Vertex-AI-LLM-Request-Type: shared" \ + -d '{ + "contents": [{"role": "user", "parts": [{"text": "Hello!"}]}] + }' +``` + +--- + +## Troubleshooting + +**Q: What does `403 Permission denied` or `IAM_PERMISSION_DENIED` mean?** +A: The service account or Application Default Credentials (ADC) user does not have the `roles/aiplatform.user` role. To resolve this, re-run the `gcloud projects add-iam-policy-binding` command as shown above in the guide. + +**Q: What should I do if I get a `429 Quota exceeded` error?** +A: This means you've hit the per-region QPM (queries per minute) or TPM (tokens per minute) quota. You can: +- Request a quota increase from the [GCP Quotas console](https://console.cloud.google.com/iam-admin/quotas) +- Add more regions to your LiteLLM configuration for load balancing (see the region balancing guide above) +- Upgrade to [Provisioned Throughput](https://cloud.google.com/vertex-ai/generative-ai/docs/provisioned-throughput) for guaranteed capacity + +**Q: How do I fix the `VERTEXAI_PROJECT not set` error?** +A: Either pass the `vertex_project` parameter explicitly in your LiteLLM call, or set the `VERTEXAI_PROJECT` environment variable before running your code. + diff --git a/docs/my-website/static/img/vertex_cost_tracking_flow.svg b/docs/my-website/static/img/vertex_cost_tracking_flow.svg new file mode 100644 index 0000000000..d808dd2e36 --- /dev/null +++ b/docs/my-website/static/img/vertex_cost_tracking_flow.svg @@ -0,0 +1,62 @@ + + + + + + + + + + + HTTP request + X-Vertex-AI-LLM-Shared-Request-Type: priority + + + + + Vertex AI + + + + + Vertex response + usageMetadata.trafficType = ON_DEMAND_PRIORITY + + + + + + + + + LiteLLM stores it + _hidden_params.provider_specific_fields.traffic_type + + + + + + + + + completion_cost() + Maps traffic_type ’ service_tier = "priority" + + + + + + + + + Pricing lookup + input/output_cost_per_token_priority + + + + ` + a + b + c + d + From 8e943929a2a3802f03036437521ce73cdd2a2c50 Mon Sep 17 00:00:00 2001 From: Sameer Kankute Date: Wed, 18 Mar 2026 17:52:51 +0530 Subject: [PATCH 2/4] docs(sidebar): add vertex PayGo tutorial under Spend Tracking Made-with: Cursor --- docs/my-website/sidebars.js | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index 79a0279bad..56d1bb8b55 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -528,6 +528,7 @@ const sidebars = { label: "Spend Tracking", items: [ "proxy/cost_tracking", + "tutorials/vertex_ai_pay_go", "proxy/request_tags", "proxy/custom_pricing", "proxy/pricing_calculator", From b56fdf188ed95d7db6457caec8f3008c4f0e97f4 Mon Sep 17 00:00:00 2001 From: Sameer Kankute Date: Wed, 18 Mar 2026 17:57:27 +0530 Subject: [PATCH 3/4] Fix greptile review --- .../docs/tutorials/vertex_ai_pay_go.md | 4 +-- .../static/img/vertex_cost_tracking_flow.svg | 27 ++++++++++--------- 2 files changed, 16 insertions(+), 15 deletions(-) diff --git a/docs/my-website/docs/tutorials/vertex_ai_pay_go.md b/docs/my-website/docs/tutorials/vertex_ai_pay_go.md index 625aff35e1..b40a8b2157 100644 --- a/docs/my-website/docs/tutorials/vertex_ai_pay_go.md +++ b/docs/my-website/docs/tutorials/vertex_ai_pay_go.md @@ -124,7 +124,7 @@ MODEL_ID="gemini-2.0-flash-001" PROJECT_ID="YOUR_PROJECT_ID" curl -X POST \ - "${LITELLM_PROXY_BASE_URL}/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:generateContent" \ + "${LITELLM_PROXY_BASE_URL}/vertex_ai/v1/projects/${PROJECT_ID}/locations/global/publishers/google/models/${MODEL_ID}:generateContent" \ -H "Authorization: Bearer sk-your-litellm-key" \ -H "Content-Type: application/json" \ -H "x-pass-X-Vertex-AI-LLM-Request-Type: shared" \ @@ -143,7 +143,7 @@ A: The service account or Application Default Credentials (ADC) user does not ha **Q: What should I do if I get a `429 Quota exceeded` error?** A: This means you've hit the per-region QPM (queries per minute) or TPM (tokens per minute) quota. You can: - Request a quota increase from the [GCP Quotas console](https://console.cloud.google.com/iam-admin/quotas) -- Add more regions to your LiteLLM configuration for load balancing (see the region balancing guide above) +- Add more regions to your LiteLLM configuration for load balancing - Upgrade to [Provisioned Throughput](https://cloud.google.com/vertex-ai/generative-ai/docs/provisioned-throughput) for guaranteed capacity **Q: How do I fix the `VERTEXAI_PROJECT not set` error?** diff --git a/docs/my-website/static/img/vertex_cost_tracking_flow.svg b/docs/my-website/static/img/vertex_cost_tracking_flow.svg index d808dd2e36..d607d072dd 100644 --- a/docs/my-website/static/img/vertex_cost_tracking_flow.svg +++ b/docs/my-website/static/img/vertex_cost_tracking_flow.svg @@ -6,7 +6,7 @@ - + HTTP request X-Vertex-AI-LLM-Shared-Request-Type: priority @@ -17,7 +17,7 @@ Vertex AI - + Vertex response usageMetadata.trafficType = ON_DEMAND_PRIORITY @@ -27,7 +27,7 @@ - + LiteLLM stores it _hidden_params.provider_specific_fields.traffic_type @@ -37,26 +37,27 @@ - + completion_cost() - Maps traffic_type ’ service_tier = "priority" + Maps traffic_type → service_tier = "priority" - + Pricing lookup input/output_cost_per_token_priority - - ` - a - b - c - d - + + â‘  + â‘¡ + â‘¢ + â‘£ + ⑤ + + \ No newline at end of file From ea80a19a3970f338b398dafb6f57881f52bad30a Mon Sep 17 00:00:00 2001 From: Sameer Kankute Date: Wed, 18 Mar 2026 18:07:25 +0530 Subject: [PATCH 4/4] Fix greptile review --- docs/my-website/docs/tutorials/vertex_ai_pay_go.md | 2 +- .../static/img/vertex_cost_tracking_flow.svg | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/my-website/docs/tutorials/vertex_ai_pay_go.md b/docs/my-website/docs/tutorials/vertex_ai_pay_go.md index b40a8b2157..87197e5bad 100644 --- a/docs/my-website/docs/tutorials/vertex_ai_pay_go.md +++ b/docs/my-website/docs/tutorials/vertex_ai_pay_go.md @@ -138,7 +138,7 @@ curl -X POST \ ## Troubleshooting **Q: What does `403 Permission denied` or `IAM_PERMISSION_DENIED` mean?** -A: The service account or Application Default Credentials (ADC) user does not have the `roles/aiplatform.user` role. To resolve this, re-run the `gcloud projects add-iam-policy-binding` command as shown above in the guide. +A: The service account or Application Default Credentials (ADC) user does not have the `roles/aiplatform.user` role. To resolve this, re-run the `gcloud projects add-iam-policy-binding`. **Q: What should I do if I get a `429 Quota exceeded` error?** A: This means you've hit the per-region QPM (queries per minute) or TPM (tokens per minute) quota. You can: diff --git a/docs/my-website/static/img/vertex_cost_tracking_flow.svg b/docs/my-website/static/img/vertex_cost_tracking_flow.svg index d607d072dd..c3b2e33a07 100644 --- a/docs/my-website/static/img/vertex_cost_tracking_flow.svg +++ b/docs/my-website/static/img/vertex_cost_tracking_flow.svg @@ -6,7 +6,7 @@ - + HTTP request X-Vertex-AI-LLM-Shared-Request-Type: priority @@ -17,7 +17,7 @@ Vertex AI - + Vertex response usageMetadata.trafficType = ON_DEMAND_PRIORITY @@ -27,7 +27,7 @@ - + LiteLLM stores it _hidden_params.provider_specific_fields.traffic_type @@ -37,7 +37,7 @@ - + completion_cost() Maps traffic_type → service_tier = "priority" @@ -47,7 +47,7 @@ - + Pricing lookup input/output_cost_per_token_priority