From f95f5563ea5501cad2ffff6ee32f8883ff0f76c5 Mon Sep 17 00:00:00 2001 From: Krrish Dholakia Date: Tue, 20 Jan 2026 10:44:49 -0800 Subject: [PATCH] docs: document input/output/total tokens behaviour Closes https://github.com/BerriAI/litellm/issues/17480 --- docs/my-website/docs/proxy/users.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/my-website/docs/proxy/users.md b/docs/my-website/docs/proxy/users.md index 3e0e00dfa5..a389f0bd44 100644 --- a/docs/my-website/docs/proxy/users.md +++ b/docs/my-website/docs/proxy/users.md @@ -545,6 +545,26 @@ You can set: - max parallel requests - rpm / tpm limits per model for a given key +### TPM Rate Limit Type (Input/Output/Total) + +By default, TPM (tokens per minute) rate limits count **total tokens** (input + output). You can configure this to count only input tokens or only output tokens instead. + +Set `token_rate_limit_type` in your `config.yaml`: + +```yaml +general_settings: + master_key: sk-1234 + token_rate_limit_type: "output" # Options: "input", "output", "total" (default) +``` + +| Value | Description | +|-------|-------------| +| `total` | Count total tokens (prompt + completion). **Default behavior.** | +| `input` | Count only prompt/input tokens | +| `output` | Count only completion/output tokens | + +This setting applies globally to all TPM rate limit checks (keys, users, teams, etc.). +