mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-14 02:22:54 +00:00
docs: document input/output/total tokens behaviour
Closes https://github.com/BerriAI/litellm/issues/17480
This commit is contained in:
@@ -545,6 +545,26 @@ You can set:
|
||||
- max parallel requests
|
||||
- rpm / tpm limits per model for a given key
|
||||
|
||||
### TPM Rate Limit Type (Input/Output/Total)
|
||||
|
||||
By default, TPM (tokens per minute) rate limits count **total tokens** (input + output). You can configure this to count only input tokens or only output tokens instead.
|
||||
|
||||
Set `token_rate_limit_type` in your `config.yaml`:
|
||||
|
||||
```yaml
|
||||
general_settings:
|
||||
master_key: sk-1234
|
||||
token_rate_limit_type: "output" # Options: "input", "output", "total" (default)
|
||||
```
|
||||
|
||||
| Value | Description |
|
||||
|-------|-------------|
|
||||
| `total` | Count total tokens (prompt + completion). **Default behavior.** |
|
||||
| `input` | Count only prompt/input tokens |
|
||||
| `output` | Count only completion/output tokens |
|
||||
|
||||
This setting applies globally to all TPM rate limit checks (keys, users, teams, etc.).
|
||||
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="per-team" label="Per Team">
|
||||
|
||||
Reference in New Issue
Block a user