tiennm99/litellm
1
0
Fork 0
mirror of https://github.com/tiennm99/litellm.git synced 2026-08-06 00:23:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add infrastructure recommendations to benchmarks documentation (#18264)

Browse Source 
Added concise PostgreSQL and Redis specifications based on benchmark results and industry standards for API gateway deployments. Includes tiered recommendations for different RPS workloads, configuration best practices, and scaling guidelines.
This commit is contained in:
Alexsander Hamir
2025-12-19 13:44:36 -08:00
committed by GitHub
parent f3b97356fb
commit 5b1fda02fb
1 changed files with 52 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/my-website/docs/benchmarks.md
+52
View File
@@ -60,6 +60,58 @@ Each machine deploying LiteLLM had the following specs:
- Database: PostgreSQL
- Redis: Not used
## Infrastructure Recommendations
Recommended specifications based on benchmark results and industry standards for API gateway deployments.
### PostgreSQL
Required for authentication, key management, and usage tracking.
| Workload | CPU | RAM | Storage | Connections |
|----------|-----|-----|---------|-------------|
| 1-2K RPS | 4-8 cores | 16GB | 200GB SSD (3000+ IOPS) | 100-200 |
| 2-5K RPS | 8 cores | 16-32GB | 500GB SSD (5000+ IOPS) | 200-500 |
| 5K+ RPS | 16+ cores | 32-64GB | 1TB+ SSD (10000+ IOPS) | 500+ |
**Configuration:** Set `proxy_batch_write_at: 60` to batch writes and reduce DB load. Total connections = pool limit × instances.
### Redis (Recommended)
Redis was not used in these benchmarks but provides significant production benefits: 60-80% reduced DB load.
| Workload | CPU | RAM |
|----------|-----|-----|
| 1-2K RPS | 2-4 cores | 8GB |
| 2-5K RPS | 4 cores | 16GB |
| 5K+ RPS | 8+ cores | 32GB+ |
**Requirements:** Redis 7.0+, AOF persistence enabled, `allkeys-lru` eviction policy.
**Configuration:**
```yaml
router_settings:
redis_host: os.environ/REDIS_HOST
redis_port: os.environ/REDIS_PORT
redis_password: os.environ/REDIS_PASSWORD
litellm_settings:
cache: True
cache_params:
type: redis
host: os.environ/REDIS_HOST
port: os.environ/REDIS_PORT
password: os.environ/REDIS_PASSWORD
```
:::tip
Use `redis_host`, `redis_port`, and `redis_password` instead of `redis_url` for ~80 RPS better performance.
:::
**Scaling:** DB connections scale linearly with instances. Consider PostgreSQL read replicas beyond 5K RPS.
See [Production Configuration](./proxy/prod) for detailed best practices.
## Locust Settings
- 1000 Users