tiennm99/litellm
1
0
Fork 0
mirror of https://github.com/tiennm99/litellm.git synced 2026-08-05 04:22:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: reorder docs

Browse Source
This commit is contained in:
Krrish Dholakia
2025-11-10 07:33:22 -08:00
parent c088483da9
commit 50f35fca47
1 changed files with 42 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/my-website/docs/benchmarks.md
+42 -41
View File
@@ -48,47 +48,6 @@ In these tests the baseline latency characteristics are measured against a fake-
- High-percentile latencies drop significantly: P95 630 ms → 150 ms, P99 1,200 ms → 240 ms.
- Setting workers equal to CPU count gives optimal performance.
## LiteLLM vs Portkey Performance Comparison
**Test Configuration**: 4 CPUs, 8 GB RAM per instance | Load: 1k concurrent users, 500 ramp-up
### Multi-Instance (4×) Performance
| Metric | Portkey (no DB) | LiteLLM (with DB) |
| ------------------- | --------------- | ----------------- |
| **Total Requests** | 293,796 | 312,405 |
| **Failed Requests** | 0 | 0 |
| **Median Latency** | 100 ms | 100 ms |
| **p95 Latency** | 230 ms | 150 ms |
| **p99 Latency** | 500 ms | 240 ms |
| **Average Latency** | 123 ms | 111 ms |
| **Current RPS** | 1,170.9 | 1,170 |
### Technical Insights
**Portkey**
**Pros**
* Low memory footprint
* Stable latency with minimal spikes
**Cons**
* CPU utilization capped around ~40%, indicating underutilization of available compute resources
* Experienced three I/O timeout outages
**LiteLLM**
**Pros**
* Fully utilizes available CPU capacity
* Strong connection handling and low latency after initial warm-up spikes
**Cons**
* High memory usage during initialization and per request
## Machine Spec used for testing
Each machine deploying LiteLLM had the following specs:
@@ -163,6 +122,48 @@ class MyUser(HttpUser):
```
## LiteLLM vs Portkey Performance Comparison
**Test Configuration**: 4 CPUs, 8 GB RAM per instance | Load: 1k concurrent users, 500 ramp-up
### Multi-Instance (4×) Performance
| Metric | Portkey (no DB) | LiteLLM (with DB) |
| ------------------- | --------------- | ----------------- |
| **Total Requests** | 293,796 | 312,405 |
| **Failed Requests** | 0 | 0 |
| **Median Latency** | 100 ms | 100 ms |
| **p95 Latency** | 230 ms | 150 ms |
| **p99 Latency** | 500 ms | 240 ms |
| **Average Latency** | 123 ms | 111 ms |
| **Current RPS** | 1,170.9 | 1,170 |
### Technical Insights
**Portkey**
**Pros**
* Low memory footprint
* Stable latency with minimal spikes
**Cons**
* CPU utilization capped around ~40%, indicating underutilization of available compute resources
* Experienced three I/O timeout outages
**LiteLLM**
**Pros**
* Fully utilizes available CPU capacity
* Strong connection handling and low latency after initial warm-up spikes
**Cons**
* High memory usage during initialization and per request
## Logging Callbacks