diff --git a/docs/my-website/docs/benchmarks.md b/docs/my-website/docs/benchmarks.md index 43ab82b8e6..be175a9212 100644 --- a/docs/my-website/docs/benchmarks.md +++ b/docs/my-website/docs/benchmarks.md @@ -16,19 +16,17 @@ model_list: api_key: "test" ``` -### 1 Instance LiteLLM Proxy +### 2 Instance LiteLLM Proxy In these tests the baseline latency characteristics are measured against a fake-openai-endpoint. #### Performance Metrics -| Metric | Value | -|--------|-------| -| **Requests per Second (RPS)** | 475 | -| **End-to-End Latency P50 (ms)** | 100 | -| **LiteLLM Overhead P50 (ms)** | 3 | -| **LiteLLM Overhead P90 (ms)** | 17 | -| **LiteLLM Overhead P99 (ms)** | 31 | +| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** | +| --- | --- | --- | --- | --- | --- | --- | +| POST | /chat/completions | 200 | 630 | 1200 | 262.46 | 1035.7 | +| Custom | LiteLLM Overhead Duration (ms) | 12 | 29 | 43 | 14.74 | 1035.7 | +| | Aggregated | 100 | 430 | 930 | 138.6 | 2071.4 | @@ -36,28 +34,32 @@ In these tests the baseline latency characteristics are measured against a fake- --> + +### 4 Instances + +| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** | +| --- | --- | --- | --- | --- | --- | --- | +| POST | /chat/completions | 100 | 150 | 240 | 111.73 | 1170 | +| Custom | LiteLLM Overhead Duration (ms) | 2 | 8 | 13 | 3.32 | 1170 | +| | Aggregated | 77 | 130 | 180 | 57.53 | 2340 | + #### Key Findings -- Single instance: 475 RPS @ 100ms median latency -- LiteLLM adds 3ms P50 overhead, 17ms P90 overhead, 31ms P99 overhead -- 2 LiteLLM instances: 950 RPS @ 100ms latency -- 4 LiteLLM instances: 1900 RPS @ 100ms latency - -### 2 Instances - -**Adding 1 instance, will double the RPS and maintain the `100ms-110ms` median latency.** - -| Metric | Litellm Proxy (2 Instances) | -|--------|------------------------| -| Median Latency (ms) | 100 | -| RPS | 950 | - +- Doubling from 2 to 4 LiteLLM instances cuts median latency from 200 ms to 100 ms. +- High-percentile latencies drop significantly: P95 from 430 ms to 130 ms, P99 from 930 ms to 180 ms. +- Setting workers equal to CPU count gives optimal performance. ## Machine Spec used for testing Each machine deploying LiteLLM had the following specs: -- 2 CPU -- 4GB RAM +- 4 CPU +- 8GB RAM + + +## Locust Settings + +- 1000 Users +- 500 user Ramp Up ## How to measure LiteLLM Overhead @@ -137,10 +139,3 @@ Using LangSmith has **no impact on latency, RPS compared to Basic Litellm Proxy* |--------|------------------------|---------------------| | RPS | 1133.2 | 1135 | | Median Latency (ms) | 140 | 132 | - - - -## Locust Settings - -- 2500 Users -- 100 user Ramp Up