tiennm99/litellm
1
0
Fork 0
mirror of https://github.com/tiennm99/litellm.git synced 2026-08-07 06:29:41 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs fix

Browse Source
This commit is contained in:
Ishaan Jaffer
2025-11-08 16:50:06 -08:00
parent 0640949333
commit 4647b41d7f
2 changed files with 47 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/my-website/release_notes/v1.79.1-stable/index.md
-56
View File
@@ -54,62 +54,6 @@ pip install litellm==1.80.0
---
### Performance – `/responses` 19× Lower Median, `/embeddings` 13× Lower p95 Latency
This update fixes two major issues. First, embeddings now go through the same optimized pipeline as chat/completions, benefiting from all performance improvements applied so far. Second, while the responses API was already using the same pipeline, it now also includes our internal network management for connection handling, drastically improving latency and reducing memory overhead from per-request setup.
As a result, end-to-end latency improved across all percentiles:
`/embeddings`
**Median latency:** 290 ms → **230 ms** (−20.7%)
**p95 latency:** 5,700 ms → **430 ms** (−92.5%, ~13× faster)
**p99 latency:** 7,200 ms → **780 ms** (−89.2%)
**Average latency:** 844 ms → **263 ms** (−68.9%)
<br/>
`/responses`
**Median latency:** 3,600 ms → **190 ms** (−94.7%, ~19× faster)
**p95 latency:** 4,300 ms → **280 ms** (−93.5%)
**p99 latency:** 4,600 ms → **590 ms** (−87.2%)
**Average latency:** 3,571 ms → **208 ms** (−94.2%)
**RPS:** 231.3 → **1,058.6** (+357.5%)
### **Test Setup**
**Locust**
- **Concurrent users:** 1,000
- **Ramp-up:** 500
**System Specs**
- **CPU:** 4 vCPUs
- **Memory:** 8 GB RAM
- **LiteLLM Workers:** 4
- **Instances**: 4
> Database used; Redis unused.
**Configuration (config.yaml)**
View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/550791675fd752befcac6a9e44024652)
**Load Script (no_cache_hits.py)**
View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/99d673bf74cdd81fd39f59fa9048f2e8)
---
## Dependency Upgrades
- **Dependencies**
docs/my-website/release_notes/v1.79.3-stable/index.md
+47
View File
@@ -53,6 +53,53 @@ pip install litellm==1.79.3.rc.1
---
--
### Performance – `/responses` 19× Lower Median Latency
This update improves latency and reduces memory overhead from per-request setup by including our internal network management for connection handling while the responses API was already using the same pipeline.
As a result, end-to-end latency improved across all percentiles:
`/responses`
**Median latency:** 3,600 ms → **190 ms** (−94.7%, ~19× faster)
**p95 latency:** 4,300 ms → **280 ms** (−93.5%)
**p99 latency:** 4,600 ms → **590 ms** (−87.2%)
**Average latency:** 3,571 ms → **208 ms** (−94.2%)
**RPS:** 231.3 → **1,058.6** (+357.5%)
#### **Test Setup**
**Locust**
- **Concurrent users:** 1,000
- **Ramp-up:** 500
**System Specs**
- **CPU:** 4 vCPUs
- **Memory:** 8 GB RAM
- **LiteLLM Workers:** 4
- **Instances**: 4
> Database used; Redis unused.
**Configuration (config.yaml)**
View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/550791675fd752befcac6a9e44024652)
**Load Script (no_cache_hits.py)**
View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/99d673bf74cdd81fd39f59fa9048f2e8)
---
## New Models / Updated Models
#### New Model Support