From 4647b41d7fb6786b93fff94b97bae0c08035fe92 Mon Sep 17 00:00:00 2001 From: Ishaan Jaffer Date: Sat, 8 Nov 2025 16:49:30 -0800 Subject: [PATCH] docs fix --- .../release_notes/v1.79.1-stable/index.md | 56 ------------------- .../release_notes/v1.79.3-stable/index.md | 47 ++++++++++++++++ 2 files changed, 47 insertions(+), 56 deletions(-) diff --git a/docs/my-website/release_notes/v1.79.1-stable/index.md b/docs/my-website/release_notes/v1.79.1-stable/index.md index 725bb35830..54cfcf7c7f 100644 --- a/docs/my-website/release_notes/v1.79.1-stable/index.md +++ b/docs/my-website/release_notes/v1.79.1-stable/index.md @@ -54,62 +54,6 @@ pip install litellm==1.80.0 --- -### Performance – `/responses` 19× Lower Median, `/embeddings` 13× Lower p95 Latency - -This update fixes two major issues. First, embeddings now go through the same optimized pipeline as chat/completions, benefiting from all performance improvements applied so far. Second, while the responses API was already using the same pipeline, it now also includes our internal network management for connection handling, drastically improving latency and reducing memory overhead from per-request setup. - -As a result, end-to-end latency improved across all percentiles: - -`/embeddings` - -**Median latency:** 290 ms → **230 ms** (−20.7%) - -**p95 latency:** 5,700 ms → **430 ms** (−92.5%, ~13× faster) - -**p99 latency:** 7,200 ms → **780 ms** (−89.2%) - -**Average latency:** 844 ms → **263 ms** (−68.9%) - -
- -`/responses` - -**Median latency:** 3,600 ms → **190 ms** (−94.7%, ~19× faster) - -**p95 latency:** 4,300 ms → **280 ms** (−93.5%) - -**p99 latency:** 4,600 ms → **590 ms** (−87.2%) - -**Average latency:** 3,571 ms → **208 ms** (−94.2%) - -**RPS:** 231.3 → **1,058.6** (+357.5%) - -### **Test Setup** - -**Locust** - -- **Concurrent users:** 1,000 -- **Ramp-up:** 500 - -**System Specs** - -- **CPU:** 4 vCPUs -- **Memory:** 8 GB RAM -- **LiteLLM Workers:** 4 -- **Instances**: 4 - -> Database used; Redis unused. - -**Configuration (config.yaml)** - -View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/550791675fd752befcac6a9e44024652) - -**Load Script (no_cache_hits.py)** - -View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/99d673bf74cdd81fd39f59fa9048f2e8) - ---- - ## Dependency Upgrades - **Dependencies** diff --git a/docs/my-website/release_notes/v1.79.3-stable/index.md b/docs/my-website/release_notes/v1.79.3-stable/index.md index 9f51b499e6..e76577be8f 100644 --- a/docs/my-website/release_notes/v1.79.3-stable/index.md +++ b/docs/my-website/release_notes/v1.79.3-stable/index.md @@ -53,6 +53,53 @@ pip install litellm==1.79.3.rc.1 --- +-- + + +### Performance – `/responses` 19× Lower Median Latency + +This update improves latency and reduces memory overhead from per-request setup by including our internal network management for connection handling while the responses API was already using the same pipeline. + +As a result, end-to-end latency improved across all percentiles: + +`/responses` + +**Median latency:** 3,600 ms → **190 ms** (−94.7%, ~19× faster) + +**p95 latency:** 4,300 ms → **280 ms** (−93.5%) + +**p99 latency:** 4,600 ms → **590 ms** (−87.2%) + +**Average latency:** 3,571 ms → **208 ms** (−94.2%) + +**RPS:** 231.3 → **1,058.6** (+357.5%) + +#### **Test Setup** + +**Locust** + +- **Concurrent users:** 1,000 +- **Ramp-up:** 500 + +**System Specs** + +- **CPU:** 4 vCPUs +- **Memory:** 8 GB RAM +- **LiteLLM Workers:** 4 +- **Instances**: 4 + +> Database used; Redis unused. + +**Configuration (config.yaml)** + +View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/550791675fd752befcac6a9e44024652) + +**Load Script (no_cache_hits.py)** + +View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/99d673bf74cdd81fd39f59fa9048f2e8) + +--- + ## New Models / Updated Models #### New Model Support