tiennm99/litellm
1
0
Fork 0
mirror of https://github.com/tiennm99/litellm.git synced 2026-08-05 06:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #21950 from ryan-crabbe/docs/v1-81-14-perf-section

Browse Source 
docs: add performance & reliability section to v1.81.14 release notes
This commit is contained in:
ryan-crabbe
2026-02-23 13:13:21 -08:00
committed by GitHub
parent 6574d8d395 67ceade162
commit 0ca9869b99
2 changed files with 34 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/my-website/img/release_notes/v1_81_14_perf.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 625 KiB

docs/my-website/release_notes/v1.81.14.md
+34
View File
@@ -96,6 +96,39 @@ The Compliance Playground lets you test any guardrail against our pre-built eval
---
## Performance & Reliability — Up to 13% Lower Latency
<Image img={require('../img/release_notes/v1_81_14_perf.png')} />
This release cuts latency across all percentiles through 20+ micro-optimizations across logging, cost calculation, routing, and connection management. See [benchmarking](../../docs/benchmarks) for more info about how to benchmark yourself.
- **Mean latency:** 78.4 ms → **70.3 ms** (−10.3%)
- **p50 latency:** 64.8 ms → **57.3 ms** (−11.7%)
- **p99 latency:** 288.9 ms → **250.0 ms** (−13.4%)
**Streaming Connection Pool Fix**
Fixed a 3-fold connection leak that caused TCP connection starvation under streaming workloads: the aiohttp transport wasn't closing connections, no `finally` blocks were calling close on disconnect, and a Uvicorn bug prevented disconnect signaling. [PR #21213](https://github.com/BerriAI/litellm/pull/21213)
```mermaid
graph LR
A[Client Disconnects] --> B[Stream Abandoned]
B --> C{Connection cleaned up?}
C -->|Before| D["❌ No — connection leaked"]
C -->|After| E["✅ Yes — connection returned to pool"]
```
**Redis Connection Pool Reliability**
Fixed 4 separate connection pool bugs to make how we use Redis more reliable. The most important change was on pools being leaked on cache expiry and the other fixes are detailed here in [PR #21717](https://github.com/BerriAI/litellm/pull/21717).
```mermaid
graph LR
A[Cache Entry Expires] --> B{Pool cleanup?}
B -->|Before| C["❌ New untracked pool created — leaked"]
B -->|After| D["✅ Pool closed on eviction"]
```
---
## New Providers and Endpoints
@@ -438,6 +471,7 @@ The Compliance Playground lets you test any guardrail against our pre-built eval
- Fix Redis connection pool reliability — prevent connection exhaustion under load - [PR #21717](https://github.com/BerriAI/litellm/pull/21717)
- Fix Prisma connection self-heal for auth and runtime reconnection (reverted, will be re-introduced with fixes) - [PR #21706](https://github.com/BerriAI/litellm/pull/21706)
- Close streaming connections to prevent connection pool exhaustion - [PR #21213](https://github.com/BerriAI/litellm/pull/21213)
- Make `PodLockManager.release_lock` atomic compare-and-delete - [PR #21226](https://github.com/BerriAI/litellm/pull/21226)
---