From f37dd6bb9503bc74219e2f99fec52c5651726e77 Mon Sep 17 00:00:00 2001 From: Ishaan Jaff Date: Sat, 13 Sep 2025 18:41:34 -0700 Subject: [PATCH] Litellm 1.77.2 stable notes (#14544) * fix release notes instructions * docs v1 * fix doc * fix highlights * docs fix * docs fix --- .../RELEASE_NOTES_GENERATION_INSTRUCTIONS.md | 256 ++++++++++++++++++ .../release_notes/v1.77.2-stable/index.md | 155 +++++++++++ 2 files changed, 411 insertions(+) create mode 100644 cookbook/misc/RELEASE_NOTES_GENERATION_INSTRUCTIONS.md create mode 100644 docs/my-website/release_notes/v1.77.2-stable/index.md diff --git a/cookbook/misc/RELEASE_NOTES_GENERATION_INSTRUCTIONS.md b/cookbook/misc/RELEASE_NOTES_GENERATION_INSTRUCTIONS.md new file mode 100644 index 0000000000..f24bc8f129 --- /dev/null +++ b/cookbook/misc/RELEASE_NOTES_GENERATION_INSTRUCTIONS.md @@ -0,0 +1,256 @@ +# LiteLLM Release Notes Generation Instructions + +This document provides comprehensive instructions for AI agents to generate release notes for LiteLLM following the established format and style. + +## Required Inputs + +1. **Release Version** (e.g., `v1.76.3-stable`) +2. **PR Diff/Changelog** - List of PRs with titles and contributors +3. **Previous Version Commit Hash** - To compare model pricing changes +4. **Reference Release Notes** - Previous release notes to follow style/format + +## Step-by-Step Process + +### 1. Initial Setup and Analysis + +```bash +# Check git diff for model pricing changes +git diff HEAD -- model_prices_and_context_window.json +``` + +**Key Analysis Points:** +- New models added (look for new entries) +- Deprecated models removed (look for deleted entries) +- Pricing updates (look for cost changes) +- Feature support changes (tool calling, reasoning, etc.) + +### 2. Release Notes Structure + +Follow this exact structure based on `docs/my-website/release_notes/v1.76.1-stable/index.md`: + +```markdown +--- +title: "v1.76.X-stable - [Key Theme]" +slug: "v1-76-X" +date: YYYY-MM-DDTHH:mm:ss +authors: [standard author block] +hide_table_of_contents: false +--- + +## Deploy this version +[Docker and pip installation tabs] + +## Key Highlights +[3-5 bullet points of major features] + +## Major Changes +[Critical changes users need to know] + +## Performance Improvements +[Performance-related changes] + +## New Models / Updated Models +[Detailed model tables and provider updates] + +## LLM API Endpoints +[API-related features and fixes] + +## Management Endpoints / UI +[Admin interface and management changes] + +## Logging / Guardrail Integrations +[Observability and security features] + +## Performance / Loadbalancing / Reliability improvements +[Infrastructure improvements] + +## General Proxy Improvements +[Other proxy-related changes] + +## New Contributors +[List of first-time contributors] + +## Full Changelog +[Link to GitHub comparison] +``` + +### 3. Categorization Rules + +**Performance Improvements:** +- RPS improvements +- Memory optimizations +- CPU usage optimizations +- Timeout controls +- Worker configuration + +**New Models/Updated Models:** +- Extract from model_prices_and_context_window.json diff +- Create tables with: Provider, Model, Context Window, Input Cost, Output Cost, Features +- Group by provider +- Note pricing corrections +- Highlight deprecated models + +**Provider Features:** +- Group by provider (Gemini, OpenAI, Anthropic, etc.) +- Link to provider docs: `../../docs/providers/[provider_name]` +- Separate features from bug fixes + +**API Endpoints:** +- Images API +- Video Generation (if applicable) +- Responses API +- Passthrough endpoints +- General chat completions + +**UI/Management:** +- Authentication changes +- Dashboard improvements +- Team management +- Key management + +**Integrations:** +- Logging providers (Datadog, Braintrust, etc.) +- Guardrails +- Cost tracking +- Observability + +### 4. Documentation Linking Strategy + +**Link to docs when:** +- New provider support added +- Significant feature additions +- API endpoint changes +- Integration additions + +**Link format:** `../../docs/[category]/[specific_doc]` + +**Common doc paths:** +- `../../docs/providers/[provider]` - Provider-specific docs +- `../../docs/image_generation` - Image generation +- `../../docs/video_generation` - Video generation (if exists) +- `../../docs/response_api` - Responses API +- `../../docs/proxy/logging` - Logging integrations +- `../../docs/proxy/guardrails` - Guardrails +- `../../docs/pass_through/[provider]` - Passthrough endpoints + +### 5. Model Table Generation + +From git diff analysis, create tables like: + +```markdown +| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features | +| -------- | ----- | -------------- | ------------------- | -------------------- | -------- | +| OpenRouter | `openrouter/openai/gpt-4.1` | 1M | $2.00 | $8.00 | Chat completions with vision | +``` + +**Extract from JSON:** +- `max_input_tokens` → Context Window +- `input_cost_per_token` × 1,000,000 → Input cost +- `output_cost_per_token` × 1,000,000 → Output cost +- `supports_*` fields → Features +- Special pricing fields (per image, per second) for generation models + +### 6. PR Categorization Logic + +**By Keywords in PR Title:** +- `[Perf]`, `Performance`, `RPS` → Performance Improvements +- `[Bug]`, `[Bug Fix]`, `Fix` → Bug Fixes section +- `[Feat]`, `[Feature]`, `Add support` → Features section +- `[Docs]` → Documentation (usually exclude from main sections) +- Provider names (Gemini, OpenAI, etc.) → Group under provider + +**By PR Content Analysis:** +- New model additions → New Models section +- UI changes → Management Endpoints/UI +- Logging/observability → Logging/Guardrail Integrations +- Rate limiting/budgets → Performance/Reliability +- Authentication → Management Endpoints + +### 7. Writing Style Guidelines + +**Tone:** +- Professional but accessible +- Focus on user impact +- Highlight breaking changes clearly +- Use active voice + +**Formatting:** +- Use consistent markdown formatting +- Include PR links: `[PR #XXXXX](https://github.com/BerriAI/litellm/pull/XXXXX)` +- Use code blocks for configuration examples +- Bold important terms and section headers + +**Warnings/Notes:** +- Add warning boxes for breaking changes +- Include migration instructions when needed +- Provide override options for default changes + +### 8. Quality Checks + +**Before finalizing:** +- Verify all PR links work +- Check documentation links are valid +- Ensure model pricing is accurate +- Confirm provider names are consistent +- Review for typos and formatting issues + +### 9. Common Patterns to Follow + +**Performance Changes:** +```markdown +- **+400 RPS Performance Boost** - Description - [PR #XXXXX](link) +``` + +**New Models:** +Always include pricing table and feature highlights + +**Breaking Changes:** +```markdown +:::warning +This release has a known issue... +::: +``` + +**Provider Features:** +```markdown +- **[Provider Name](../../docs/providers/provider)** + - Feature description - [PR #XXXXX](link) +``` + +### 10. Missing Documentation Check + +**Review for missing docs:** +- New providers without documentation +- New API endpoints without examples +- Complex features without guides +- Integration setup instructions + +**Flag for documentation needs:** +- New provider integrations +- Significant API changes +- Complex configuration options +- Migration requirements + +## Example Command Workflow + +```bash +# 1. Get model changes +git diff HEAD -- model_prices_and_context_window.json + +# 2. Analyze PR list for categorization +# 3. Create release notes following template +# 4. Link to appropriate documentation +# 5. Review for missing documentation needs +``` + +## Output Requirements + +- Follow exact markdown structure from reference +- Include all PR links and contributors +- Provide accurate model pricing tables +- Link to relevant documentation +- Highlight breaking changes with warnings +- Include deployment instructions +- End with full changelog link + +This process ensures consistent, comprehensive release notes that help users understand changes and upgrade smoothly. diff --git a/docs/my-website/release_notes/v1.77.2-stable/index.md b/docs/my-website/release_notes/v1.77.2-stable/index.md new file mode 100644 index 0000000000..cdbe6595fe --- /dev/null +++ b/docs/my-website/release_notes/v1.77.2-stable/index.md @@ -0,0 +1,155 @@ +--- +title: "v1.77.2-stable - Bedrock Batches API" +slug: "v1-77-2" +date: 2025-09-13T10:00:00 +authors: + - name: Krrish Dholakia + title: CEO, LiteLLM + url: https://www.linkedin.com/in/krish-d/ + image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg + - name: Ishaan Jaffer + title: CTO, LiteLLM + url: https://www.linkedin.com/in/reffajnaahsi/ + image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg + +hide_table_of_contents: false +--- + +import Image from '@theme/IdealImage'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Deploy this version + + + + +``` showLineNumbers title="docker run litellm" +docker run \ +-e STORE_MODEL_IN_DB=True \ +-p 4000:4000 \ +ghcr.io/berriai/litellm:v1.77.2 +``` + + + + +``` showLineNumbers title="pip install litellm" +pip install litellm==1.77.2 +``` + + + + +--- + +## Key Highlights + +- **Bedrock Batches API** - Support for creating Batch Inference Jobs on Bedrock using LiteLLM's unified batch API (OpenAI compatible) +- **Qwen API Tiered Pricing** - Cost tracking support for Dashscope (Qwen) models with multiple pricing tiers + +## New Models / Updated Models + +#### New Model Support + +| Provider | Model | Context Window | Pricing ($/1M tokens) | Features | +| ----------- | ------------------------------- | -------------- | --------------------- | -------- | +| DeepInfra | `deepinfra/deepseek-ai/DeepSeek-R1` | 164K | **Input:** $0.70
**Output:** $2.40 | Chat completions, tool calling | +| Heroku | `heroku/claude-4-sonnet` | 8K | Contact provider for pricing | Function calling, tool choice | +| Heroku | `heroku/claude-3-7-sonnet` | 8K | Contact provider for pricing | Function calling, tool choice | +| Heroku | `heroku/claude-3-5-sonnet-latest` | 8K | Contact provider for pricing | Function calling, tool choice | +| Heroku | `heroku/claude-3-5-haiku` | 4K | Contact provider for pricing | Function calling, tool choice | +| Dashscope | `dashscope/qwen-plus-latest` | 1M | **Tiered Pricing:**
• 0-256K tokens: $0.40 / $1.20
• 256K-1M tokens: $1.20 / $3.60 | Function calling, reasoning | +| Dashscope | `dashscope/qwen3-max-preview` | 262K | **Tiered Pricing:**
• 0-32K tokens: $1.20 / $6.00
• 32K-128K tokens: $2.40 / $12.00
• 128K-252K tokens: $3.00 / $15.00 | Function calling, reasoning | +| Dashscope | `dashscope/qwen-flash` | 1M | **Tiered Pricing:**
• 0-256K tokens: $0.05 / $0.40
• 256K-1M tokens: $0.25 / $2.00 | Function calling, reasoning | +| Dashscope | `dashscope/qwen3-coder-plus` | 1M | **Tiered Pricing:**
• 0-32K tokens: $1.00 / $5.00
• 32K-128K tokens: $1.80 / $9.00
• 128K-256K tokens: $3.00 / $15.00
• 256K-1M tokens: $6.00 / $60.00 | Function calling, reasoning, caching | +| Dashscope | `dashscope/qwen3-coder-flash` | 1M | **Tiered Pricing:**
• 0-32K tokens: $0.30 / $1.50
• 32K-128K tokens: $0.50 / $2.50
• 128K-256K tokens: $0.80 / $4.00
• 256K-1M tokens: $1.60 / $9.60 | Function calling, reasoning, caching | + +--- + +#### Features + +- **[Bedrock](../../docs/providers/bedrock_batches)** + - Bedrock Batches API - batch processing support with file upload and request transformation - [PR #14518](https://github.com/BerriAI/litellm/pull/14518), [PR #14522](https://github.com/BerriAI/litellm/pull/14522) +- **[VLLM](../../docs/providers/vllm)** + - Added transcription endpoint support - [PR #14523](https://github.com/BerriAI/litellm/pull/14523) +- **[Ollama](../../docs/providers/ollama)** + - `ollama_chat/` - images, thinking, and content as list handling - [PR #14523](https://github.com/BerriAI/litellm/pull/14523) +- **General** + - New debug flag for detailed request/response logging [PR #14482](https://github.com/BerriAI/litellm/pull/14482) + +#### Bug Fixes + +- **[Azure OpenAI](../../docs/providers/azure)** + - Fixed extra_body injection causing payload rejection in image generation - [PR #14475](https://github.com/BerriAI/litellm/pull/14475) +- **[LM Studio](../../docs/providers/lm-studio)** + - Resolved illegal Bearer header value issue - [PR #14512](https://github.com/BerriAI/litellm/pull/14512) + +--- + +## LLM API Endpoints + +#### Bug Fixes + +- **[/messages](../../docs/anthropic_unified)** + - Don't send content block after message w/ finish reason + usage block - [PR #14477](https://github.com/BerriAI/litellm/pull/14477) +- **[/generateContent](../../docs/generateContent)** + - Gemini CLI Integration - Fixed token count errors - [PR #14451](https://github.com/BerriAI/litellm/pull/14451), [PR #14417](https://github.com/BerriAI/litellm/pull/14417) + +--- + +## Spend Tracking, Budgets and Rate Limiting + +#### Features + +- **[Qwen API Tiered Pricing](../../docs/providers/dashscope)** - Added comprehensive tiered cost tracking for Dashscope/Qwen models - [PR #14471](https://github.com/BerriAI/litellm/pull/14471), [PR #14479](https://github.com/BerriAI/litellm/pull/14479) + +#### Bug Fixes + +- **Provider Budgets** - Fixed provider budget calculations - [PR #14459](https://github.com/BerriAI/litellm/pull/14459) + +--- + +## Management Endpoints / UI + +#### Features + +- **User Headers Mapping** - New X-LiteLLM Users mapping feature for enhanced user tracking - [PR #14485](https://github.com/BerriAI/litellm/pull/14485) +- **Key Unblocking** - Support for hashed tokens in `/key/unblock` endpoint - [PR #14477](https://github.com/BerriAI/litellm/pull/14477) +- **Model Group Header Forwarding** - Enhanced wildcard model support with documentation - [PR #14528](https://github.com/BerriAI/litellm/pull/14528) + +#### Bug Fixes + +- **Log Tab Key Alias** - Fixed filtering inaccuracies for failed logs - [PR #14469](https://github.com/BerriAI/litellm/pull/14469), [PR #14529](https://github.com/BerriAI/litellm/pull/14529) + +--- + +## Logging / Guardrail Integrations + +#### Features + +- **Noma Integration** - Added non-blocking monitor mode with anonymize input support - [PR #14401](https://github.com/BerriAI/litellm/pull/14401) + +--- + +## Performance / Loadbalancing / Reliability improvements + +#### Performance +- Removed dynamic creation of static values - [PR #14538](https://github.com/BerriAI/litellm/pull/14538) +- Using `_PROXY_MaxParallelRequestsHandler_v3` by default for optimal throughput - [PR #14450](https://github.com/BerriAI/litellm/pull/14450) +- Improved execution context propagation into logging tasks - [PR #14455](https://github.com/BerriAI/litellm/pull/14455) + +--- + + + +## New Contributors +* @Sameerlite made their first contribution in [PR #14460](https://github.com/BerriAI/litellm/pull/14460) +* @holzman made their first contribution in [PR #14459](https://github.com/BerriAI/litellm/pull/14459) +* @sashank5644 made their first contribution in [PR #14469](https://github.com/BerriAI/litellm/pull/14469) +* @TomAlon made their first contribution in [PR #14401](https://github.com/BerriAI/litellm/pull/14401) +* @AlexsanderHamir made their first contribution in [PR #14538](https://github.com/BerriAI/litellm/pull/14538) + +--- + +## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.77.1.dev.2...v1.77.2.dev)**