mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-12 15:05:01 +00:00
stuxf
dedaf74a5e
* chore(auth): validate clientside api_base against SSRF guard; clear admin secrets on base override Two related issues with how the proxy handles client-supplied ``api_base`` / ``base_url`` overrides on chat-completion requests: 1. **SSRF gate bypass** — ``check_complete_credentials()`` returned ``True`` for any non-empty ``api_key``, allowing the ``is_request_body_safe`` ``banned_params`` loop to admit ``api_base`` / ``base_url`` values that point at private (RFC 1918), loopback, link-local, or cloud-metadata addresses. Now: when the gate sees a client-supplied ``api_base`` / ``base_url``, it runs the URL through ``litellm_core_utils.url_utils.validate_url`` (DNS-resolves, blocks internal/IMDS/LL networks, defends against rebinding). Rejection raises with a clear message. 2. **Admin-config leak on base override** — ``get_dynamic_litellm_params`` only carried the three clientside keys (``api_key``, ``api_base``, ``base_url``) from request to upstream call. Other admin-configured fields on ``litellm_params`` — ``organization``, ``extra_body``, ``extra_headers``, ``api_version``, ``azure_ad_token``, AWS / Vertex creds, etc. — flowed through unchanged. With base redirected to a client-controlled server, those admin secrets were sent to the attacker. Now: when ``api_base`` / ``base_url`` is in ``request_kwargs``, drop those admin-config fields from ``litellm_params`` unless the caller re-supplied them. Tests cover the SSRF-target rejection per URL field, the admin-secret clearing on base override, the don't-clear case when only ``api_key`` is overridden (BYOK pattern), and the don't-overwrite case when the caller resupplies fields like ``organization`` themselves. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(vertex-batches): wrap api_base GET in safe_get for defense-in-depth The vertex batches status-poll fetches an attacker-influenceable ``api_base`` URL with a raw ``sync_handler.get()``. The proxy auth gate already validates clientside ``api_base`` before reaching this sink, so the proxy flow is covered. This adds the per-sink wrap so SDK callers and any future code path that bypasses the proxy gate pick up the same SSRF defense from ``url_utils.safe_get``. Operators with a legitimate private Vertex base can either allowlist the host via ``litellm.user_url_allowed_hosts`` or disable validation with ``litellm.user_url_validation = False``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(auth): hoist url_utils import; derive admin-config field list from CredentialLiteLLMParams /simplify pass: - Move ``from litellm.litellm_core_utils.url_utils import SSRFError, validate_url`` to module top in ``proxy/auth/auth_utils.py``. CLAUDE.md prefers module-level imports unless avoiding a circular dependency, and there's no cycle here (``url_utils`` doesn't depend on ``proxy.auth``). - Replace the hardcoded ``_ADMIN_CONFIG_FIELDS_TO_CLEAR_ON_BASE_OVERRIDE`` literal with ``_admin_config_fields_to_clear_on_base_override()`` that derives the typed-field portion from ``CredentialLiteLLMParams.model_fields``. Adds three fields the hardcoded list missed (``aws_bedrock_runtime_endpoint``, ``watsonx_region_name``, ``region_name``) and stays in sync as new provider fields are declared on the model. The kwargs-only set (``organization``, ``extra_body``, ``azure_ad_token``, ``aws_session_token``, ``aws_sts_endpoint``, ``aws_web_identity_token``, ``aws_role_name``, …) remains explicit since those fields aren't on the typed model. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(auth): close field-echo bypass; gate URL check on toggle; cover async batch path Three issues from review: 1. ``get_dynamic_litellm_params`` used ``if field not in request_kwargs: pop`` to clear admin-set provider config when the caller redirected ``api_base``. A caller could *echo* any clear-list field name (with any value, including an empty string) to skip the pop, leaving the admin's value in ``litellm_params`` to be forwarded to the redirected upstream. Fix: always pop, then write the caller's value back if they resupplied the field. 2. ``check_complete_credentials`` called ``validate_url`` directly. That helper doesn't itself consult ``litellm.user_url_validation``; the toggle is honoured by ``safe_get`` / ``async_safe_get``. Mirror that here so admins who explicitly disabled URL validation aren't blocked at the proxy boundary. 3. ``VertexAIBatchesHandler._async_retrieve_batch`` still used a bare ``await client.get(api_base, ...)`` while the sync sibling was wrapped in ``safe_get``. Wrap the async call in ``async_safe_get`` so SDK callers on the async path get the same DNS-rebind / private / cloud-metadata defenses as the sync path. Tests: - ``TestCheckCompleteCredentialsBlocksSSRF`` is now mock-only; an autouse fixture flips the toggle on, ``validate_url`` is patched in the parametrized blocking tests, and the positive path no longer makes a real DNS call to api.openai.com. - ``test_skips_url_validation_when_toggle_is_off`` documents the new toggle-off behaviour and asserts ``validate_url`` is not called. - ``test_caller_resupplied_value_overrides_admin_value_on_base_override`` replaces the prior test that asserted the buggy preserve-admin-value-on-echo behaviour. - ``test_field_echo_does_not_preserve_admin_value`` is a focused regression test for the empty-string echo vector. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(auth): close provider-confusion credential exfil; expand banned-params; cover OCI Three additions on top of the entry-point URL gate so the cluster is fully closed against caller-supplied ``api_base`` redirection: 1. ``get_llm_provider_logic.py`` matched registered openai-compatible endpoints against ``api_base`` with an unanchored substring search (``if endpoint in api_base:``). A caller could pass an api_base like ``https://attacker.com/api.groq.com/openai/v1`` to coerce the proxy into reading ``GROQ_API_KEY`` from the environment and forwarding it as a Bearer credential to the attacker's host. Replaced with parsed- URL semantics (hostname exact-match plus segment-bounded path-prefix) in a new ``_endpoint_matches_api_base`` helper. 2. ``is_request_body_safe`` rejects ``api_base`` / ``base_url`` / ``user_config`` / a handful of AWS / vertex fields, but the list omitted three other endpoint-targeting fields: * ``aws_bedrock_runtime_endpoint`` — Bedrock endpoint redirect * ``langsmith_base_url`` / ``langfuse_host`` — observability callback hostnames; attacker-controlled values exfiltrate the entire request payload (incl. message content) via the logging hook. Added all three to the blocklist. 3. ``_admin_config_fields_to_clear_on_base_override`` derives its typed- field list from ``CredentialLiteLLMParams.model_fields``, which does not declare any of the OCI provider's auth fields. Added ``oci_signer``, ``oci_user``, ``oci_fingerprint``, ``oci_tenancy``, ``oci_key``, and ``oci_key_file`` to the kwargs-only fixed list so they are cleared on caller-redirected ``api_base`` like the AWS / Azure / Vertex equivalents. Tests: - ``TestEndpointMatchesApiBase`` — direct unit tests on the new matcher: legitimate provider URLs (5 shapes) match; attacker smuggling via path injection, suffix label, prefix label, userinfo ``@`` injection, and path-segment lookalikes (7 shapes) do not. - ``TestGetLlmProviderRejectsAttackerSmuggledApiBase`` — end-to-end invariant that ``GROQ_API_KEY`` is never read against an attacker- controlled host while the legitimate ``api.groq.com`` path still resolves the provider correctly. - ``TestIsRequestBodySafeBlocksEndpointTargetingFields`` — parametrized coverage that each of the three new banned-params raises a clear rejection naming the offending field. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(auth): remove implicit api-key bypass + add posthog/braintrust/slack to blocklist The historical ``check_complete_credentials`` clause inside ``is_request_body_safe`` was a third, *implicit*, *caller-controlled* BYOK path: any caller that supplied a non-empty ``api_key`` caused the entire banned-params blocklist to be skipped. That turned every missing entry on the blocklist into an exploitable SSRF / credential-exfil hole and is the root cause of the chain of api_base advisories that have been re-discovered with each new integration: * GHSA-jh89-88fc-qrfp (critical, triage) — env-var exfil via api_base * GHSA-3frq-6r6h-7j64 (high, triage) — admin org / extra_body leak * veria-admin Dv_m860l, b_yRJeQ5, stN90yjP, LBlyOAc8, U2TD78kg — variations on "list X is missing field Y" Two explicit, admin-controlled BYOK paths already exist and remain: ``general_settings.allow_client_side_credentials = true`` (proxy-wide) and ``configurable_clientside_auth_params: [...]`` per deployment. Removing the implicit bypass converts the failure mode of a missing blocklist entry from "live credential leak" to "predictable 400 with a clear remediation message," which is the structural fix. Also adds the three remaining endpoint-targeting fields the dynamic callback layer reads from request body: ``posthog_host``, ``braintrust_host``, ``slack_webhook_url``. ``slack_webhook_url`` in particular was a direct exfil channel (caller-set webhook → proxy mirrors every request to attacker's Slack). Tests: - ``test_api_key_does_not_bypass_blocklist`` — parametrized regression asserting api_key=anything no longer skips the gate for any of the five highest-risk fields. - ``test_admin_opt_in_proxy_wide_still_allows`` — confirms the documented BYOK opt-in still works. - Extends ``test_endpoint_targeting_field_in_request_body_is_rejected`` to cover posthog / braintrust / slack. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(auth): block sagemaker_base_url, s3_endpoint_url, deployment_url Provider-specific endpoint overrides surfaced by a wider audit of ``optional_params`` consumers in ``litellm/llms/``. Same threat as ``api_base``: a caller-supplied value redirects the outbound request to an attacker host. * ``s3_endpoint_url`` — read in ``litellm/llms/bedrock/files/transformation.py`` to build the S3 upload URL for Bedrock files. Caller redirects file uploads to attacker-controlled S3. * ``sagemaker_base_url`` — read in ``litellm/llms/sagemaker/{chat,completion}/*``. Caller redirects SageMaker traffic. This is the primary vector described in veria-admin mNqEBBtG. * ``deployment_url`` — popped in ``litellm/llms/sap/chat/transformation.py``. Caller redirects SAP deployment requests. Tests parametrize ``test_endpoint_targeting_field_in_request_body_is_rejected`` to cover the three new fields. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🚅 LiteLLM
LiteLLM AI Gateway
Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.
LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier | Website
What is LiteLLM
LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format.
Use it as a Python SDK for direct library integration, or deploy the AI Gateway (Proxy Server) as a centralized service for your team or organization.
Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers
Why LiteLLM
Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:
- Unified API — one interface for 100+ LLMs, no provider-specific SDK juggling
- Drop-in OpenAI compatibility — swap providers without rewriting your code
- Production-ready gateway — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box
- 8ms P95 latency at 1k RPS (benchmarks)
OSS Adopters
Netflix |
Features
LLMs - Call 100+ LLMs (Python SDK + AI Gateway)
All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.
Python SDK
uv add litellm
from litellm import completion
import os
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"
# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])
# Anthropic
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])
AI Gateway (Proxy Server)
Getting Started - E2E Tutorial - Setup virtual keys, make your first request
uv tool install 'litellm[proxy]'
litellm --model gpt-4o
import openai
client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}]
)
Agents - Invoke A2A Agents (Python SDK + AI Gateway)
Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI
Python SDK - A2A Protocol
from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4
client = A2AClient(base_url="http://localhost:10001")
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello!"}],
"messageId": uuid4().hex,
}
)
)
response = await client.send_message(request)
AI Gateway (Proxy Server)
Step 1. Add your Agent to the AI Gateway
Step 2. Call Agent via A2A SDK
from a2a.client import A2ACardResolver, A2AClient
from a2a.types import MessageSendParams, SendMessageRequest
from uuid import uuid4
import httpx
base_url = "http://localhost:4000/a2a/my-agent" # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer sk-1234"} # LiteLLM Virtual Key
async with httpx.AsyncClient(headers=headers) as httpx_client:
resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url)
agent_card = await resolver.get_agent_card()
client = A2AClient(httpx_client=httpx_client, agent_card=agent_card)
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello!"}],
"messageId": uuid4().hex,
}
)
)
response = await client.send_message(request)
MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway)
Python SDK - MCP Bridge
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm
server_params = StdioServerParameters(command="python", args=["mcp_server.py"])
async with stdio_client(server_params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
# Load MCP tools in OpenAI format
tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")
# Use with any LiteLLM model
response = await litellm.acompletion(
model="gpt-4o",
messages=[{"role": "user", "content": "What's 3 + 5?"}],
tools=tools
)
AI Gateway - MCP Gateway
Step 1. Add your MCP Server to the AI Gateway
Step 2. Call MCP tools via /chat/completions
curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Summarize the latest open PR"}],
"tools": [{
"type": "mcp",
"server_url": "litellm_proxy/mcp/github",
"server_label": "github_mcp",
"require_approval": "never"
}]
}'
Use with Cursor IDE
{
"mcpServers": {
"LiteLLM": {
"url": "http://localhost:4000/mcp/",
"headers": {
"x-litellm-api-key": "Bearer sk-1234"
}
}
}
}
Supported Providers (Website Supported Models | Docs)
Get Started
You can use LiteLLM through either the Proxy Server or Python SDK. Both give you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:
| LiteLLM AI Gateway | LiteLLM Python SDK | |
|---|---|---|
| Use Case | Central service (LLM Gateway) to access multiple LLMs | Use LiteLLM directly in your Python code |
| Who Uses It? | Gen AI Enablement / ML Platform Teams | Developers building LLM projects |
| Key Features | Centralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and management | Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.) |
Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published. More information about the release cycle here
Support for more providers. Missing a provider or LLM Platform, raise a feature request.
Run in Developer Mode
Services
- Setup .env file in root
- Run dependant services
docker-compose up db prometheus
Backend
- (In root) create virtual environment
python -m venv .venv - Activate virtual environment
source .venv/bin/activate - Install dependencies
uv sync --all-extras --group proxy-dev uv run prisma generateprisma generate- Start proxy backend
python litellm/proxy/proxy_cli.py
Frontend
- Navigate to
ui/litellm-dashboard - Install dependencies
npm install - Run
npm run devto start the dashboard
Verify Docker Image Signatures
All LiteLLM Docker images published to GHCR are signed with cosign. Every release is signed with the same key introduced in commit 0112e53.
Verify using the pinned commit hash (recommended):
A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
ghcr.io/berriai/litellm:<release-tag>
Verify using a release tag (convenience):
Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
ghcr.io/berriai/litellm:<release-tag>
Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).
Enterprise
For companies that need better security, user management and professional support
Get an Enterprise License Talk to founders
This covers:
- ✅ Features under the LiteLLM Commercial License:
- ✅ Feature Prioritization
- ✅ Custom Integrations
- ✅ Professional Support - Dedicated discord + slack
- ✅ Custom SLAs
- ✅ Secure access with Single Sign-On
Contributing
We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.
Quick Start for Contributors
This requires uv to be installed.
git clone https://github.com/BerriAI/litellm.git
cd litellm
make install-dev # Install development dependencies
make format # Format your code
make lint # Run all linting checks
make test-unit # Run unit tests
make format-check # Check formatting only
For detailed contributing guidelines, see CONTRIBUTING.md.
📖 Contributing to documentation? The LiteLLM docs have moved to a separate repository: BerriAI/litellm-docs. Please open doc PRs there. Docs are served at docs.litellm.ai.
Code Quality / Linting
LiteLLM follows the Google Python Style Guide.
Our automated checks include:
- Black for code formatting
- Ruff for linting and code quality
- MyPy for type checking
- Circular import detection
- Import safety checks
All these checks must pass before your PR can be merged.
Support / talk with founders
- Schedule Demo 👋
- Community Discord 💭
- Community Slack 💭
- Our emails ✉️ ishaan@berri.ai / krrish@berri.ai