mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-17 10:17:46 +00:00
8e61b32b8e
* feat(xai): add grok-4.20 beta 2 models with pricing (#23900)
Add three grok-4.20 beta 2 model variants from xAI:
- grok-4.20-multi-agent-beta-0309 (reasoning + multi-agent)
- grok-4.20-beta-0309-reasoning (reasoning)
- grok-4.20-beta-0309-non-reasoning
Pricing (from https://docs.x.ai/docs/models):
- Input: $2.00/1M tokens ($0.20/1M cached)
- Output: $6.00/1M tokens
- Context: 2M tokens
All variants support vision, function calling, tool choice, and web search.
Closes LIT-2171
* docs: add Quick Install section for litellm --setup wizard (#23905)
* docs: add Quick Install section for litellm --setup wizard
* docs: clarify setup wizard is for local/beginner use
* feat(setup): interactive setup wizard + install.sh (#23644)
* feat(setup): add interactive setup wizard + install.sh
Adds `litellm --setup` — a Claude Code-style TUI onboarding wizard that
guides users through provider selection, API key entry, and proxy config
generation, then optionally starts the proxy immediately.
- litellm/setup_wizard.py: wizard with ASCII art, numbered provider menu
(OpenAI, Anthropic, Azure, Gemini, Bedrock, Ollama), API key prompts,
port/master-key config, and litellm_config.yaml generation
- litellm/proxy/proxy_cli.py: adds --setup flag that invokes the wizard
- scripts/install.sh: curl-installable script (detect OS/Python, pip
install litellm[proxy], launch wizard)
Usage:
curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | sh
litellm --setup
* fix(install.sh): remove orange color, add LITELLM_BRANCH env var for branch installs
* fix(install.sh): install from git branch so --setup is available for QA
* fix(install.sh): remove stale LITELLM_BRANCH reference that caused unbound variable error
* fix(install.sh): force-reinstall from git to bypass cached PyPI version
* fix(install.sh): show pip progress bar during install
* fix(install.sh): always launch wizard via $PYTHON_BIN -m litellm, not PATH binary
* fix(install.sh): use litellm.proxy.proxy_cli module (no __main__.py exists)
* fix(install.sh): suppress RuntimeWarning from module invocation
* fix(install.sh): use Python bin-dir litellm binary to avoid CWD sys.path shadowing
* fix(install.sh): use sysconfig.get_path('scripts') to find pip-installed litellm binary
* fix(install.sh): redirect stdin from /dev/tty on exec so wizard gets terminal, not exhausted pipe
* fix(install.sh): warn about git clone duration, drop --no-cache-dir so re-runs are faster
* feat(setup_wizard): arrow-key selector, updated model names
* fix(setup_wizard): use sysconfig binary to start proxy, not python -m litellm
* feat(setup_wizard): credential validation after key entry + clear next-steps after proxy start
* style(install.sh): show git clone warning in blue
* refactor(setup_wizard): class with static methods, use check_valid_key from litellm.utils
* address greptile review: fix yaml escaping, port validation, display name collisions, tests
- setup_wizard.py: add _yaml_escape() for safe YAML embedding of API keys
- setup_wizard.py: add _styled_input() with readline ANSI ignore markers
- setup_wizard.py: change DIVIDER to _divider() fn to avoid import-time color capture
- setup_wizard.py: validate port range 1-65535, initialize before loop
- setup_wizard.py: qualify azure display names (azure-gpt-4o) to avoid collision with openai
- setup_wizard.py: work on env_copy in _build_config to avoid mutating caller's dict
- setup_wizard.py: skip model_list entries for providers with no credentials
- setup_wizard.py: prompt for azure deployment name
- setup_wizard.py: wrap os.execlp in try/except with friendly fallback
- setup_wizard.py: wrap config write in try/except OSError
- setup_wizard.py: fix _validate_and_report to use two print lines (no \r overwrite)
- setup_wizard.py: add .gitignore tip next to key storage notice
- setup_wizard.py: fix run_setup_wizard() return type annotation to None
- scripts/install.sh: drop pipefail (not supported by dash on Ubuntu when invoked as sh)
- scripts/install.sh: use litellm[proxy] from PyPI (not hardcoded dev branch)
- scripts/install.sh: guard /dev/tty read with -r check for Docker/CI compat
- scripts/install.sh: remove --force-reinstall to avoid downgrading dependencies
- tests/test_litellm/test_setup_wizard.py: 13 unit tests for _build_config and _yaml_escape
* style: black format setup_wizard.py
* fix: address remaining greptile issues - Windows compat, YAML quoting, credential flow
- guard termios/tty imports with try/except ImportError for Windows compat
- quote master_key as YAML double-quoted scalar (same as env vars)
- remove unused port param from _build_config signature
- _validate_and_report now returns the final key so re-entered creds are stored
- add test for master_key YAML quoting
* fix: add --port to suggested command, guard /dev/tty exec in install.sh
* fix: quote api_base in YAML, skip azure if no deployment, only redraw on state change
* fix: address greptile review comments
- _yaml_escape: add control character escaping (\n, \r, \t)
- test: fix tautological assertion in test_build_config_azure_no_deployment_skipped
- test: add tests for control character escaping in _yaml_escape
* feat(ui): remove Chat UI page link and banner from sidebar and playground (#23908)
* feat(guardrails): MCPJWTSigner - built-in guardrail for zero trust MCP auth (#23897)
* Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers
* Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present.
* Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings.
* Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers
* Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present.
* Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings.
* feat(guardrails): add MCPJWTSigner built-in guardrail for zero trust MCP auth
Signs outbound MCP tool calls with a LiteLLM-issued RS256 JWT so MCP servers
can trust a single signing authority instead of every upstream IdP.
Enable in config.yaml:
guardrails:
- guardrail_name: mcp-jwt-signer
litellm_params:
guardrail: mcp_jwt_signer
mode: pre_mcp_call
default_on: true
JWT carries sub (user_id), act.sub (team_id, RFC 8693), tool-level scope, iss,
aud, iat/exp/nbf. RSA-2048 keypair auto-generated at startup unless
MCP_JWT_SIGNING_KEY env var is set.
Adds /.well-known/jwks.json endpoint and jwks_uri to /.well-known/openid-configuration
so MCP servers can verify LiteLLM-issued tokens via OIDC discovery.
* Update MCPServerManager to raise HTTPException with status code 400 for extra headers in OpenAPI-backed servers. Adjust tests to verify the correct status code and exception message.
* fix: address P1 issues in MCPJWTSigner
- OpenAPI servers: warn + skip header injection instead of 500
- JWKS Cache-Control: 5min for auto-generated keys, 1h for persistent
- sub claim: fallback to apikey:{token_hash} for anonymous callers
- ttl_seconds: validate > 0 at init time
* docs: add MCP zero trust auth guide with architecture diagram
* docs: add FastMCP JWT verification guide to zero trust doc
* fix: address remaining Greptile review issues (round 2)
- mcp_server_manager: warn when hook Authorization overwrites existing header
- __init__: remove _mcp_jwt_signer_instance from __all__ (private internal)
- discoverable_endpoints: copy dict instead of mutating in-place on OIDC augmentation
- test docstring: reflect warn-and-continue behavior for OpenAPI servers
- test: update scope assertions for least-privilege (no mcp:tools/list on tool-call JWTs)
* fix: address Greptile round 3 feedback
- initialize_guardrail: validate mode='pre_mcp_call' at init time — misconfigured
mode silently bypasses JWT injection, which is a zero-trust bypass
- _build_claims: remove duplicate inline 'import re' (module-level import already present)
- _types.py: add TODO comment explaining jwt_claims is forward-compat plumbing
for a follow-up PR that will forward upstream IdP claims into outbound MCP JWTs
* feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes
Addresses all missing pieces from the scoping doc review:
FR-5 (Verify + re-sign): MCPJWTSigner now accepts access_token_discovery_uri
and token_introspection_endpoint. When set, the incoming Bearer token is
extracted from raw_headers (threaded through pre_call_tool_check), verified
against the IdP's JWKS (JWT) or introspected (opaque), and only re-signed if
valid. Falls back to user_api_key_dict.jwt_claims for LiteLLM JWT-auth mode.
FR-12 (Configurable end-user identity mapping): end_user_claim_sources
ordered list drives sub resolution — sources: token:<claim>, litellm:user_id,
litellm:email, litellm:end_user_id, litellm:team_id.
FR-13 (Claim operations): add_claims (insert-if-absent), set_claims (always
override), remove_claims (delete) applied in that order.
FR-14 (Two-token model): channel_token_audience + channel_token_ttl issue a
second JWT injected as x-mcp-channel-token: Bearer <token>.
FR-15 (Incoming claim validation): required_claims raises HTTP 403 when any
listed claim is absent; optional_claims passes listed claims from verified
token into the outbound JWT.
FR-9 (Debug headers): debug_headers: true emits x-litellm-mcp-debug with kid,
sub, iss, exp, scope.
FR-10 (Configurable scopes): allowed_scopes replaces auto-generation. Also
fixed: tool-call JWTs no longer grant mcp:tools/list (overpermission).
P1 fixes:
- proxy/utils.py: _convert_mcp_hook_response_to_kwargs merges rather than
replaces extra_headers, preserving headers from prior guardrails.
- mcp_server_manager.py: warns when hook injects Authorization alongside a
server-configured authentication_token (previously silent).
- mcp_server_manager.py: pre_call_tool_check now accepts raw_headers and
extracts incoming_bearer_token so FR-5 verification has the raw token.
- proxy/utils.py: remove stray inline import inspect inside loop (pre-existing
lint error, now cleaned up).
Tests: 43 passing (28 new tests covering all FR flags + P1 fixes).
* feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes (core)
Remaining files from the FR implementation:
mcp_jwt_signer.py — full rewrite with all new params:
FR-5: access_token_discovery_uri, token_introspection_endpoint,
verify_issuer, verify_audience + _verify_incoming_jwt(),
_introspect_opaque_token()
FR-12: end_user_claim_sources ordered resolution chain
FR-13: add_claims, set_claims, remove_claims
FR-14: channel_token_audience, channel_token_ttl → x-mcp-channel-token
FR-15: required_claims (raises 403), optional_claims (passthrough)
FR-9: debug_headers → x-litellm-mcp-debug
FR-10: allowed_scopes; tool-call JWTs no longer over-grant tools/list
mcp_server_manager.py:
- pre_call_tool_check gains raw_headers param to extract incoming_bearer_token
- Silent Authorization override warning fixed: now fires when server has
authentication_token AND hook injects Authorization
tests/test_mcp_jwt_signer.py:
28 new tests covering all FR flags + P1 fixes (43 total, all passing)
* fix(mcp_jwt_signer): address pre-landing review issues
- Remove stale TODO comment on UserAPIKeyAuth.jwt_claims — the field is
already populated and consumed by MCPJWTSigner in the same PR
- Fix _get_oidc_discovery to only cache the OIDC discovery doc when
jwks_uri is present; a malformed/empty doc now retries on the next
request instead of being permanently cached until proxy restart
- Add FR-5 test coverage for _fetch_jwks (cache hit/miss),
_get_oidc_discovery (cache/no-cache on bad doc), _verify_incoming_jwt
(valid token, expired token), _introspect_opaque_token (active,
inactive, no endpoint), and the end-to-end 401 hook path — 53 tests
total, all passing
* docs(mcp_zero_trust): rewrite as use-case guide covering all new JWT signer features
Add scenario-driven sections for each new config area:
- Verify+re-sign with Okta/Azure AD (access_token_discovery_uri,
end_user_claim_sources, token_introspection_endpoint)
- Enforcing caller attributes with required_claims / optional_claims
- Adding metadata via add_claims / set_claims / remove_claims
- Two-token model for AWS Bedrock AgentCore Gateway
(channel_token_audience / channel_token_ttl)
- Controlling scopes with allowed_scopes
- Debugging JWT rejections with debug_headers
Update JWT claims table to reflect configurable sub (end_user_claim_sources)
* fix(mcp_jwt_signer): wire all config.yaml params through initialize_guardrail
The factory was only passing issuer/audience/ttl_seconds to MCPJWTSigner.
All FR-5/9/10/12/13/14/15 params (access_token_discovery_uri,
end_user_claim_sources, add/set/remove_claims, channel_token_audience,
required/optional_claims, debug_headers, allowed_scopes, etc.) were
silently dropped, making every advertised advanced feature non-functional
when loaded from config.yaml.
Add regression test that asserts every param is wired through correctly.
* docs(mcp_zero_trust): add hero image
* docs(mcp_zero_trust): apply Linear-style edits
- Lead with the problem (unsigned direct calls bypass access controls)
- Shorter statement section headers instead of question-form headers
- Move diagram/OIDC discovery block after the reader is bought in
- Add 'read further only if you need to' callout after basic setup
- Two-token section now opens from the user problem not product jargon
- Add concrete 403 error response example in required_claims section
- Debug section opens from the symptom (MCP server returning 401)
- Lowercase claims reference header for consistency
* fix(mcp_jwt_signer): fix algorithm confusion attack + add OIDC discovery 24h TTL
- Remove alg from unverified JWT header; use signing_jwk.algorithm_name from JWKS key instead.
Reading alg from attacker-controlled headers enables alg:none / HS256 confusion attacks.
- Add _oidc_discovery_fetched_at timestamp and _OIDC_DISCOVERY_TTL = 86400 (24h).
Without a TTL the cached discovery doc never refreshes, so IdP key rotation is invisible.
---------
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
* fix(ci): stabilize CI - formatting, type errors, test polling, security CVEs, router bug, batch resolution
Fix 1: Run Black formatter on 35 files
Fix 2: Fix MyPy type errors:
- setup_wizard.py: add type annotation for 'selected' set variable
- user_api_key_auth.py: remove redundant type annotation on jwt_claims reassignment
Fix 3: Fix spend accuracy test burst 2 polling to wait for expected total
spend instead of just 'any increase' from burst 2
Fix 4: Bump Next.js 16.1.6 -> 16.1.7 to fix CVE-2026-27978, CVE-2026-27979,
CVE-2026-27980, CVE-2026-29057
Fix 5: Fix router _pre_call_checks model variable being overwritten inside
loop, causing wrong model lookups on subsequent deployments. Use local
_deployment_model variable instead.
Fix 6: Add missing resolve_output_file_ids_to_unified call in batch retrieve
non-terminal-to-terminal path (matching the terminal path behavior)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* chore: regenerate poetry.lock to sync with pyproject.toml
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: format merged files from main and regenerate poetry.lock
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(mypy): annotate jwt_claims as Optional[dict] to fix type incompatibility
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): update router region test to use gpt-4.1-mini (fix flaky model lookup)
Replace deprecated gpt-3.5-turbo-1106 with gpt-4.1-mini + mock_response in
test_router_region_pre_call_check, following the same pattern used in commit
717d37cc5b for test_router_context_window_check_pre_call_check_out_group.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* ci: retry flaky logging_testing (async event loop race condition)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): aggregate all mock calls in langfuse e2e test to fix race condition
The _verify_langfuse_call helper only inspected the last mock call
(mock_post.call_args), but the Langfuse SDK may split trace-create and
generation-create events across separate HTTP flush cycles. This caused
an IndexError when the last call's batch contained only one event type.
Fix: iterate over mock_post.call_args_list to collect batch items from
ALL calls. Also add a safety assertion after filtering by trace_id and
mark all langfuse e2e tests with @pytest.mark.flaky(retries=3) as an
extra safety net for any residual timing issues.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): black formatting + update OpenAPI compliance tests for spec changes
- Apply Black 26.x formatting to litellm_logging.py (parenthesized style)
- Update test_input_types_match_spec to follow $ref to InteractionsInput schema
(Google updated their OpenAPI spec to use $ref instead of inline oneOf)
- Update test_content_schema_uses_discriminator to handle discriminator without
explicit mapping (Google removed the mapping key from Content discriminator)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* revert: undo incorrect Black 26.x formatting on litellm_logging.py
The file was correctly formatted for Black 23.12.1 (the version pinned
in pyproject.toml). The previous commit applied Black 26.x formatting
which was incompatible with the CI's Black version.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): deduplicate and sort langfuse batch events after aggregation
The Langfuse SDK may send the same event (e.g., trace-create) in
multiple flush cycles, causing duplicates when we aggregate from all
mock calls. After filtering by trace_id, deduplicate by keeping only
the first event of each type, then sort to ensure trace-create is at
index 0 and generation-create at index 1.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
1452 lines
43 KiB
JavaScript
1452 lines
43 KiB
JavaScript
/**
|
|
* Creating a sidebar enables you to:
|
|
- create an ordered group of docs
|
|
- render a sidebar for each doc of that group
|
|
- provide next/previous navigation
|
|
|
|
The sidebars can be generated from the filesystem, or explicitly defined here.
|
|
|
|
Create as many sidebars as you want.
|
|
*/
|
|
|
|
// @ts-check
|
|
|
|
/** @type {import('@docusaurus/plugin-content-docs').SidebarsConfig} */
|
|
const sidebars = {
|
|
// // By default, Docusaurus generates a sidebar from the docs folder structure
|
|
integrationsSidebar: [
|
|
{ type: "doc", id: "integrations/index" },
|
|
{ type: "doc", id: "integrations/community" },
|
|
{
|
|
type: "category",
|
|
label: "Observability",
|
|
link: { type: "doc", id: "integrations/observability_index" },
|
|
items: [
|
|
{
|
|
type: "category",
|
|
label: "Contributing to Integrations",
|
|
items: [
|
|
{
|
|
type: "autogenerated",
|
|
dirName: "contribute_integration"
|
|
}
|
|
],
|
|
},
|
|
{
|
|
type: "autogenerated",
|
|
dirName: "observability"
|
|
},
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Guardrail Providers",
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Guardrail Providers",
|
|
description: "Add safety and content filtering to LLM calls",
|
|
slug: "/guardrail_providers"
|
|
},
|
|
items: [
|
|
...[
|
|
"proxy/guardrails/qualifire",
|
|
"proxy/guardrails/aim_security",
|
|
"proxy/guardrails/onyx_security",
|
|
"proxy/guardrails/aporia_api",
|
|
"proxy/guardrails/azure_content_guardrail",
|
|
"proxy/guardrails/bedrock",
|
|
"proxy/guardrails/crowdstrike_aidr",
|
|
"proxy/guardrails/enkryptai",
|
|
"proxy/guardrails/ibm_guardrails",
|
|
"proxy/guardrails/grayswan",
|
|
"proxy/guardrails/hiddenlayer",
|
|
"proxy/guardrails/lasso_security",
|
|
"proxy/guardrails/guardrails_ai",
|
|
"proxy/guardrails/lakera_ai",
|
|
"proxy/guardrails/model_armor",
|
|
"proxy/guardrails/noma_security",
|
|
"proxy/guardrails/dynamoai",
|
|
"proxy/guardrails/openai_moderation",
|
|
"proxy/guardrails/pangea",
|
|
"proxy/guardrails/pillar_security",
|
|
"proxy/guardrails/pii_masking_v2",
|
|
"proxy/guardrails/panw_prisma_airs",
|
|
"proxy/guardrails/secret_detection",
|
|
"proxy/guardrails/custom_guardrail",
|
|
"proxy/guardrails/custom_code_guardrail",
|
|
"proxy/guardrails/prompt_injection",
|
|
"proxy/guardrails/tool_permission",
|
|
"proxy/guardrails/zscaler_ai_guard",
|
|
"proxy/guardrails/javelin"
|
|
].sort(),
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Alerting & Monitoring",
|
|
items: [
|
|
"proxy/alerting",
|
|
"proxy/pagerduty",
|
|
"proxy/prometheus",
|
|
"proxy/pyroscope_profiling"
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "[Beta] Prompt Management",
|
|
items: [
|
|
{
|
|
type: "category",
|
|
label: "Contributing to Prompt Management",
|
|
items: [
|
|
"adding_provider/generic_prompt_management_api",
|
|
]
|
|
},
|
|
"proxy/litellm_prompt_management",
|
|
"proxy/custom_prompt_management",
|
|
"proxy/native_litellm_prompt",
|
|
"proxy/prompt_management",
|
|
"proxy/arize_phoenix_prompts"
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "AI Tools",
|
|
link: {
|
|
type: "generated-index",
|
|
title: "AI Tools",
|
|
description: "Integrate LiteLLM with AI tools like OpenWebUI, Claude Code, and more",
|
|
slug: "/ai_tools"
|
|
},
|
|
items: [
|
|
"tutorials/openweb_ui",
|
|
{
|
|
type: "category",
|
|
label: "Claude Code",
|
|
items: [
|
|
"tutorials/claude_responses_api",
|
|
"tutorials/claude_code_max_subscription",
|
|
"tutorials/claude_code_byok",
|
|
"tutorials/claude_code_customer_tracking",
|
|
"tutorials/claude_code_prompt_cache_routing",
|
|
"tutorials/claude_code_websearch",
|
|
"tutorials/claude_mcp",
|
|
"tutorials/claude_non_anthropic_models",
|
|
"tutorials/claude_code_plugin_marketplace",
|
|
"tutorials/claude_code_beta_headers",
|
|
]
|
|
},
|
|
"tutorials/opencode_integration",
|
|
"tutorials/openclaw_integration",
|
|
"tutorials/cursor_integration",
|
|
"tutorials/github_copilot_integration",
|
|
"tutorials/litellm_gemini_cli",
|
|
"tutorials/litellm_qwen_code_cli",
|
|
"tutorials/openai_codex",
|
|
"tutorials/retool_assist",
|
|
"tutorials/cost_tracking_coding"
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Agent SDKs",
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Agent SDKs",
|
|
description: "Use LiteLLM with agent frameworks and SDKs",
|
|
slug: "/agent_sdks"
|
|
},
|
|
items: [
|
|
"tutorials/openai_agents_sdk",
|
|
"tutorials/claude_agent_sdk",
|
|
"tutorials/copilotkit_sdk",
|
|
"tutorials/google_adk",
|
|
"tutorials/google_genai_sdk",
|
|
"tutorials/livekit_xai_realtime",
|
|
"integrations/letta",
|
|
{ type: "doc", id: "tutorials/instructor", label: "Instructor with LiteLLM" },
|
|
{ type: "doc", id: "langchain/langchain", label: "LangChain with LiteLLM" },
|
|
"projects/openai-agents"
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Manage with AI Agents",
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Manage with AI Agents",
|
|
description: "Use AI agents to manage your LiteLLM deployment — create users, teams, keys, models, and more via natural language.",
|
|
slug: "/manage_with_ai_agents"
|
|
},
|
|
items: [
|
|
"tutorials/claude_code_skills",
|
|
]
|
|
},
|
|
|
|
],
|
|
// But you can create a sidebar manually
|
|
tutorialSidebar: [
|
|
// ════════════════════════════════════════════════════════════
|
|
// GET STARTED
|
|
// ════════════════════════════════════════════════════════════
|
|
{
|
|
type: "category",
|
|
label: "Get Started",
|
|
collapsible: false,
|
|
collapsed: false,
|
|
items: [
|
|
{ type: "doc", id: "index", label: "Quickstart" },
|
|
{ type: "link", label: "Models & Pricing", href: "https://models.litellm.ai" },
|
|
{ type: "link", label: "Changelog", href: "/release_notes" },
|
|
],
|
|
},
|
|
|
|
{
|
|
type: "category",
|
|
label: "LiteLLM Python SDK",
|
|
items: [
|
|
{
|
|
type: "link",
|
|
label: "Quick Start",
|
|
href: "/docs/#litellm-python-sdk",
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "SDK Functions",
|
|
items: [
|
|
{
|
|
type: "doc",
|
|
id: "completion/input",
|
|
label: "completion()",
|
|
},
|
|
{
|
|
type: "doc",
|
|
id: "embedding/supported_embedding",
|
|
label: "embedding()",
|
|
},
|
|
{
|
|
type: "doc",
|
|
id: "response_api",
|
|
label: "responses()",
|
|
},
|
|
{
|
|
type: "doc",
|
|
id: "text_completion",
|
|
label: "text_completion()",
|
|
},
|
|
{
|
|
type: "doc",
|
|
id: "image_generation",
|
|
label: "image_generation()",
|
|
},
|
|
{
|
|
type: "doc",
|
|
id: "audio_transcription",
|
|
label: "transcription()",
|
|
},
|
|
{
|
|
type: "doc",
|
|
id: "text_to_speech",
|
|
label: "speech()",
|
|
},
|
|
{
|
|
type: "link",
|
|
label: "All Supported Endpoints →",
|
|
href: "https://docs.litellm.ai/docs/supported_endpoints",
|
|
},
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Configuration",
|
|
items: [
|
|
"set_keys",
|
|
"proxy_auth",
|
|
"caching/all_caches",
|
|
],
|
|
},
|
|
"completion/token_usage",
|
|
"exception_mapping",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "LiteLLM AI Gateway (Proxy)",
|
|
link: {
|
|
type: "generated-index",
|
|
title: "LiteLLM AI Gateway (LLM Proxy)",
|
|
description: `OpenAI Proxy Server (LLM Gateway) to call 100+ LLMs in a unified interface & track spend, set budgets per virtual key/user`,
|
|
slug: "/simple_proxy",
|
|
},
|
|
items: [
|
|
{ type: "doc", id: "proxy/docker_quick_start", label: "Getting Started Tutorial" },
|
|
{
|
|
type: "category",
|
|
label: "Agent & MCP Gateway",
|
|
items: [
|
|
{
|
|
type: "category",
|
|
label: "A2A Agent Gateway",
|
|
items: [
|
|
"a2a",
|
|
"a2a_invoking_agents",
|
|
"a2a_agent_headers",
|
|
"a2a_cost_tracking",
|
|
"a2a_agent_permissions",
|
|
"a2a_iteration_budgets",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "MCP Gateway",
|
|
items: [
|
|
"mcp",
|
|
"mcp_usage",
|
|
"mcp_openapi",
|
|
"mcp_oauth",
|
|
"mcp_aws_sigv4",
|
|
"mcp_zero_trust",
|
|
"mcp_public_internet",
|
|
"mcp_semantic_filter",
|
|
"mcp_control",
|
|
"mcp_cost",
|
|
"mcp_guardrail",
|
|
{
|
|
type: "link",
|
|
label: "MCP Troubleshooting Guide",
|
|
href: "/docs/mcp_troubleshoot"
|
|
},
|
|
],
|
|
},
|
|
],
|
|
},
|
|
{
|
|
"type": "category",
|
|
"label": "Config.yaml",
|
|
"items": ["proxy/configs", "proxy/config_management", "proxy/config_settings"]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Setup & Deployment",
|
|
items: [
|
|
"proxy/quick_start",
|
|
"proxy/cli",
|
|
"proxy/debugging",
|
|
"proxy/error_diagnosis",
|
|
"proxy/deploy",
|
|
"proxy/health",
|
|
"proxy/master_key_rotations",
|
|
"proxy/model_management",
|
|
"proxy/prod",
|
|
"proxy/worker_startup_hooks",
|
|
"proxy/release_cycle",
|
|
],
|
|
},
|
|
{
|
|
"type": "link",
|
|
"label": "Demo LiteLLM Cloud",
|
|
"href": "https://www.litellm.ai/cloud"
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Admin UI",
|
|
items: [
|
|
"proxy/ui",
|
|
{
|
|
type: "category",
|
|
label: "Setup & SSO",
|
|
items: [
|
|
"proxy/admin_ui_sso",
|
|
"proxy/ui/ui_edit_logo",
|
|
"proxy/custom_sso",
|
|
"proxy/custom_root_ui",
|
|
"tutorials/scim_litellm",
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Models",
|
|
items: [
|
|
"proxy/ui_credentials",
|
|
"proxy/ai_hub",
|
|
"proxy/model_compare_ui",
|
|
"proxy/ui_store_model_db_setting",
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Teams & Organizations",
|
|
items: [
|
|
{
|
|
type: "link",
|
|
label: "Role-based Access Controls (RBAC) →",
|
|
href: "/docs/proxy/access_control"
|
|
},
|
|
"proxy/self_serve",
|
|
"proxy/public_teams",
|
|
"proxy/ui_project_management",
|
|
"proxy/ui/bulk_edit_users",
|
|
"proxy/ui/page_visibility",
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Observability: Usage",
|
|
items: [
|
|
"proxy/customer_usage",
|
|
"proxy/endpoint_activity",
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Logs",
|
|
items: [
|
|
"proxy/ui_logs",
|
|
"proxy/ui_spend_log_settings",
|
|
"proxy/ui_logs_sessions",
|
|
"proxy/deleted_keys_teams",
|
|
]
|
|
}
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Architecture",
|
|
items: [
|
|
"proxy/architecture",
|
|
"proxy/multi_tenant_architecture",
|
|
"proxy/control_plane_and_data_plane",
|
|
"proxy/db_deadlocks",
|
|
"proxy/db_info",
|
|
"proxy/image_handling",
|
|
"proxy/jwt_auth_arch",
|
|
"proxy/spend_logs_deletion",
|
|
"proxy/user_management_heirarchy",
|
|
"router_architecture"
|
|
],
|
|
},
|
|
{
|
|
type: "link",
|
|
label: "All Endpoints (Swagger)",
|
|
href: "https://litellm-api.up.railway.app/",
|
|
},
|
|
"proxy/enterprise",
|
|
{
|
|
type: "category",
|
|
label: "Authentication",
|
|
items: [
|
|
"proxy/virtual_keys",
|
|
"proxy/token_auth",
|
|
"proxy/service_accounts",
|
|
"proxy/access_control",
|
|
"proxy/cli_sso",
|
|
"proxy/custom_auth",
|
|
"proxy/ip_address",
|
|
"proxy/multiple_admins",
|
|
"proxy/public_routes",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Budgets + Rate Limits",
|
|
items: [
|
|
"proxy/users",
|
|
"proxy/team_budgets",
|
|
"proxy/project_management",
|
|
"proxy/ui_team_soft_budget_alerts",
|
|
"proxy/tag_budgets",
|
|
"proxy/customers",
|
|
"proxy/dynamic_rate_limit",
|
|
"proxy/rate_limit_tiers",
|
|
"proxy/temporary_budget_increase",
|
|
"proxy/budget_reset_and_tz",
|
|
],
|
|
},
|
|
"proxy/caching",
|
|
{
|
|
type: "category",
|
|
label: "Guardrails",
|
|
items: [
|
|
"proxy/guardrails/quick_start",
|
|
"proxy/guardrails/team_based_guardrails",
|
|
"proxy/guardrails/guardrail_load_balancing",
|
|
"proxy/guardrails/test_playground",
|
|
"proxy/guardrails/litellm_content_filter",
|
|
"proxy/guardrails/realtime_guardrails",
|
|
{
|
|
type: "link",
|
|
label: "Providers →",
|
|
href: "/docs/guardrail_providers",
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Contributing to Guardrails",
|
|
items: [
|
|
"adding_provider/generic_guardrail_api",
|
|
"adding_provider/simple_guardrail_tutorial",
|
|
"adding_provider/adding_guardrail_support",
|
|
]
|
|
},
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Policies",
|
|
items: [
|
|
"proxy/guardrails/guardrail_policies",
|
|
"proxy/guardrails/policy_flow_builder",
|
|
"proxy/guardrails/policy_templates",
|
|
"proxy/guardrails/policy_tags",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Create Custom Plugins",
|
|
description: "Modify requests, responses, and more",
|
|
items: [
|
|
"proxy/call_hooks",
|
|
"proxy/rules",
|
|
]
|
|
},
|
|
"proxy/management_cli",
|
|
{
|
|
type: "link",
|
|
label: "Load Balancing, Routing, Fallbacks",
|
|
href: "https://docs.litellm.ai/docs/routing-load-balancing",
|
|
},
|
|
"traffic_mirroring",
|
|
{
|
|
type: "category",
|
|
label: "Logging, Alerting, Metrics",
|
|
items: [
|
|
"proxy/dynamic_logging",
|
|
"proxy/logging",
|
|
"proxy/logging_spec",
|
|
"proxy/team_logging",
|
|
"proxy/email",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Making LLM Requests",
|
|
items: [
|
|
"proxy/user_keys",
|
|
"proxy/clientside_auth",
|
|
"proxy/request_headers",
|
|
"proxy/response_headers",
|
|
"proxy/forward_client_headers",
|
|
"proxy/model_discovery",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Model Access",
|
|
items: [
|
|
"proxy/model_access_guide",
|
|
"proxy/model_access",
|
|
"proxy/model_access_groups",
|
|
"proxy/access_groups",
|
|
"proxy/team_model_add"
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Secret Managers",
|
|
items: [
|
|
"secret_managers/overview",
|
|
"secret_managers/aws_secret_manager",
|
|
"secret_managers/aws_kms",
|
|
"secret_managers/azure_key_vault",
|
|
"secret_managers/cyberark",
|
|
"secret_managers/google_secret_manager",
|
|
"secret_managers/google_kms",
|
|
"secret_managers/hashicorp_vault",
|
|
"secret_managers/custom_secret_manager",
|
|
"oidc"
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Spend Tracking",
|
|
items: [
|
|
"proxy/cost_tracking",
|
|
"proxy/request_tags",
|
|
"proxy/custom_pricing",
|
|
"proxy/pricing_calculator",
|
|
"proxy/provider_margins",
|
|
"proxy/provider_discounts",
|
|
"proxy/sync_models_github",
|
|
"proxy/billing",
|
|
],
|
|
},
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Supported Endpoints",
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Supported Endpoints",
|
|
description:
|
|
"Learn how to deploy + call models from different providers on LiteLLM",
|
|
slug: "/supported_endpoints",
|
|
},
|
|
items: [
|
|
{
|
|
type: "link",
|
|
label: "/a2a - A2A Agent Gateway",
|
|
href: "/docs/a2a",
|
|
},
|
|
"assistants",
|
|
"audio_transcription",
|
|
"text_to_speech",
|
|
{
|
|
type: "category",
|
|
label: "/batches",
|
|
items: [
|
|
"batches",
|
|
"proxy/managed_batches",
|
|
]
|
|
},
|
|
"containers",
|
|
"container_files",
|
|
{
|
|
type: "category",
|
|
label: "/chat/completions",
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Chat Completions",
|
|
description: "Details on the completion() function",
|
|
slug: "/completion",
|
|
},
|
|
items: [
|
|
"completion/input",
|
|
"completion/output",
|
|
"completion/usage",
|
|
"completion/http_handler_config",
|
|
],
|
|
},
|
|
"text_completion",
|
|
"bedrock_converse",
|
|
"embedding/supported_embedding",
|
|
{
|
|
type: "category",
|
|
label: "/files",
|
|
items: [
|
|
"files_endpoints",
|
|
"proxy/litellm_managed_files",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "/fine_tuning",
|
|
items: [
|
|
"fine_tuning",
|
|
"proxy/managed_finetuning",
|
|
]
|
|
},
|
|
"evals_api",
|
|
"generateContent",
|
|
"apply_guardrail",
|
|
"bedrock_invoke",
|
|
"interactions",
|
|
"image_edits",
|
|
"image_generation",
|
|
"image_variations",
|
|
"videos",
|
|
"vector_store_files",
|
|
"vector_stores/create",
|
|
"vector_stores/search",
|
|
{
|
|
type: "category",
|
|
label: "/mcp - Model Context Protocol",
|
|
items: [
|
|
"mcp",
|
|
"mcp_usage",
|
|
"mcp_openapi",
|
|
"mcp_oauth",
|
|
"mcp_aws_sigv4",
|
|
"mcp_zero_trust",
|
|
"mcp_public_internet",
|
|
"mcp_semantic_filter",
|
|
"mcp_control",
|
|
"mcp_cost",
|
|
"mcp_guardrail",
|
|
"mcp_zero_trust",
|
|
"mcp_troubleshoot",
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "/v1/messages",
|
|
items: [
|
|
"anthropic_unified/index",
|
|
"anthropic_unified/structured_output",
|
|
"anthropic_unified/messages_to_responses_mapping",
|
|
]
|
|
},
|
|
"anthropic_count_tokens",
|
|
"moderation",
|
|
"ocr",
|
|
{
|
|
type: "category",
|
|
label: "Pass-through Endpoints (Anthropic SDK, etc.)",
|
|
items: [
|
|
"pass_through/intro",
|
|
"pass_through/anthropic_completion",
|
|
"pass_through/assembly_ai",
|
|
"pass_through/bedrock",
|
|
"pass_through/azure_passthrough",
|
|
"pass_through/cohere",
|
|
"pass_through/cursor",
|
|
"pass_through/google_ai_studio",
|
|
"pass_through/langfuse",
|
|
"pass_through/mistral",
|
|
"pass_through/openai_passthrough",
|
|
{
|
|
type: "category",
|
|
label: "Vertex AI",
|
|
items: [
|
|
"pass_through/vertex_ai",
|
|
"pass_through/vertex_ai_live_websocket",
|
|
"pass_through/vertex_ai_search_datastores",
|
|
]
|
|
},
|
|
"pass_through/vllm",
|
|
"proxy/pass_through",
|
|
"proxy/pass_through_guardrails"
|
|
]
|
|
},
|
|
"rag_ingest",
|
|
"rag_query",
|
|
"realtime",
|
|
"proxy/realtime_webrtc",
|
|
"rerank",
|
|
"response_api",
|
|
"response_api_compact",
|
|
{
|
|
type: "category",
|
|
label: "/search",
|
|
items: [
|
|
"search/index",
|
|
"search/perplexity",
|
|
"search/tavily",
|
|
"search/exa_ai",
|
|
"search/brave",
|
|
"search/parallel_ai",
|
|
"search/google_pse",
|
|
"search/dataforseo",
|
|
"search/firecrawl",
|
|
"search/searxng",
|
|
"search/linkup",
|
|
"search/serper",
|
|
]
|
|
},
|
|
"skills",
|
|
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Supported Models & Providers",
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Providers",
|
|
description:
|
|
"Learn how to deploy + call models from different providers on LiteLLM",
|
|
slug: "/providers",
|
|
},
|
|
items: [
|
|
{
|
|
type: "doc",
|
|
id: "provider_registration/index",
|
|
label: "Integrate as a Model Provider",
|
|
},
|
|
{
|
|
type: "doc",
|
|
id: "contributing/adding_openai_compatible_providers",
|
|
label: "Add OpenAI-Compatible Provider (JSON)",
|
|
},
|
|
{
|
|
type: "doc",
|
|
id: "provider_registration/add_model_pricing",
|
|
label: "Add Model Pricing & Context Window",
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "OpenAI",
|
|
items: [
|
|
"providers/openai",
|
|
"providers/openai/responses_api",
|
|
"providers/openai/text_to_speech",
|
|
"providers/openai/videos",
|
|
]
|
|
},
|
|
"providers/text_completion_openai",
|
|
"providers/openai_compatible",
|
|
{
|
|
type: "category",
|
|
label: "Azure OpenAI",
|
|
items: [
|
|
"providers/azure/azure",
|
|
"providers/azure/azure_responses",
|
|
"providers/azure/azure_embedding",
|
|
"providers/azure/azure_speech",
|
|
"providers/azure/videos",
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Azure AI",
|
|
items: [
|
|
"providers/azure_ai",
|
|
"providers/azure_ai/azure_model_router",
|
|
"providers/azure_ai_agents",
|
|
"providers/azure_ocr",
|
|
"providers/azure_document_intelligence",
|
|
"providers/azure_ai_speech",
|
|
"providers/azure_ai_img",
|
|
"providers/azure_ai_vector_stores",
|
|
"providers/azure_ai/azure_ai_vector_stores_passthrough",
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Vertex AI",
|
|
items: [
|
|
"providers/vertex",
|
|
"providers/vertex_ai/videos",
|
|
"providers/vertex_partner",
|
|
"providers/vertex_self_deployed",
|
|
"providers/vertex_embedding",
|
|
"providers/vertex_image",
|
|
"providers/vertex_speech",
|
|
"providers/vertex_batch",
|
|
"providers/vertex_ocr",
|
|
"providers/vertex_ai_agent_engine",
|
|
"providers/vertex_realtime",
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Google AI Studio",
|
|
items: [
|
|
"providers/gemini",
|
|
"providers/gemini/videos",
|
|
"providers/google_ai_studio/files",
|
|
"providers/google_ai_studio/image_gen",
|
|
"providers/google_ai_studio/realtime",
|
|
]
|
|
},
|
|
"providers/anthropic",
|
|
"providers/aws_sagemaker",
|
|
{
|
|
type: "category",
|
|
label: "Bedrock",
|
|
items: [
|
|
"providers/bedrock",
|
|
"providers/bedrock_embedding",
|
|
"providers/bedrock_imported",
|
|
"providers/bedrock_image_gen",
|
|
"providers/bedrock_rerank",
|
|
"providers/bedrock_agentcore",
|
|
"providers/bedrock_agents",
|
|
"providers/bedrock_writer",
|
|
"providers/bedrock_batches",
|
|
"providers/bedrock_realtime_with_audio",
|
|
"providers/aws_polly",
|
|
"providers/bedrock_vector_store",
|
|
"providers/bedrock_mantle",
|
|
]
|
|
},
|
|
"providers/litellm_proxy",
|
|
"providers/abliteration",
|
|
"providers/ai21",
|
|
"providers/aiml",
|
|
"providers/aleph_alpha",
|
|
"providers/amazon_nova",
|
|
"providers/anyscale",
|
|
"providers/apertis",
|
|
"providers/baseten",
|
|
"providers/black_forest_labs",
|
|
"providers/black_forest_labs_img_edit",
|
|
"providers/bytez",
|
|
"providers/cerebras",
|
|
"providers/chutes",
|
|
"providers/clarifai",
|
|
"providers/cloudflare_workers",
|
|
"providers/codestral",
|
|
"providers/cohere",
|
|
"providers/cometapi",
|
|
"providers/compactifai",
|
|
"providers/custom_llm_server",
|
|
"providers/dashscope",
|
|
"providers/databricks",
|
|
"providers/datarobot",
|
|
"providers/deepgram",
|
|
"providers/deepinfra",
|
|
"providers/deepseek",
|
|
"providers/docker_model_runner",
|
|
"providers/elevenlabs",
|
|
"providers/fal_ai",
|
|
"providers/featherless_ai",
|
|
"providers/fireworks_ai",
|
|
"providers/friendliai",
|
|
"providers/galadriel",
|
|
"providers/github",
|
|
"providers/github_copilot",
|
|
"providers/gmi",
|
|
"providers/chatgpt",
|
|
"providers/gradient_ai",
|
|
"providers/groq",
|
|
"providers/helicone",
|
|
"providers/heroku",
|
|
{
|
|
type: "category",
|
|
label: "HuggingFace",
|
|
items: [
|
|
"providers/huggingface",
|
|
"providers/huggingface_rerank",
|
|
]
|
|
},
|
|
"providers/hyperbolic",
|
|
"providers/infinity",
|
|
"providers/jina_ai",
|
|
"providers/lambda_ai",
|
|
"providers/langgraph",
|
|
"providers/lemonade",
|
|
"providers/llamafile",
|
|
"providers/llamagate",
|
|
"providers/lm_studio",
|
|
"providers/manus",
|
|
"providers/meta_llama",
|
|
"providers/milvus_vector_stores",
|
|
"providers/mistral",
|
|
"providers/minimax",
|
|
"providers/moonshot",
|
|
"providers/morph",
|
|
"providers/nebius",
|
|
"providers/nlp_cloud",
|
|
"providers/nano-gpt",
|
|
"providers/novita",
|
|
{ type: "doc", id: "providers/nscale", label: "Nscale (EU Sovereign)" },
|
|
{
|
|
type: "category",
|
|
label: "Nvidia NIM",
|
|
items: [
|
|
"providers/nvidia_nim",
|
|
"providers/nvidia_nim_rerank",
|
|
]
|
|
},
|
|
"providers/oci",
|
|
"providers/ollama",
|
|
"providers/openrouter",
|
|
"providers/sarvam",
|
|
"providers/ovhcloud",
|
|
{
|
|
type: "category",
|
|
label: "Perplexity AI",
|
|
items: [
|
|
"providers/perplexity",
|
|
"providers/perplexity_embedding",
|
|
]
|
|
},
|
|
"providers/petals",
|
|
"providers/poe",
|
|
"providers/publicai",
|
|
"providers/predibase",
|
|
"providers/pydantic_ai_agent",
|
|
"providers/ragflow",
|
|
"providers/recraft",
|
|
"providers/replicate",
|
|
{
|
|
type: "category",
|
|
label: "RunwayML",
|
|
items: [
|
|
"providers/runwayml/images",
|
|
"providers/runwayml/videos",
|
|
]
|
|
},
|
|
"providers/sambanova",
|
|
"providers/sap",
|
|
"providers/scaleway",
|
|
"providers/stability",
|
|
"providers/synthetic",
|
|
"providers/snowflake",
|
|
"providers/togetherai",
|
|
"providers/topaz",
|
|
"providers/triton-inference-server",
|
|
"providers/v0",
|
|
"providers/vercel_ai_gateway",
|
|
{
|
|
type: "category",
|
|
label: "vLLM",
|
|
items: [
|
|
"providers/vllm",
|
|
"providers/vllm_batches",
|
|
]
|
|
},
|
|
"providers/volcano",
|
|
"providers/voyage",
|
|
"providers/wandb_inference",
|
|
{
|
|
type: "category",
|
|
label: "WatsonX",
|
|
items: [
|
|
"providers/watsonx/index",
|
|
"providers/watsonx/audio_transcription",
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "xAI",
|
|
items: [
|
|
"providers/xai",
|
|
"providers/xai_realtime",
|
|
]
|
|
},
|
|
"providers/xiaomi_mimo",
|
|
"providers/xinference",
|
|
"providers/zai",
|
|
],
|
|
},
|
|
|
|
|
|
{
|
|
type: "category",
|
|
label: "Routing & Load Balancing",
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Routing & Load Balancing",
|
|
description: "Learn how to load balance, route, and set fallbacks for your LLM requests",
|
|
slug: "/routing-load-balancing",
|
|
},
|
|
items: [
|
|
"routing",
|
|
"scheduler",
|
|
"proxy/auto_routing",
|
|
"proxy/load_balancing",
|
|
"proxy/keys_teams_router_settings",
|
|
"proxy/provider_budget_routing",
|
|
"proxy/reliability",
|
|
"proxy/fallback_management",
|
|
"proxy/tag_routing",
|
|
"proxy/timeout",
|
|
"wildcard_routing"
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Load Testing",
|
|
items: [
|
|
"benchmarks",
|
|
"load_test_advanced",
|
|
"load_test_sdk",
|
|
"load_test_rpm",
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Contributing",
|
|
items: [
|
|
"extras/contributing_code",
|
|
{
|
|
type: "category",
|
|
label: "Adding Providers",
|
|
items: [
|
|
"contributing/adding_openai_compatible_providers",
|
|
"adding_provider/directory_structure",
|
|
"adding_provider/new_rerank_provider",
|
|
]
|
|
},
|
|
"extras/contributing",
|
|
"contributing",
|
|
]
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Extras",
|
|
items: [
|
|
"sdk_custom_pricing",
|
|
"migration",
|
|
"data_security",
|
|
"data_retention",
|
|
"proxy/security_encryption_faq",
|
|
"migration_policy",
|
|
{
|
|
type: "category",
|
|
label: "❤️ 🚅 Projects built on LiteLLM",
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Projects built on LiteLLM",
|
|
description:
|
|
"Learn how to deploy + call models from different providers on LiteLLM",
|
|
slug: "/project",
|
|
},
|
|
items: [
|
|
"projects/smolagents",
|
|
"projects/mini-swe-agent",
|
|
"projects/openai-agents",
|
|
"projects/Google ADK",
|
|
"projects/Agent Lightning",
|
|
"projects/Harbor",
|
|
"projects/GraphRAG",
|
|
"projects/Docq.AI",
|
|
"projects/PDL",
|
|
"projects/OpenInterpreter",
|
|
"projects/Elroy",
|
|
"projects/dbally",
|
|
"projects/FastREPL",
|
|
"projects/PROMPTMETHEUS",
|
|
"projects/Codium PR Agent",
|
|
"projects/Prompt2Model",
|
|
"projects/SalesGPT",
|
|
"projects/Softgen",
|
|
"projects/Quivr",
|
|
"projects/Langstream",
|
|
"projects/Otter",
|
|
"projects/YiVal",
|
|
"projects/llm_cord",
|
|
"projects/pgai",
|
|
"projects/GPTLocalhost",
|
|
"projects/HolmesGPT",
|
|
"projects/Railtracks",
|
|
],
|
|
},
|
|
"extras/code_quality",
|
|
"rules",
|
|
"proxy/team_based_routing",
|
|
"proxy/customer_routing",
|
|
"proxy_server",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Troubleshooting",
|
|
items: [
|
|
"troubleshoot/ui_issues",
|
|
"mcp_troubleshoot",
|
|
{
|
|
type: "category",
|
|
label: "Performance / Latency",
|
|
items: [
|
|
"troubleshoot/latency_overhead",
|
|
"troubleshoot/cpu_issues",
|
|
"troubleshoot/memory_issues",
|
|
"troubleshoot/spend_queue_warnings",
|
|
"troubleshoot/max_callbacks",
|
|
"troubleshoot/prisma_migrations",
|
|
],
|
|
},
|
|
"troubleshoot/pip_venv_upgrade",
|
|
"troubleshoot/rollback",
|
|
"troubleshoot",
|
|
],
|
|
},
|
|
],
|
|
};
|
|
|
|
const learnSidebar = {
|
|
learnSidebar: [
|
|
// ── Landing page ──────────────────────────────────────────────────
|
|
{ type: "doc", id: "learn/index", label: "Learn" },
|
|
{
|
|
type: "category",
|
|
label: "Start Here",
|
|
collapsible: true,
|
|
collapsed: false,
|
|
items: [
|
|
"learn/sdk_quickstart",
|
|
"learn/gateway_quickstart",
|
|
],
|
|
},
|
|
|
|
// ── Guides ────────────────────────────────────────────────────────
|
|
{
|
|
type: "category",
|
|
label: "Guides",
|
|
collapsible: true,
|
|
collapsed: false,
|
|
link: { type: "doc", id: "guides/index" },
|
|
items: [
|
|
{
|
|
type: "category",
|
|
label: "Core Requests",
|
|
collapsible: true,
|
|
collapsed: false,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Core Requests",
|
|
description: "Streaming, batching, structured outputs, and reasoning behavior",
|
|
slug: "/guides/core_request_response_patterns"
|
|
},
|
|
items: [
|
|
"completion/stream",
|
|
"completion/batching",
|
|
"completion/json_mode",
|
|
"reasoning_content",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Tool Calling",
|
|
collapsible: true,
|
|
collapsed: true,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Tool Calling",
|
|
description: "Function calling, web tools, interception patterns, computer use, code interpreter, and tool-call hygiene",
|
|
slug: "/guides/tools_integrations"
|
|
},
|
|
items: [
|
|
"completion/function_call",
|
|
"completion/web_search",
|
|
{
|
|
type: "doc",
|
|
id: "integrations/websearch_interception",
|
|
label: "Web Search Interception",
|
|
},
|
|
"completion/web_fetch",
|
|
"completion/computer_use",
|
|
"guides/code_interpreter",
|
|
"completion/message_sanitization",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Multimodal I/O",
|
|
collapsible: true,
|
|
collapsed: true,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Multimodal I/O",
|
|
description: "Vision, audio, PDFs, image generation, and video generation",
|
|
slug: "/guides/multimodal_io"
|
|
},
|
|
items: [
|
|
"completion/vision",
|
|
"completion/audio",
|
|
"completion/document_understanding",
|
|
"completion/image_generation_chat",
|
|
"proxy/veo_video_generation",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Retrieval & Knowledge",
|
|
collapsible: true,
|
|
collapsed: true,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Retrieval & Knowledge",
|
|
description: "Vector stores, file search, citations, and knowledge-base routing",
|
|
slug: "/guides/retrieval_knowledge"
|
|
},
|
|
items: [
|
|
"completion/knowledgebase",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Prompts & Context",
|
|
collapsible: true,
|
|
collapsed: true,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Prompts & Context",
|
|
description: "Prompt caching, trimming, formatting, assistant prefill, and predicted outputs",
|
|
slug: "/guides/prompts_context"
|
|
},
|
|
items: [
|
|
"completion/prefix",
|
|
"completion/predict_outputs",
|
|
"completion/message_trimming",
|
|
"completion/prompt_caching",
|
|
"completion/prompt_formatting",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Compatibility & Extensibility",
|
|
collapsible: true,
|
|
collapsed: true,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Compatibility & Extensibility",
|
|
description: "Provider-specific params, model aliases, fine-tuned models, and adapters",
|
|
slug: "/guides/compatibility_extensibility"
|
|
},
|
|
items: [
|
|
"completion/provider_specific_params",
|
|
"completion/drop_params",
|
|
"completion/model_alias",
|
|
"guides/finetuned_models",
|
|
"extras/creating_adapters",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Reliability, Testing & Spend",
|
|
collapsible: true,
|
|
collapsed: true,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Reliability, Testing & Spend",
|
|
description: "Retries, fallbacks, mock responses, and budget controls",
|
|
slug: "/guides/reliability_testing_spend"
|
|
},
|
|
items: [
|
|
"completion/mock_requests",
|
|
"completion/reliable_completions",
|
|
"budget_manager",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Security & Network",
|
|
collapsible: true,
|
|
collapsed: true,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Security & Network",
|
|
description: "SSL, custom CA bundles, HTTP proxy settings, and per-service verification",
|
|
slug: "/guides/security_network"
|
|
},
|
|
items: [
|
|
"guides/security_settings",
|
|
],
|
|
},
|
|
],
|
|
},
|
|
|
|
// ── Tutorials ─────────────────────────────────────────────────────
|
|
{
|
|
type: "category",
|
|
label: "Tutorials",
|
|
collapsible: true,
|
|
collapsed: false,
|
|
link: { type: "doc", id: "tutorials/index" },
|
|
items: [
|
|
{
|
|
type: "category",
|
|
label: "Getting Started",
|
|
collapsed: false,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Getting Started",
|
|
description: "Installation, playground, text completion, and mock completions",
|
|
slug: "/tutorials/getting_started"
|
|
},
|
|
items: [
|
|
"tutorials/installation",
|
|
"tutorials/first_playground",
|
|
"tutorials/text_completion",
|
|
"tutorials/mock_completion",
|
|
],
|
|
},
|
|
{
|
|
type: "link",
|
|
label: "Agent SDKs & Frameworks",
|
|
href: "/docs/agent_sdks",
|
|
},
|
|
{
|
|
type: "link",
|
|
label: "AI Coding Tools",
|
|
href: "/docs/ai_tools",
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Python SDK",
|
|
collapsed: true,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Python SDK",
|
|
description: "Tutorials using only the Python SDK — no proxy server required",
|
|
slug: "/tutorials/python_sdk"
|
|
},
|
|
items: [
|
|
"tutorials/gradio_integration",
|
|
"tutorials/provider_specific_params",
|
|
"tutorials/model_fallbacks",
|
|
"tutorials/fallbacks",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Provider Setup",
|
|
collapsed: true,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Provider Setup",
|
|
description: "Connect LiteLLM to Azure OpenAI, HuggingFace, TogetherAI, local models, and more",
|
|
slug: "/tutorials/provider_tutorials"
|
|
},
|
|
items: [
|
|
"tutorials/azure_openai",
|
|
"tutorials/TogetherAI_liteLLM",
|
|
"tutorials/huggingface_tutorial",
|
|
"tutorials/huggingface_codellama",
|
|
"tutorials/finetuned_chat_gpt",
|
|
"tutorials/oobabooga",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Proxy: Admin & Access",
|
|
collapsed: true,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Proxy: Admin & Access",
|
|
description: "User and team management, SSO, SCIM, and routing rules",
|
|
slug: "/tutorials/proxy_admin_access"
|
|
},
|
|
items: [
|
|
"tutorials/default_team_self_serve",
|
|
"tutorials/msft_sso",
|
|
"tutorials/scim_litellm",
|
|
"tutorials/tag_management",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Proxy: Features & Safety",
|
|
collapsed: true,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Proxy: Features & Safety",
|
|
description: "Prompt caching, passthrough APIs, realtime, guardrails, and PII masking",
|
|
slug: "/tutorials/proxy_features_safety"
|
|
},
|
|
items: [
|
|
"tutorials/prompt_caching",
|
|
"tutorials/anthropic_file_usage",
|
|
"tutorials/gemini_realtime_with_audio",
|
|
"tutorials/litellm_proxy_aporia",
|
|
"tutorials/presidio_pii_masking",
|
|
],
|
|
},
|
|
{
|
|
type: "category",
|
|
label: "Observability & Evaluation",
|
|
collapsed: true,
|
|
link: {
|
|
type: "generated-index",
|
|
title: "Observability & Evaluation",
|
|
description: "Logging, monitoring, benchmarking, and evaluation suites",
|
|
slug: "/tutorials/observability_evaluation"
|
|
},
|
|
items: [
|
|
"tutorials/elasticsearch_logging",
|
|
"tutorials/compare_llms",
|
|
"tutorials/litellm_Test_Multiple_Providers",
|
|
"tutorials/eval_suites",
|
|
"tutorials/lm_evaluation_harness",
|
|
],
|
|
},
|
|
],
|
|
},
|
|
],
|
|
};
|
|
|
|
module.exports = { ...sidebars, ...learnSidebar };
|