mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-02 06:22:48 +00:00
Revert "Merge pull request #14769 from TeddyAmkie/doc-updates-sept-2025"
This reverts commit666bcac2af, reversing changes made tob7803bcbb0.
This commit is contained in:
Ishaan Jaffer
@@ -26,7 +26,6 @@ response = completion(
|
||||
|
||||
print(response.usage)
|
||||
```
|
||||
> **Note:** LiteLLM supports endpoint bridging—if a model does not natively support a requested endpoint, LiteLLM will automatically route the call to the correct supported endpoint (such as bridging `/chat/completions` to `/responses` or vice versa) based on the model's `mode`set in `model_prices_and_context_window`.
|
||||
|
||||
## Streaming Usage
|
||||
|
||||
|
||||
@@ -1,11 +1,6 @@
|
||||
import Image from '@theme/IdealImage';
|
||||
|
||||
# Enterprise
|
||||
|
||||
:::info
|
||||
✨ SSO is free for up to 5 users. After that, an enterprise license is required. [Get Started with Enterprise here](https://www.litellm.ai/enterprise)
|
||||
:::
|
||||
|
||||
For companies that need SSO, user management and professional support for LiteLLM Proxy
|
||||
|
||||
:::info
|
||||
|
||||
@@ -13,8 +13,6 @@ This is an Enterprise only endpoint [Get Started with Enterprise here](https://c
|
||||
| Feature | Supported | Notes |
|
||||
|-------|-------|-------|
|
||||
| Supported Providers | OpenAI, Azure OpenAI, Vertex AI | - |
|
||||
|
||||
#### ⚡️See an exhaustive list of supported models and providers at [models.litellm.ai](https://models.litellm.ai/)
|
||||
| Cost Tracking | 🟡 | [Let us know if you need this](https://github.com/BerriAI/litellm/issues) |
|
||||
| Logging | ✅ | Works across all logging integrations |
|
||||
|
||||
|
||||
@@ -32,8 +32,7 @@ Next Steps 👉 [Call all supported models - e.g. Claude-2, Llama2-70b, etc.](./
|
||||
More details 👉
|
||||
|
||||
- [Completion() function details](./completion/)
|
||||
- [Overview of supported models / providers on LiteLLM](./providers/)
|
||||
- [Search all models / providers](https://models.litellm.ai/)
|
||||
- [All supported models / providers on LiteLLM](./providers/)
|
||||
- [Build your own OpenAI proxy](https://github.com/BerriAI/liteLLM-proxy/tree/main)
|
||||
|
||||
## streaming
|
||||
|
||||
@@ -18,9 +18,6 @@ LiteLLM provides image editing functionality that maps to OpenAI's `/images/edit
|
||||
| Supported LiteLLM Proxy Versions | 1.71.1+ | |
|
||||
| Supported LLM providers | **OpenAI** | Currently only `openai` is supported |
|
||||
|
||||
#### ⚡️See all supported models and providers at [models.litellm.ai](https://models.litellm.ai/)
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
### LiteLLM Python SDK
|
||||
|
||||
@@ -279,8 +279,6 @@ print(f"response: {response}")
|
||||
|
||||
## Supported Providers
|
||||
|
||||
#### ⚡️See all supported models and providers at [models.litellm.ai](https://models.litellm.ai/)
|
||||
|
||||
| Provider | Documentation Link |
|
||||
|----------|-------------------|
|
||||
| OpenAI | [OpenAI Image Generation →](./providers/openai) |
|
||||
|
||||
@@ -524,15 +524,6 @@ try:
|
||||
except OpenAIError as e:
|
||||
print(e)
|
||||
```
|
||||
### See How LiteLLM Transforms Your Requests
|
||||
|
||||
Want to understand how LiteLLM parses and normalizes your LLM API requests? Use the `/utils/transform_request` endpoint to see exactly how your request is transformed internally.
|
||||
|
||||
You can try it out now directly on our Demo App!
|
||||
Go to the [LiteLLM API docs for transform_request](https://litellm-api.up.railway.app/#/llm%20utils/transform_request_utils_transform_request_post)
|
||||
|
||||
LiteLLM will show you the normalized, provider-agnostic version of your request. This is useful for debugging, learning, and understanding how LiteLLM handles different providers and options.
|
||||
|
||||
|
||||
### Logging Observability - Log LLM Input/Output ([Docs](https://docs.litellm.ai/docs/observability/callbacks))
|
||||
LiteLLM exposes pre defined callbacks to send data to Lunary, MLflow, Langfuse, Helicone, Promptlayer, Traceloop, Slack
|
||||
|
||||
@@ -130,8 +130,6 @@ Here's the exact json output and type you can expect from all moderation calls:
|
||||
|
||||
## **Supported Providers**
|
||||
|
||||
#### ⚡️See all supported models and providers at [models.litellm.ai](https://models.litellm.ai/)
|
||||
|
||||
| Provider |
|
||||
|-------------|
|
||||
| OpenAI |
|
||||
|
||||
@@ -5,15 +5,13 @@
|
||||
liteLLM provides `input_callbacks`, `success_callbacks` and `failure_callbacks`, making it easy for you to send data to a particular provider depending on the status of your responses.
|
||||
|
||||
:::tip
|
||||
**New to LiteLLM Callbacks?**
|
||||
|
||||
- For proxy/server logging and observability, see the [Proxy Logging Guide](https://docs.litellm.ai/docs/proxy/logging).
|
||||
- To write your own callback logic, see the [Custom Callbacks Guide](https://docs.litellm.ai/docs/observability/custom_callback).
|
||||
**New to LiteLLM Callbacks?** Check out our comprehensive [Callback Management Guide](./callback_management.md) to understand when to use different callback hooks like `async_log_success_event` vs `async_post_call_success_hook`.
|
||||
:::
|
||||
|
||||
liteLLM supports:
|
||||
|
||||
### Supported Callback Integrations
|
||||
|
||||
- [Custom Callback Functions](https://docs.litellm.ai/docs/observability/custom_callback)
|
||||
- [Callback Management Guide](./callback_management.md) - **Comprehensive guide for choosing the right hooks**
|
||||
- [Lunary](https://lunary.ai/docs)
|
||||
- [Langfuse](https://langfuse.com/docs)
|
||||
- [LangSmith](https://www.langchain.com/langsmith)
|
||||
@@ -23,20 +21,9 @@ liteLLM provides `input_callbacks`, `success_callbacks` and `failure_callbacks`,
|
||||
- [Sentry](https://docs.sentry.io/platforms/python/)
|
||||
- [PostHog](https://posthog.com/docs/libraries/python)
|
||||
- [Slack](https://slack.dev/bolt-python/concepts)
|
||||
- [Arize](https://docs.arize.com/)
|
||||
- [PromptLayer](https://docs.promptlayer.com/)
|
||||
|
||||
This is **not** an extensive list. Please check the dropdown for all logging integrations.
|
||||
|
||||
### Related Cookbooks
|
||||
Try out our cookbooks for code snippets and interactive demos:
|
||||
|
||||
- [Langfuse Callback Example (Colab)](https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/logging_observability/LiteLLM_Langfuse.ipynb)
|
||||
- [Lunary Callback Example (Colab)](https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/logging_observability/LiteLLM_Lunary.ipynb)
|
||||
- [Arize Callback Example (Colab)](https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/logging_observability/LiteLLM_Arize.ipynb)
|
||||
- [Proxy + Langfuse Callback Example (Colab)](https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/logging_observability/LiteLLM_Proxy_Langfuse.ipynb)
|
||||
- [PromptLayer Callback Example (Colab)](https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/LiteLLM_PromptLayer.ipynb)
|
||||
|
||||
### Quick Start
|
||||
|
||||
```python
|
||||
|
||||
@@ -67,23 +67,6 @@ asyncio.run(completion())
|
||||
- `async_post_call_success_hook` - Access user data + modify responses
|
||||
- `async_pre_call_hook` - Modify requests before sending
|
||||
|
||||
### Example: Modifying the Response in async_post_call_success_hook
|
||||
|
||||
You can use `async_post_call_success_hook` to add custom headers or metadata to the response before it is returned to the client. For example:
|
||||
|
||||
```python
|
||||
async def async_post_call_success_hook(data, user_api_key_dict, response):
|
||||
# Add a custom header to the response
|
||||
additional_headers = getattr(response, "_hidden_params", {}).get("additional_headers", {}) or {}
|
||||
additional_headers["x-litellm-custom-header"] = "my-value"
|
||||
if not hasattr(response, "_hidden_params"):
|
||||
response._hidden_params = {}
|
||||
response._hidden_params["additional_headers"] = additional_headers
|
||||
return response
|
||||
```
|
||||
|
||||
This allows you to inject custom metadata or headers into the response for downstream consumers. You can use this pattern to pass information to clients, proxies, or observability tools.
|
||||
|
||||
## Callback Functions
|
||||
If you just want to log on a specific event (e.g. on input) - you can use callback functions.
|
||||
|
||||
|
||||
@@ -2340,39 +2340,6 @@ response = completion(
|
||||
|
||||
Make the bedrock completion call
|
||||
|
||||
---
|
||||
|
||||
### Required AWS IAM Policy for AssumeRole
|
||||
|
||||
To use `aws_role_name` (STS AssumeRole) with LiteLLM, your IAM user or role **must** have permission to call `sts:AssumeRole` on the target role. If you see an error like:
|
||||
|
||||
```
|
||||
An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::...:assumed-role/litellm-ecs-task-role/... is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::...:role/Enterprise/BedrockCrossAccountConsumer
|
||||
```
|
||||
|
||||
This means the IAM identity running LiteLLM does **not** have permission to assume the target role. You must update your IAM policy to allow this action.
|
||||
|
||||
#### Example IAM Policy
|
||||
|
||||
Replace `<TARGET_ROLE_ARN>` with the ARN of the role you want to assume (e.g., `arn:aws:iam::123456789012:role/Enterprise/BedrockCrossAccountConsumer`).
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "sts:AssumeRole",
|
||||
"Resource": "<TARGET_ROLE_ARN>"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Note:** The target role itself must also trust the calling IAM identity (via its trust policy) for AssumeRole to succeed. See [AWS AssumeRole docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-api.html) for more details.
|
||||
|
||||
---
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="sdk" label="SDK">
|
||||
|
||||
|
||||
@@ -196,19 +196,7 @@ model_list:
|
||||
vertex_location: "us-central1"
|
||||
vertex_credentials: "/path/to/service_account.json" # [OPTIONAL] Do this OR `!gcloud auth application-default login` - run this to add vertex credentials to your env
|
||||
```
|
||||
or
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gemini-pro
|
||||
litellm_params:
|
||||
model: vertex_ai/gemini-1.5-pro
|
||||
litellm_credential_name: vertex-global
|
||||
vertex_project: project-name-here
|
||||
vertex_location: global
|
||||
base_model: gemini
|
||||
model_info:
|
||||
provider: Vertex
|
||||
```
|
||||
|
||||
2. Start Proxy
|
||||
|
||||
```
|
||||
|
||||
@@ -958,19 +958,6 @@ curl http://localhost:4000/v1/chat/completions \
|
||||
|
||||
</Tabs>
|
||||
|
||||
|
||||
## Redis max_connections
|
||||
|
||||
You can set the `max_connections` parameter in your `cache_params` for Redis. This is passed directly to the Redis client and controls the maximum number of simultaneous connections in the pool. If you see errors like `No connection available`, try increasing this value:
|
||||
|
||||
```yaml
|
||||
litellm_settings:
|
||||
cache: true
|
||||
cache_params:
|
||||
type: redis
|
||||
max_connections: 100
|
||||
```
|
||||
|
||||
## Supported `cache_params` on proxy config.yaml
|
||||
|
||||
```yaml
|
||||
@@ -979,7 +966,6 @@ cache_params:
|
||||
ttl: Optional[float]
|
||||
default_in_memory_ttl: Optional[float]
|
||||
default_in_redis_ttl: Optional[float]
|
||||
max_connections: Optional[Int]
|
||||
|
||||
# Type of cache (options: "local", "redis", "s3")
|
||||
type: s3
|
||||
|
||||
@@ -50,7 +50,6 @@ litellm_settings:
|
||||
port: 6379 # The port number for the Redis cache. Required if type is "redis".
|
||||
password: "your_password" # The password for the Redis cache. Required if type is "redis".
|
||||
namespace: "litellm.caching.caching" # namespace for redis cache
|
||||
max_connections: 100 # [OPTIONAL] Set Maximum number of Redis connections. Passed directly to redis-py.
|
||||
|
||||
# Optional - Redis Cluster Settings
|
||||
redis_startup_nodes: [{"host": "127.0.0.1", "port": "7001"}]
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
# ✨ Event Hooks for SSO Login
|
||||
|
||||
:::info
|
||||
✨ SSO is free for up to 5 users. After that, an enterprise license is required. [Get Started with Enterprise here](https://www.litellm.ai/enterprise)
|
||||
|
||||
✨ This is an Enterprise only feature [Get Started with Enterprise here](https://www.litellm.ai/enterprise)
|
||||
|
||||
:::
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -84,29 +84,3 @@ LiteLLM emits the following prometheus metrics to monitor the health/status of t
|
||||
| `litellm_in_memory_spend_update_queue_size` | In-memory aggregate spend values for keys, users, teams, team members, etc.| In-Memory |
|
||||
| `litellm_redis_spend_update_queue_size` | Redis aggregate spend values for keys, users, teams, etc. | Redis |
|
||||
|
||||
|
||||
## Troubleshooting: Redis Connection Errors
|
||||
|
||||
You may see errors like:
|
||||
|
||||
```
|
||||
LiteLLM Redis Caching: async async_increment() - Got exception from REDIS No connection available., Writing value=21
|
||||
LiteLLM Redis Caching: async set_cache_pipeline() - Got exception from REDIS No connection available., Writing value=None
|
||||
```
|
||||
|
||||
This means all available Redis connections are in use, and LiteLLM cannot obtain a new connection from the pool. This can happen under high load or with many concurrent proxy requests.
|
||||
|
||||
**Solution:**
|
||||
|
||||
- Increase the `max_connections` parameter in your Redis config section in `proxy_config.yaml` to allow more simultaneous connections. For example:
|
||||
|
||||
```yaml
|
||||
litellm_settings:
|
||||
cache: True
|
||||
cache_params:
|
||||
type: redis
|
||||
max_connections: 100 # Increase as needed for your traffic
|
||||
```
|
||||
|
||||
Adjust this value based on your expected concurrency and Redis server capacity.
|
||||
|
||||
|
||||
@@ -4,10 +4,6 @@ import TabItem from '@theme/TabItem';
|
||||
|
||||
# Bedrock Guardrails
|
||||
|
||||
:::tip ⚡️
|
||||
If you haven't set up or authenticated your Bedrock provider yet, see the [Bedrock Provider Setup & Authentication Guide](../../providers/bedrock.md).
|
||||
:::
|
||||
|
||||
LiteLLM supports Bedrock guardrails via the [Bedrock ApplyGuardrail API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ApplyGuardrail.html).
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -172,9 +172,6 @@ router_settings:
|
||||
redis_host: <your redis host>
|
||||
redis_password: <your redis password>
|
||||
redis_port: 1992
|
||||
cache_params:
|
||||
type: redis
|
||||
max_connections: 100 # maximum Redis connections in the pool; tune based on expected concurrency/load
|
||||
```
|
||||
|
||||
## Router settings on config - routing_strategy, model_group_alias
|
||||
|
||||
@@ -227,7 +227,7 @@ export PROXY_LOGOUT_URL="https://www.google.com"
|
||||
<Image img={require('../../img/ui_logout.png')} style={{ width: '400px', height: 'auto' }} />
|
||||
|
||||
|
||||
### Set default max budget for internal users
|
||||
### Set max budget for internal users
|
||||
|
||||
Automatically apply budget per internal user when they sign up. By default the table will be checked every 10 minutes, for users to reset. To modify this, [see this](./users.md#reset-budgets)
|
||||
|
||||
@@ -239,10 +239,6 @@ litellm_settings:
|
||||
|
||||
This sets a max budget of $10 USD for internal users when they sign up.
|
||||
|
||||
You can also manage these settings visually in the UI:
|
||||
|
||||
<Image img={require('../../img/default_user_settings_admin_ui.png')} style={{ width: '700px', height: 'auto' }} />
|
||||
|
||||
This budget only applies to personal keys created by that user - seen under `Default Team` on the UI.
|
||||
|
||||
<Image img={require('../../img/max_budget_for_internal_users.png')} style={{ width: '500px', height: 'auto' }} />
|
||||
|
||||
@@ -27,7 +27,7 @@ Email us @ krrish@berri.ai
|
||||
## Supported Models for LiteLLM Key
|
||||
These are the models that currently work with the "sk-litellm-.." keys.
|
||||
|
||||
For a complete list of models/providers that you can call with LiteLLM, [check out our provider list](./providers/) or check out [models.litellm.ai](https://models.litellm.ai/)
|
||||
For a complete list of models/providers that you can call with LiteLLM, [check out our provider list](./providers/)
|
||||
|
||||
* OpenAI models - [OpenAI docs](./providers/openai.md)
|
||||
* gpt-4
|
||||
|
||||
@@ -109,8 +109,6 @@ curl http://0.0.0.0:4000/rerank \
|
||||
|
||||
## **Supported Providers**
|
||||
|
||||
#### ⚡️See all supported models and providers at [models.litellm.ai](https://models.litellm.ai/)
|
||||
|
||||
| Provider | Link to Usage |
|
||||
|-------------|--------------------|
|
||||
| Cohere (v1 + v2 clients) | [Usage](#quick-start) |
|
||||
|
||||
@@ -3,11 +3,8 @@ import TabItem from '@theme/TabItem';
|
||||
|
||||
# /responses [Beta]
|
||||
|
||||
|
||||
LiteLLM provides a BETA endpoint in the spec of [OpenAI's `/responses` API](https://platform.openai.com/docs/api-reference/responses)
|
||||
|
||||
Requests to /chat/completions may be bridged here automatically when the provider lacks support for that endpoint. The model’s default `mode` determines how bridging works.(see `model_prices_and_context_window`)
|
||||
|
||||
| Feature | Supported | Notes |
|
||||
|---------|-----------|--------|
|
||||
| Cost Tracking | ✅ | Works with all supported models |
|
||||
@@ -81,43 +78,6 @@ print(retrieved_response)
|
||||
# retrieved_response = await litellm.aget_responses(response_id=response_id)
|
||||
```
|
||||
|
||||
#### CANCEL a Response
|
||||
You can cancel an in-progress response (if supported by the provider):
|
||||
|
||||
```python showLineNumbers title="Cancel Response by ID"
|
||||
import litellm
|
||||
|
||||
# First, create a response
|
||||
response = litellm.responses(
|
||||
model="openai/o1-pro",
|
||||
input="Tell me a three sentence bedtime story about a unicorn.",
|
||||
max_output_tokens=100
|
||||
)
|
||||
|
||||
# Get the response ID
|
||||
response_id = response.id
|
||||
|
||||
# Cancel the response by ID
|
||||
cancel_response = litellm.cancel_responses(
|
||||
response_id=response_id
|
||||
)
|
||||
|
||||
print(cancel_response)
|
||||
|
||||
# For async usage
|
||||
# cancel_response = await litellm.acancel_responses(response_id=response_id)
|
||||
```
|
||||
|
||||
|
||||
**REST API:**
|
||||
```bash
|
||||
curl -X POST http://localhost:4000/v1/responses/response_id/cancel \
|
||||
-H "Authorization: Bearer sk-1234"
|
||||
```
|
||||
|
||||
This will attempt to cancel the in-progress response with the given ID.
|
||||
**Note:** Not all providers support response cancellation. If unsupported, an error will be raised.
|
||||
|
||||
#### DELETE a Response
|
||||
```python showLineNumbers title="Delete Response by ID"
|
||||
import litellm
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 234 KiB |
+160
-189
@@ -57,31 +57,31 @@ const sidebars = {
|
||||
type: "category",
|
||||
label: "Alerting & Monitoring",
|
||||
items: [
|
||||
"proxy/prometheus",
|
||||
"proxy/alerting",
|
||||
"proxy/pagerduty",
|
||||
"proxy/prometheus"
|
||||
]
|
||||
"proxy/pagerduty"
|
||||
].sort()
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "[Beta] Prompt Management",
|
||||
items: [
|
||||
"proxy/custom_prompt_management",
|
||||
"proxy/prompt_management",
|
||||
"proxy/native_litellm_prompt",
|
||||
"proxy/prompt_management"
|
||||
]
|
||||
"proxy/custom_prompt_management"
|
||||
].sort()
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "AI Tools (OpenWebUI, Claude Code, etc.)",
|
||||
items: [
|
||||
"tutorials/claude_responses_api",
|
||||
"tutorials/cost_tracking_coding",
|
||||
"tutorials/github_copilot_integration",
|
||||
"tutorials/openweb_ui",
|
||||
"tutorials/openai_codex",
|
||||
"tutorials/litellm_gemini_cli",
|
||||
"tutorials/litellm_qwen_code_cli",
|
||||
"tutorials/openai_codex",
|
||||
"tutorials/openweb_ui"
|
||||
"tutorials/github_copilot_integration",
|
||||
"tutorials/claude_responses_api",
|
||||
"tutorials/cost_tracking_coding",
|
||||
]
|
||||
},
|
||||
|
||||
@@ -111,63 +111,41 @@ const sidebars = {
|
||||
label: "Setup & Deployment",
|
||||
items: [
|
||||
"proxy/quick_start",
|
||||
"proxy/cli",
|
||||
"proxy/debugging",
|
||||
"proxy/deploy",
|
||||
"proxy/health",
|
||||
"proxy/master_key_rotations",
|
||||
"proxy/model_management",
|
||||
"proxy/prod",
|
||||
"proxy/cli",
|
||||
"proxy/release_cycle",
|
||||
"proxy/model_management",
|
||||
"proxy/health",
|
||||
"proxy/debugging",
|
||||
"proxy/master_key_rotations",
|
||||
],
|
||||
},
|
||||
"proxy/demo",
|
||||
{
|
||||
type: "category",
|
||||
label: "Admin UI",
|
||||
items: [
|
||||
"proxy/admin_ui_sso",
|
||||
"proxy/custom_root_ui",
|
||||
"proxy/custom_sso",
|
||||
"proxy/model_hub",
|
||||
"proxy/public_teams",
|
||||
"proxy/self_serve",
|
||||
"proxy/ui",
|
||||
"proxy/ui/bulk_edit_users",
|
||||
"proxy/ui_credentials",
|
||||
"tutorials/scim_litellm",
|
||||
{
|
||||
type: "category",
|
||||
label: "UI Logs",
|
||||
items: [
|
||||
"proxy/ui_logs",
|
||||
"proxy/ui_logs_sessions"
|
||||
]
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Architecture",
|
||||
items: [
|
||||
"proxy/architecture",
|
||||
"proxy/control_plane_and_data_plane",
|
||||
"proxy/db_deadlocks",
|
||||
"proxy/db_info",
|
||||
"proxy/image_handling",
|
||||
"proxy/jwt_auth_arch",
|
||||
"proxy/spend_logs_deletion",
|
||||
"proxy/user_management_heirarchy",
|
||||
"router_architecture"
|
||||
],
|
||||
items: ["proxy/architecture", "proxy/control_plane_and_data_plane", "proxy/db_info", "proxy/db_deadlocks", "router_architecture", "proxy/user_management_heirarchy", "proxy/jwt_auth_arch", "proxy/image_handling", "proxy/spend_logs_deletion"],
|
||||
},
|
||||
{
|
||||
type: "link",
|
||||
label: "All Endpoints (Swagger)",
|
||||
href: "https://litellm-api.up.railway.app/",
|
||||
},
|
||||
"proxy/enterprise",
|
||||
"proxy/management_cli",
|
||||
"proxy/enterprise",
|
||||
"proxy/management_cli",
|
||||
{
|
||||
type: "category",
|
||||
label: "Making LLM Requests",
|
||||
items: [
|
||||
"proxy/user_keys",
|
||||
"proxy/clientside_auth",
|
||||
"proxy/request_headers",
|
||||
"proxy/response_headers",
|
||||
"proxy/forward_client_headers",
|
||||
"proxy/model_discovery",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Authentication",
|
||||
@@ -185,25 +163,45 @@ const sidebars = {
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Budgets + Rate Limits",
|
||||
label: "Model Access",
|
||||
items: [
|
||||
"proxy/customers",
|
||||
"proxy/dynamic_rate_limit",
|
||||
"proxy/rate_limit_tiers",
|
||||
"proxy/team_budgets",
|
||||
"proxy/temporary_budget_increase",
|
||||
"proxy/users"
|
||||
],
|
||||
"proxy/model_access",
|
||||
"proxy/team_model_add"
|
||||
]
|
||||
},
|
||||
"proxy/caching",
|
||||
{
|
||||
type: "category",
|
||||
label: "Create Custom Plugins",
|
||||
description: "Modify requests, responses, and more",
|
||||
label: "Admin UI",
|
||||
items: [
|
||||
"proxy/call_hooks",
|
||||
"proxy/rules",
|
||||
]
|
||||
"proxy/ui",
|
||||
"proxy/admin_ui_sso",
|
||||
"proxy/custom_root_ui",
|
||||
"proxy/model_hub",
|
||||
"proxy/self_serve",
|
||||
"proxy/public_teams",
|
||||
"tutorials/scim_litellm",
|
||||
"proxy/custom_sso",
|
||||
"proxy/ui_credentials",
|
||||
"proxy/ui/bulk_edit_users",
|
||||
{
|
||||
type: "category",
|
||||
label: "UI Logs",
|
||||
items: [
|
||||
"proxy/ui_logs",
|
||||
"proxy/ui_logs_sessions"
|
||||
]
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Spend Tracking",
|
||||
items: ["proxy/cost_tracking", "proxy/custom_pricing", "proxy/billing",],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Budgets + Rate Limits",
|
||||
items: ["proxy/users", "proxy/temporary_budget_increase", "proxy/rate_limit_tiers", "proxy/team_budgets", "proxy/dynamic_rate_limit", "proxy/customers"],
|
||||
},
|
||||
{
|
||||
type: "link",
|
||||
@@ -214,32 +212,13 @@ const sidebars = {
|
||||
type: "category",
|
||||
label: "Logging, Alerting, Metrics",
|
||||
items: [
|
||||
"proxy/dynamic_logging",
|
||||
"proxy/logging",
|
||||
"proxy/logging_spec",
|
||||
"proxy/team_logging"
|
||||
"proxy/team_logging",
|
||||
"proxy/dynamic_logging"
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Making LLM Requests",
|
||||
items: [
|
||||
"proxy/user_keys",
|
||||
"proxy/clientside_auth",
|
||||
"proxy/request_headers",
|
||||
"proxy/response_headers",
|
||||
"proxy/forward_client_headers",
|
||||
"proxy/model_discovery",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Model Access",
|
||||
items: [
|
||||
"proxy/model_access",
|
||||
"proxy/team_model_add"
|
||||
]
|
||||
},
|
||||
|
||||
{
|
||||
type: "category",
|
||||
label: "Secret Managers",
|
||||
@@ -250,13 +229,14 @@ const sidebars = {
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Spend Tracking",
|
||||
label: "Create Custom Plugins",
|
||||
description: "Modify requests, responses, and more",
|
||||
items: [
|
||||
"proxy/billing",
|
||||
"proxy/cost_tracking",
|
||||
"proxy/custom_pricing"
|
||||
],
|
||||
"proxy/call_hooks",
|
||||
"proxy/rules",
|
||||
]
|
||||
},
|
||||
"proxy/caching",
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -270,23 +250,6 @@ const sidebars = {
|
||||
slug: "/supported_endpoints",
|
||||
},
|
||||
items: [
|
||||
"assistants",
|
||||
{
|
||||
type: "category",
|
||||
label: "/audio",
|
||||
items: [
|
||||
"audio_transcription",
|
||||
"text_to_speech",
|
||||
]
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "/batches",
|
||||
items: [
|
||||
"batches",
|
||||
"proxy/managed_batches",
|
||||
]
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "/chat/completions",
|
||||
@@ -303,8 +266,57 @@ const sidebars = {
|
||||
"completion/http_handler_config",
|
||||
],
|
||||
},
|
||||
"response_api",
|
||||
"text_completion",
|
||||
"embedding/supported_embedding",
|
||||
"anthropic_unified",
|
||||
"mcp",
|
||||
"generateContent",
|
||||
{
|
||||
type: "category",
|
||||
label: "/images",
|
||||
items: [
|
||||
"image_generation",
|
||||
"image_edits",
|
||||
"image_variations",
|
||||
]
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "/audio",
|
||||
"items": [
|
||||
"audio_transcription",
|
||||
"text_to_speech",
|
||||
]
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "/vector_stores",
|
||||
items: [
|
||||
"vector_stores/search",
|
||||
]
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Pass-through Endpoints (Anthropic SDK, etc.)",
|
||||
items: [
|
||||
"pass_through/intro",
|
||||
"pass_through/vertex_ai",
|
||||
"pass_through/google_ai_studio",
|
||||
"pass_through/cohere",
|
||||
"pass_through/vllm",
|
||||
"pass_through/mistral",
|
||||
"pass_through/openai_passthrough",
|
||||
"pass_through/anthropic_completion",
|
||||
"pass_through/bedrock",
|
||||
"pass_through/assembly_ai",
|
||||
"pass_through/langfuse",
|
||||
"proxy/pass_through",
|
||||
],
|
||||
},
|
||||
"rerank",
|
||||
"assistants",
|
||||
|
||||
{
|
||||
type: "category",
|
||||
label: "/files",
|
||||
@@ -313,6 +325,15 @@ const sidebars = {
|
||||
"proxy/litellm_managed_files",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "/batches",
|
||||
items: [
|
||||
"batches",
|
||||
"proxy/managed_batches",
|
||||
]
|
||||
},
|
||||
"realtime",
|
||||
{
|
||||
type: "category",
|
||||
label: "/fine_tuning",
|
||||
@@ -321,48 +342,8 @@ const sidebars = {
|
||||
"proxy/managed_finetuning",
|
||||
]
|
||||
},
|
||||
"generateContent",
|
||||
"apply_guardrail",
|
||||
{
|
||||
type: "category",
|
||||
label: "/images",
|
||||
items: [
|
||||
"image_edits",
|
||||
"image_generation",
|
||||
"image_variations",
|
||||
]
|
||||
},
|
||||
"mcp",
|
||||
"moderation",
|
||||
{
|
||||
type: "category",
|
||||
label: "Pass-through Endpoints (Anthropic SDK, etc.)",
|
||||
items: [
|
||||
"pass_through/intro",
|
||||
"pass_through/anthropic_completion",
|
||||
"pass_through/assembly_ai",
|
||||
"pass_through/bedrock",
|
||||
"pass_through/cohere",
|
||||
"pass_through/google_ai_studio",
|
||||
"pass_through/langfuse",
|
||||
"pass_through/mistral",
|
||||
"pass_through/openai_passthrough",
|
||||
"pass_through/vertex_ai",
|
||||
"pass_through/vllm",
|
||||
"proxy/pass_through"
|
||||
]
|
||||
},
|
||||
"realtime",
|
||||
"rerank",
|
||||
"response_api",
|
||||
"anthropic_unified",
|
||||
{
|
||||
type: "category",
|
||||
label: "/vector_stores",
|
||||
items: [
|
||||
"vector_stores/search",
|
||||
]
|
||||
},
|
||||
"apply_guardrail",
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -519,32 +500,33 @@ const sidebars = {
|
||||
type: "category",
|
||||
label: "Guides",
|
||||
items: [
|
||||
"completion/audio",
|
||||
"completion/batching",
|
||||
"completion/computer_use",
|
||||
"completion/document_understanding",
|
||||
"completion/drop_params",
|
||||
"completion/function_call",
|
||||
"completion/image_generation_chat",
|
||||
"completion/json_mode",
|
||||
"completion/knowledgebase",
|
||||
"completion/message_trimming",
|
||||
"completion/model_alias",
|
||||
"completion/mock_requests",
|
||||
"completion/predict_outputs",
|
||||
"completion/prefix",
|
||||
"completion/prompt_caching",
|
||||
"completion/prompt_formatting",
|
||||
"completion/reliable_completions",
|
||||
"completion/stream",
|
||||
"completion/provider_specific_params",
|
||||
"completion/vision",
|
||||
"completion/web_search",
|
||||
"exception_mapping",
|
||||
"completion/provider_specific_params",
|
||||
"guides/finetuned_models",
|
||||
"guides/security_settings",
|
||||
"completion/audio",
|
||||
"completion/image_generation_chat",
|
||||
"completion/web_search",
|
||||
"completion/document_understanding",
|
||||
"completion/vision",
|
||||
"completion/json_mode",
|
||||
"reasoning_content",
|
||||
"completion/computer_use",
|
||||
"completion/prompt_caching",
|
||||
"completion/predict_outputs",
|
||||
"completion/knowledgebase",
|
||||
"completion/prefix",
|
||||
"completion/drop_params",
|
||||
"completion/prompt_formatting",
|
||||
"completion/stream",
|
||||
"completion/message_trimming",
|
||||
"completion/function_call",
|
||||
"completion/model_alias",
|
||||
"completion/batching",
|
||||
"completion/mock_requests",
|
||||
"completion/reliable_completions",
|
||||
"proxy/veo_video_generation",
|
||||
"reasoning_content"
|
||||
|
||||
]
|
||||
},
|
||||
|
||||
@@ -557,37 +539,26 @@ const sidebars = {
|
||||
description: "Learn how to load balance, route, and set fallbacks for your LLM requests",
|
||||
slug: "/routing-load-balancing",
|
||||
},
|
||||
items: [
|
||||
"routing",
|
||||
"scheduler",
|
||||
"proxy/auto_routing",
|
||||
"proxy/load_balancing",
|
||||
"proxy/provider_budget_routing",
|
||||
"proxy/reliability",
|
||||
"proxy/tag_routing",
|
||||
"proxy/timeout",
|
||||
"wildcard_routing"
|
||||
],
|
||||
items: ["routing", "scheduler", "proxy/load_balancing", "proxy/reliability", "proxy/timeout", "proxy/auto_routing", "proxy/tag_routing", "proxy/provider_budget_routing", "wildcard_routing"],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "LiteLLM Python SDK",
|
||||
items: [
|
||||
"set_keys",
|
||||
"budget_manager",
|
||||
"caching/all_caches",
|
||||
"completion/token_usage",
|
||||
"sdk/headers",
|
||||
"sdk_custom_pricing",
|
||||
"embedding/async_embedding",
|
||||
"embedding/moderation",
|
||||
"budget_manager",
|
||||
"caching/all_caches",
|
||||
"migration",
|
||||
"sdk_custom_pricing",
|
||||
{
|
||||
type: "category",
|
||||
label: "LangChain, LlamaIndex, Instructor Integration",
|
||||
items: ["langchain/langchain", "tutorials/instructor"],
|
||||
}
|
||||
},
|
||||
],
|
||||
},
|
||||
|
||||
|
||||
@@ -88,11 +88,11 @@ class ModelArmorGuardrail(CustomGuardrail, VertexBase):
|
||||
def _create_sanitize_request(
|
||||
self, content: str, source: Literal["user_prompt", "model_response"]
|
||||
) -> dict:
|
||||
"""Create request body for Model Armor API."""
|
||||
"""Create request body for Model Armor API with correct camelCase field names."""
|
||||
if source == "user_prompt":
|
||||
return {"user_prompt_data": {"text": content}}
|
||||
return {"userPromptData": {"text": content}}
|
||||
else:
|
||||
return {"model_response_data": {"text": content}}
|
||||
return {"modelResponseData": {"text": content}}
|
||||
|
||||
def _extract_content_from_response(
|
||||
self, response: Union[Any, ModelResponse]
|
||||
@@ -119,11 +119,16 @@ class ModelArmorGuardrail(CustomGuardrail, VertexBase):
|
||||
|
||||
async def make_model_armor_request(
|
||||
self,
|
||||
content: str,
|
||||
source: Literal["user_prompt", "model_response"],
|
||||
content: Optional[str] = None,
|
||||
source: Literal["user_prompt", "model_response"] = "user_prompt",
|
||||
request_data: Optional[dict] = None,
|
||||
file_bytes: Optional[bytes] = None,
|
||||
file_type: Optional[str] = None,
|
||||
) -> dict:
|
||||
"""Make request to Model Armor API."""
|
||||
"""
|
||||
Make request to Model Armor API. Supports both text and file prompt sanitization.
|
||||
If file_bytes and file_type are provided, file prompt sanitization is performed.
|
||||
"""
|
||||
# Get access token using VertexBase auth
|
||||
access_token, resolved_project_id = await self._ensure_access_token_async(
|
||||
credentials=self.credentials,
|
||||
@@ -143,7 +148,14 @@ class ModelArmorGuardrail(CustomGuardrail, VertexBase):
|
||||
url = f"{endpoint}/v1/projects/{self.project_id}/locations/{self.location}/templates/{self.template_id}:sanitizeModelResponse"
|
||||
|
||||
# Create request body
|
||||
body = self._create_sanitize_request(content, source)
|
||||
if file_bytes is not None and file_type is not None:
|
||||
body = self.sanitize_file_prompt(file_bytes, file_type, source)
|
||||
elif content is not None:
|
||||
body = self._create_sanitize_request(content, source)
|
||||
else:
|
||||
raise ValueError(
|
||||
"Either content or file_bytes and file_type must be provided."
|
||||
)
|
||||
|
||||
# Set headers
|
||||
headers = {
|
||||
@@ -189,57 +201,110 @@ class ModelArmorGuardrail(CustomGuardrail, VertexBase):
|
||||
return await json_response
|
||||
return json_response
|
||||
|
||||
def sanitize_file_prompt(
|
||||
self, file_bytes: bytes, file_type: str, source: str = "user_prompt"
|
||||
) -> dict:
|
||||
"""
|
||||
Helper to build the request body for file prompt sanitization for Model Armor.
|
||||
file_type should be one of: PLAINTEXT_UTF8, PDF, WORD_DOCUMENT, EXCEL_DOCUMENT, POWERPOINT_DOCUMENT, TXT, CSV
|
||||
Returns the request body dict.
|
||||
"""
|
||||
import base64
|
||||
|
||||
base64_data = base64.b64encode(file_bytes).decode("utf-8")
|
||||
if source == "user_prompt":
|
||||
return {
|
||||
"userPromptData": {
|
||||
"byteItem": {"byteDataType": file_type, "byteData": base64_data}
|
||||
}
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"modelResponseData": {
|
||||
"byteItem": {"byteDataType": file_type, "byteData": base64_data}
|
||||
}
|
||||
}
|
||||
|
||||
def _should_block_content(self, armor_response: dict) -> bool:
|
||||
"""Check if Model Armor response indicates content should be blocked."""
|
||||
# Check the sanitizationResult from Model Armor API
|
||||
"""Check if Model Armor response indicates content should be blocked, including both inspectResult and deidentifyResult."""
|
||||
sanitization_result = armor_response.get("sanitizationResult", {})
|
||||
filter_results = sanitization_result.get("filterResults", {})
|
||||
|
||||
# Check blocking filters (these should cause the request to be blocked)
|
||||
# RAI (Responsible AI) filters
|
||||
rai_results = filter_results.get("rai", {}).get("raiFilterResult", {})
|
||||
if rai_results.get("matchState") == "MATCH_FOUND":
|
||||
return True
|
||||
|
||||
# Prompt injection and jailbreak filters
|
||||
pi_jailbreak = filter_results.get("piAndJailbreakFilterResult", {})
|
||||
if pi_jailbreak.get("matchState") == "MATCH_FOUND":
|
||||
return True
|
||||
|
||||
# Malicious URI filters
|
||||
malicious_uri = filter_results.get("maliciousUriFilterResult", {})
|
||||
if malicious_uri.get("matchState") == "MATCH_FOUND":
|
||||
return True
|
||||
|
||||
# CSAM filters
|
||||
csam = filter_results.get("csamFilterFilterResult", {})
|
||||
if csam.get("matchState") == "MATCH_FOUND":
|
||||
return True
|
||||
|
||||
# Virus scan filters
|
||||
virus_scan = filter_results.get("virusScanFilterResult", {})
|
||||
if virus_scan.get("matchState") == "MATCH_FOUND":
|
||||
return True
|
||||
# filterResults can be a dict (named keys) or a list (array of filter result dicts)
|
||||
filter_result_items = []
|
||||
if isinstance(filter_results, dict):
|
||||
filter_result_items = [filter_results]
|
||||
elif isinstance(filter_results, list):
|
||||
filter_result_items = filter_results
|
||||
|
||||
for filt in filter_result_items:
|
||||
# Check RAI, PI/Jailbreak, Malicious URI, CSAM, Virus scan as before
|
||||
if filt.get("raiFilterResult", {}).get("matchState") == "MATCH_FOUND":
|
||||
return True
|
||||
if (
|
||||
filt.get("piAndJailbreakFilterResult", {}).get("matchState")
|
||||
== "MATCH_FOUND"
|
||||
):
|
||||
return True
|
||||
if (
|
||||
filt.get("maliciousUriFilterResult", {}).get("matchState")
|
||||
== "MATCH_FOUND"
|
||||
):
|
||||
return True
|
||||
if (
|
||||
filt.get("csamFilterFilterResult", {}).get("matchState")
|
||||
== "MATCH_FOUND"
|
||||
):
|
||||
return True
|
||||
if filt.get("virusScanFilterResult", {}).get("matchState") == "MATCH_FOUND":
|
||||
return True
|
||||
# Check sdpFilterResult for both inspectResult and deidentifyResult
|
||||
sdp = filt.get("sdpFilterResult")
|
||||
if sdp:
|
||||
if sdp.get("inspectResult", {}).get("matchState") == "MATCH_FOUND":
|
||||
return True
|
||||
if sdp.get("deidentifyResult", {}).get("matchState") == "MATCH_FOUND":
|
||||
return True
|
||||
# Fallback dict code removed; all cases handled above
|
||||
return False
|
||||
|
||||
def _get_sanitized_content(self, armor_response: dict) -> Optional[str]:
|
||||
"""Extract sanitized content from Model Armor response."""
|
||||
# Model Armor returns sanitized content in the sanitizationResult
|
||||
sanitization_result = armor_response.get("sanitizationResult", {})
|
||||
"""
|
||||
Get the sanitized content from a Model Armor response, if available.
|
||||
Looks for sanitized text in deidentifyResult, and falls back to root-level fields if not found.
|
||||
"""
|
||||
result = armor_response.get("sanitizationResult", {})
|
||||
filter_results = result.get("filterResults", {})
|
||||
|
||||
# Check for sdp structure (for deidentification)
|
||||
filter_results = sanitization_result.get("filterResults", {})
|
||||
sdp = filter_results.get("sdp", {}).get("sdpFilterResult")
|
||||
# filterResults can be a dict (single filter) or a list (multiple filters)
|
||||
filters = (
|
||||
[filter_results]
|
||||
if isinstance(filter_results, dict)
|
||||
else filter_results
|
||||
if isinstance(filter_results, list)
|
||||
else []
|
||||
)
|
||||
|
||||
if sdp is not None:
|
||||
# Model Armor returns sanitized text under deidentifyResult in sdp
|
||||
deidentify_result = sdp.get("deidentifyResult", {})
|
||||
sanitized_text = deidentify_result.get("data", {}).get("text", "")
|
||||
if deidentify_result.get("matchState") == "MATCH_FOUND" and sanitized_text:
|
||||
return sanitized_text
|
||||
# Prefer sanitized text from deidentifyResult if present
|
||||
for filter_entry in filters:
|
||||
sdp = filter_entry.get("sdpFilterResult")
|
||||
if sdp:
|
||||
deid = sdp.get("deidentifyResult", {})
|
||||
sanitized = deid.get("data", {}).get("text", "")
|
||||
# If Model Armor found something and returned a sanitized version, use it
|
||||
if deid.get("matchState") == "MATCH_FOUND" and sanitized:
|
||||
return sanitized
|
||||
|
||||
# Fallback to checking root level
|
||||
# If no deidentifyResult, optionally check for inspectResult (rare, but could have findings)
|
||||
for filter_entry in filters:
|
||||
sdp = filter_entry.get("sdpFilterResult")
|
||||
if sdp:
|
||||
inspect = sdp.get("inspectResult", {})
|
||||
# If Model Armor flagged something but didn't sanitize, return None
|
||||
if inspect.get("matchState") == "MATCH_FOUND":
|
||||
return None
|
||||
|
||||
# Fallback: if Model Armor put sanitized text at the root, use it
|
||||
return armor_response.get("sanitizedText") or armor_response.get("text")
|
||||
|
||||
def _process_response(
|
||||
|
||||
@@ -12,11 +12,6 @@
|
||||
- For installation and configuration, see: [Self-hosting guided](https://docs.litellm.ai/docs/proxy/deploy)
|
||||
- **Telemetry** We run no telemetry when you self host LiteLLM
|
||||
|
||||
|
||||
:::info
|
||||
✨ SSO is free for up to 5 users. After that, an enterprise license is required. [Get Started with Enterprise here](https://www.litellm.ai/enterprise)
|
||||
:::
|
||||
|
||||
### LiteLLM Cloud
|
||||
|
||||
- We encrypt all data stored using your `LITELLM_MASTER_KEY` and in transit using TLS.
|
||||
|
||||
Reference in New Issue
Block a user