diff --git a/docs/my-website/docs/proxy/pass_through.md b/docs/my-website/docs/proxy/pass_through.md index 1dfdf4166e..3a9e4028c1 100644 --- a/docs/my-website/docs/proxy/pass_through.md +++ b/docs/my-website/docs/proxy/pass_through.md @@ -6,51 +6,47 @@ import TabItem from '@theme/TabItem'; Route requests from your LiteLLM proxy to any external API. Perfect for custom models, image generation APIs, or any service you want to proxy through LiteLLM. -Onboard third-party endpoints like Bria API and Mistral OCR, set a cost per request, and give your developers access +Onboard third-party endpoints like Bria API and Mistral OCR, set a cost per request, and give your developers access. -## Usage +## Quick Start with UI (Recommended) -In this example we will onboard the [Bria API](https://docs.bria.ai/image-generation/endpoints/text-to-image-base) and set a cost per request. +The easiest way to create pass through endpoints is through the LiteLLM UI. In this example, we'll onboard the [Bria API](https://docs.bria.ai/image-generation/endpoints/text-to-image-base) and set a cost per request. -### 1. Create a pass through route on LiteLLM +### Step 1: Create Route Mappings -### Add route mappings - -`path`: This is the route clients shoudl use when calling LiteLLM Proxy. +`path`: This is the route clients should use when calling LiteLLM Proxy. `target`: This is the URL the request will be forwarded to. -This allows for the following route mappings: +This creates the following route mappings: -- `https:///bria` will forward requests to `https://engine.prod.bria-api.com` -- `https:///v1/text-to-image/base/model` will forward requests to `https://engine.prod.bria-api.com/v1/text-to-image/base/model` -- `https:///v1/enhance_image` will forward requests to `https://engine.prod.bria-api.com/v1/enhance_image` +- `https:///bria` → `https://engine.prod.bria-api.com` +- `https:///v1/text-to-image/base/model` → `https://engine.prod.bria-api.com/v1/text-to-image/base/model` +- `https:///v1/enhance_image` → `https://engine.prod.bria-api.com/v1/enhance_image` - -### 2. Add custom headers and cost per request +### Step 2: Configure Headers and Pricing -For making requests to the Bria API, we need to add the following headers: +For the Bria API, add the required header: +- `api_token: string` -- `'api_token: string'` +### Step 3: Test Your Endpoint -### 3. Test it! - -Make the following request to the Bria API through LiteLLM Proxy +Make a request to the Bria API through your LiteLLM Proxy: ```shell curl -i -X POST \ 'http://localhost:4000/bria/v1/text-to-image/base/2.3' \ -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer ' \ + -H 'Authorization: Bearer ' \ -d '{ "prompt": "a book", "num_results": 2, @@ -58,322 +54,185 @@ curl -i -X POST \ }' ``` +--- +## Config.yaml setup +You can also create pass through endpoints using the `config.yaml` file. Here's how to add a `/v1/rerank` route that forwards to Cohere's API: - - - -### 4. View Request/Response Logs - - - - -**Example:** Add a route `/v1/rerank` that forwards requests to `https://api.cohere.com/v1/rerank` through LiteLLM Proxy - - -💡 This allows making the following Request to LiteLLM Proxy -```shell -curl --request POST \ - --url http://localhost:4000/v1/rerank \ - --header 'accept: application/json' \ - --header 'content-type: application/json' \ - --data '{ - "model": "rerank-english-v3.0", - "query": "What is the capital of the United States?", - "top_n": 3, - "documents": ["Carson City is the capital city of the American state of Nevada."] - }' -``` - -## Tutorial - Create Pass Through on Proxy config.yaml - -**Step 1** Define pass through routes on [litellm config.yaml](configs.md) +### Example Configuration ```yaml general_settings: master_key: sk-1234 pass_through_endpoints: - - path: "/v1/rerank" # route you want to add to LiteLLM Proxy Server - target: "https://api.cohere.com/v1/rerank" # URL this route should forward requests to - headers: # headers to forward to this URL - Authorization: "bearer os.environ/COHERE_API_KEY" # (Optional) Auth Header to forward to your Endpoint - content-type: application/json # (Optional) Extra Headers to pass to this endpoint + - path: "/v1/rerank" # Route on LiteLLM Proxy + target: "https://api.cohere.com/v1/rerank" # Target endpoint + headers: # Headers to forward + Authorization: "bearer os.environ/COHERE_API_KEY" + content-type: application/json accept: application/json - forward_headers: True # (Optional) Forward all headers from the incoming request to the target endpoint + forward_headers: true # Forward all incoming headers ``` -**Step 2** Start Proxy Server in detailed_debug mode +### Start and Test -```shell -litellm --config config.yaml --detailed_debug -``` -**Step 3** Make Request to pass through endpoint +1. **Start the proxy:** + ```shell + litellm --config config.yaml --detailed_debug + ``` -Here `http://localhost:4000` is your litellm proxy endpoint +2. **Make a test request:** + ```shell + curl --request POST \ + --url http://localhost:4000/v1/rerank \ + --header 'accept: application/json' \ + --header 'content-type: application/json' \ + --data '{ + "model": "rerank-english-v3.0", + "query": "What is the capital of the United States?", + "top_n": 3, + "documents": ["Carson City is the capital city of the American state of Nevada."] + }' + ``` -```shell -curl --request POST \ - --url http://localhost:4000/v1/rerank \ - --header 'accept: application/json' \ - --header 'content-type: application/json' \ - --data '{ - "model": "rerank-english-v3.0", - "query": "What is the capital of the United States?", - "top_n": 3, - "documents": ["Carson City is the capital city of the American state of Nevada.", - "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.", - "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.", - "Capitalization or capitalisation in English grammar is the use of a capital letter at the start of a word. English usage varies from capitalization in other languages.", - "Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states."] - }' -``` - - -🎉 **Expected Response** - -This request got forwarded from LiteLLM Proxy -> Defined Target URL (with headers) - -```shell +### Expected Response +```json { "id": "37103a5b-8cfb-48d3-87c7-da288bedd429", "results": [ { "index": 2, "relevance_score": 0.999071 - }, - { - "index": 4, - "relevance_score": 0.7867867 - }, - { - "index": 0, - "relevance_score": 0.32713068 } ], "meta": { - "api_version": { - "version": "1" - }, - "billed_units": { - "search_units": 1 - } + "api_version": {"version": "1"}, + "billed_units": {"search_units": 1} } } ``` +--- -## ✨ [Enterprise] - Use LiteLLM keys/authentication on Pass Through Endpoints +## ✨ Enterprise Features -Use this if you want the pass through endpoint to honour LiteLLM keys/authentication +### Authentication & Rate Limiting -This also enforces the key's rpm limits on pass-through endpoints. +Enable LiteLLM authentication and rate limiting on pass through endpoints: -Usage - set `auth: true` on the config ```yaml general_settings: master_key: sk-1234 pass_through_endpoints: - path: "/v1/rerank" target: "https://api.cohere.com/v1/rerank" - auth: true # 👈 Key change to use LiteLLM Auth / Keys + auth: true # Enable LiteLLM auth headers: Authorization: "bearer os.environ/COHERE_API_KEY" content-type: application/json - accept: application/json ``` -Test Request with LiteLLM Key - +Test with LiteLLM key: ```shell curl --request POST \ --url http://localhost:4000/v1/rerank \ - --header 'accept: application/json' \ - --header 'Authorization: Bearer sk-1234'\ + --header 'Authorization: Bearer sk-1234' \ --header 'content-type: application/json' \ - --data '{ - "model": "rerank-english-v3.0", - "query": "What is the capital of the United States?", - "top_n": 3, - "documents": ["Carson City is the capital city of the American state of Nevada.", - "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.", - "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.", - "Capitalization or capitalisation in English grammar is the use of a capital letter at the start of a word. English usage varies from capitalization in other languages.", - "Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states."] - }' + --data '{"model": "rerank-english-v3.0", "query": "test"}' ``` -## `pass_through_endpoints` Spec on config.yaml +--- -All possible values for `pass_through_endpoints` and what they mean +## Configuration Reference + +### Complete Specification -**Example config** ```yaml general_settings: pass_through_endpoints: - - path: "/v1/rerank" # route you want to add to LiteLLM Proxy Server - target: "https://api.cohere.com/v1/rerank" # URL this route should forward requests to - headers: # headers to forward to this URL - Authorization: "bearer os.environ/COHERE_API_KEY" # (Optional) Auth Header to forward to your Endpoint - content-type: application/json # (Optional) Extra Headers to pass to this endpoint - accept: application/json + - path: string # Route on LiteLLM Proxy Server + target: string # Target URL for forwarding + auth: boolean # Enable LiteLLM authentication (Enterprise) + forward_headers: boolean # Forward all incoming headers + headers: # Custom headers to add + Authorization: string # Auth header for target API + content-type: string # Request content type + accept: string # Expected response format + LANGFUSE_PUBLIC_KEY: string # For Langfuse endpoints + LANGFUSE_SECRET_KEY: string # For Langfuse endpoints + : string # Any custom header ``` -**Spec** +### Header Options +- **Authorization**: Authentication for the target API +- **content-type**: Request body format specification +- **accept**: Expected response format +- **LANGFUSE_PUBLIC_KEY/SECRET_KEY**: For Langfuse integration +- **Custom headers**: Any additional key-value pairs -* `pass_through_endpoints` *list*: A collection of endpoint configurations for request forwarding. - * `path` *string*: The route to be added to the LiteLLM Proxy Server. - * `target` *string*: The URL to which requests for this path should be forwarded. - * `headers` *object*: Key-value pairs of headers to be forwarded with the request. You can set any key value pair here and it will be forwarded to your target endpoint - * `Authorization` *string*: The authentication header for the target API. - * `content-type` *string*: The format specification for the request body. - * `accept` *string*: The expected response format from the server. - * `LANGFUSE_PUBLIC_KEY` *string*: Your Langfuse account public key - only set this when forwarding to Langfuse. - * `LANGFUSE_SECRET_KEY` *string*: Your Langfuse account secret key - only set this when forwarding to Langfuse. - * `` *string*: Pass any custom header key/value pair - * `forward_headers` *Optional(boolean)*: If true, all headers from the incoming request will be forwarded to the target endpoint. Default is `False`. +--- +## Advanced: Custom Adapters -## Custom Chat Endpoints (Anthropic/Bedrock/Vertex) +For complex integrations (like Anthropic/Bedrock clients), you can create custom adapters that translate between different API schemas. -Allow developers to call the proxy with Anthropic/boto3/etc. client sdk's. - -Test our [Anthropic Adapter](../anthropic_completion.md) for reference [**Code**](https://github.com/BerriAI/litellm/blob/fd743aaefd23ae509d8ca64b0c232d25fe3e39ee/litellm/adapters/anthropic_adapter.py#L50) - -### 1. Write an Adapter - -Translate the request/response from your custom API schema to the OpenAI schema (used by litellm.completion()) and back. - -For provider-specific params 👉 [**Provider-Specific Params**](../completion/provider_specific_params.md) +### 1. Create an Adapter ```python from litellm import adapter_completion -import litellm -from litellm import ChatCompletionRequest, verbose_logger from litellm.integrations.custom_logger import CustomLogger from litellm.types.llms.anthropic import AnthropicMessagesRequest, AnthropicResponse -import os -# What is this? -## Translates OpenAI call to Anthropic `/v1/messages` format -import json -import os -import traceback -import uuid -from typing import Literal, Optional - -import dotenv -import httpx -from pydantic import BaseModel - - -################### -# CUSTOM ADAPTER ## -################### - class AnthropicAdapter(CustomLogger): - def __init__(self) -> None: - super().__init__() - - def translate_completion_input_params( - self, kwargs - ) -> Optional[ChatCompletionRequest]: - """ - - translate params, where needed - - pass rest, as is - """ - request_body = AnthropicMessagesRequest(**kwargs) # type: ignore - - translated_body = litellm.AnthropicConfig().translate_anthropic_to_openai( + def translate_completion_input_params(self, kwargs): + """Translate Anthropic format to OpenAI format""" + request_body = AnthropicMessagesRequest(**kwargs) + return litellm.AnthropicConfig().translate_anthropic_to_openai( anthropic_message_request=request_body ) - return translated_body - - def translate_completion_output_params( - self, response: litellm.ModelResponse - ) -> Optional[AnthropicResponse]: - + def translate_completion_output_params(self, response): + """Translate OpenAI response back to Anthropic format""" return litellm.AnthropicConfig().translate_openai_response_to_anthropic( response=response ) - def translate_completion_output_params_streaming(self) -> Optional[BaseModel]: - return super().translate_completion_output_params_streaming() - - anthropic_adapter = AnthropicAdapter() - -########### -# TEST IT # -########### - -## register CUSTOM ADAPTER -litellm.adapters = [{"id": "anthropic", "adapter": anthropic_adapter}] - -## set ENV variables -os.environ["OPENAI_API_KEY"] = "your-openai-key" -os.environ["COHERE_API_KEY"] = "your-cohere-key" - -messages = [{ "content": "Hello, how are you?","role": "user"}] - -# openai call -response = adapter_completion(model="gpt-3.5-turbo", messages=messages, adapter_id="anthropic") - -# cohere call -response = adapter_completion(model="command-nightly", messages=messages, adapter_id="anthropic") -print(response) ``` -### 2. Create new endpoint - -We pass the custom callback class defined in Step1 to the config.yaml. Set callbacks to python_filename.logger_instance_name - -In the config below, we pass - -python_filename: `custom_callbacks.py` -logger_instance_name: `anthropic_adapter`. This is defined in Step 1 - -`target: custom_callbacks.proxy_handler_instance` +### 2. Configure the Endpoint ```yaml model_list: - - model_name: my-fake-claude-endpoint + - model_name: my-claude-endpoint litellm_params: model: gpt-3.5-turbo api_key: os.environ/OPENAI_API_KEY - general_settings: master_key: sk-1234 pass_through_endpoints: - - path: "/v1/messages" # route you want to add to LiteLLM Proxy Server - target: custom_callbacks.anthropic_adapter # Adapter to use for this route + - path: "/v1/messages" + target: custom_callbacks.anthropic_adapter headers: - litellm_user_api_key: "x-api-key" # Field in headers, containing LiteLLM Key + litellm_user_api_key: "x-api-key" ``` -### 3. Test it! - -**Start proxy** - -```bash -litellm --config /path/to/config.yaml -``` - -**Curl** +### 3. Test Custom Endpoint ```bash curl --location 'http://0.0.0.0:4000/v1/messages' \ --H 'x-api-key: sk-1234' \ --H 'anthropic-version: 2023-06-01' \ # ignored --H 'content-type: application/json' \ --D '{ - "model": "my-fake-claude-endpoint", + -H 'x-api-key: sk-1234' \ + -H 'anthropic-version: 2023-06-01' \ + -H 'content-type: application/json' \ + -d '{ + "model": "my-claude-endpoint", "max_tokens": 1024, - "messages": [ - {"role": "user", "content": "Hello, world"} - ] -}' + "messages": [{"role": "user", "content": "Hello, world"}] + }' ``` +--- + +Need help? Check out our [provider-specific parameters guide](../completion/provider_specific_params.md) for more advanced configurations.