diff --git a/docs/my-website/docs/proxy/deploy.md b/docs/my-website/docs/proxy/deploy.md index 496bde05f1..655d93b6b0 100644 --- a/docs/my-website/docs/proxy/deploy.md +++ b/docs/my-website/docs/proxy/deploy.md @@ -28,7 +28,7 @@ docker run ghcr.io/berriai/litellm:main-latest -### Run with LiteLLM CLI args +#### Run with LiteLLM CLI args See all supported CLI args [here](https://docs.litellm.ai/docs/proxy/cli): @@ -129,13 +129,25 @@ spec: name: litellm-config-file ``` -> [!TIP] -> To avoid issues with predictability, difficulties in rollback, and inconsistent environments, use versioning or SHA digests (for example, `litellm:main-v1.30.3` or `litellm@sha256:12345abcdef...`) instead of `litellm:main-latest`. +:::info +To avoid issues with predictability, difficulties in rollback, and inconsistent environments, use versioning or SHA digests (for example, `litellm:main-v1.30.3` or `litellm@sha256:12345abcdef...`) instead of `litellm:main-latest`. +::: +**That's it ! That's the quick start to deploy litellm** + +## Options to deploy LiteLLM + +| Method | When to Use | Docs | +| --- | --- | --- | +| LiteLLM - Quick Start | call 100+ LLMs + Load Balancing | [Quick Start](#quick-start) | +| LiteLLM Database container + PostgresDB | + use Virtual Keys + Track Spend | [Deploy with Database](#deploy-with-database) | +| LiteLLM container + Redis | + load balance across multiple litellm containers | [LiteLLM container + Redis](#litellm-container--redis) | +| LiteLLM Database container + PostgresDB + Redis | + use Virtual Keys + Track Spend + load balance across multiple litellm containers | [LiteLLM container + Redis](#litellm-container--redis) | + ## Deploy with Database We maintain a [seperate Dockerfile](https://github.com/BerriAI/litellm/pkgs/container/litellm-database) for reducing build time when running LiteLLM proxy with a connected Postgres Database @@ -159,7 +171,7 @@ Your OpenAI proxy server is now running on `http://0.0.0.0:4000`. -### Step 1. Create deployment.yaml +#### Step 1. Create deployment.yaml ```yaml apiVersion: apps/v1 @@ -188,7 +200,7 @@ Your OpenAI proxy server is now running on `http://0.0.0.0:4000`. kubectl apply -f /path/to/deployment.yaml ``` -### Step 2. Create service.yaml +#### Step 2. Create service.yaml ```yaml apiVersion: v1 @@ -209,7 +221,7 @@ spec: kubectl apply -f /path/to/service.yaml ``` -### Step 3. Start server +#### Step 3. Start server ``` kubectl port-forward service/litellm-service 4000:4000 @@ -220,13 +232,13 @@ Your OpenAI proxy server is now running on `http://0.0.0.0:4000`. -### Step 1. Clone the repository +#### Step 1. Clone the repository ```bash git clone https://github.com/BerriAI/litellm.git ``` -### Step 2. Deploy with Helm +#### Step 2. Deploy with Helm ```bash helm install \ @@ -235,7 +247,7 @@ helm install \ deploy/charts/litellm ``` -### Step 3. Expose the service to localhost +#### Step 3. Expose the service to localhost ```bash kubectl \ @@ -249,6 +261,38 @@ Your OpenAI proxy server is now running on `http://127.0.0.1:8000`. +## LiteLLM container + Redis +Use Redis when you need litellm to load balance across multiple litellm containers + +The only change required is setting Redis on your `config.yaml` +LiteLLM Proxy supports sharing rpm/tpm shared across multiple litellm instances, pass `redis_host`, `redis_password` and `redis_port` to enable this. (LiteLLM will use Redis to track rpm/tpm usage ) + +```yaml +model_list: + - model_name: gpt-3.5-turbo + litellm_params: + model: azure/ + api_base: + api_key: + rpm: 6 # Rate limit for this deployment: in requests per minute (rpm) + - model_name: gpt-3.5-turbo + litellm_params: + model: azure/gpt-turbo-small-ca + api_base: https://my-endpoint-canada-berri992.openai.azure.com/ + api_key: + rpm: 6 +router_settings: + redis_host: + redis_password: + redis_port: 1992 +``` + +Start docker container with config + +```shell +docker run ghcr.io/berriai/litellm:main-latest --config your_config.yaml +``` + ## Best Practices for Deploying to Production ### 1. Switch of debug logs in production don't use [`--detailed-debug`, `--debug`](https://docs.litellm.ai/docs/proxy/debugging#detailed-debug) or `litellm.set_verbose=True`. We found using debug logs can add 5-10% latency per LLM API call diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index 33720b43eb..62a2d38424 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -42,10 +42,6 @@ const sidebars = { "proxy/team_based_routing", "proxy/ui", "proxy/budget_alerts", - "proxy/model_management", - "proxy/health", - "proxy/debugging", - "proxy/pii_masking", { "type": "category", "label": "🔥 Load Balancing", @@ -54,6 +50,10 @@ const sidebars = { "proxy/reliability", ] }, + "proxy/model_management", + "proxy/health", + "proxy/debugging", + "proxy/pii_masking", "proxy/caching", { "type": "category",