mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-04 00:24:18 +00:00
(docs) deploying litell
This commit is contained in:
ishaan-jaff
@@ -28,7 +28,7 @@ docker run ghcr.io/berriai/litellm:main-latest
|
||||
|
||||
<TabItem value="cli" label="With CLI Args">
|
||||
|
||||
### Run with LiteLLM CLI args
|
||||
#### Run with LiteLLM CLI args
|
||||
|
||||
See all supported CLI args [here](https://docs.litellm.ai/docs/proxy/cli):
|
||||
|
||||
@@ -129,13 +129,25 @@ spec:
|
||||
name: litellm-config-file
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> To avoid issues with predictability, difficulties in rollback, and inconsistent environments, use versioning or SHA digests (for example, `litellm:main-v1.30.3` or `litellm@sha256:12345abcdef...`) instead of `litellm:main-latest`.
|
||||
:::info
|
||||
To avoid issues with predictability, difficulties in rollback, and inconsistent environments, use versioning or SHA digests (for example, `litellm:main-v1.30.3` or `litellm@sha256:12345abcdef...`) instead of `litellm:main-latest`.
|
||||
:::
|
||||
|
||||
</TabItem>
|
||||
|
||||
</Tabs>
|
||||
|
||||
**That's it ! That's the quick start to deploy litellm**
|
||||
|
||||
## Options to deploy LiteLLM
|
||||
|
||||
| Method | When to Use | Docs |
|
||||
| --- | --- | --- |
|
||||
| LiteLLM - Quick Start | call 100+ LLMs + Load Balancing | [Quick Start](#quick-start) |
|
||||
| LiteLLM Database container + PostgresDB | + use Virtual Keys + Track Spend | [Deploy with Database](#deploy-with-database) |
|
||||
| LiteLLM container + Redis | + load balance across multiple litellm containers | [LiteLLM container + Redis](#litellm-container--redis) |
|
||||
| LiteLLM Database container + PostgresDB + Redis | + use Virtual Keys + Track Spend + load balance across multiple litellm containers | [LiteLLM container + Redis](#litellm-container--redis) |
|
||||
|
||||
## Deploy with Database
|
||||
|
||||
We maintain a [seperate Dockerfile](https://github.com/BerriAI/litellm/pkgs/container/litellm-database) for reducing build time when running LiteLLM proxy with a connected Postgres Database
|
||||
@@ -159,7 +171,7 @@ Your OpenAI proxy server is now running on `http://0.0.0.0:4000`.
|
||||
</TabItem>
|
||||
<TabItem value="kubernetes-deploy" label="Kubernetes">
|
||||
|
||||
### Step 1. Create deployment.yaml
|
||||
#### Step 1. Create deployment.yaml
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
@@ -188,7 +200,7 @@ Your OpenAI proxy server is now running on `http://0.0.0.0:4000`.
|
||||
kubectl apply -f /path/to/deployment.yaml
|
||||
```
|
||||
|
||||
### Step 2. Create service.yaml
|
||||
#### Step 2. Create service.yaml
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
@@ -209,7 +221,7 @@ spec:
|
||||
kubectl apply -f /path/to/service.yaml
|
||||
```
|
||||
|
||||
### Step 3. Start server
|
||||
#### Step 3. Start server
|
||||
|
||||
```
|
||||
kubectl port-forward service/litellm-service 4000:4000
|
||||
@@ -220,13 +232,13 @@ Your OpenAI proxy server is now running on `http://0.0.0.0:4000`.
|
||||
</TabItem>
|
||||
<TabItem value="helm-deploy" label="Helm">
|
||||
|
||||
### Step 1. Clone the repository
|
||||
#### Step 1. Clone the repository
|
||||
|
||||
```bash
|
||||
git clone https://github.com/BerriAI/litellm.git
|
||||
```
|
||||
|
||||
### Step 2. Deploy with Helm
|
||||
#### Step 2. Deploy with Helm
|
||||
|
||||
```bash
|
||||
helm install \
|
||||
@@ -235,7 +247,7 @@ helm install \
|
||||
deploy/charts/litellm
|
||||
```
|
||||
|
||||
### Step 3. Expose the service to localhost
|
||||
#### Step 3. Expose the service to localhost
|
||||
|
||||
```bash
|
||||
kubectl \
|
||||
@@ -249,6 +261,38 @@ Your OpenAI proxy server is now running on `http://127.0.0.1:8000`.
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## LiteLLM container + Redis
|
||||
Use Redis when you need litellm to load balance across multiple litellm containers
|
||||
|
||||
The only change required is setting Redis on your `config.yaml`
|
||||
LiteLLM Proxy supports sharing rpm/tpm shared across multiple litellm instances, pass `redis_host`, `redis_password` and `redis_port` to enable this. (LiteLLM will use Redis to track rpm/tpm usage )
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gpt-3.5-turbo
|
||||
litellm_params:
|
||||
model: azure/<your-deployment-name>
|
||||
api_base: <your-azure-endpoint>
|
||||
api_key: <your-azure-api-key>
|
||||
rpm: 6 # Rate limit for this deployment: in requests per minute (rpm)
|
||||
- model_name: gpt-3.5-turbo
|
||||
litellm_params:
|
||||
model: azure/gpt-turbo-small-ca
|
||||
api_base: https://my-endpoint-canada-berri992.openai.azure.com/
|
||||
api_key: <your-azure-api-key>
|
||||
rpm: 6
|
||||
router_settings:
|
||||
redis_host: <your redis host>
|
||||
redis_password: <your redis password>
|
||||
redis_port: 1992
|
||||
```
|
||||
|
||||
Start docker container with config
|
||||
|
||||
```shell
|
||||
docker run ghcr.io/berriai/litellm:main-latest --config your_config.yaml
|
||||
```
|
||||
|
||||
## Best Practices for Deploying to Production
|
||||
### 1. Switch of debug logs in production
|
||||
don't use [`--detailed-debug`, `--debug`](https://docs.litellm.ai/docs/proxy/debugging#detailed-debug) or `litellm.set_verbose=True`. We found using debug logs can add 5-10% latency per LLM API call
|
||||
|
||||
@@ -42,10 +42,6 @@ const sidebars = {
|
||||
"proxy/team_based_routing",
|
||||
"proxy/ui",
|
||||
"proxy/budget_alerts",
|
||||
"proxy/model_management",
|
||||
"proxy/health",
|
||||
"proxy/debugging",
|
||||
"proxy/pii_masking",
|
||||
{
|
||||
"type": "category",
|
||||
"label": "🔥 Load Balancing",
|
||||
@@ -54,6 +50,10 @@ const sidebars = {
|
||||
"proxy/reliability",
|
||||
]
|
||||
},
|
||||
"proxy/model_management",
|
||||
"proxy/health",
|
||||
"proxy/debugging",
|
||||
"proxy/pii_masking",
|
||||
"proxy/caching",
|
||||
{
|
||||
"type": "category",
|
||||
|
||||
Reference in New Issue
Block a user