mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-23 18:25:22 +00:00
doc High Availability Setup
This commit is contained in:
@@ -8,34 +8,49 @@ Resolve any Database Deadlocks you see in high traffic by using this setup
|
||||
|
||||
## What causes the problem?
|
||||
|
||||
LiteLLM writes `UPDATE` and `UPSERT` queries to the DB. When using 10+ pods of LiteLLM, these queries can cause deadlocks since each pod could simultaneously attempt to update the same `user_id`, `team_id`, `key` etc.
|
||||
LiteLLM writes `UPDATE` and `UPSERT` queries to the DB. When using 10+ instances of LiteLLM, these queries can cause deadlocks since each instance could simultaneously attempt to update the same `user_id`, `team_id`, `key` etc.
|
||||
|
||||
## How the high availability setup fixes the problem
|
||||
- All pods will write to a Redis queue instead of the DB.
|
||||
- A single pod will acquire a lock on the DB and flush the redis queue to the DB.
|
||||
- All instances will write to a Redis queue instead of the DB.
|
||||
- A single instance will acquire a lock on the DB and flush the redis queue to the DB.
|
||||
|
||||
|
||||
## How it works
|
||||
|
||||
### Stage 1. Each pod writes updates to redis
|
||||
### Stage 1. Each instance writes updates to redis
|
||||
|
||||
Each pod will accumlate the spend updates for a key, user, team, etc and write the updates to a redis queue.
|
||||
Each instance will accumlate the spend updates for a key, user, team, etc and write the updates to a redis queue.
|
||||
|
||||
<Image img={require('../../img/deadlock_fix_1.png')} style={{ width: '900px', height: 'auto' }} />
|
||||
<p style={{textAlign: 'left', color: '#666'}}>
|
||||
Each instance writes updates to redis
|
||||
</p>
|
||||
|
||||
|
||||
### Stage 2. A single pod flushes the redis queue to the DB
|
||||
### Stage 2. A single instance flushes the redis queue to the DB
|
||||
|
||||
A single pod will acquire a lock on the DB and flush all elements in the redis queue to the DB.
|
||||
A single instance will acquire a lock on the DB and flush all elements in the redis queue to the DB.
|
||||
|
||||
|
||||
<Image img={require('../../img/deadlock_fix_2.png')} style={{ width: '900px', height: 'auto' }} />
|
||||
<p style={{textAlign: 'left', color: '#666'}}>
|
||||
A single instance flushes the redis queue to the DB
|
||||
</p>
|
||||
|
||||
|
||||
## Setup
|
||||
## Usage
|
||||
|
||||
## Required components
|
||||
|
||||
- Redis
|
||||
- Postgres
|
||||
|
||||
### Setup on LiteLLM config
|
||||
|
||||
You can enable using the redis buffer by setting `use_redis_transaction_buffer: true` in the `general_settings` section of your `proxy_config.yaml` file.
|
||||
|
||||
Note: This setup requires a redis instance to be running.
|
||||
|
||||
```yaml showLineNumbers title="litellm proxy_config.yaml"
|
||||
general_settings:
|
||||
use_redis_transaction_buffer: true
|
||||
@@ -44,5 +59,5 @@ litellm_settings:
|
||||
cache: True
|
||||
cache_params:
|
||||
type: redis
|
||||
supported_call_types: []
|
||||
supported_call_types: [] # Optional: Set cache for proxy, but not on the actual llm api call
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user