doc High Availability Setup

This commit is contained in:
Ishaan Jaff
2025-04-02 13:38:49 -07:00
parent b48b8366c2
commit 6ab1eba7b6
+24 -9
View File
@@ -8,34 +8,49 @@ Resolve any Database Deadlocks you see in high traffic by using this setup
## What causes the problem?
LiteLLM writes `UPDATE` and `UPSERT` queries to the DB. When using 10+ pods of LiteLLM, these queries can cause deadlocks since each pod could simultaneously attempt to update the same `user_id`, `team_id`, `key` etc.
LiteLLM writes `UPDATE` and `UPSERT` queries to the DB. When using 10+ instances of LiteLLM, these queries can cause deadlocks since each instance could simultaneously attempt to update the same `user_id`, `team_id`, `key` etc.
## How the high availability setup fixes the problem
- All pods will write to a Redis queue instead of the DB.
- A single pod will acquire a lock on the DB and flush the redis queue to the DB.
- All instances will write to a Redis queue instead of the DB.
- A single instance will acquire a lock on the DB and flush the redis queue to the DB.
## How it works
### Stage 1. Each pod writes updates to redis
### Stage 1. Each instance writes updates to redis
Each pod will accumlate the spend updates for a key, user, team, etc and write the updates to a redis queue.
Each instance will accumlate the spend updates for a key, user, team, etc and write the updates to a redis queue.
<Image img={require('../../img/deadlock_fix_1.png')} style={{ width: '900px', height: 'auto' }} />
<p style={{textAlign: 'left', color: '#666'}}>
Each instance writes updates to redis
</p>
### Stage 2. A single pod flushes the redis queue to the DB
### Stage 2. A single instance flushes the redis queue to the DB
A single pod will acquire a lock on the DB and flush all elements in the redis queue to the DB.
A single instance will acquire a lock on the DB and flush all elements in the redis queue to the DB.
<Image img={require('../../img/deadlock_fix_2.png')} style={{ width: '900px', height: 'auto' }} />
<p style={{textAlign: 'left', color: '#666'}}>
A single instance flushes the redis queue to the DB
</p>
## Setup
## Usage
## Required components
- Redis
- Postgres
### Setup on LiteLLM config
You can enable using the redis buffer by setting `use_redis_transaction_buffer: true` in the `general_settings` section of your `proxy_config.yaml` file.
Note: This setup requires a redis instance to be running.
```yaml showLineNumbers title="litellm proxy_config.yaml"
general_settings:
use_redis_transaction_buffer: true
@@ -44,5 +59,5 @@ litellm_settings:
cache: True
cache_params:
type: redis
supported_call_types: []
supported_call_types: [] # Optional: Set cache for proxy, but not on the actual llm api call
```