mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-06 16:24:46 +00:00
docs(prod.md): add litellm spend logs server to docs
This commit is contained in:
@@ -44,8 +44,74 @@ CMD ["--port", "4000", "--config", "./proxy_server_config.yaml"]
|
||||
|
||||
Writing each spend log to the db can slow down your proxy. In testing we saw a 70% improvement in median response time, by moving writing spend logs to a separate server.
|
||||
|
||||
👉 [LiteLLM Spend Logs Server](https://github.com/BerriAI/litellm/tree/main/litellm-js/spend-logs)
|
||||
|
||||
|
||||
**1. Start the server**
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 \
|
||||
-e DATABASE_URL="postgres://.." \
|
||||
ghcr.io/berriai/litellm:litellm-spend_logs-latest
|
||||
|
||||
# RUNNING on http://0.0.0.0:3000
|
||||
```
|
||||
|
||||
**2. Connect to proxy**
|
||||
|
||||
Add 'SPEND_LOGS_URL' as an environment variable when starting the proxy
|
||||
|
||||
Example litellm_config.yaml
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: fake-openai-endpoint
|
||||
litellm_params:
|
||||
model: openai/my-fake-model
|
||||
api_key: my-fake-key
|
||||
api_base: https://exampleopenaiendpoint-production.up.railway.app/
|
||||
|
||||
general_settings:
|
||||
master_key: sk-1234
|
||||
proxy_batch_write_at: 5 # 👈 Frequency of batch writing logs to server (in seconds)
|
||||
```
|
||||
```bash
|
||||
docker run \
|
||||
-v $(pwd)/litellm_config.yaml:/app/config.yaml \
|
||||
-e DATABASE_URL=postgres://... \
|
||||
-e SPEND_LOGS_URL="http://0.0.0.0:3000" \ # 👈 KEY CHANGE
|
||||
-p 4000:4000 \
|
||||
ghcr.io/berriai/litellm:main-latest \
|
||||
--config /app/config.yaml --detailed_debug
|
||||
|
||||
# Running on http://0.0.0.0:4000
|
||||
```
|
||||
|
||||
**3. Test it!**
|
||||
|
||||
```bash
|
||||
curl --location 'http://0.0.0.0:4000/v1/chat/completions' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header 'Authorization: Bearer sk-1234' \
|
||||
--data '{
|
||||
"model": "fake-openai-endpoint",
|
||||
"messages": [
|
||||
{"role": "system", "content": "Be helpful"},
|
||||
{"role": "user", "content": "What do you know?"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
In your LiteLLM Spend Logs Server, you should see
|
||||
|
||||
**Expected Response**
|
||||
|
||||
```
|
||||
Received and stored 1 logs. Total logs in memory: 1
|
||||
...
|
||||
Flushed 1 log to the DB.
|
||||
```
|
||||
|
||||
## 4. Switch off resetting budgets
|
||||
|
||||
Add this to your config.yaml. (Only spend per Key, User and Team will be tracked - spend per API Call will not be written to the LiteLLM Database)
|
||||
|
||||
@@ -4,19 +4,6 @@ model_list:
|
||||
model: openai/my-fake-model
|
||||
api_key: my-fake-key
|
||||
api_base: https://exampleopenaiendpoint-production.up.railway.app/
|
||||
- model_name: gpt-3.5-turbo
|
||||
litellm_params:
|
||||
model: gpt-3.5-turbo-1106
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
|
||||
# litellm_settings:
|
||||
# cache: true
|
||||
# cache_params:
|
||||
# type: redis
|
||||
# callbacks: ["batch_redis_requests"]
|
||||
# # success_callbacks: ["langfuse"]
|
||||
|
||||
general_settings:
|
||||
master_key: sk-1234
|
||||
disable_spend_logs: true
|
||||
database_url: "postgresql://neondb_owner:hz8tyUlJ5ivV@ep-cool-sunset-a5ywubeh.us-east-2.aws.neon.tech/neondb?sslmode=require"
|
||||
master_key: sk-1234
|
||||
Reference in New Issue
Block a user