diff --git a/docs/my-website/docs/proxy/prod.md b/docs/my-website/docs/proxy/prod.md index f287aea644..6f25e26367 100644 --- a/docs/my-website/docs/proxy/prod.md +++ b/docs/my-website/docs/proxy/prod.md @@ -44,8 +44,74 @@ CMD ["--port", "4000", "--config", "./proxy_server_config.yaml"] Writing each spend log to the db can slow down your proxy. In testing we saw a 70% improvement in median response time, by moving writing spend logs to a separate server. +👉 [LiteLLM Spend Logs Server](https://github.com/BerriAI/litellm/tree/main/litellm-js/spend-logs) +**1. Start the server** + +```bash +docker run -p 3000:3000 \ + -e DATABASE_URL="postgres://.." \ + ghcr.io/berriai/litellm:litellm-spend_logs-latest + +# RUNNING on http://0.0.0.0:3000 +``` + +**2. Connect to proxy** + +Add 'SPEND_LOGS_URL' as an environment variable when starting the proxy + +Example litellm_config.yaml + +```yaml +model_list: +- model_name: fake-openai-endpoint + litellm_params: + model: openai/my-fake-model + api_key: my-fake-key + api_base: https://exampleopenaiendpoint-production.up.railway.app/ + +general_settings: + master_key: sk-1234 + proxy_batch_write_at: 5 # 👈 Frequency of batch writing logs to server (in seconds) +``` +```bash +docker run \ + -v $(pwd)/litellm_config.yaml:/app/config.yaml \ + -e DATABASE_URL=postgres://... \ + -e SPEND_LOGS_URL="http://0.0.0.0:3000" \ # 👈 KEY CHANGE + -p 4000:4000 \ + ghcr.io/berriai/litellm:main-latest \ + --config /app/config.yaml --detailed_debug + +# Running on http://0.0.0.0:4000 +``` + +**3. Test it!** + +```bash +curl --location 'http://0.0.0.0:4000/v1/chat/completions' \ +--header 'Content-Type: application/json' \ +--header 'Authorization: Bearer sk-1234' \ +--data '{ + "model": "fake-openai-endpoint", + "messages": [ + {"role": "system", "content": "Be helpful"}, + {"role": "user", "content": "What do you know?"} + ] +}' +``` + +In your LiteLLM Spend Logs Server, you should see + +**Expected Response** + +``` +Received and stored 1 logs. Total logs in memory: 1 +... +Flushed 1 log to the DB. +``` + ## 4. Switch off resetting budgets Add this to your config.yaml. (Only spend per Key, User and Team will be tracked - spend per API Call will not be written to the LiteLLM Database) diff --git a/litellm/proxy/_new_secret_config.yaml b/litellm/proxy/_new_secret_config.yaml index 07a24dd7e8..2e656b4cfd 100644 --- a/litellm/proxy/_new_secret_config.yaml +++ b/litellm/proxy/_new_secret_config.yaml @@ -4,19 +4,6 @@ model_list: model: openai/my-fake-model api_key: my-fake-key api_base: https://exampleopenaiendpoint-production.up.railway.app/ -- model_name: gpt-3.5-turbo - litellm_params: - model: gpt-3.5-turbo-1106 - api_key: os.environ/OPENAI_API_KEY - -# litellm_settings: -# cache: true -# cache_params: -# type: redis -# callbacks: ["batch_redis_requests"] -# # success_callbacks: ["langfuse"] general_settings: - master_key: sk-1234 - disable_spend_logs: true - database_url: "postgresql://neondb_owner:hz8tyUlJ5ivV@ep-cool-sunset-a5ywubeh.us-east-2.aws.neon.tech/neondb?sslmode=require" \ No newline at end of file + master_key: sk-1234 \ No newline at end of file