3a086cee06
* fix test_moderations_bad_model * use async_post_call_failure_hook * basic logging errors in DB * show status on ui * show status on ui * ui show request / response side by side * stash fixes * working, track raw request * track error info in metadata * fix showing error / request / response logs * show traceback on error viewer * ui with traceback of error * fix async_post_call_failure_hook * fix(http_parsing_utils.py): orjson can throw errors on some emoji's in text, default to json.loads * test_get_error_information * fix code quality * rename proxy track cost callback test * _should_store_errors_in_spend_logs * feature flag error logs * Revert "_should_store_errors_in_spend_logs" This reverts commit 7f345df47762ff3be04e6fde2f13e70019ede4ee. * Revert "feature flag error logs" This reverts commit 0e90c022bbea3550f169118d81e60d711a4024fe. * test_spend_logs_payload * fix OTEL log_db_metrics * fix import json * fix ui linting error * test_async_post_call_failure_hook * test_chat_completion_bad_model_with_spend_logs --------- Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
2.6 KiB
import Image from '@theme/IdealImage'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';
Life of a Request
High Level architecture
<Image img={require('../../img/litellm_gateway.png')} style={{ width: '100%', maxWidth: '4000px' }} />
Request Flow
-
User Sends Request: The process begins when a user sends a request to the LiteLLM Proxy Server (Gateway).
-
Virtual Keys: At this stage the
Bearertoken in the request is checked to ensure it is valid and under it's budget. Here is the list of checks that run for each request- 2.1 Check if the Virtual Key exists in Redis Cache or In Memory Cache
- 2.2 If not in Cache, Lookup Virtual Key in DB
-
Rate Limiting: The MaxParallelRequestsHandler checks the rate limit (rpm/tpm) for the the following components:
- Global Server Rate Limit
- Virtual Key Rate Limit
- User Rate Limit
- Team Limit
-
LiteLLM
proxy_server.py: Contains the/chat/completionsand/embeddingsendpoints. Requests to these endpoints are sent through the LiteLLM Router -
LiteLLM Router: The LiteLLM Router handles Load balancing, Fallbacks, Retries for LLM API deployments.
-
litellm.completion() / litellm.embedding(): The litellm Python SDK is used to call the LLM in the OpenAI API format (Translation and parameter mapping)
-
Post-Request Processing: After the response is sent back to the client, the following asynchronous tasks are performed:
- Logging to Lunary, MLflow, LangFuse or other logging destinations
- The MaxParallelRequestsHandler updates the rpm/tpm usage for the
- Global Server Rate Limit
- Virtual Key Rate Limit
- User Rate Limit
- Team Limit
- The
_ProxyDBLoggerupdates spend / usage in the LiteLLM database. Here is everything tracked in the DB per request
Frequently Asked Questions
- Is a db transaction tied to the lifecycle of request?
- No, a db transaction is not tied to the lifecycle of a request.
- The check if a virtual key is valid relies on a DB read if it's not in cache.
- All other DB transactions are async in background tasks