mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-20 18:20:33 +00:00
docs: Update CLI arguments documentation with all available options (#20437)
- Reorganized documentation into logical sections (Server Configuration, Server Backend Options, SSL/TLS Configuration, Model Configuration, Model Parameters, Database Configuration, Debugging, Testing & Health Checks) - Added missing CLI arguments: - --iam_token_db_auth: RDS IAM token authentication for database connections - --run_gunicorn: Start proxy with gunicorn backend - --run_hypercorn: Start proxy with hypercorn backend (HTTP/2 support) - --ssl_keyfile_path, --ssl_certfile_path, --ciphers: SSL/TLS configuration - --keepalive_timeout: Uvicorn keepalive timeout setting - --max_requests_before_restart: Worker recycling for memory management - --use_prisma_db_push: Alternative database schema update method - --test_async, --num_requests: Async endpoint testing - --version/-v: Print version - --max_budget: Set budget limits for API calls - --headers, --add_key, --save: Model configuration options - --use_queue: Celery workers for async endpoints - --local: Local debugging flag - Updated default value for --num_workers to reflect actual behavior - Added environment variable documentation for applicable options Co-authored-by: Cursor Agent <cursoragent@cursor.com>
This commit is contained in:
Krish Dholakia
co-authored by
Cursor Agent
Cursor Agent
parent
b74afec2be
commit
f65eec8fa8
@@ -1,7 +1,10 @@
|
||||
# CLI Arguments
|
||||
Cli arguments, --host, --port, --num_workers
|
||||
|
||||
## --host
|
||||
This page documents all command-line interface (CLI) arguments available for the LiteLLM proxy server.
|
||||
|
||||
## Server Configuration
|
||||
|
||||
### --host
|
||||
- **Default:** `'0.0.0.0'`
|
||||
- The host for the server to listen on.
|
||||
- **Usage:**
|
||||
@@ -14,7 +17,7 @@ Cli arguments, --host, --port, --num_workers
|
||||
litellm
|
||||
```
|
||||
|
||||
## --port
|
||||
### --port
|
||||
- **Default:** `4000`
|
||||
- The port to bind the server to.
|
||||
- **Usage:**
|
||||
@@ -27,9 +30,9 @@ Cli arguments, --host, --port, --num_workers
|
||||
litellm
|
||||
```
|
||||
|
||||
## --num_workers
|
||||
- **Default:** `1`
|
||||
- The number of uvicorn workers to spin up.
|
||||
### --num_workers
|
||||
- **Default:** Number of logical CPUs in the system, or `4` if that cannot be determined
|
||||
- The number of uvicorn / gunicorn workers to spin up.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --num_workers 4
|
||||
@@ -40,55 +43,273 @@ Cli arguments, --host, --port, --num_workers
|
||||
litellm
|
||||
```
|
||||
|
||||
## --api_base
|
||||
### --config
|
||||
- **Short form:** `-c`
|
||||
- **Default:** `None`
|
||||
- The API base for the model litellm should call.
|
||||
- Path to the proxy configuration file (e.g., config.yaml).
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --config path/to/config.yaml
|
||||
```
|
||||
|
||||
### --log_config
|
||||
- **Default:** `None`
|
||||
- **Type:** `str`
|
||||
- Path to the logging configuration file for uvicorn.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --log_config path/to/log_config.conf
|
||||
```
|
||||
|
||||
### --keepalive_timeout
|
||||
- **Default:** `None`
|
||||
- **Type:** `int`
|
||||
- Set the uvicorn keepalive timeout in seconds (uvicorn timeout_keep_alive parameter).
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --keepalive_timeout 30
|
||||
```
|
||||
- **Usage - set Environment Variable:** `KEEPALIVE_TIMEOUT`
|
||||
```shell
|
||||
export KEEPALIVE_TIMEOUT=30
|
||||
litellm
|
||||
```
|
||||
|
||||
### --max_requests_before_restart
|
||||
- **Default:** `None`
|
||||
- **Type:** `int`
|
||||
- Restart worker after this many requests. This is useful for mitigating memory growth over time.
|
||||
- For uvicorn: maps to `limit_max_requests`
|
||||
- For gunicorn: maps to `max_requests`
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --max_requests_before_restart 10000
|
||||
```
|
||||
- **Usage - set Environment Variable:** `MAX_REQUESTS_BEFORE_RESTART`
|
||||
```shell
|
||||
export MAX_REQUESTS_BEFORE_RESTART=10000
|
||||
litellm
|
||||
```
|
||||
|
||||
## Server Backend Options
|
||||
|
||||
### --run_gunicorn
|
||||
- **Default:** `False`
|
||||
- **Type:** `bool` (Flag)
|
||||
- Starts proxy via gunicorn instead of uvicorn. Better for managing multiple workers in production.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --run_gunicorn
|
||||
```
|
||||
|
||||
### --run_hypercorn
|
||||
- **Default:** `False`
|
||||
- **Type:** `bool` (Flag)
|
||||
- Starts proxy via hypercorn instead of uvicorn. Supports HTTP/2.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --run_hypercorn
|
||||
```
|
||||
|
||||
### --skip_server_startup
|
||||
- **Default:** `False`
|
||||
- **Type:** `bool` (Flag)
|
||||
- Skip starting the server after setup (useful for database migrations only).
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --skip_server_startup
|
||||
```
|
||||
|
||||
## SSL/TLS Configuration
|
||||
|
||||
### --ssl_keyfile_path
|
||||
- **Default:** `None`
|
||||
- **Type:** `str`
|
||||
- Path to the SSL keyfile. Use this when you want to provide SSL certificate when starting proxy.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --ssl_keyfile_path /path/to/key.pem --ssl_certfile_path /path/to/cert.pem
|
||||
```
|
||||
- **Usage - set Environment Variable:** `SSL_KEYFILE_PATH`
|
||||
```shell
|
||||
export SSL_KEYFILE_PATH=/path/to/key.pem
|
||||
litellm
|
||||
```
|
||||
|
||||
### --ssl_certfile_path
|
||||
- **Default:** `None`
|
||||
- **Type:** `str`
|
||||
- Path to the SSL certfile. Use this when you want to provide SSL certificate when starting proxy.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --ssl_certfile_path /path/to/cert.pem --ssl_keyfile_path /path/to/key.pem
|
||||
```
|
||||
- **Usage - set Environment Variable:** `SSL_CERTFILE_PATH`
|
||||
```shell
|
||||
export SSL_CERTFILE_PATH=/path/to/cert.pem
|
||||
litellm
|
||||
```
|
||||
|
||||
### --ciphers
|
||||
- **Default:** `None`
|
||||
- **Type:** `str`
|
||||
- Ciphers to use for the SSL setup. Only used with `--run_hypercorn`.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --run_hypercorn --ssl_keyfile_path /path/to/key.pem --ssl_certfile_path /path/to/cert.pem --ciphers "ECDHE+AESGCM"
|
||||
```
|
||||
|
||||
## Model Configuration
|
||||
|
||||
### --model or -m
|
||||
- **Default:** `None`
|
||||
- The model name to pass to LiteLLM.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --model gpt-3.5-turbo
|
||||
```
|
||||
|
||||
### --alias
|
||||
- **Default:** `None`
|
||||
- An alias for the model, for user-friendly reference. Use this to give a litellm model name (e.g., "huggingface/codellama/CodeLlama-7b-Instruct-hf") a more user-friendly name ("codellama").
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --alias my-gpt-model
|
||||
```
|
||||
|
||||
### --api_base
|
||||
- **Default:** `None`
|
||||
- The API base for the model LiteLLM should call.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --model huggingface/tinyllama --api_base https://k58ory32yinf1ly0.us-east-1.aws.endpoints.huggingface.cloud
|
||||
```
|
||||
|
||||
## --api_version
|
||||
- **Default:** `None`
|
||||
### --api_version
|
||||
- **Default:** `2024-07-01-preview`
|
||||
- For Azure services, specify the API version.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --model azure/gpt-deployment --api_version 2023-08-01 --api_base https://<your api base>"
|
||||
```
|
||||
|
||||
## --model or -m
|
||||
### --headers
|
||||
- **Default:** `None`
|
||||
- The model name to pass to Litellm.
|
||||
- Headers for the API call (as JSON string).
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --model gpt-3.5-turbo
|
||||
litellm --model my-model --headers '{"Authorization": "Bearer token"}'
|
||||
```
|
||||
|
||||
## --test
|
||||
- **Type:** `bool` (Flag)
|
||||
- Proxy chat completions URL to make a test request.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --test
|
||||
```
|
||||
|
||||
## --health
|
||||
- **Type:** `bool` (Flag)
|
||||
- Runs a health check on all models in config.yaml
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --health
|
||||
```
|
||||
|
||||
## --alias
|
||||
### --add_key
|
||||
- **Default:** `None`
|
||||
- An alias for the model, for user-friendly reference.
|
||||
- Add a key to the model configuration.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --alias my-gpt-model
|
||||
litellm --add_key my-api-key
|
||||
```
|
||||
|
||||
## --debug
|
||||
### --save
|
||||
- **Type:** `bool` (Flag)
|
||||
- Save the model-specific config.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --model gpt-3.5-turbo --save
|
||||
```
|
||||
|
||||
## Model Parameters
|
||||
|
||||
### --temperature
|
||||
- **Default:** `None`
|
||||
- **Type:** `float`
|
||||
- Set the temperature for the model.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --temperature 0.7
|
||||
```
|
||||
|
||||
### --max_tokens
|
||||
- **Default:** `None`
|
||||
- **Type:** `int`
|
||||
- Set the maximum number of tokens for the model output.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --max_tokens 50
|
||||
```
|
||||
|
||||
### --request_timeout
|
||||
- **Default:** `None`
|
||||
- **Type:** `int`
|
||||
- Set the timeout in seconds for completion calls.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --request_timeout 300
|
||||
```
|
||||
|
||||
### --max_budget
|
||||
- **Default:** `None`
|
||||
- **Type:** `float`
|
||||
- Set max budget for API calls. Works for hosted models like OpenAI, TogetherAI, Anthropic, etc.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --max_budget 100.0
|
||||
```
|
||||
|
||||
### --drop_params
|
||||
- **Type:** `bool` (Flag)
|
||||
- Drop any unmapped params.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --drop_params
|
||||
```
|
||||
|
||||
### --add_function_to_prompt
|
||||
- **Type:** `bool` (Flag)
|
||||
- If a function passed but unsupported, pass it as a part of the prompt.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --add_function_to_prompt
|
||||
```
|
||||
|
||||
## Database Configuration
|
||||
|
||||
### --iam_token_db_auth
|
||||
- **Default:** `False`
|
||||
- **Type:** `bool` (Flag)
|
||||
- Connects to an RDS database using IAM token authentication instead of a password. This is useful for AWS RDS instances that are configured to use IAM database authentication.
|
||||
- When enabled, LiteLLM will generate an IAM authentication token to connect to the database.
|
||||
- **Required Environment Variables:**
|
||||
- `DATABASE_HOST` - The RDS database host
|
||||
- `DATABASE_PORT` - The database port
|
||||
- `DATABASE_USER` - The database user
|
||||
- `DATABASE_NAME` - The database name
|
||||
- `DATABASE_SCHEMA` (optional) - The database schema
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --iam_token_db_auth
|
||||
```
|
||||
- **Usage - set Environment Variable:** `IAM_TOKEN_DB_AUTH`
|
||||
```shell
|
||||
export IAM_TOKEN_DB_AUTH=True
|
||||
export DATABASE_HOST=mydb.us-east-1.rds.amazonaws.com
|
||||
export DATABASE_PORT=5432
|
||||
export DATABASE_USER=mydbuser
|
||||
export DATABASE_NAME=mydb
|
||||
litellm
|
||||
```
|
||||
|
||||
### --use_prisma_db_push
|
||||
- **Default:** `False`
|
||||
- **Type:** `bool` (Flag)
|
||||
- Use `prisma db push` instead of `prisma migrate` for database schema updates. This is useful when you want to quickly sync your database schema without creating migration files.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --use_prisma_db_push
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
### --debug
|
||||
- **Default:** `False`
|
||||
- **Type:** `bool` (Flag)
|
||||
- Enable debugging mode for the input.
|
||||
@@ -102,10 +323,10 @@ Cli arguments, --host, --port, --num_workers
|
||||
litellm
|
||||
```
|
||||
|
||||
## --detailed_debug
|
||||
### --detailed_debug
|
||||
- **Default:** `False`
|
||||
- **Type:** `bool` (Flag)
|
||||
- Enable debugging mode for the input.
|
||||
- Enable detailed debugging mode to view verbose debug logs.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --detailed_debug
|
||||
@@ -116,80 +337,76 @@ Cli arguments, --host, --port, --num_workers
|
||||
litellm
|
||||
```
|
||||
|
||||
#### --temperature
|
||||
- **Default:** `None`
|
||||
- **Type:** `float`
|
||||
- Set the temperature for the model.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --temperature 0.7
|
||||
```
|
||||
|
||||
## --max_tokens
|
||||
- **Default:** `None`
|
||||
- **Type:** `int`
|
||||
- Set the maximum number of tokens for the model output.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --max_tokens 50
|
||||
```
|
||||
|
||||
## --request_timeout
|
||||
- **Default:** `6000`
|
||||
- **Type:** `int`
|
||||
- Set the timeout in seconds for completion calls.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --request_timeout 300
|
||||
```
|
||||
|
||||
## --drop_params
|
||||
### --local
|
||||
- **Default:** `False`
|
||||
- **Type:** `bool` (Flag)
|
||||
- Drop any unmapped params.
|
||||
- For local debugging purposes.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --drop_params
|
||||
litellm --local
|
||||
```
|
||||
|
||||
## --add_function_to_prompt
|
||||
## Testing & Health Checks
|
||||
|
||||
### --test
|
||||
- **Type:** `bool` (Flag)
|
||||
- If a function passed but unsupported, pass it as a part of the prompt.
|
||||
- Proxy chat completions URL to make a test request to.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --add_function_to_prompt
|
||||
litellm --test
|
||||
```
|
||||
|
||||
## --config
|
||||
- Configure Litellm by providing a configuration file path.
|
||||
### --test_async
|
||||
- **Default:** `False`
|
||||
- **Type:** `bool` (Flag)
|
||||
- Calls async endpoints `/queue/requests` and `/queue/response`.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --config path/to/config.yaml
|
||||
litellm --test_async
|
||||
```
|
||||
|
||||
## --telemetry
|
||||
### --num_requests
|
||||
- **Default:** `10`
|
||||
- **Type:** `int`
|
||||
- Number of requests to hit async endpoint with (used with `--test_async`).
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --test_async --num_requests 100
|
||||
```
|
||||
|
||||
### --health
|
||||
- **Type:** `bool` (Flag)
|
||||
- Runs a health check on all models in config.yaml.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --health
|
||||
```
|
||||
|
||||
## Other Options
|
||||
|
||||
### --version
|
||||
- **Short form:** `-v`
|
||||
- **Type:** `bool` (Flag)
|
||||
- Print LiteLLM version and exit.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --version
|
||||
```
|
||||
|
||||
### --telemetry
|
||||
- **Default:** `True`
|
||||
- **Type:** `bool`
|
||||
- Help track usage of this feature.
|
||||
- Help track usage of this feature. Turn off for privacy.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --telemetry False
|
||||
```
|
||||
|
||||
|
||||
## --log_config
|
||||
- **Default:** `None`
|
||||
- **Type:** `str`
|
||||
- Specify a log configuration file for uvicorn.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --log_config path/to/log_config.conf
|
||||
```
|
||||
|
||||
## --skip_server_startup
|
||||
### --use_queue
|
||||
- **Default:** `False`
|
||||
- **Type:** `bool` (Flag)
|
||||
- Skip starting the server after setup (useful for DB migrations only).
|
||||
- To use celery workers for async endpoints.
|
||||
- **Usage:**
|
||||
```shell
|
||||
litellm --skip_server_startup
|
||||
```
|
||||
litellm --use_queue
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user