mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-02 20:22:10 +00:00
7d73ab5d88
* Add `litellm-proxy` CLI (#10478) * First cut at a Python client module for proxy * Add UnauthorizedError + add_model method * Add delete_model method * Add example model_id to delete_model docstring * Make delete_model raise NotFoundError * Add get_model * Add get_all_model_info * Rename models.list_models to models.list * Rename models.get_all_model_info to models.info * Move ModelsManagementClient.get_all_model_group_info to ModelGroupsManagementClient.info * Rename get_model to get * Rename add_model to new * Rename delete_model to delete * In client classes, rename base_url attribute to _base_url and api_key attribute to _api_key * Add ModelsManagementClient.updae method * Add client.chat.completions (ChatClient) * ruff format litellm/proxy/client * ruff format tests/litellm/proxy/client/*.py * Add latest changes * Rename KeysManagementClient.create to KeysManagementClient.generate * Add new parameters to KeysManagementClient.generate * Add CredentialsManagementClient * Remove api_key parameter from KeysManagementClient.generate * Fix lint errors * Add litellm/proxy/client/README.md * README.md: Remove api_key param to client.keys.generate * Fix mypy errors * First cut at litellm-proxy cli * Add test for `litellm-proxy models list` * Nicer get_models_info * get_models_info: --columns option * Use format_timestamp in list_models * ruff format litellm/proxy/client * Simpler JSON printing with rich.print_json * Move models-related commands to separate file From `cli.py` to `groups/models.py` * Improve directory structure * Cleanup cli/groups/models.py - esp. usage of rich * Refactoring * Refactor mocking in cli/test_main.py * Dedup models commands tests * Update poetry.lock * Fix mypy errors * ruff format litellm/proxy/client/cli * ruff format tests/litellm/proxy/client/*.py * Fix timezone issue in test_models_list_table_format * Add cli/README.md * Small README.md tweaks * README.md enhancements * Add credentials commands * Add chat commands * Add http commands * ruff format litellm/proxy/client/cli * Fix lint errors in credentials and http commands * json => json_lib * test-key => sk-test-key * Mock HTTP responses so http command tests pass * Fix mypy error in credentials.py * bump: version 1.67.5 → 1.67.6 * build: update litellm version * cli/main.py: show_envvar=True * Increase test job timeout to 8 minutes because it looks like maybe the job is getting canceled because it takes too long with the additional tests? This probably could be reverted once #10484 is merged, since that speeds up pytest runs greatly. * Add keys functionality to library/CLI * Add info about keys commands to litellm/proxy/client/cli/README.md * Move Model Information section in CLI README * Make Model Information a level 4 heading * Move rich to extras as suggested by @ishaan-jaff --------- Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com> * pin rich=13.7.1 --------- Co-authored-by: Marc Abramowitz <abramowi@adobe.com> Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
litellm-proxy
A local, fast, and lightweight OpenAI-compatible server to call 100+ LLM APIs.
usage
$ pip install litellm
$ litellm --model ollama/codellama
#INFO: Ollama running on http://0.0.0.0:8000
replace openai base
import openai # openai v1.0.0+
client = openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:8000") # set proxy to base_url
# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
{
"role": "user",
"content": "this is a test request, write a short poem"
}
])
print(response)
See how to call Huggingface,Bedrock,TogetherAI,Anthropic, etc.
Folder Structure
Routes
proxy_server.py- all openai-compatible routes -/v1/chat/completion,/v1/embedding+ model info routes -/v1/models,/v1/model/info,/v1/model_group_inforoutes.health_endpoints/-/health,/health/liveliness,/health/readinessmanagement_endpoints/key_management_endpoints.py- all/key/*routesmanagement_endpoints/team_endpoints.py- all/team/*routesmanagement_endpoints/internal_user_endpoints.py- all/user/*routesmanagement_endpoints/ui_sso.py- all/sso/*routes