mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-25 14:27:26 +00:00
1cd1d23fdf
* docs(bedrock.md): clarify bedrock auth in litellm docs * fix(convert_dict_to_response.py): Fixes https://github.com/BerriAI/litellm/issues/6387 * feat(pattern_match_deployments.py): more robust handling for wildcard routes (model_name: custom_route/* -> openai/*) Enables user to expose custom routes to users with dynamic handling * test: add more testing * docs(custom_pricing.md): add debug tutorial for custom pricing * test: skip codestral test - unreachable backend * test: fix test * fix(pattern_matching_deployments.py): fix typing * test: cleanup codestral tests - backend api unavailable * (refactor) prometheus async_log_success_event to be under 100 LOC (#6416) * unit testig for prometheus * unit testing for success metrics * use 1 helper for _increment_token_metrics * use helper for _increment_remaining_budget_metrics * use _increment_remaining_budget_metrics * use _increment_top_level_request_and_spend_metrics * use helper for _set_latency_metrics * remove noqa violation * fix test prometheus * test prometheus * unit testing for all prometheus helper functions * fix prom unit tests * fix unit tests prometheus * fix unit test prom * (refactor) router - use static methods for client init utils (#6420) * use InitalizeOpenAISDKClient * use InitalizeOpenAISDKClient static method * fix # noqa: PLR0915 * (code cleanup) remove unused and undocumented logging integrations - litedebugger, berrispend (#6406) * code cleanup remove unused and undocumented code files * fix unused logging integrations cleanup * bump: version 1.50.3 → 1.50.4 --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
2.3 KiB
2.3 KiB
import Image from '@theme/IdealImage';
Custom LLM Pricing
Use this to register custom pricing for models.
There's 2 ways to track cost:
- cost per token
- cost per second
By default, the response cost is accessible in the logging object via kwargs["response_cost"] on success (sync + async). Learn More
:::info
LiteLLM already has pricing for any model in our model cost map.
:::
Cost Per Second (e.g. Sagemaker)
Usage with LiteLLM Proxy Server
Step 1: Add pricing to config.yaml
model_list:
- model_name: sagemaker-completion-model
litellm_params:
model: sagemaker/berri-benchmarking-Llama-2-70b-chat-hf-4
input_cost_per_second: 0.000420
- model_name: sagemaker-embedding-model
litellm_params:
model: sagemaker/berri-benchmarking-gpt-j-6b-fp16
input_cost_per_second: 0.000420
Step 2: Start proxy
litellm /path/to/config.yaml
Step 3: View Spend Logs
<Image img={require('../../img/spend_logs_table.png')} />
Cost Per Token (e.g. Azure)
Usage with LiteLLM Proxy Server
model_list:
- model_name: azure-model
litellm_params:
model: azure/<your_deployment_name>
api_key: os.environ/AZURE_API_KEY
api_base: os.environ/AZURE_API_BASE
api_version: os.envrion/AZURE_API_VERSION
input_cost_per_token: 0.000421 # 👈 ONLY to track cost per token
output_cost_per_token: 0.000520 # 👈 ONLY to track cost per token
Debugging
If you're custom pricing is not being used or you're seeing errors, please check the following:
- Run the proxy with
LITELLM_LOG="DEBUG"or the--detailed_debugcli flag
litellm --config /path/to/config.yaml --detailed_debug
- Check logs for this line:
LiteLLM:DEBUG: utils.py:263 - litellm.acompletion
- Check if 'input_cost_per_token' and 'output_cost_per_token' are top-level keys in the acompletion function.
acompletion(
...,
input_cost_per_token: my-custom-price,
output_cost_per_token: my-custom-price,
)
If these keys are not present, LiteLLM will not use your custom pricing.
If the problem persists, please file an issue on GitHub.