Marc Abramowitz
a79fd772f4
Simplify mock_patch_acompletion
2024-05-02 12:47:27 -07:00
Marc Abramowitz
6ec058711a
Make unnecessary to pass extra arg for mock object
...
Modify `mock_patch_acompletion` to be a context manager instead of a
function that returns a mock object. This way, the mock object is
created and yielded by the context manager, and the test function
doesn't need to pass the mock object as an argument.
2024-05-02 12:41:30 -07:00
Marc Abramowitz
4dfadb0cf4
mock_patch_acompletion in test_proxy_server.py
2024-05-02 12:24:49 -07:00
Marc Abramowitz
152b5c8ceb
Add test_openai_deployments_model_chat_completions_azure
2024-05-02 10:27:32 -07:00
Marc Abramowitz
dd166680d1
Move chat_completions before completions
...
so that the `chat_completions` route is defined before the `completions` route.
This is necessary because the `chat_completions` route is more
specific than the `completions` route, and the order of route definitions
matters in FastAPI.
Without this, doing a request to
`/openai/deployments/{model_in_url}/chat/completions` might trigger
`completions` being called (with `model` set to `{model_in_url}/chat` instead of
`chat_completions` getting called, which is the correct function.
Fixes: GH-3372
2024-04-30 15:07:10 -07:00
Krrish Dholakia
285a3733a9
test(test_image_generation.py): fix test
2024-04-30 12:14:29 -07:00
Krrish Dholakia
398d503590
build(model_prices_and_context_window.json): add bedrock llama3 pricing
2024-04-30 11:36:29 -07:00
Krrish Dholakia
00d1440d0d
test(test_image_generation.py): change img model for test - bedrock EOL
2024-04-30 08:55:40 -07:00
Krrish Dholakia
d717fa2588
test(test_tpm_rpm_routing_v2.py): fix test - bump number of iteration s
2024-04-30 08:48:55 -07:00
Krrish Dholakia
1cd24d8906
bump: version 1.35.32 → 1.35.33
2024-04-30 07:20:50 -07:00
Krrish Dholakia
020b175ef4
fix(lowest_tpm_rpm_v2.py): skip if item_tpm is None
2024-04-29 21:34:25 -07:00
Ishaan Jaff
81df36b298
docs - slack alerting
2024-04-29 21:33:03 -07:00
Ishaan Jaff
b1e888edad
docs example logging to langfuse
2024-04-29 21:26:27 -07:00
Ishaan Jaff
0cad58f5c6
docs logging to langfuse on proxy
2024-04-29 21:26:15 -07:00
Ishaan Jaff
0c99ae9451
docs - fix kub.yaml config on docs
2024-04-29 21:20:29 -07:00
Krrish Dholakia
b46db8b891
feat(utils.py): json logs for raw request sent by litellm
...
make it easier to view verbose logs in datadog
2024-04-29 19:21:19 -07:00
Krrish Dholakia
f0e48cdd53
fix(router.py): raise better exception when no deployments are available
...
Fixes https://github.com/BerriAI/litellm/issues/3355
2024-04-29 18:48:04 -07:00
Krrish Dholakia
1e53c06064
test(test_router_caching.py): remove unstable test
...
test would fail due to timing issues
2024-04-29 18:37:31 -07:00
Krrish Dholakia
e7b4882e97
fix(router.py): fix high-traffic bug for usage-based-routing-v2
2024-04-29 16:48:01 -07:00
Krish Dholakia and GitHub
09bae3d8ad
Merge pull request #3351 from elisalimli/main
...
Fix Cohere tool calling
2024-04-29 16:45:48 -07:00
Krish Dholakia and GitHub
32534b5e91
Merge pull request #3358 from sumanth13131/usage-based-routing-RPM-fix
...
usage based routing RPM count fix
2024-04-29 16:45:25 -07:00
Krrish Dholakia
bd79e8b516
docs(langfuse_integration.md): add 'existing_trace_id' to langfuse docs
2024-04-29 16:40:38 -07:00
Krrish Dholakia
853b70aba9
fix(langfuse.py): support 'existing_trace_id' param
...
allow user to call out a trace as pre-existing, this prevents creating a default trace name, and potentially overwriting past traces
2024-04-29 16:39:17 -07:00
Krrish Dholakia
2cf069befb
fix(langfuse.py): don't set default trace_name if trace_id given
2024-04-29 16:39:17 -07:00
Ishaan Jaff and GitHub
d58dd2cbeb
Merge pull request #3360 from BerriAI/litellm_random_pick_lowest_latency
...
[Fix] Lowest Latency routing - random pick deployments when all latencies=0
2024-04-29 16:31:32 -07:00
Krrish Dholakia
77f155d158
docs(load_test.md): cleanup docs
2024-04-29 16:27:58 -07:00
Krrish Dholakia
af6a21f27c
docs(load_test.md): add multi-instance router load test to docs
2024-04-29 16:25:56 -07:00
Ishaan Jaff
4cb4a7f06d
fix - lowest latency routing
2024-04-29 16:02:57 -07:00
Krrish Dholakia
8f830bd948
docs(load_test.md): simplify doc
2024-04-29 16:00:02 -07:00
Krrish Dholakia
fcb83781ec
docs(load_test.md): formatting
2024-04-29 15:58:41 -07:00
Krrish Dholakia
5fe0f38558
docs(load_test.md): load test multiple instances of the proxy w/ tpm/rpm limits on deployments
2024-04-29 15:58:14 -07:00
Ishaan Jaff
3b0aa05378
fix lowest latency - routing
2024-04-29 15:51:52 -07:00
Ishaan Jaff
5247d7b6a5
test - lowest latency router
2024-04-29 15:51:01 -07:00
Krrish Dholakia
cef2d95bb4
docs(routing.md): add max parallel requests to router docs
2024-04-29 15:37:48 -07:00
Krrish Dholakia
a978f2d881
fix(lowest_tpm_rpm_v2.py): shuffle deployments with same tpm values
2024-04-29 15:23:47 -07:00
Krrish Dholakia
f10a066d36
fix(lowest_tpm_rpm_v2.py): add more detail to 'No deployments available' error message
2024-04-29 15:04:37 -07:00
Ishaan Jaff and GitHub
de3e642999
Merge pull request #3359 from BerriAI/litellm_docs_trackin_cost
...
docs - update track cost with custom callbacks
2024-04-29 13:19:25 -07:00
Ishaan Jaff
8d26030b99
docs - track cost custom callbacks
2024-04-29 13:15:08 -07:00
sumanth
89e655c79e
usage based routing RPM count fix
2024-04-30 00:29:38 +05:30
Krrish Dholakia
4b04b017df
bump: version 1.35.31 → 1.35.32
2024-04-29 09:16:44 -07:00
Krish Dholakia and GitHub
ec2510029a
Merge pull request #3354 from BerriAI/litellm_replicate_cost_tracking
...
fix(utils.py): replicate now also has token based pricing for some models
2024-04-29 09:13:41 -07:00
Krrish Dholakia
3725732c4d
fix(utils.py): default to time-based tracking for unmapped replicate models. fix time-based cost calc for replicate
2024-04-29 08:36:01 -07:00
Krrish Dholakia
a18844b230
fix(utils.py): use llama tokenizer for replicate models
2024-04-29 08:28:31 -07:00
Krrish Dholakia
dc5c175406
build(model_prices_and_context_window.json): add token-based replicate costs to model cost map
2024-04-29 08:20:44 -07:00
Krrish Dholakia
ab954243e8
fix(utils.py): fix watson streaming
2024-04-29 08:09:59 -07:00
Krrish Dholakia
2cfb97141d
fix(utils.py): replicate now also has token based pricing for some models
2024-04-29 08:06:15 -07:00
alisalim17
0aa8b94ff5
test: completion with Cohere command-r-plus model
2024-04-29 18:38:12 +04:00
Krrish Dholakia
0a6b6302f1
fix(router.py): fix typing error
2024-04-29 07:25:39 -07:00
Krrish Dholakia
7b617e666d
fix(proxy_server.py): return more detailed auth error message.
2024-04-29 07:24:19 -07:00
Krish Dholakia and GitHub
ffc6af0b22
Merge pull request #3334 from CyanideByte/main
...
protected_namespaces warning fixed for model_name & model_info
2024-04-29 07:16:05 -07:00
Marc Abramowitz