Commit Graph

526 Commits

Author SHA1 Message Date
Krish Dholakia e1f7bcb47d Fix VertexAI Credential Caching issue (#9756)
* refactor(vertex_llm_base.py): Prevent credential misrouting for projects

Fixes https://github.com/BerriAI/litellm/issues/7904

* fix: passing unit tests

* fix(vertex_llm_base.py): common auth logic across sync + async vertex ai calls

prevents credential caching issue across both flows

* test: fix test

* fix(vertex_llm_base.py): handle project id in default cause

* fix(factory.py): don't pass cache control if not set

bedrock invoke does not support this

* test: fix test

* fix(vertex_llm_base.py): add .exception message in load_auth

* fix: fix ruff error
2025-04-04 16:38:08 -07:00
Ishaan Jaff 888446256c fix vertex failing test 2025-04-04 15:37:48 -07:00
Ishaan Jaff afcd00bdc0 test_redis_caching_llm_caching_ttl 2025-04-02 21:54:35 -07:00
Ishaan Jaff acf920a41a Merge branch 'main' into litellm_fix_azure_o_series 2025-04-02 20:58:52 -07:00
Ishaan Jaff c3341a1e18 test fixes - azure deprecated dall-e-2 2025-04-02 20:56:20 -07:00
Ishaan Jaff 8f372ea243 test_completion_invalid_param_cohere 2025-04-02 06:49:11 -07:00
Ishaan Jaff 61b609f320 Merge pull request #9673 from BerriAI/litellm_qa_deadlock_fixes
[Reliability] - Ensure new Redis + DB architecture tracks spend accurately
2025-04-01 12:04:03 -07:00
Ishaan Jaff 7a2442d6c0 test_batch_update_spend 2025-04-01 07:12:29 -07:00
Krish Dholakia 722f3ff0e6 fix(cost_calculator.py): allows checking received + sent model name when checking for cost calculation (#9669)
Fixes issue introduced by https://github.com/BerriAI/litellm/commit/dfb838eaff82301d4101d09982fbbb251bbc1ce1#r154667517
2025-03-31 21:29:48 -07:00
Krish Dholakia 5ac61a7572 Add bedrock latency optimized inference support (#9623)
* fix(converse_transformation.py): add performanceConfig param support on bedrock

Closes https://github.com/BerriAI/litellm/issues/7606

* fix(converse_transformation.py): refactor to use more flexible single getter for params which are separate config blocks

* test(test_main.py): add e2e mock test for bedrock performance config

* build(model_prices_and_context_window.json): add versioned multimodal embedding

* refactor(multimodal_embeddings/): migrate to config pattern

* feat(vertex_ai/multimodalembeddings): calculate usage for multimodal embedding calls

Enables cost calculation for multimodal embeddings

* feat(vertex_ai/multimodalembeddings): get usage object for embedding calls

ensures accurate cost tracking for vertexai multimodal embedding calls

* fix(embedding_handler.py): remove unused imports

* fix: fix linting errors

* fix: handle response api usage calculation

* test(test_vertex_ai_multimodal_embedding_transformation.py): update tests

* test: mark flaky test

* feat(vertex_ai/multimodal_embeddings/transformation.py): support text+image+video input

* docs(vertex.md): document sending text + image to vertex multimodal embeddings

* test: remove incorrect file

* fix(multimodal_embeddings/transformation.py): fix linting error

* style: remove unused import
2025-03-29 00:23:09 -07:00
Ishaan Jaff 7e8a02099c Merge branch 'main' into litellm_use_redis_for_updates 2025-03-28 20:12:29 -07:00
Krrish Dholakia 28a9edb547 test(test_caching_handler.py): move to in-memory cache - prevent redis flakiness from impacting ci/cd 2025-03-28 13:32:04 -07:00
Ishaan Jaff 758182fc7f fix typo on codebase 2025-03-27 22:36:00 -07:00
Krish Dholakia 63c9f59373 Allow team admins to add/update/delete models on UI + show api base and model id on request logs (#9572)
* feat(view_logs.tsx): show model id + api base in request logs

easier debugging

* fix(index.tsx): fix length of api base

easier viewing

* refactor(leftnav.tsx): show models tab to team admin

* feat(model_dashboard.tsx): add explainer for what the 'models' page is for team admin

helps them understand how they can use it

* feat(model_management_endpoints.py): restrict model add by team to just team admin

allow team admin to add models via non-team keys (e.g. ui token)

* test(test_add_update_models.py): update unit testing for new behaviour

* fix(model_dashboard.tsx): show user the models

* feat(proxy_server.py): add new query param 'user_models_only' to `/v2/model/info`

Allows user to retrieve just the models they've added

Used in UI to show internal users just the models they've added

* feat(model_dashboard.tsx): allow team admins to view their own models

* fix: allow ui user to fetch model cost map

* feat(add_model_tab.tsx): require team admins to specify team when onboarding models

* fix(_types.py): add `/v1/model/info` to info route

`/model/info` was already there

* fix(model_info_view.tsx): allow user to edit a model they created

* fix(model_management_endpoints.py): allow team admin to update team model

* feat(model_managament_endpoints.py): allow team admin to delete team models

* fix(model_management_endpoints.py): don't require team id to be set when adding a model

* fix(proxy_server.py): fix linting error

* fix: fix ui linting error

* fix(model_management_endpoints.py): ensure consistent auth checks on all model calls

* test: remove old test - function no longer exists in same form

* test: add updated mock testing
2025-03-27 12:06:31 -07:00
Krish Dholakia c0845fec1f Add OpenAI gpt-4o-transcribe support (#9517)
* refactor: introduce new transformation config for gpt-4o-transcribe models

* refactor: expose new transformation configs for audio transcription

* ci: fix config yml

* feat(openai/transcriptions): support provider config transformation on openai audio transcriptions

allows gpt-4o and whisper audio transformation to work as expected

* refactor: migrate fireworks ai + deepgram to new transform request pattern

* feat(openai/): working support for gpt-4o-audio-transcribe

* build(model_prices_and_context_window.json): add gpt-4o-transcribe to model cost map

* build(model_prices_and_context_window.json): specify what endpoints are supported for `/audio/transcriptions`

* fix(get_supported_openai_params.py): fix return

* refactor(deepgram/): migrate unit test to deepgram handler

* refactor: cleanup unused imports

* fix(get_supported_openai_params.py): fix linting error

* test: update test
2025-03-26 23:10:25 -07:00
Krrish Dholakia 109add7946 build(model_prices_and_context_window.json): add gemini multimodal embedding cost 2025-03-26 23:04:24 -07:00
Krish Dholakia 4351c77253 Support Gemini audio token cost tracking + fix openai audio input token cost tracking (#9535)
* fix(vertex_and_google_ai_studio_gemini.py): log gemini audio tokens in usage object

enables accurate cost tracking

* refactor(vertex_ai/cost_calculator.py): refactor 128k+ token cost calculation to only run if model info has it

Google has moved away from this for gemini-2.0 models

* refactor(vertex_ai/cost_calculator.py): migrate to usage object for more flexible data passthrough

* fix(llm_cost_calc/utils.py): support audio token cost tracking in generic cost per token

enables vertex ai cost tracking to work with audio tokens

* fix(llm_cost_calc/utils.py): default to total prompt tokens if text tokens field not set

* refactor(llm_cost_calc/utils.py): move openai cost tracking to generic cost per token

more consistent behaviour across providers

* test: add unit test for gemini audio token cost calculation

* ci: bump ci config

* test: fix test
2025-03-26 17:26:25 -07:00
Ishaan Jaff 8eaf4c55c0 test_gemini_fine_tuned_model_request_consistency 2025-03-26 14:18:11 -07:00
Ishaan Jaff da9d849348 test_gemini_fine_tuned_model_request_consistency 2025-03-26 14:10:32 -07:00
Ishaan Jaff baa9b34950 Merge branch 'main' into litellm_fix_vertex_ai_ft_models 2025-03-26 11:11:54 -07:00
Ishaan Jaff bbe69a47a9 _is_model_gemini_gemini_spec_model 2025-03-26 10:53:23 -07:00
Ishaan Jaff e7a8b5a809 run ci/cd again 2025-03-26 08:12:51 -07:00
Ishaan Jaff efce84815a test_gemini_fine_tuned_model_request_consistency 2025-03-25 23:54:06 -07:00
Krish Dholakia 6fd18651d1 Support litellm.api_base for vertex_ai + gemini/ across completion, embedding, image_generation (#9516)
* test(tests): add unit testing for litellm_proxy integration

* fix(cost_calculator.py): fix tracking cost in sdk when calling proxy

* fix(main.py): respect litellm.api_base on `vertex_ai/` and `gemini/` routes

* fix(main.py): consistently support custom api base across gemini + vertexai on embedding + completion

* feat(vertex_ai/): test

* fix: fix linting error

* test: set api base as None before starting loadtest
2025-03-25 23:46:20 -07:00
Ishaan Jaff 9aec7c3878 test_create_delete_assistants 2025-03-25 22:08:06 -07:00
Krrish Dholakia 75994d0bf0 test: improve flaky test 2025-03-24 23:15:04 -07:00
Tyler Hutcherson 7864cd1f76 update redisvl dependency 2025-03-24 08:42:11 -04:00
Ishaan Jaff 69c9a782b2 add supports_web_search 2025-03-22 13:32:22 -07:00
Ishaan Jaff 78c371d2e8 search_context_cost_per_query test 2025-03-22 13:08:57 -07:00
Ishaan Jaff 1bdb94a314 add search_context_cost_per_1k_calls to model cost map spec 2025-03-22 12:56:21 -07:00
Krrish Dholakia 48e6a7036b test: mock sagemaker tests 2025-03-21 16:21:18 -07:00
Krrish Dholakia 46d68a61c8 fix: fix testing 2025-03-20 14:37:58 -07:00
Krish Dholakia 706bcf4432 Merge pull request #9366 from JamesGuthrie/jg/vertex-output-dimensionality
fix: VertexAI outputDimensionality configuration
2025-03-20 13:55:33 -07:00
Ishaan Jaff 247e4d09ee Merge branch 'main' into litellm_fix_ssl_verify 2025-03-19 21:03:06 -07:00
James Guthrie 437dbe7246 fix: VertexAI outputDimensionality configuration
VertexAI's API documentation [1] is an absolute mess. In it, they
describe the parameter to configure output dimensionality as
`output_dimensionality`. In the API example, they switch to using snake
case `outputDimensionality`, which is the correct variant.

[1]: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings-api#generative-ai-get-text-embedding-drest
2025-03-19 11:07:36 +01:00
Ishaan Jaff e32aee9124 Merge pull request #9353 from BerriAI/litellm_arize_dynamic_logging
[Feat] - API - Allow using dynamic Arize AI Spaces on LiteLLM
2025-03-18 23:35:28 -07:00
Krish Dholakia 6347b694ee Merge pull request #9335 from BerriAI/litellm_dev_03_17_2025_p3
Contributor PR: Fix sagemaker too little data for content error
2025-03-18 23:24:07 -07:00
Ishaan Jaff 57e5c94360 Merge branch 'main' into litellm_arize_dynamic_logging 2025-03-18 22:13:35 -07:00
Ishaan Jaff c101fe9b5d Merge pull request #9352 from BerriAI/litellm_arize_mar_18
[Bug Fix] Arize AI Logging Integration with LiteLLM
2025-03-18 22:12:46 -07:00
Ishaan Jaff 412ad0d64e test_arize_callback 2025-03-18 20:21:23 -07:00
Ishaan Jaff 19a7bfa6b5 test_arize_callback 2025-03-18 18:49:06 -07:00
Krrish Dholakia a34cc2031d fix(response_metadata.py): log the litellm_model_name
make it easier to track the model sent to the provider
2025-03-18 17:46:33 -07:00
Krrish Dholakia 8ed3483adb test(test_tpm_rpm_routing_v2.py): initial test, for asserting async pre call check works as expected 2025-03-18 17:36:55 -07:00
Ishaan Jaff c010cdef59 test_dynamic_azure_params 2025-03-18 17:26:23 -07:00
Krrish Dholakia 39ac9e3eca fix(lowest_tpm_rpm_v2.py): fix updating limits 2025-03-18 17:10:17 -07:00
Ishaan Jaff 41d9cc14be test_async_dynamic_arize_config 2025-03-18 15:02:22 -07:00
Ishaan Jaff 3e0df0fc1e test_arize_set_attributes 2025-03-18 14:31:34 -07:00
Ishaan Jaff 959299c8ea Merge pull request #9338 from nate-mar/arize-integration-fixes
Arize integration Fix
2025-03-18 14:28:25 -07:00
Krrish Dholakia 267084a1af test(test_get_llm_provider.py): cover scenario where xai not in model name 2025-03-18 11:04:59 -07:00
Krrish Dholakia aeec703c4e test(test_get_llm_provider.py): Minimal repro for https://github.com/BerriAI/litellm/issues/9291 2025-03-18 10:35:50 -07:00