Commit Graph

156 Commits

Author SHA1 Message Date
Ishaan Jaff 489f1a6c25 [Feat] v2 Custom Logger API Endpoints (#10575)
* fixes for generic api logger

* tests for generic api logger

* test_generic_api_callback_multiple_logs

* allow health checking generic api endpoints

* docs generic api endpoint for logging

* allow setting headers for generic api  callback

* fix for test_init_custom_logger_compatible_class_as_callback

* fix linting
2025-05-05 16:57:55 -07:00
Ishaan Jaff 3f45200802 test_openai_responses_api_web_search_cost_tracking 2025-05-03 20:22:42 -07:00
Ishaan Jaff 8faa7dbc39 fixes for web search tests 2025-05-03 18:10:30 -07:00
Ishaan Jaff a37e4cb0b8 fix typo (#10535) 2025-05-03 11:39:40 -07:00
Ishaan Jaff 988e20aa36 [QA] Bedrock Vector Stores Integration - Allow using with registry + in OpenAI API spec with tools (#10516)
* refactor KB implementation to use central registry

* allow passing tools when making KB calls

* test fixes

* linting fix

* fix kb tests

* QA for KB stored in DB

* fix, use litellm_credential_name when adding KB on litellm UI

* QA list endpoint vector stores

* allow using UI creds with KBs
2025-05-03 08:30:38 -07:00
Ishaan Jaff 28cb7cc0ed [Feat] KnowledgeBase/Vector Store - Log StandardLoggingVectorStoreRequest for requests made when a vector store is used (#10509)
* ensure vector store results are logged in SLP

* fix tests

* fix tests with vector_store_request_metadata

* fix linting
2025-05-02 13:43:20 -07:00
Ishaan Jaff 96e75628d6 [Fix + Refactor] Trigger Soft Budget Webhooks When Key Crosses Threshold (#10491)
* fix slack alerting with webhooks

* emit correct event group/entity on webhooks

* refactor to use a common class of alerts with abc methods

* fixes for tests

* refactor to use a common class of alerts with abc methods

* Send a budget alert on slack or webhook

* unit test slack alerting

* fix code qa
2025-05-02 07:06:07 -07:00
Ishaan Jaff be885e4a17 [Feat] Vector Stores/KnowledgeBases - Allow defining Vector Store Configs (#10448)
* init vector store configs

* working kb init

* add vector store endpoints

* use litellm_credential_name

* working CRUD vector stores litellm

* working creds with vector DB

* ui cleanup

* clean up vector store id

* fix delete button

* refactored vector store component

* working selector for KBs

* ui vector stores

* add vector store tool calls usage on chat ui

* fixes for vector stores litellm

* test fix

* docs Knowledge Bases

* fixes for vector stores litellm

* fix linting

* add managed vectorstores

* fix orjson ci/cd test

* fix linting

* add types.tsx file
2025-04-30 21:49:59 -07:00
Ishaan Jaff f30871ef13 [Feat] Add support for using Bedrock Knowledge Bases with LiteLLM /chat/completions requests (#10413)
* add make_bedrock_kb_retrieve_request

* working bedrock KB hook

* working bedrock KB hook

* test_openai_with_knowledge_base_mock_openai

* fix linting

* fix BedrockKnowledgeBaseHook

* docs using bedrock kb with litellm

* docs kb with litellm

* fix bedrock kb test

* DynamicPromptManagementParamLiteral

* fix _should_run_prompt_management_hooks_without_prompt_id

* test_init_custom_logger_compatible_class_as_callback
2025-04-29 17:29:02 -07:00
Ishaan Jaff 34cdefbb77 [Feat] Add logging callback support for /moderations API (#10390)
* add OpenAIModerationResponse as a type

* Add typing for litellm.moderation response obj

* Add typing for litellm.moderation response obj

* test_moderations_api_logging

* add assertions for logged slp object

* test_moderations_api_logging

* refactor moderations api usage

* Revert "test_moderations_api_logging"

This reverts commit 5a9001f04c592b063f016d2bb854416255ec6cdf.

* Revert "refactor moderations api usage"

This reverts commit b076d3174ca14251ee3fc28bd4744e25685c1e5f.

* test_moderations_api_logging
2025-04-28 20:24:17 -07:00
Ishaan Jaff 331e784db4 [Feat] Responses API - Add session management support for non-openai models (#10321)
* add session id in spendLogs

* fix log proxy server request as independant field

* use trace id for SpendLogs

* add _ENTERPRISE_ResponsesSessionHandler

* use _ENTERPRISE_ResponsesSessionHandler

* working session_ids

* working session management

* working session_ids

* test_async_gcs_pub_sub_v1

* test_spend_logs_payload_e2e

* working session_ids

* test_get_standard_logging_payload_trace_id

* test_get_standard_logging_payload_trace_id

* test_gcs_pub_sub.py

* fix all linting errors

* test_spend_logs_payload_with_prompts_enabled

* _ENTERPRISE_ResponsesSessionHandler

* _ENTERPRISE_ResponsesSessionHandler

* expose session id on ui

* get spend logs by session

* add sessionSpendLogsCall

* add session handling

* session logs

* ui session details

* fix on rowExpandDetails

* ui working sessions
2025-04-25 23:24:24 -07:00
Ishaan Jaff 96e31d205c feat: Added Missing Attributes For Arize & Phoenix Integration (#10043) (#10215)
* feat: Added Missing Attributes For Arize & Phoenix Integration

* chore: Added noqa for PLR0915 to suppress warning

* chore: Moved Contributor Test to Correct Location

* chore: Removed Redundant Fallback

Co-authored-by: Ali Saleh <saleh.a@turing.com>
2025-04-22 21:34:51 -07:00
Ishaan Jaff 44264ab6d6 fix failing agent ops test 2025-04-22 14:39:50 -07:00
Krrish Dholakia b4f2b3dad1 test: update test to be more robust to usage updates 2025-04-19 09:26:26 -07:00
Krish Dholakia 1ea046cc61 test: update tests to new deployment model (#10142)
* test: update tests to new deployment model

* test: update model name

* test: skip cohere rbac issue test

* test: update test - replace gpt-4o model
2025-04-18 14:22:12 -07:00
Ishaan Jaff c1a642ce20 [UI] Allow setting prompt cache_control_injection_points (#10000)
* test_anthropic_cache_control_hook_system_message

* test_anthropic_cache_control_hook.py

* should_run_prompt_management_hooks

* fix should_run_prompt_management_hooks

* test_anthropic_cache_control_hook_specific_index

* fix test

* fix linting errors

* ChatCompletionCachedContent

* initial commit for cache control

* fixes ui design

* fix inserting cache_control_injection_points

* fix entering cache control points

* fixes for using cache control on ui + backend

* update cache control settings on edit model page

* fix init custom logger compatible class

* fix linting errors

* fix linting errors

* fix get_chat_completion_prompt
2025-04-14 21:17:42 -07:00
Ishaan Jaff f9ce754817 [Feat] Add litellm.supports_reasoning() util to track if an llm supports reasoning (#9923)
* add supports_reasoning for xai models

* add "supports_reasoning": true for o1 series models

* add supports_reasoning util

* add litellm.supports_reasoning

* add supports reasoning for claude 3-7 models

* add deepseek as supports reasoning

* test_supports_reasoning

* add supports reasoning to model group info

* add supports_reasoning

* docs supports reasoning

* fix supports_reasoning test

* "supports_reasoning": false,

* fix test

* supports_reasoning
2025-04-11 17:56:04 -07:00
Ishaan Jaff 94a553dbb2 [Feat] Emit Key, Team Budget metrics on a cron job schedule (#9528)
* _initialize_remaining_budget_metrics

* initialize_budget_metrics_cron_job

* initialize_budget_metrics_cron_job

* initialize_budget_metrics_cron_job

* test_initialize_budget_metrics_cron_job

* LITELLM_PROXY_ADMIN_NAME

* fix code qa checks

* test_initialize_budget_metrics_cron_job

* test_initialize_budget_metrics_cron_job

* pod lock manager allow dynamic cron job ID

* fix pod lock manager

* require cronjobid for PodLockManager

* fix DB_SPEND_UPDATE_JOB_NAME acquire / release lock

* add comment on prometheus logger

* add debug statements for emitting key, team budget metrics

* test_pod_lock_manager.py

* test_initialize_budget_metrics_cron_job

* initialize_budget_metrics_cron_job

* initialize_remaining_budget_metrics

* remove outdated test
2025-04-10 16:59:14 -07:00
Krrish Dholakia b168f8b744 test: update test 2025-04-10 14:04:57 -07:00
Krish Dholakia 0c5b4aa96d feat(realtime/): add token tracking + log usage object in spend logs … (#9843)
* feat(realtime/): add token tracking + log usage object in spend logs metadata

* test: fix test

* test: update tests

* test: update testing

* test: update test

* test: update test

* test: update test

* test: update test

* test: update tesdt

* test: update test
2025-04-09 22:11:00 -07:00
Krish Dholakia 4a128cfd64 Realtime API Cost tracking (#9795)
* fix(proxy_server.py): log realtime calls to spendlogs

Fixes https://github.com/BerriAI/litellm/issues/8410

* feat(realtime/): OpenAI Realtime API cost tracking

Closes https://github.com/BerriAI/litellm/issues/8410

* test: add unit testing for coverage

* test: add more unit testing

* fix: handle edge cases
2025-04-07 16:43:12 -07:00
Ishaan Jaff b7cd4cef07 test_get_exception_class_name 2025-04-04 21:32:55 -07:00
Ishaan Jaff df4593d58b test prom unit tests 2025-04-04 21:30:05 -07:00
Ishaan Jaff a3df0269bb fix tests 2025-03-29 17:38:24 -07:00
Ishaan Jaff 047d767947 fix tests for gcs pub sub 2025-03-29 17:06:36 -07:00
Ishaan Jaff d17ab7da2a Merge pull request #9508 from BerriAI/litellm_fix_gcs_pub_sub
[Fix] Use StandardLoggingPayload for GCS Pub Sub Logging Integration
2025-03-24 18:22:43 -07:00
Ishaan Jaff 36d17c23ac fix flaky test test_openai_responses_api_web_search_cost_tracking 2025-03-24 15:51:33 -07:00
Ishaan Jaff 6a0ff944e4 assert_gcs_pubsub_request_matches_expected_standard_logging_payload 2025-03-24 15:41:05 -07:00
Ishaan Jaff 43a6c4c635 test_openai_responses_api_web_search_cost_tracking 2025-03-22 17:29:17 -07:00
Ishaan Jaff 81033dc8c9 test_openai_responses_api_web_search_cost_tracking 2025-03-22 17:21:41 -07:00
Ishaan Jaff beb68037d0 move web search cost tracking 2025-03-22 17:02:43 -07:00
Ishaan Jaff 475dfaa156 fixes for web search cost tracking 2025-03-22 16:56:32 -07:00
Ishaan Jaff cf22d31b2b search_context_cost_per_query 2025-03-22 14:52:58 -07:00
Ishaan Jaff bddbeff717 test_openai_web_search_logging_cost_tracking 2025-03-22 14:29:02 -07:00
Ishaan Jaff 8690873488 test_arize_dynamic_params 2025-03-18 23:22:55 -07:00
Ishaan Jaff 8568caf532 test_arize_dynamic_params 2025-03-18 23:18:07 -07:00
Ishaan Jaff 19a7bfa6b5 test_arize_callback 2025-03-18 18:49:06 -07:00
Ishaan Jaff 00795f2bbf test arize logging 2025-03-18 18:40:39 -07:00
Ishaan Jaff 3e0df0fc1e test_arize_set_attributes 2025-03-18 14:31:34 -07:00
Ishaan Jaff 959299c8ea Merge pull request #9338 from nate-mar/arize-integration-fixes
Arize integration Fix
2025-03-18 14:28:25 -07:00
Nate Mar a1d188ba5e Fix test and add comments 2025-03-18 03:46:53 -07:00
Nate Mar 434e262b8c revert space_key change and add tests for arize integration 2025-03-18 01:40:10 -07:00
Nate Mar 35e0856f11 Fix wrong import and use space_id instead of space_key for Arize integration 2025-03-17 20:37:28 -07:00
Krrish Dholakia 359b8298f8 test(test_assemble_streaming_responses.py): update test to use correct type 2025-03-17 18:17:01 -07:00
Krrish Dholakia a5b497667c fix(logging_utils.py): revert change 2025-03-16 21:04:41 -07:00
Krish Dholakia 4330ef8e81 Fix batches api cost tracking + Log batch models in spend logs / standard logging payload (#9077)
* feat(batches/): fix batch cost calculation - ensure it's accurate

use the correct cost value - prev. defaulting to non-batch cost

* feat(batch_utils.py): log batch models to spend logs + standard logging payload

makes it easy to understand how cost was calculated

* fix: fix stored payload for test

* test: fix test
2025-03-08 11:47:25 -08:00
Krrish Dholakia 4418e6dd14 build: merge branch 2025-03-02 08:31:57 -08:00
Ishaan Jaff 1008da7c41 (UI) Error Logs improvements - Store Raw proxy server request for success and failure (#8917)
* ui render error logs

* add proxy_server_request to spend logs metadata

* _add_proxy_server_request_to_metadata

* proxy track cost callback fixes

* _add_proxy_server_request_to_metadata

* test spend logs

* test_async_post_call_failure_hook
2025-03-01 16:26:47 -08:00
Ishaan Jaff bc9b3e4847 (Bug fix) - don't log messages in model_parameters in StandardLoggingPayload (#8932)
* define model param helper

* use ModelParamHelper

* get_standard_logging_model_parameters

* fix code quality

* get_standard_logging_model_parameters

* StandardLoggingPayload

* test_get_kwargs_for_cache_key

* test_langsmith_key_based_logging

* fix code qa

* fix linting
2025-03-01 13:39:45 -08:00
Ishaan Jaff 3a086cee06 (Feat) - Show Error Logs on LiteLLM UI (#8904)
* fix test_moderations_bad_model

* use async_post_call_failure_hook

* basic logging errors in DB

* show status on ui

* show status on ui

* ui show request / response side by side

* stash fixes

* working, track raw request

* track error info in metadata

* fix showing error / request / response logs

* show traceback on error viewer

* ui with traceback of error

* fix async_post_call_failure_hook

* fix(http_parsing_utils.py): orjson can throw errors on some emoji's in text, default to json.loads

* test_get_error_information

* fix code quality

* rename proxy track cost callback test

* _should_store_errors_in_spend_logs

* feature flag error logs

* Revert "_should_store_errors_in_spend_logs"

This reverts commit 7f345df47762ff3be04e6fde2f13e70019ede4ee.

* Revert "feature flag error logs"

This reverts commit 0e90c022bbea3550f169118d81e60d711a4024fe.

* test_spend_logs_payload

* fix OTEL log_db_metrics

* fix import json

* fix ui linting error

* test_async_post_call_failure_hook

* test_chat_completion_bad_model_with_spend_logs

---------

Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
2025-02-28 20:10:09 -08:00