Commit Graph

174 Commits

Author SHA1 Message Date
Krish Dholakia 010a4d44af Fix passing standard optional params (#11124)
* fix(main.py): use processed non-default-params as standard input params for langfuse

Fixes https://github.com/BerriAI/litellm/issues/11072

 Fixes https://github.com/BerriAI/litellm/issues/11096

* fix(main.py): rename variable to be more accurate

* test(test_langfuse_e2e_test.py): add router unit test for langfuse e2e testing

Prevent https://github.com/BerriAI/litellm/issues/11072 from happening again

* build: update lock

* fix(utils.py): refactor optional params function

make it easier to get the standardized non default params

* fix(utils.py): improve process non default params function

* fix(main.py): include provider specific params in processed non default params used in logging

ensures user can see any provider specific params on langfuse

 ensures user can see any provider specific params on langfus e
2025-05-24 12:12:31 -07:00
Ishaan Jaff 86cdb8382b [Feat] Use aiohttp transport by default - 97% lower median latency (#11097)
* fix: add flag for disabling use_aiohttp_transport

* feat: add _create_async_transport

* feat: fixes for transport

* add httpx-aiohttp

* feat: fixes for transport

* refactor: fixes for transport

* build: fix deps

* fixes: test fixes

* fix: ensure aiohttp does not auto set content type

* test: test fixes

* feat: add LiteLLMAiohttpTransport

* fix: fixes for responses API handling

* test: fixes for responses API handling

* test: fixes for responses API handling

* feat: fixes for transport

* fix: base embedding handler

* test: test_async_http_handler_force_ipv4

* test: fix failing deepeval test

* fix: add YARL for bedrock urls

* fix: issues with transport

* fix: comment out linting issues

* test fix

* test: XAI is unstable

* test: fixes for using respx

* test: XAI fixes

* test: XAI fixes

* test: infinity testing fixes

* docs(config_settings.md): document param

* test: test_openai_image_edit_litellm_sdk

* test: remove deprecated test

* bump respx==0.22.0

* test: test_xai_message_name_filtering

* test: fix anthropic test after bumping httpx

* use n 4 for mapped tests (#11109)

* fix: use 1 session per event loop

* test: test_client_session_helper

* fix: linting error

* fix: resolving GET requests on httpx 0.28.1

* test fixes proxy unit tests

* fix: add ssl verify settings

* fix: proxy unit tests

* fix: refactor

* tests: basic unit tests for aiohttp transports

* tests: fixes xai

---------

Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
2025-05-23 22:55:35 -07:00
Krish Dholakia 2efaa3cf36 Expose /list and /info endpoints for Audit Log events (#11102)
* feat(audit_logging_endpoints.py): expose list endpoint to show all audit logs

make it easier for user to retrieve individual endpoints

* feat(enterprise/): add audit logging endpoint

* feat(audit_logging_endpoints.py): expose new GET `/audit/{id}` endpoint

make it easier to retrieve view individual audit logs

* feat(key_management_event_hooks.py): correctly show the key of the user who initiated the change

* fix(key_management_event_hooks.py): add key rotations as an audit log event

'

* test(test_audit_logging_endpoints.py): add simple unit testing for audit log endpoint

* fix: testing fixes

* fix: fix ruff check
2025-05-23 22:54:59 -07:00
Ishaan Jaff 13bbf11ab0 test: fix failing deepeval test 2025-05-23 14:40:39 -07:00
Ishaan Jaff 754a94db97 Revert "Revert "Support passing prompt_label to langfuse (#11018)""
This reverts commit 0be7e7d088.
2025-05-22 14:14:39 -07:00
Ishaan Jaff 0be7e7d088 Revert "Support passing prompt_label to langfuse (#11018)"
This reverts commit 2b50b43ae2.
2025-05-22 14:11:19 -07:00
Krish Dholakia 2b50b43ae2 Support passing prompt_label to langfuse (#11018)
* fix: add prompt label support to prompt management hook

* feat: support 'prompt_label' parameter for langfuse prompt management

Closes https://github.com/BerriAI/litellm/discussions/9003#discussioncomment-13221555

* fix(litellm_logging.py): deep copy optional params to avoid mutation while logging

* fix(log-consistent-optional-param-values-across-providers): ensures params can be used for finetuning from providers

* fix: fix linting error

* test: update test

* test: update langfuse tests

* fix(litellm_logging.py): avoid deepcopying optional params

might contain thread object
2025-05-21 22:27:36 -07:00
Ishaan Jaff 14321a2708 [Feat] Prometheus - Track route on proxy_* metrics (#10992)
* fix: trace route on prometheus metrics

* fix: show route on prometheus metrics for total fails

* test: trace route on metrics

* fix: tests for route in prom metrics

* test: fix test metrics

* test: fix test_proxy_failure_metrics
2025-05-20 22:55:55 -07:00
Ishaan Jaff 298912bd38 [UI] - Add Guardrail Tracing to LiteLLM SpendLogs (#10893)
* feat: trace guardrail SLP in spendLogs

* test: trace guardrail SLP in spendLogs

* add guardrail viewer

* checkpoint - working guardrail view on logs

* ui add guardrail view to SpendLogs

* test: fixes guardrails

* trace: fixes guardrails
2025-05-16 12:20:20 -07:00
Ishaan Jaff 42e6e664b2 [Refactor] Make Pagerduty a free feature (#10857)
* refactor: make pagerduty free

* refactor: make pagerduty free

* fix: pagerduty loc

* fix: linting error
2025-05-15 10:12:06 -07:00
Ishaan Jaff 2a994d7016 [Feat] Presidio Improvements - Allow adding presidio api base on UI, Test presidio on Test Key, fixes for running presidio hook (#10840)
* feat: add GuardrailProviderSpecificParams

* feat: add add apply_guardrail helper for presidio

* ui cleanup

* fixes pii config on ui

* fixes for adding presidio pii

* refactor: InitializeGuardrails

* feat: init guardrails from DB

* allow running guardrails from test key pg

* fix: running a request with guardrails on UI

* fix: types/guardrails.py

* test: test_presidio_pre_call_hook_with_different_call_types

* test: test_initialize_presidio_guardrail

* test: fix custom guardrail tests
2025-05-14 17:41:33 -07:00
Krish Dholakia a421316e56 fix(litellm_logging.py): log custom headers in requester metadata (#10818)
* fix(litellm_logging.py): log custom headers in requester metadata

allows passing along custom headers from client to logging integration - e.g. `x-correlation-id`

* refactor: move enterprise code out of OSS package

work towards simplified CE version of docker image

* test: update test

* fix: fix linting error
2025-05-13 23:04:37 -07:00
Ishaan Jaff a4fb1da2d9 fix: pass application/json for GenericAPILogger (#10772)
* fix: pass application/json for GenericAPILogger

* fix: test_generic_api_callback
2025-05-12 14:15:33 -07:00
Ishaan Jaff d13117992c fix: test_init_custom_logger_compatible_class_as_callback 2025-05-10 17:26:12 -07:00
Ishaan Jaff 3731ee436a [Refactor] Use pip package for enterprise/ folder (#10709)
* init enterprise pip

* init enterprise pip

* init enterprise pip

* test: enterprise pip

* add litellm-enterprise to pip

* litellm ent check

* litellm ent check

* fix import email router

* fix setup_litellm_enterprise_pip

* fix local testing with enterprise pip
2025-05-09 17:18:48 -07:00
Ishaan Jaff d9f2415f65 [Refactor - Filtering Spend Logs] Add status to root of SpendLogs table (#10661)
* add status to spendlogs

* add status to spendlogs

* add status to spendlogs

* add status to spendlogs

* fix tests with spendlogs
2025-05-08 11:29:25 -07:00
Ishaan Jaff dd32860d62 [Feat] V2 Emails - Fixes for sending emails when creating keys + Resend API support (#10602)
* working email integration

* fix get_custom_loggers_for_type

* add SendKeyCreatedEmailEvent type

* bug fix, only send 1 email when creating key for user

* polish for emails for key created

* polish for key created email

* fix test_init_custom_logger_compatible_class_as_callback

* testing resend email integration

* testing fixes for email integration
2025-05-06 22:50:48 -07:00
Krish Dholakia 500e6cddf5 Fix otel - follow genai semantic conventions + support 'instructions' param for tts (#10608)
* fix(opentelemetry.py): fix passing otel semantic conventions

Fixes SpanAttributes.LLM_PROMPTS to SpanAttributes.LLM_PROMPTS.value

* fix(opentelemetry.py): ensure spanattributes always pass the actual enum value

* fix(main.py): support passing 'instructions' param for gpt-4o-mini-tts

* test: update tests
2025-05-06 21:57:01 -07:00
Ishaan Jaff 489f1a6c25 [Feat] v2 Custom Logger API Endpoints (#10575)
* fixes for generic api logger

* tests for generic api logger

* test_generic_api_callback_multiple_logs

* allow health checking generic api endpoints

* docs generic api endpoint for logging

* allow setting headers for generic api  callback

* fix for test_init_custom_logger_compatible_class_as_callback

* fix linting
2025-05-05 16:57:55 -07:00
Ishaan Jaff 3f45200802 test_openai_responses_api_web_search_cost_tracking 2025-05-03 20:22:42 -07:00
Ishaan Jaff 8faa7dbc39 fixes for web search tests 2025-05-03 18:10:30 -07:00
Ishaan Jaff a37e4cb0b8 fix typo (#10535) 2025-05-03 11:39:40 -07:00
Ishaan Jaff 988e20aa36 [QA] Bedrock Vector Stores Integration - Allow using with registry + in OpenAI API spec with tools (#10516)
* refactor KB implementation to use central registry

* allow passing tools when making KB calls

* test fixes

* linting fix

* fix kb tests

* QA for KB stored in DB

* fix, use litellm_credential_name when adding KB on litellm UI

* QA list endpoint vector stores

* allow using UI creds with KBs
2025-05-03 08:30:38 -07:00
Ishaan Jaff 28cb7cc0ed [Feat] KnowledgeBase/Vector Store - Log StandardLoggingVectorStoreRequest for requests made when a vector store is used (#10509)
* ensure vector store results are logged in SLP

* fix tests

* fix tests with vector_store_request_metadata

* fix linting
2025-05-02 13:43:20 -07:00
Ishaan Jaff 96e75628d6 [Fix + Refactor] Trigger Soft Budget Webhooks When Key Crosses Threshold (#10491)
* fix slack alerting with webhooks

* emit correct event group/entity on webhooks

* refactor to use a common class of alerts with abc methods

* fixes for tests

* refactor to use a common class of alerts with abc methods

* Send a budget alert on slack or webhook

* unit test slack alerting

* fix code qa
2025-05-02 07:06:07 -07:00
Ishaan Jaff be885e4a17 [Feat] Vector Stores/KnowledgeBases - Allow defining Vector Store Configs (#10448)
* init vector store configs

* working kb init

* add vector store endpoints

* use litellm_credential_name

* working CRUD vector stores litellm

* working creds with vector DB

* ui cleanup

* clean up vector store id

* fix delete button

* refactored vector store component

* working selector for KBs

* ui vector stores

* add vector store tool calls usage on chat ui

* fixes for vector stores litellm

* test fix

* docs Knowledge Bases

* fixes for vector stores litellm

* fix linting

* add managed vectorstores

* fix orjson ci/cd test

* fix linting

* add types.tsx file
2025-04-30 21:49:59 -07:00
Ishaan Jaff f30871ef13 [Feat] Add support for using Bedrock Knowledge Bases with LiteLLM /chat/completions requests (#10413)
* add make_bedrock_kb_retrieve_request

* working bedrock KB hook

* working bedrock KB hook

* test_openai_with_knowledge_base_mock_openai

* fix linting

* fix BedrockKnowledgeBaseHook

* docs using bedrock kb with litellm

* docs kb with litellm

* fix bedrock kb test

* DynamicPromptManagementParamLiteral

* fix _should_run_prompt_management_hooks_without_prompt_id

* test_init_custom_logger_compatible_class_as_callback
2025-04-29 17:29:02 -07:00
Ishaan Jaff 34cdefbb77 [Feat] Add logging callback support for /moderations API (#10390)
* add OpenAIModerationResponse as a type

* Add typing for litellm.moderation response obj

* Add typing for litellm.moderation response obj

* test_moderations_api_logging

* add assertions for logged slp object

* test_moderations_api_logging

* refactor moderations api usage

* Revert "test_moderations_api_logging"

This reverts commit 5a9001f04c592b063f016d2bb854416255ec6cdf.

* Revert "refactor moderations api usage"

This reverts commit b076d3174ca14251ee3fc28bd4744e25685c1e5f.

* test_moderations_api_logging
2025-04-28 20:24:17 -07:00
Ishaan Jaff 331e784db4 [Feat] Responses API - Add session management support for non-openai models (#10321)
* add session id in spendLogs

* fix log proxy server request as independant field

* use trace id for SpendLogs

* add _ENTERPRISE_ResponsesSessionHandler

* use _ENTERPRISE_ResponsesSessionHandler

* working session_ids

* working session management

* working session_ids

* test_async_gcs_pub_sub_v1

* test_spend_logs_payload_e2e

* working session_ids

* test_get_standard_logging_payload_trace_id

* test_get_standard_logging_payload_trace_id

* test_gcs_pub_sub.py

* fix all linting errors

* test_spend_logs_payload_with_prompts_enabled

* _ENTERPRISE_ResponsesSessionHandler

* _ENTERPRISE_ResponsesSessionHandler

* expose session id on ui

* get spend logs by session

* add sessionSpendLogsCall

* add session handling

* session logs

* ui session details

* fix on rowExpandDetails

* ui working sessions
2025-04-25 23:24:24 -07:00
Ishaan Jaff 96e31d205c feat: Added Missing Attributes For Arize & Phoenix Integration (#10043) (#10215)
* feat: Added Missing Attributes For Arize & Phoenix Integration

* chore: Added noqa for PLR0915 to suppress warning

* chore: Moved Contributor Test to Correct Location

* chore: Removed Redundant Fallback

Co-authored-by: Ali Saleh <saleh.a@turing.com>
2025-04-22 21:34:51 -07:00
Ishaan Jaff 44264ab6d6 fix failing agent ops test 2025-04-22 14:39:50 -07:00
Krrish Dholakia b4f2b3dad1 test: update test to be more robust to usage updates 2025-04-19 09:26:26 -07:00
Krish Dholakia 1ea046cc61 test: update tests to new deployment model (#10142)
* test: update tests to new deployment model

* test: update model name

* test: skip cohere rbac issue test

* test: update test - replace gpt-4o model
2025-04-18 14:22:12 -07:00
Ishaan Jaff c1a642ce20 [UI] Allow setting prompt cache_control_injection_points (#10000)
* test_anthropic_cache_control_hook_system_message

* test_anthropic_cache_control_hook.py

* should_run_prompt_management_hooks

* fix should_run_prompt_management_hooks

* test_anthropic_cache_control_hook_specific_index

* fix test

* fix linting errors

* ChatCompletionCachedContent

* initial commit for cache control

* fixes ui design

* fix inserting cache_control_injection_points

* fix entering cache control points

* fixes for using cache control on ui + backend

* update cache control settings on edit model page

* fix init custom logger compatible class

* fix linting errors

* fix linting errors

* fix get_chat_completion_prompt
2025-04-14 21:17:42 -07:00
Ishaan Jaff f9ce754817 [Feat] Add litellm.supports_reasoning() util to track if an llm supports reasoning (#9923)
* add supports_reasoning for xai models

* add "supports_reasoning": true for o1 series models

* add supports_reasoning util

* add litellm.supports_reasoning

* add supports reasoning for claude 3-7 models

* add deepseek as supports reasoning

* test_supports_reasoning

* add supports reasoning to model group info

* add supports_reasoning

* docs supports reasoning

* fix supports_reasoning test

* "supports_reasoning": false,

* fix test

* supports_reasoning
2025-04-11 17:56:04 -07:00
Ishaan Jaff 94a553dbb2 [Feat] Emit Key, Team Budget metrics on a cron job schedule (#9528)
* _initialize_remaining_budget_metrics

* initialize_budget_metrics_cron_job

* initialize_budget_metrics_cron_job

* initialize_budget_metrics_cron_job

* test_initialize_budget_metrics_cron_job

* LITELLM_PROXY_ADMIN_NAME

* fix code qa checks

* test_initialize_budget_metrics_cron_job

* test_initialize_budget_metrics_cron_job

* pod lock manager allow dynamic cron job ID

* fix pod lock manager

* require cronjobid for PodLockManager

* fix DB_SPEND_UPDATE_JOB_NAME acquire / release lock

* add comment on prometheus logger

* add debug statements for emitting key, team budget metrics

* test_pod_lock_manager.py

* test_initialize_budget_metrics_cron_job

* initialize_budget_metrics_cron_job

* initialize_remaining_budget_metrics

* remove outdated test
2025-04-10 16:59:14 -07:00
Krrish Dholakia b168f8b744 test: update test 2025-04-10 14:04:57 -07:00
Krish Dholakia 0c5b4aa96d feat(realtime/): add token tracking + log usage object in spend logs … (#9843)
* feat(realtime/): add token tracking + log usage object in spend logs metadata

* test: fix test

* test: update tests

* test: update testing

* test: update test

* test: update test

* test: update test

* test: update test

* test: update tesdt

* test: update test
2025-04-09 22:11:00 -07:00
Krish Dholakia 4a128cfd64 Realtime API Cost tracking (#9795)
* fix(proxy_server.py): log realtime calls to spendlogs

Fixes https://github.com/BerriAI/litellm/issues/8410

* feat(realtime/): OpenAI Realtime API cost tracking

Closes https://github.com/BerriAI/litellm/issues/8410

* test: add unit testing for coverage

* test: add more unit testing

* fix: handle edge cases
2025-04-07 16:43:12 -07:00
Ishaan Jaff b7cd4cef07 test_get_exception_class_name 2025-04-04 21:32:55 -07:00
Ishaan Jaff df4593d58b test prom unit tests 2025-04-04 21:30:05 -07:00
Ishaan Jaff a3df0269bb fix tests 2025-03-29 17:38:24 -07:00
Ishaan Jaff 047d767947 fix tests for gcs pub sub 2025-03-29 17:06:36 -07:00
Ishaan Jaff d17ab7da2a Merge pull request #9508 from BerriAI/litellm_fix_gcs_pub_sub
[Fix] Use StandardLoggingPayload for GCS Pub Sub Logging Integration
2025-03-24 18:22:43 -07:00
Ishaan Jaff 36d17c23ac fix flaky test test_openai_responses_api_web_search_cost_tracking 2025-03-24 15:51:33 -07:00
Ishaan Jaff 6a0ff944e4 assert_gcs_pubsub_request_matches_expected_standard_logging_payload 2025-03-24 15:41:05 -07:00
Ishaan Jaff 43a6c4c635 test_openai_responses_api_web_search_cost_tracking 2025-03-22 17:29:17 -07:00
Ishaan Jaff 81033dc8c9 test_openai_responses_api_web_search_cost_tracking 2025-03-22 17:21:41 -07:00
Ishaan Jaff beb68037d0 move web search cost tracking 2025-03-22 17:02:43 -07:00
Ishaan Jaff 475dfaa156 fixes for web search cost tracking 2025-03-22 16:56:32 -07:00