2bc5d93f23
use_callback_in_llm_call
2025-10-01 18:32:37 -07:00
82948b9a1a
test: update test
2025-09-27 16:30:52 -07:00
6aa35ec999
test text-embedding-ada-002
2025-09-27 12:41:35 -07:00
c27beb74b9
test fix
2025-09-27 12:40:34 -07:00
ddf918cb42
use_callback_in_llm_call
2025-09-27 11:55:41 -07:00
3a8e4f3160
test_async_embedding_azure_caching
2025-09-27 09:54:54 -07:00
1f42ac1aec
test_async_completion_azure_caching_streaming
2025-09-27 09:48:24 -07:00
9790451cce
test_custom_callback_router.py
2025-09-27 09:23:49 -07:00
afbebcd626
test_async_chat_azure
2025-09-27 09:12:37 -07:00
8510c70416
test fixes
2025-09-27 09:11:43 -07:00
e04df1416d
test_async_chat_azure
2025-09-27 09:07:44 -07:00
198777f2d2
Merge branch 'main' into feature/sqs_pushes_errors
2025-09-27 08:51:35 -04:00
3d0ec68798
Make errors go through SQS
2025-09-27 08:51:29 -04:00
2dd38420a7
[Feat] - Cost Tracking - show input, output, tool call cost breakdown in StandardLoggingPayload ( #14921 )
...
* add new CostBreakdown typed dict
* fix CostBreakdown type
* fix fix _store_cost_breakdown_in_logging_obj
* fix CostBreakdown
* test_cost_breakdown_in_standard_logging_payload
2025-09-25 15:48:22 -07:00
eaa04cd8ce
fix: use fastuuid helper ( #14903 )
...
* fix: use fastuuid helper across the codebase
First batch of changes, simple drop in replacement.
* second batch of changes
* fixed: script mistake on helper file
2025-09-25 15:47:01 -07:00
ab5fb704c2
[Feat] Logging - datadog callback Log message content w/o sending to datadog ( #14909 )
...
* add DatadogInitParams
* fix _get_datadog_params
* test_datadog_message_redaction
2025-09-25 15:46:22 -07:00
df88b0656e
test: update .json
2025-09-24 16:13:51 -07:00
d739d226ed
fix: test
2025-09-19 16:28:09 -07:00
3018c66867
fix: test_awesome_otel_with_message_logging_off
2025-09-18 07:49:39 -07:00
ff36dfdc76
Merge pull request #14637 from akraines/feature/middle-truncate-spend-logs
...
feat: implement middle-truncation for spend log payloads
2025-09-17 23:47:04 -07:00
8b804303ed
fix: ci/cd tests + lint errors ( #14646 )
...
* fix: lint errors + tests
* fixed ci tests
* fixed tests
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com >
2025-09-17 17:06:43 -07:00
e168161e64
Feat/add posthog observability ( #14610 )
...
* feat: add posthog observability
* docs: add posthog logging docs
* docs: posthog integration in proxy mode
2025-09-17 08:24:04 -07:00
115a3e9ded
feat: implement middle-truncation for spend log payloads
...
- Change truncation strategy from head-only to middle-truncation (35% start, 65% end)
- Preserve both beginning and end of long strings for better debugging context
- Apply same sanitization to response payloads when store_prompts_in_spend_logs is enabled
- Increase default MAX_STRING_LENGTH_PROMPT_IN_DB from 1000 to 2048 characters
- Update tests to verify new truncation behavior with 35%-65% split
This provides better diagnostic value by keeping the more important end context
while still maintaining storage limits.
2025-09-17 16:30:25 +03:00
0721345703
Switch to string constant based truncation
2025-08-28 16:28:36 -06:00
b9132968b2
[Perf] Improvements for Async Success Handler (Logging Callbacks) - Approx +130 RPS ( #13905 )
...
* [Performance] Reduce Significant CPU overhead from litellm_logging.py (#13895 )
* fix: litellm.configured_cold_storage_logger
* fix Session Management - Non-OpenAI Models docs
* ruff fix
* test fix
* create LoggingWorker
* add GLOBAL_LOGGING_WORKER for async task handling
* fix logging tests
* add conftest
* fix conftest
* test fix location of encode bedrock runtime modelid arn
* fix conftest.py
* tuning LoggingWorker
* conftest.py
* fix conftest batches/
* test_async_chat_azure
* event_loop
* test_bedrock_streaming_passthrough_test2
* fix GLOBAL_LOGGING_WORKER
* logging worker
* add flush for global logging worker
* Revert "fix GLOBAL_LOGGING_WORKER"
This reverts commit d254f508f48935652f054777652938ad71976cce.
* fix conftest clear_queue
* fix conftest clear_queue
* setup_and_teardown for llm translation
* docs AWS_REGION
* test_async_chat_azure
* change test DIR
* run ci/cd again
* use 1 job for litellm_router_unit_testing
* fix space
* fix litellm_router_unit_testing
* test_aaarouter_dynamic_cooldown_message_retry_time
* litellm_router_unit_testing
* conftest.py clearing qu
* fixes litellm_router_unit_testing
* fixes clear_queue
* fix router_unit_tests
* remove conftest
* add back conftest for router
* fix event loop test
* test fix
* fixes for LoggingWorker
* ruff fix
2025-08-23 13:13:23 -07:00
d96df5e9be
Revert "test_stream_token_counting_anthropic_with_include_usage"
...
This reverts commit c3aee1194b
2025-08-16 13:07:00 -07:00
c3aee1194b
test_stream_token_counting_anthropic_with_include_usage
2025-08-16 13:06:31 -07:00
8bb5ee2ba5
test: update unit test
2025-08-16 12:39:03 -07:00
4d941c914e
[Feat] Responses API Session Handling - Multi media support ( #13347 )
...
* rename ResponsesSessionHandler
* use ResponsesSessionHandler
* test session handler
* refactor ResponsesSessionHandler
* fix get_proxy_server_request_from_spend_log
* use constant for LITELLM_TRUNCATED_PAYLOAD_FIELD
* add _should_check_cold_storage_for_full_payload
* add get_class_type_for_custom_logger_name
* get_active_custom_logger_for_callback_name
* add get_proxy_server_request_from_cold_storage to CustomLogger
* add ColdStorageHandler
* start using cold storage integration
* add get_proxy_server_request_from_cold_storage
* fixes from manual testing
* s3 v2 fix getting region name
* ChatCompletionImageUrlObject
* use _get_configured_cold_storage_custom_logger
* fixes for _should_check_cold_storage_for_full_payload
* fix _download_object_from_s3
* test_s3_v2_with_cold_storage
* add cold_storage_object_key to StandardLoggingMetadata
* use get_proxy_server_request_from_cold_storage_with_object_key
* add cold_storage_object_key to SpendLogsMetadata
* add cold_storage_object_key
* get_proxy_server_request_from_cold_storage_with_object_key
* use get_proxy_server_request_from_cold_storage_with_object_key
* test responses API
* add get_proxy_server_request_from_cold_storage_with_object_key
* session handler fixes
* test session handler
* fix ruff checks
* _download_object_from_s3
* cleanup
* test
* lint fix
* test_e2e_cold_storage_successful_retrieval
* test_e2e_generate_cold_storage_object_key_successful
* test_async_gcs_pub_sub_v1
* test fix
* test fix
* test fix
* test_standard_logging_metadata_has_cold_storage_object_key_field
* test_sanitize_request_body_for_spend_logs_payload_basic
* test_transform_input_image_item_to_image_item_with_image_data
2025-08-07 10:59:53 -07:00
30fc5b871c
feat(integrations): allow setting of braintrust callback base url ( #13368 )
...
* feat(integrations): allow setting of braintrust callback base url
* chore(misc): remove extra additions due to merge
2025-08-07 08:40:11 -07:00
ee70d593c1
[Feat] Allow redacting message / response content for specific logging integrations - DD LLM Observability ( #13158 )
...
* fix redact_standard_logging_payload
* add StandardCustomLoggerInitParams
* allow defining DatadogLLMObsInitParams
* fix init DataDogLLMObsLogger
* fix import
* update redact_standard_logging_payload_from_model_call_details
* test_dd_llms_obs_redaction
* docs DD logging
* docs DD
* docs DD
* Redacting Messages, Response docs DD LLM Obs
* fix redaction logic
* fix create_llm_obs_payload
* fix logging response
* fixes
* ruff fix
* fix test
* test_dd_llms_obs_redaction
* test_create_llm_obs_payload
* redact_standard_logging_payload_from_model_call_details
* img - dd_llm_obs
* docs DD
* fix linting
* fix linting
* fix mypy
* test_create_llm_obs_payload
* test_create_llm_obs_payload
* fix mock_env_vars
* fix _handle_anthropic_messages_response_logging
2025-07-31 16:44:16 -07:00
c7774ba495
add fix for redaction ( #13005 )
2025-07-25 17:48:48 -07:00
a46b9d376f
[Prometheus] Move Prometheus to enterprise folder ( #12659 )
...
* fix tools fetch for keys
* add promethues to enterprise
* remove old prom
* remove old prom
* fix tests
* safe imports
* add if
* fix enterprise test
* rename imports
* added label import
* added label import
* move tests to enterprise
* fix tests
* add log
* build: update versions
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com >
2025-07-18 11:54:47 -07:00
33c84846e9
[Refactor] Vector Stores - Use class VectorStorePreCallHook for all Vector Store Integrations ( #12715 )
...
* add VectorStorePreCallHook
* vector_store_pre_call_hook
* add pop_vector_stores_to_run
* async_get_chat_completion_prompt
* working e2e tests
* test_e2e_bedrock_knowledgebase_retrieval_with_completion
* delete old files
* fix logging test
* VectorStorePreCallHook
* fix ruff check
* vector_store_pre_call_hook
* linting error fixes
2025-07-17 16:31:58 -07:00
d8327b4740
[Bug Fix] [Bug]: Knowledge Base Call returning error ( #12628 )
...
* bug fix using vector stores as tools
* test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call_with_tools
2025-07-15 21:33:49 -07:00
749051105b
Team Members - reset budget, if duration set + Prometheus - support tag based metrics ( #12534 )
...
* fix(internal_user_endpoints.py): initial commit removing logic to create new budget for new user if default max budget in team set
* feat(proxy_setting_endpoints.py): update team member budget when set via default internal user endpoint
removes need to create a unique budget per user
* feat(proxy_server.py): set team max member budget on startup, if set on config.yaml
* fix(prometheus.py): support custom tags for tracking on prometheus
Allows tracking user agent values on prometheus metrics
* test(test_internal_user_endpoints.py): fix test
2025-07-11 22:54:16 -07:00
c4af2eb5e2
MCP - usage tracking ( #12397 )
...
* fix(common_daily_activity.py): initial commit with working mock BE endpoint for mcp usage
* feat(ui/): show mcp server activity on UI
allows admin to know which mcp's are being used
* feat(common_daily_activity.py): return activity by key
* feat(ui/): show top api keys for a given model / mcp server
allow user to know which key is driving spend
* fix(common_daily_activity.py): use known mcp server names
* feat(server.py): log the namespaced tool name (includes server prefix)
allow accurate cost tracking
* feat(db_spend_update_writer.py): log by mcp_namespaced_tool_name
store aggregate daily activity by mcp_namespaced_tool_name
Enables cost / usage tracking by mcp tool name
* fix(server.py): add key/user metadata to mcp calls
* refactor(common_daily_activity.py): update to return mcp activity in API
* fix(common_daily_activity.py): handle empty key
* fix(common_daily_activity.py): track when api key is empty
* test(test_spend_management_endpoints.py): update tests
* fix: fix ui linting error
* fix: fix linting errors
* test: add missing key
* build(schema.prisma): add mcp tool tracking
* fix(migration.sql): add schema migration file
* feat(server.py): add request logging for mcp calls
enables storing the mcp calls
* fix(new_usage.tsx): fix linting errors
* fix: fix code qa errors
* fix(activity_metrics.tsx): fix ui linting errors post-merge
* fix(types/utils.py): fix linting error
* fix(server.py): always have name
2025-07-08 22:08:16 -07:00
6f27385edf
Ensure message redaction works for responses API logging ( #12291 )
...
* add fixes to choice implementation redaction
* add isInstance check on responses API
* change datadog to revert back
* change datadog to revert back
* fix type errors
* Redaction test changes
* Redaction test changes
* Redaction test changes - remove changes
2025-07-04 15:11:20 -07:00
3d71b49d11
[Feat] Add failure logging support for s3 logger ( #12299 )
...
* add async_log_failure_event
* test_basic_s3_v2_logging_failure
2025-07-04 11:11:30 -07:00
66fafa3a7f
[Feat] Polish - add better error validation when users configure prometheus metrics and labels to control cardinality ( #12182 )
...
* self._pretty_print_invalid_metric_error
* docs prometheus.md
* test prom validation checks
* update metric name
* fix _pretty_print_validation_errors
* fix linting
* test prometheus
* test fixes - prometheus
2025-07-01 20:17:17 -07:00
d727d63a81
[Feat] Add new AWS SQS Logging Integration ( #12176 )
...
* add aws_sqs
* add sqs controls
* add SQS to registry
* fix url lib parse
* fixes AWS SQS
* test_async_sqs_logger_flush
* fix test
* fix SQS logger auth
* add AWS SQS
* add aws sqs
* docs logging
* test_async_sqs_logger_flush
* test_async_sqs_logger_flush
* add SQS logger
* update SQS logging
* use constants for SQS
2025-06-30 14:02:49 -07:00
2bb8048864
[Feat] Add OpenAI Search Vector Store Operation ( #12018 )
...
* add BaseVectorStoreTransformation
* fix BaseVectorStoreTransformation
* add OpenAIVectorStoreTransformation
* fix transform
* add search, asearch vector stores
* add skeleton for vector store searching
* fix VectorStoreSearchOptionalRequestParams
* fix VectorStoreRequestUtils
* fix litellm.asearch/litellm.search
* fix BaseVectorStoreConfig
* add vector_store_search_handler to llm http handler
* use llm http handler for searching vector stores
* fix base vector store config
* fix vector_store_search_handler
* async_vector_store_search_handler
* add conftest
* add BaseVectorStoreTest
* move litellm.integrations.vector_store_integrations
* fix working OAI OpenAIVectorStoreConfig
* add Search vector store
* add OpenAI Vector Stores
2025-06-24 15:52:43 -07:00
8c5fb6f539
[Feat] Enterprise - Allow dynamically disabling callbacks in request headers ( #11985 )
...
* Add support for disabling callbacks via x-litellm-disable-callbacks header
* add _is_callback_disabled_via_headers
* add get_proxy_server_request_headers
* _is_callback_disabled_via_headers
* X_LITELLM_DISABLE_CALLBACKS
* add EnterpriseCallbackControls
* use EnterpriseCallbackControls
* use CustomLoggerRegistry
* use CustomLoggerRegistry
* CustomLoggerRegistry
* EnterpriseCallbackControls
* TestEnterpriseCallbackControls
* docs clean up
* docs dynamic callbacks
* doc fixes
* fix code qa checks
* fix CustomLoggerRegistry
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
2025-06-23 14:32:05 -07:00
931b2e4875
[Bug Fix] Fix model_group tracked for /v1/messages and /moderations ( #11933 )
...
* fixes _get_router_metadata_variable_name
* fixes _update_kwargs_before_fallbacks
* test_anthropic_messages_litellm_router_non_streaming_with_logging
* test_moderations_api_logging
* fix _pass_through_moderation_endpoint_factory
2025-06-20 14:51:50 -07:00
3540984965
test: fix tests
2025-06-19 16:35:31 -07:00
bcdb53920a
[Fix] Bug Fix for using prom metrics config ( #11779 )
...
* fixes for using config for prom metrics
* test_set_llm_deployment_success_metrics_with_label_filtering
* fixes for deployment failure
* fix code qa checks
* test_async_post_call_success_hook
2025-06-17 14:44:24 -07:00
bb907b5ecc
test: fix test
2025-06-16 18:49:41 -07:00
7a128e2017
VertexAI Anthropic - streaming passthrough cost tracking ( #11734 )
...
* feat(vertex_passthrough_logging_handler.py): initial anthropic passthrough streaming cost tracking support
* fix: fix linting errors
* test: update test
2025-06-15 01:16:43 -07:00
c92b6c175c
Prometheus - fix request increment + add route tracking for streaming requests ( #11731 )
...
* fix(prometheus.py): remove request increment from inside the log success event
it's only done on post-call success/failure
* fix(litellm_logging.py): add additional validation step for checking if 'stream' is true
prevent double counting on non-stream requests
* test: add unit testing to ensure stream is not incorrectly set to true
* feat(litellm_logging.py): emit request route in standard logging payload
used by prometheus streaming metrics for route
* fix: fix otel test
* fix: fix linting errors
* test: update test
* fix: fix linting error
2025-06-14 16:26:48 -07:00
ad82792c4b
fix lf OTEL
2025-06-14 15:43:38 -07:00
Ishaan Jaffer