f77882948d
test_init_custom_logger_compatible_class_as_callback
2025-01-24 21:27:22 -08:00
4911cd80a1
fix(utils.py): move adding custom logger callback to success event in… ( #7905 )
...
* fix(utils.py): move adding custom logger callback to success event into separate function + don't add success callback to failure event
if user is explicitly choosing 'success' callback, don't log failure as well
* test(test_utils.py): add unit test to ensure custom logger callback only adds callback to specific event
* fix(utils.py): remove string from list of callbacks once corresponding callback class is added
prevents floating values - simplifies testing
* fix(utils.py): fix linting error
* test: cleanup args before test
* test: fix test
* test: update test
* test: fix test
2025-01-22 21:49:09 -08:00
55546f403b
Revert "fix: fix test"
...
This reverts commit 4e672f6269
2025-01-22 18:21:07 -08:00
4e672f6269
fix: fix test
2025-01-22 18:19:49 -08:00
5c870c0c51
(performance improvement - litellm sdk + proxy) - ensure litellm does not create unnecessary threads when running async functions ( #7680 )
...
* fix handle_sync_success_callbacks_for_async_calls
* fix handle_sync_success_callbacks_for_async_calls
* fix linting / testing errors
* use handle_sync_success_callbacks_for_async_calls
* add unit testing for logging fixes
2025-01-10 17:57:22 -08:00
03b1db5a7d
(Feat) - Add PagerDuty Alerting Integration ( #7478 )
...
* define basic types
* fix verbose_logger.exception statement
* fix basic alerting
* test pager duty alerting
* test_pagerduty_alerting_high_failure_rate
* PagerDutyAlerting
* async_log_failure_event
* use pre_call_hook
* add _request_is_completed helper util
* update AlertingConfig
* rename PagerDutyInternalEvent
* _send_alert_if_thresholds_crossed
* use pagerduty as _custom_logger_compatible_callbacks_literal
* fix slack alerting imports
* fix imports in slack alerting
* PagerDutyAlerting
* fix _load_alerting_settings
* test_pagerduty_hanging_request_alerting
* working pager duty alerting
* fix linting
* doc pager duty alerting
* update hanging_response_handler
* fix import location
* update failure_threshold
* update async_pre_call_hook
* docs pagerduty
* test - callback_class_str_to_classType
* fix linting errors
* fix linting + testing error
* PagerDutyAlerting
* test_pagerduty_hanging_request_alerting
* fix unused imports
* docs pager duty
* @pytest.mark.flaky(retries=6, delay=2)
* test_model_info_bedrock_converse_enforcement
2025-01-01 07:12:51 -08:00
41e5b3aa8d
HumanLoop integration for Prompt Management ( #7479 )
...
* feat(humanloop.py): initial commit for humanloop prompt management integration
Closes https://github.com/BerriAI/litellm/issues/213
* feat(humanloop.py): working e2e humanloop prompt management integration
Closes https://github.com/BerriAI/litellm/issues/213
* fix(humanloop.py): fix linting errors
* fix: fix linting erro
* fix: fix test
* test: handle filenotfound error
2024-12-30 22:26:03 -08:00
17d5ff2fa4
(fix) initializing OTEL Logging on LiteLLM Proxy - ensure OTEL logger is initialized only once ( #7435 )
...
* add otel to _custom_logger_compatible_callbacks_literal
* remove extra code
* fix _get_custom_logger_settings_from_proxy_server
* update unit tests
2024-12-26 21:17:19 -08:00
a790d43116
[Bug Fix]: ImportError: cannot import name 'T' from 're' ( #7314 )
...
* fix unused imports
* add test for python 3.12
* re introduce error - as a test
* update config for ci/cd
* fix python 13 install
* bump pyyaml
* bump numpy
* fix embedding requests
* bump pillow dep
* bump version
* bump pydantic
* bump tiktoken
* fix import
* fix python 3.13 import
* fix unused imports in tests/*
2024-12-19 13:09:30 -08:00
3c984ed60e
(feat) Add Azure Blob Storage Logging Integration ( #7265 )
...
* add path to http handler
* AzureBlobStorageLogger
* test_azure_blob_storage
* use constants for Azure storage
* use helper get_azure_ad_token_from_entrata_id
* azure blob storage support
* get_azure_ad_token_from_azure_storage
* fix import
* azure logging
* docs azure storage
* add docs on azure blobs
* add premium user check
* add azure_storage as identified logging callback
* async_upload_payload_to_azure_blob_storage
* docs azure storage
* callback_class_str_to_classType
2024-12-16 22:18:22 -08:00
19a4273fda
feat(langfuse/): support langfuse prompt management ( #7073 )
...
* feat(langfuse/): support langfuse prompt management
Initial working commit for langfuse prompt management support
Closes https://github.com/BerriAI/litellm/issues/6269
* test: update test
* fix(litellm_logging.py): suppress linting error
2024-12-06 23:10:22 -08:00
7e9d8b58f6
LiteLLM Minor Fixes & Improvements (11/23/2024) ( #6870 )
...
* feat(pass_through_endpoints/): support logging anthropic/gemini pass through calls to langfuse/s3/etc.
* fix(utils.py): allow disabling end user cost tracking with new param
Allows proxy admin to disable cost tracking for end user - keeps prometheus metrics small
* docs(configs.md): add disable_end_user_cost_tracking reference to docs
* feat(key_management_endpoints.py): add support for restricting access to `/key/generate` by team/proxy level role
Enables admin to restrict key creation, and assign team admins to handle distributing keys
* test(test_key_management.py): add unit testing for personal / team key restriction checks
* docs: add docs on restricting key creation
* docs(finetuned_models.md): add new guide on calling finetuned models
* docs(input.md): cleanup anthropic supported params
Closes https://github.com/BerriAI/litellm/issues/6856
* test(test_embedding.py): add test for passing extra headers via embedding
* feat(cohere/embed): pass client to async embedding
* feat(rerank.py): add `/v1/rerank` if missing for cohere base url
Closes https://github.com/BerriAI/litellm/issues/6844
* fix(main.py): pass extra_headers param to openai
Fixes https://github.com/BerriAI/litellm/issues/6836
* fix(litellm_logging.py): don't disable global callbacks when dynamic callbacks are set
Fixes issue where global callbacks - e.g. prometheus were overriden when langfuse was set dynamically
* fix(handler.py): fix linting error
* fix: fix typing
* build: add conftest to proxy_admin_ui_tests/
* test: fix test
* fix: fix linting errors
* test: fix test
* fix: fix pass through testing
2024-11-23 15:17:40 +05:30
3beecfb0d4
LiteLLM Minor Fixes & Improvements (11/13/2024) ( #6729 )
...
* fix(utils.py): add logprobs support for together ai
Fixes
https://github.com/BerriAI/litellm/issues/6724
* feat(pass_through_endpoints/): add anthropic/ pass-through endpoint
adds new `anthropic/` pass-through endpoint + refactors docs
* feat(spend_management_endpoints.py): allow /global/spend/report to query team + customer id
enables seeing spend for a customer in a team
* Add integration with MLflow Tracing (#6147 )
* Add MLflow logger
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com >
* Streaming handling
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com >
* lint
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com >
* address comments and fix issues
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com >
* address comments and fix issues
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com >
* Move logger construction code
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com >
* Add docs
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com >
* async handlers
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com >
* new picture
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com >
---------
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com >
* fix(mlflow.py): fix ruff linting errors
* ci(config.yml): add mlflow to ci testing
* fix: fix test
* test: fix test
* Litellm key update fix (#6710 )
* fix(caching): convert arg to equivalent kwargs in llm caching handler
prevent unexpected errors
* fix(caching_handler.py): don't pass args to caching
* fix(caching): remove all *args from caching.py
* fix(caching): consistent function signatures + abc method
* test(caching_unit_tests.py): add unit tests for llm caching
ensures coverage for common caching scenarios across different implementations
* refactor(litellm_logging.py): move to using cache key from hidden params instead of regenerating one
* fix(router.py): drop redis password requirement
* fix(proxy_server.py): fix faulty slack alerting check
* fix(langfuse.py): avoid copying functions/thread lock objects in metadata
fixes metadata copy error when parent otel span in metadata
* test: update test
* fix(key_management_endpoints.py): fix /key/update with metadata update
* fix(key_management_endpoints.py): fix key_prepare_update helper
* fix(key_management_endpoints.py): reset value to none if set in key update
* fix: update test
'
* Litellm dev 11 11 2024 (#6693 )
* fix(__init__.py): add 'watsonx_text' as mapped llm api route
Fixes https://github.com/BerriAI/litellm/issues/6663
* fix(opentelemetry.py): fix passing parallel tool calls to otel
Fixes https://github.com/BerriAI/litellm/issues/6677
* refactor(test_opentelemetry_unit_tests.py): create a base set of unit tests for all logging integrations - test for parallel tool call handling
reduces bugs in repo
* fix(__init__.py): update provider-model mapping to include all known provider-model mappings
Fixes https://github.com/BerriAI/litellm/issues/6669
* feat(anthropic): support passing document in llm api call
* docs(anthropic.md): add pdf anthropic call to docs + expose new 'supports_pdf_input' function
* fix(factory.py): fix linting error
* add clear doc string for GCS bucket logging
* Add docs to export logs to Laminar (#6674 )
* Add docs to export logs to Laminar
* minor fix: newline at end of file
* place laminar after http and grpc
* (Feat) Add langsmith key based logging (#6682 )
* add langsmith_api_key to StandardCallbackDynamicParams
* create a file for langsmith types
* langsmith add key / team based logging
* add key based logging for langsmith
* fix langsmith key based logging
* fix linting langsmith
* remove NOQA violation
* add unit test coverage for all helpers in test langsmith
* test_langsmith_key_based_logging
* docs langsmith key based logging
* run langsmith tests in logging callback tests
* fix logging testing
* test_langsmith_key_based_logging
* test_add_callback_via_key_litellm_pre_call_utils_langsmith
* add debug statement langsmith key based logging
* test_langsmith_key_based_logging
* (fix) OpenAI's optional messages[].name does not work with Mistral API (#6701 )
* use helper for _transform_messages mistral
* add test_message_with_name to base LLMChat test
* fix linting
* add xAI on Admin UI (#6680 )
* (docs) add benchmarks on 1K RPS (#6704 )
* docs litellm proxy benchmarks
* docs GCS bucket
* doc fix - reduce clutter on logging doc title
* (feat) add cost tracking stable diffusion 3 on Bedrock (#6676 )
* add cost tracking for sd3
* test_image_generation_bedrock
* fix get model info for image cost
* add cost_calculator for stability 1 models
* add unit testing for bedrock image cost calc
* test_cost_calculator_with_no_optional_params
* add test_cost_calculator_basic
* correctly allow size Optional
* fix cost_calculator
* sd3 unit tests cost calc
* fix raise correct error 404 when /key/info is called on non-existent key (#6653 )
* fix raise correct error on /key/info
* add not_found_error error
* fix key not found in DB error
* use 1 helper for checking token hash
* fix error code on key info
* fix test key gen prisma
* test_generate_and_call_key_info
* test fix test_call_with_valid_model_using_all_models
* fix key info tests
* bump: version 1.52.4 → 1.52.5
* add defaults used for GCS logging
* LiteLLM Minor Fixes & Improvements (11/12/2024) (#6705 )
* fix(caching): convert arg to equivalent kwargs in llm caching handler
prevent unexpected errors
* fix(caching_handler.py): don't pass args to caching
* fix(caching): remove all *args from caching.py
* fix(caching): consistent function signatures + abc method
* test(caching_unit_tests.py): add unit tests for llm caching
ensures coverage for common caching scenarios across different implementations
* refactor(litellm_logging.py): move to using cache key from hidden params instead of regenerating one
* fix(router.py): drop redis password requirement
* fix(proxy_server.py): fix faulty slack alerting check
* fix(langfuse.py): avoid copying functions/thread lock objects in metadata
fixes metadata copy error when parent otel span in metadata
* test: update test
* bump: version 1.52.5 → 1.52.6
* (feat) helm hook to sync db schema (#6715 )
* v0 migration job
* fix job
* fix migrations job.yml
* handle standalone DB on helm hook
* fix argo cd annotations
* fix db migration helm hook
* fix migration job
* doc fix Using Http/2 with Hypercorn
* (fix proxy redis) Add redis sentinel support (#6154 )
* add sentinel_password support
* add doc for setting redis sentinel password
* fix redis sentinel - use sentinel password
* Fix: Update gpt-4o costs to that of gpt-4o-2024-08-06 (#6714 )
Fixes #6713
* (fix) using Anthropic `response_format={"type": "json_object"}` (#6721 )
* add support for response_format=json anthropic
* add test_json_response_format to baseLLM ChatTest
* fix test_litellm_anthropic_prompt_caching_tools
* fix test_anthropic_function_call_with_no_schema
* test test_create_json_tool_call_for_response_format
* (feat) Add cost tracking for Azure Dall-e-3 Image Generation + use base class to ensure basic image generation tests pass (#6716 )
* add BaseImageGenTest
* use 1 class for unit testing
* add debugging to BaseImageGenTest
* TestAzureOpenAIDalle3
* fix response_cost_calculator
* test_basic_image_generation
* fix img gen basic test
* fix _select_model_name_for_cost_calc
* fix test_aimage_generation_bedrock_with_optional_params
* fix undo changes cost tracking
* fix response_cost_calculator
* fix test_cost_azure_gpt_35
* fix remove dup test (#6718 )
* (build) update db helm hook
* (build) helm db pre sync hook
* (build) helm db sync hook
* test: run test_team_logging firdst
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com >
Co-authored-by: Dinmukhamed Mailibay <47117969+dinmukhamedm@users.noreply.github.com >
Co-authored-by: Kilian Lieret <kilian.lieret@posteo.de >
* test: update test
* test: skip anthropic overloaded error
* test: cleanup test
* test: update tests
* test: fix test
* test: handle gemini overloaded model error
* test: handle internal server error
* test: handle anthropic overloaded error
* test: handle claude instability
---------
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com >
Co-authored-by: Yuki Watanabe <31463517+B-Step62@users.noreply.github.com >
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com >
Co-authored-by: Dinmukhamed Mailibay <47117969+dinmukhamedm@users.noreply.github.com >
Co-authored-by: Kilian Lieret <kilian.lieret@posteo.de >
2024-11-15 11:18:31 +05:30
030ece8c3f
(Feat) New Logging integration - add Datadog LLM Observability support ( #6449 )
...
* add type for dd llm obs request ob
* working dd llm obs
* datadog use well defined type
* clean up
* unit test test_create_llm_obs_payload
* fix linting
* add datadog_llm_observability
* add datadog_llm_observability
* docs DD LLM obs
* run testing again
* document DD_ENV
* test_create_llm_obs_payload
2024-10-28 22:01:32 +05:30
cdda7c243f
(refactor) prometheus async_log_success_event to be under 100 LOC ( #6416 )
...
* unit testig for prometheus
* unit testing for success metrics
* use 1 helper for _increment_token_metrics
* use helper for _increment_remaining_budget_metrics
* use _increment_remaining_budget_metrics
* use _increment_top_level_request_and_spend_metrics
* use helper for _set_latency_metrics
* remove noqa violation
* fix test prometheus
* test prometheus
* unit testing for all prometheus helper functions
* fix prom unit tests
* fix unit tests prometheus
* fix unit test prom
2024-10-24 16:41:09 +04:00
d1f457d17a
(testing) add test coverage for init custom logger class ( #6341 )
...
* working test for init custom logger
* add test coverage for custom_logger_compatible_class_as_callback
2024-10-21 15:56:32 +05:30
bd9e29b8b9
working test for init custom logger
2024-10-21 14:33:52 +05:30
24a3090ff6
fix init logger tests
2024-10-21 14:25:19 +05:30
11adc12326
add unit tests for init callbacks
2024-10-21 14:20:37 +05:30
Ishaan Jaff