- Add @pytest.mark.flaky(retries=3, delay=1) decorator to handle intermittent Anthropic API failures
- Add error handling to skip test when Anthropic API returns InternalServerError
- Prevents false test failures due to external API 500 errors
- Reset mock call counts at start of test to ensure clean state
- Add span method to mock trace to handle log_provider_specific_information_as_span calls
- Re-establish mock chain before test call to ensure fresh state
- Add exception handling to catch and report errors during test execution
- Add verification that trace was called before checking generation
This should fix the flaky test that was failing intermittently with
'Expected generation to have been called once. Called 0 times.'
- Remove network dependency by mocking HuggingFace template fetch
- Use mock template that produces correct format for test validation
- Test now focuses on transformation logic, not network calls
- Fixes flaky test failures due to network timeouts/rate limits
The test verifies that prompt transformation occurs (not simple
concatenation), which doesn't require the actual HuggingFace template.
Mocking makes the test deterministic and faster while still validating
the core behavior.
- Add missing @pytest.mark.asyncio decorator
- Implement retry logic with exponential backoff (3 retries)
- Only retry on transient Azure internal server errors
- Fail immediately on non-transient errors
This fixes the flaky test_azure_img_gen_health_check which was failing
due to transient Azure internal server errors that are outside our control.
Fixed three flaky tests that were intermittently failing in CI:
1. test_no_duplicate_spend_logs (test_litellm/responses/test_no_duplicate_spend_logs.py)
Problem: Used await asyncio.sleep(1) to wait for async logging completion,
which created race conditions. The async logging worker queues tasks
in the background, and sleep() doesn't guarantee completion.
Fix: Replaced sleep() with GLOBAL_LOGGING_WORKER.flush() which properly waits
for the logging queue to empty, ensuring all async logging tasks complete
before assertions run.
2. test_log_langfuse_v2_handles_null_usage_values (test_litellm/integrations/test_langfuse.py)
Problem: Used datetime.datetime.now() twice for start_time and end_time, which
could cause timing inconsistencies between test runs, especially in
CI environments with variable execution speeds.
Fix: Use fixed timestamps instead of datetime.now() to ensure consistent timing
across all test runs, eliminating timing-related flakiness.
3. test_watsonx_gpt_oss_prompt_transformation (test_litellm/llms/watsonx/test_watsonx.py)
Problem: Directly accessed mock_post.call_args without checking if it exists,
which could be None if the mock wasn't called or if an exception
occurred before the POST request. The test catches exceptions and
continues, making this a potential failure point.
Fix: Added proper assertions and use call_args_list[0] for safer access:
- Assert that call_args_list has at least one call
- Assert that call_args is not None
- Assert that 'data' key exists in kwargs
This ensures the test fails with clear error messages rather than
intermittent AttributeError exceptions.
All fixes maintain the original test intent while making them deterministic
and reliable in CI environments.
Fixed bug in add_guardrail_to_applied_guardrails_header where guardrail
information was lost when request_data didn't have a metadata key. The
function would create a new metadata dict but never assign it back to
request_data, causing the x-litellm-applied-guardrails header to be
missing from responses.
This fixes the failing test_guardrails_with_api_key_controls test.
- Filter async_log_success_event calls by expected input message
- Bridge models (openai/codex-mini-latest) may make internal calls that also log
- Test now asserts exactly one call with the expected input 'Hey' instead of asserting total call count
- Makes test robust to bridge-related double logging while still validating core behavior
Reapplies the fix from commit a885e21543 that was
reverted in 6c9556be67.
The original revert was done because the test was flaky and giving false
negatives. This fix properly mocks the Langfuse client to ensure the test
can correctly verify that _log_langfuse_v2 converts None usage values to 0.
Changes:
- Add mock_langfuse_client.client attribute to prevent errors during init
- Add trace_id to mock_langfuse_generation for proper return value handling
- Remove redundant mock setup code
- Explicitly set logger.Langfuse to mock client after initialization
- Set logger.langfuse_sdk_version to ensure _supports_* methods work correctly
- Skip usage recalculation if usage was already set from response JSON
- Fixes test_bedrock_openai_response_parsing which expected usage values from JSON response
- Prevents overwriting correct usage values with token counting for OpenAI imported models
* Fix test_log_langfuse_v2_handles_null_usage_values test failure
The test was failing because the logger's Langfuse client wasn't properly
mocked. Even though sys.modules was mocked, the logger's __init__ method
creates its own Langfuse client instance that wasn't using the test's mock.
Changes:
- Explicitly set logger.Langfuse to the mock client after initialization
- Set logger.langfuse_sdk_version to ensure _supports_* methods work correctly
- Added mock_langfuse_client.client attribute to prevent errors during init
- Added trace_id to mock_langfuse_generation for proper return value handling
- Removed redundant mock setup code
This ensures the test can properly verify that _log_langfuse_v2 correctly
converts None usage values to 0 by allowing the mock's generation method
to be called and asserted.
Fixes: AssertionError: Expected 'generation' to have been called once. Called 0 times.
yuneng-jiang