* attempt to implement the passthrough feature
* Formatting and small change
* Fix formatting
* feat: grayswan guardrail overwrite ModelResponse in passthrough mode
* fix missing exception error catching on certain
endpoints
* fix wrong call site
* fix: patch anthropic endpoint internal error on streaming obj
* fix grayswan testcase
* feat: update the violation response to more natural
* Formatting
* move passthrough exception definition to custom_guardrail.
* Enhancement: show whether the blocked at input or output
* update exception name
* fix a typo in testing unit.
---------
Co-authored-by: Xiaohan Fu <xiaohan@grayswan.ai>
Add detection for Cerebras's context window exceeded error format:
"Current length is X while limit is Y"
This ensures LiteLLM raises ContextWindowExceededError instead of
generic BadRequestError when Cerebras API calls exceed the model's
context limit, enabling downstream libraries like DSPy to properly
catch and handle these errors for automatic context management.
The 'user' parameter was being ignored when using responses API models
(e.g., model="openai/responses/gpt-4.1") because the model name check
in get_supported_openai_params() didn't account for the "responses/" prefix.
Fix: Normalize the model name by stripping "responses/" prefix before
checking if the model is in the list of supported OpenAI models.
This is a minimal, non-breaking change that:
- Adds 2 lines of code in gpt_transformation.py
- Only affects the parameter support check, not the model variable itself
- Includes unit and integration tests
When OpenAI Responses API returns both text AND tool_calls, the bridge
transformation was emitting is_finished=True after the text message completed,
causing subsequent tool_call chunks to be dropped.
The fix:
- response.output_item.done for messages no longer emits is_finished=True
- Added handler for response.completed to properly signal stream end
* refactor: remove api-key conversion logic for Azure Anthropic
Co-authored-by: Erdem Halil <erdemhalil@users.noreply.github.com>
* fix(passthrough): pass custom_llm_provider to completion_cost for Azure AI Anthropic
The passthrough logging for Anthropic was failing when using Azure AI Anthropic
because the completion_cost function was not receiving the custom_llm_provider
parameter, causing it to fail with "LLM Provider NOT provided" error.
This fix:
- Retrieves custom_llm_provider from logging_obj.model_call_details
- Prepends provider prefix to model name for cost calculation
- Passes both formatted model and custom_llm_provider to completion_cost
- Centralizes provider prefix logic in _create_anthropic_response_logging_payload
This ensures cost calculation works correctly for Azure AI Anthropic requests
with models like azure_ai/claude-sonnet-4-5_gb_20250929.
Co-authored-by: Erdem Halil <erdemhalil@users.noreply.github.com>
* test: add unit tests for Azure AI Anthropic fixes
- Add tests for custom_llm_provider cost calculation in passthrough logging
- Add tests for ProviderConfigManager returning AzureAnthropicMessagesConfig
- Update existing tests to reflect removal of api-key to x-api-key conversion
Co-authored-by: Erdem Halil <erdemhalil@users.noreply.github.com>
---------
Co-authored-by: Erdem Halil <erdemhalil@users.noreply.github.com>
Fixes#17473 - Anthropic streaming fails with JSONDecodeError when
network fragmentation causes SSE data to arrive in partial chunks.
Changes:
- Add accumulated_json buffer and chunk_type to ModelResponseIterator
- Add _handle_accumulated_json_chunk() to accumulate partial JSON
- Add _parse_sse_data() to handle both complete and partial chunks
- Modify __next__ and __anext__ to use accumulation logic
- Add unit tests for partial chunk handling
The async_post_call_streaming_iterator_hook function was broken:
1. Was a sync function (def) not async generator
2. Returned AsyncGenerator without iterating it
3. Callback generators were chained but never consumed
This fix:
1. Makes the function an async generator (async def + yield)
2. Actually iterates through the chained callbacks with 'async for'
3. Properly yields chunks to the caller
Fixes#9639
* Add Amazon Nova as a first party provider
* Added new provider folder under llms/ to outline the openai supported params
* Updated supported endpoints on the documnetation
- Reset mock call counts at start of test to ensure clean state
- Add span method to mock trace to handle log_provider_specific_information_as_span calls
- Re-establish mock chain before test call to ensure fresh state
- Add exception handling to catch and report errors during test execution
- Add verification that trace was called before checking generation
This should fix the flaky test that was failing intermittently with
'Expected generation to have been called once. Called 0 times.'
- Remove network dependency by mocking HuggingFace template fetch
- Use mock template that produces correct format for test validation
- Test now focuses on transformation logic, not network calls
- Fixes flaky test failures due to network timeouts/rate limits
The test verifies that prompt transformation occurs (not simple
concatenation), which doesn't require the actual HuggingFace template.
Mocking makes the test deterministic and faster while still validating
the core behavior.
Fixed three flaky tests that were intermittently failing in CI:
1. test_no_duplicate_spend_logs (test_litellm/responses/test_no_duplicate_spend_logs.py)
Problem: Used await asyncio.sleep(1) to wait for async logging completion,
which created race conditions. The async logging worker queues tasks
in the background, and sleep() doesn't guarantee completion.
Fix: Replaced sleep() with GLOBAL_LOGGING_WORKER.flush() which properly waits
for the logging queue to empty, ensuring all async logging tasks complete
before assertions run.
2. test_log_langfuse_v2_handles_null_usage_values (test_litellm/integrations/test_langfuse.py)
Problem: Used datetime.datetime.now() twice for start_time and end_time, which
could cause timing inconsistencies between test runs, especially in
CI environments with variable execution speeds.
Fix: Use fixed timestamps instead of datetime.now() to ensure consistent timing
across all test runs, eliminating timing-related flakiness.
3. test_watsonx_gpt_oss_prompt_transformation (test_litellm/llms/watsonx/test_watsonx.py)
Problem: Directly accessed mock_post.call_args without checking if it exists,
which could be None if the mock wasn't called or if an exception
occurred before the POST request. The test catches exceptions and
continues, making this a potential failure point.
Fix: Added proper assertions and use call_args_list[0] for safer access:
- Assert that call_args_list has at least one call
- Assert that call_args is not None
- Assert that 'data' key exists in kwargs
This ensures the test fails with clear error messages rather than
intermittent AttributeError exceptions.
All fixes maintain the original test intent while making them deterministic
and reliable in CI environments.
- Filter async_log_success_event calls by expected input message
- Bridge models (openai/codex-mini-latest) may make internal calls that also log
- Test now asserts exactly one call with the expected input 'Hey' instead of asserting total call count
- Makes test robust to bridge-related double logging while still validating core behavior
Reapplies the fix from commit a885e21543 that was
reverted in 6c9556be67.
The original revert was done because the test was flaky and giving false
negatives. This fix properly mocks the Langfuse client to ensure the test
can correctly verify that _log_langfuse_v2 converts None usage values to 0.
Changes:
- Add mock_langfuse_client.client attribute to prevent errors during init
- Add trace_id to mock_langfuse_generation for proper return value handling
- Remove redundant mock setup code
- Explicitly set logger.Langfuse to mock client after initialization
- Set logger.langfuse_sdk_version to ensure _supports_* methods work correctly
yuneng-jiang