* attempt to implement the passthrough feature
* Formatting and small change
* Fix formatting
* feat: grayswan guardrail overwrite ModelResponse in passthrough mode
* fix missing exception error catching on certain
endpoints
* fix wrong call site
* fix: patch anthropic endpoint internal error on streaming obj
* fix grayswan testcase
* feat: update the violation response to more natural
* Formatting
* move passthrough exception definition to custom_guardrail.
* Enhancement: show whether the blocked at input or output
* update exception name
* fix a typo in testing unit.
---------
Co-authored-by: Xiaohan Fu <xiaohan@grayswan.ai>
Add detection for Cerebras's context window exceeded error format:
"Current length is X while limit is Y"
This ensures LiteLLM raises ContextWindowExceededError instead of
generic BadRequestError when Cerebras API calls exceed the model's
context limit, enabling downstream libraries like DSPy to properly
catch and handle these errors for automatic context management.
The 'user' parameter was being ignored when using responses API models
(e.g., model="openai/responses/gpt-4.1") because the model name check
in get_supported_openai_params() didn't account for the "responses/" prefix.
Fix: Normalize the model name by stripping "responses/" prefix before
checking if the model is in the list of supported OpenAI models.
This is a minimal, non-breaking change that:
- Adds 2 lines of code in gpt_transformation.py
- Only affects the parameter support check, not the model variable itself
- Includes unit and integration tests
When OpenAI Responses API returns both text AND tool_calls, the bridge
transformation was emitting is_finished=True after the text message completed,
causing subsequent tool_call chunks to be dropped.
The fix:
- response.output_item.done for messages no longer emits is_finished=True
- Added handler for response.completed to properly signal stream end
* refactor: remove api-key conversion logic for Azure Anthropic
Co-authored-by: Erdem Halil <erdemhalil@users.noreply.github.com>
* fix(passthrough): pass custom_llm_provider to completion_cost for Azure AI Anthropic
The passthrough logging for Anthropic was failing when using Azure AI Anthropic
because the completion_cost function was not receiving the custom_llm_provider
parameter, causing it to fail with "LLM Provider NOT provided" error.
This fix:
- Retrieves custom_llm_provider from logging_obj.model_call_details
- Prepends provider prefix to model name for cost calculation
- Passes both formatted model and custom_llm_provider to completion_cost
- Centralizes provider prefix logic in _create_anthropic_response_logging_payload
This ensures cost calculation works correctly for Azure AI Anthropic requests
with models like azure_ai/claude-sonnet-4-5_gb_20250929.
Co-authored-by: Erdem Halil <erdemhalil@users.noreply.github.com>
* test: add unit tests for Azure AI Anthropic fixes
- Add tests for custom_llm_provider cost calculation in passthrough logging
- Add tests for ProviderConfigManager returning AzureAnthropicMessagesConfig
- Update existing tests to reflect removal of api-key to x-api-key conversion
Co-authored-by: Erdem Halil <erdemhalil@users.noreply.github.com>
---------
Co-authored-by: Erdem Halil <erdemhalil@users.noreply.github.com>
* Fix test_delete_polling_removes_from_cache mock setup
- Mock async_delete_cache to properly execute the real implementation path
- Ensures init_async_client() is called and delete() is invoked on the returned client
- Fixes AssertionError: Expected 'delete' to be called once. Called 0 times.
* fix: resolve timeout in add_model_tab test by mocking useProviderFields hook
- Mock useProviderFields hook to prevent network calls and React Query delays
- Use waitFor to properly handle async operations
- Test now passes reliably without 10s timeout
* fix: add test timeout to prevent CI timeout failure
- Add 15 second timeout to 'should display Test Connect and Add Model buttons' test
- Test takes ~6 seconds locally, but CI was timing out at default 5 second limit
- Ensures test has sufficient time to complete in CI environment
* test: quarantine flaky test_oidc_circleci_with_azure
Quarantine test that fails with 401 Unauthorized from Azure OAuth.
The test is flaky and blocks CI builds. Marked with @pytest.mark.skip
until Azure authentication can be fixed or migrated to our own account.
Fixes#17473 - Anthropic streaming fails with JSONDecodeError when
network fragmentation causes SSE data to arrive in partial chunks.
Changes:
- Add accumulated_json buffer and chunk_type to ModelResponseIterator
- Add _handle_accumulated_json_chunk() to accumulate partial JSON
- Add _parse_sse_data() to handle both complete and partial chunks
- Modify __next__ and __anext__ to use accumulation logic
- Add unit tests for partial chunk handling
The async_post_call_streaming_iterator_hook function was broken:
1. Was a sync function (def) not async generator
2. Returned AsyncGenerator without iterating it
3. Callback generators were chained but never consumed
This fix:
1. Makes the function an async generator (async def + yield)
2. Actually iterates through the chained callbacks with 'async for'
3. Properly yields chunks to the caller
Fixes#9639
* Fix: Support generic_api_compatible_callbacks.json in callback initialization
- Added check in _add_custom_callback_generic_api_str to load callbacks from generic_api_compatible_callbacks.json
- Added SumoLogic webhook integration to generic_api_compatible_callbacks.json
- Fixes bug where callbacks in JSON file were not being loaded
* Added 3 unit tests for JSON callback loading
yuneng-jiang