Calculate `total_tokens` in usage data in Response manually if:
- `total_tokens` is missing
- `total_tokens` can be calculated from input and output tokens
Run the test for this feature with:
`poetry run pytest tests/test_litellm/responses/test_responses_utils.py -k "test_transform_response_api_usage_calculates_total_from_input_and_output_tokens_if_available" -v`
Fixed three flaky tests that were intermittently failing in CI:
1. test_no_duplicate_spend_logs (test_litellm/responses/test_no_duplicate_spend_logs.py)
Problem: Used await asyncio.sleep(1) to wait for async logging completion,
which created race conditions. The async logging worker queues tasks
in the background, and sleep() doesn't guarantee completion.
Fix: Replaced sleep() with GLOBAL_LOGGING_WORKER.flush() which properly waits
for the logging queue to empty, ensuring all async logging tasks complete
before assertions run.
2. test_log_langfuse_v2_handles_null_usage_values (test_litellm/integrations/test_langfuse.py)
Problem: Used datetime.datetime.now() twice for start_time and end_time, which
could cause timing inconsistencies between test runs, especially in
CI environments with variable execution speeds.
Fix: Use fixed timestamps instead of datetime.now() to ensure consistent timing
across all test runs, eliminating timing-related flakiness.
3. test_watsonx_gpt_oss_prompt_transformation (test_litellm/llms/watsonx/test_watsonx.py)
Problem: Directly accessed mock_post.call_args without checking if it exists,
which could be None if the mock wasn't called or if an exception
occurred before the POST request. The test catches exceptions and
continues, making this a potential failure point.
Fix: Added proper assertions and use call_args_list[0] for safer access:
- Assert that call_args_list has at least one call
- Assert that call_args is not None
- Assert that 'data' key exists in kwargs
This ensures the test fails with clear error messages rather than
intermittent AttributeError exceptions.
All fixes maintain the original test intent while making them deterministic
and reliable in CI environments.
* fix(responses): Add image generation support for Responses API
Fixes#16227
## Problem
When using Gemini 2.5 Flash Image with /responses endpoint, image generation
outputs were not being returned correctly. The response contained only text
with empty content instead of the generated images.
## Solution
1. Created new `OutputImageGenerationCall` type for image generation outputs
2. Modified `_extract_message_output_items()` to detect images in completion responses
3. Added `_extract_image_generation_output_items()` to transform images from
completion format (data URL) to responses format (pure base64)
4. Added `_extract_base64_from_data_url()` helper to extract base64 from data URLs
5. Updated `ResponsesAPIResponse.output` type to include `OutputImageGenerationCall`
## Changes
- litellm/types/responses/main.py: Added OutputImageGenerationCall type
- litellm/types/llms/openai.py: Updated ResponsesAPIResponse.output type
- litellm/responses/litellm_completion_transformation/transformation.py:
Added image detection and extraction logic
- tests/test_litellm/responses/litellm_completion_transformation/test_image_generation_output.py:
Added comprehensive unit tests (16 tests, all passing)
## Result
/responses endpoint now correctly returns:
```json
{
"output": [{
"type": "image_generation_call",
"id": "..._img_0",
"status": "completed",
"result": "iVBORw0KGgo..." // Pure base64, no data: prefix
}]
}
```
This matches OpenAI Responses API specification where image generation
outputs have type "image_generation_call" with base64 data in "result" field.
* docs(responses): Add image generation documentation and tests
- Add comprehensive image generation documentation to response_api.md
- Include examples for Gemini (no tools param) and OpenAI (with tools param)
- Document response format and base64 handling
- Add supported models table with provider-specific requirements
- Add unit tests for image generation output transformation
- Test base64 extraction from data URLs
- Test image generation output item creation
- Test status mapping and integration scenarios
- Verify proper transformation from completions to responses format
Related to #16227
* fix(responses): Correct status type for image generation output
- Add _map_finish_reason_to_image_generation_status() helper function
- Fix MyPy type error: OutputImageGenerationCall.status only accepts
['in_progress', 'completed', 'incomplete', 'failed'], not the full
ResponsesAPIStatus union which includes 'cancelled' and 'queued'
Fixes MyPy error in transformation.py:838
* fix: prevent duplicate spend logs in Responses API for non-OpenAI providers
Fixes#15740
This fixes a logging duplication bug where using kwargs.pop() removed
the litellm_logging_obj before passing kwargs to internal acompletion()
calls, causing duplicate spend log entries for providers without native
Responses API support (Anthropic, Gemini, etc).
By changing from pop() to get(), the logging object is preserved and
reused across the internal completion call, preventing duplicate entries
and maintaining correct cost tracking.
* test: add test for logging object preservation in responses API
Verify that litellm_logging_obj is preserved in kwargs when calling
responses(), ensuring no duplicate spend log entries are created.
- Change variable name in litellm/__init__.py from configured_cold_storage_logger to cold_storage_custom_logger
- Update all references across the codebase to use the new variable name
- This fixes silent failure of cold storage logging due to variable name mismatch
- Configuration files use cold_storage_custom_logger, code should match
Files updated:
- litellm/__init__.py
- litellm/litellm_core_utils/litellm_logging.py
- litellm/proxy/spend_tracking/cold_storage_handler.py
- litellm/responses/litellm_completion_transformation/session_handler.py
- tests/test_litellm/litellm_core_utils/test_litellm_logging.py
- tests/test_litellm/responses/litellm_completion_transformation/test_session_handler.py
* fix: pass extra_headers parameter through responses API transformation chain
Ensure extra_headers parameter is properly forwarded from the responses() function
through the transformation handler and config to maintain header propagation in
litellm_completion_request dict.
* Add tests
* fix proxy config
* fix(responses api): fix streaming ID consistency and tool format handling (#12640)
* fix(responses): ensure streaming chunk IDs use consistent encoding format
Fixes streaming ID inconsistency where streaming responses used raw provider IDs
while non-streaming responses used properly encoded IDs with provider context.
Changes:
- Updated LiteLLMCompletionStreamingIterator to accept provider context
- Added _encode_chunk_id() method using same logic as non-streaming responses
- Modified chunk transformation to encode all streaming item_ids with resp_ prefix
- Updated handlers to pass custom_llm_provider and litellm_metadata to streaming iterator
Impact:
- Streaming chunk IDs now format: resp_<base64_encoded_provider_context>
- Enables session continuity when using streaming response IDs as previous_response_id
- Allows provider detection and load balancing with streaming responses
- Maintains backward compatibility with existing streaming functionality
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(types): add explicit Optional[str] type annotation for model_id
This resolves MyPy type checking error where model_id could be None
but wasn't explicitly typed as Optional[str].
* fix(types): handle None case for litellm_metadata access
Prevents 'Item None has no attribute get' error by checking for None
before accessing litellm_metadata dictionary.
* test: add comprehensive tests for streaming ID consistency
Adds unit and E2E tests to verify streaming chunk IDs are properly encoded
with consistent format across streaming responses.
## Tests Added
### Unit Test (test_reasoning_content_transformation.py)
- `test_streaming_chunk_id_encoding()`: Validates the `_encode_chunk_id()` method
correctly encodes chunk IDs with `resp_` prefix and provider context
### E2E Tests (test_e2e_openai_responses_api.py)
- `test_streaming_id_consistency_across_chunks()`: Tests that all streaming chunk IDs
are properly encoded across multiple chunks in a real streaming response
- `test_streaming_response_id_as_previous_response_id()`: Tests the core use case -
using streaming response IDs for session continuity with `previous_response_id`
## Key Testing Approach
- Uses **Gemini** (non-OpenAI model) to test the transformation logic rather than
OpenAI passthrough, since the streaming ID consistency issue occurs when LiteLLM
transforms responses rather than just passing through to native OpenAI responses API
- Tests validate that streaming chunk IDs now use same encoding as non-streaming responses
- Verifies session continuity works with streaming responses
Addresses @ishaan-jaff's request for unit tests covering the streaming ID consistency fix.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(lint): remove unused imports in transformation.py
Removes unused imports to fix CI linting errors:
- GenericResponseOutputItem
- OutputFunctionToolCall
* test: remove E2E tests from openai_endpoints_tests
Remove streaming ID consistency E2E tests as requested by @ishaan-jaff.
Keep only the mock/unit test in test_reasoning_content_transformation.py
* revert: remove streaming chunk ID encoding to original behavior
This reverts the streaming chunk ID encoding changes to understand the original issue better.
Original behavior was:
- Streaming chunks: raw provider IDs
- Streaming final response: raw IDs (PROBLEM!)
- Non-streaming final response: encoded IDs (correct)
The real issue: streaming final response IDs were not encoded, breaking session continuity.
* fix(responses): encode streaming final response IDs to match OpenAI behavior
Fixes streaming ID inconsistency to match OpenAI's Responses API behavior:
- Streaming chunks: raw message IDs (like OpenAI's msg_xxx)
- Final response: encoded IDs (like OpenAI's resp_xxx)
This enables session continuity by ensuring streaming final response IDs
have the same encoded format as non-streaming responses, allowing them
to be used as previous_response_id in follow-up requests.
Changes:
- Add custom_llm_provider and litellm_metadata to LiteLLMCompletionStreamingIterator
- Update handlers to pass provider context to streaming iterator
- Apply _update_responses_api_response_id_with_model_id to final streaming response
- Keep streaming chunks as raw IDs to match OpenAI format
Impact:
- Session continuity works with streaming responses
- Load balancing can detect provider from streaming final response IDs
- Format matches OpenAI's Responses API exactly
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* test: update unit test to match correct OpenAI-compatible behavior
Updates the unit test to verify streaming chunk IDs are raw (not encoded)
to match OpenAI's responses API format:
- Streaming chunks: raw message IDs (like msg_xxx)
- Final response: encoded IDs (like resp_xxx)
This reflects the correct behavior implemented in the fix.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* cleanup
* TestBaseResponsesAPIStreamingIterator
---------
Co-authored-by: Javier de la Torre <jatorre@carto.com>
Co-authored-by: Claude <noreply@anthropic.com>
* add _transform_responses_api_function_call_to_chat_completion_message
* test_responses_api_with_tool_calls
* TestFunctionCallTransformation
* fixes for responses API testing google ai studio
* TestGoogleAIStudioResponsesAPITest
* test_responses_api_with_tool_calls
* test_responses_api_with_tool_calls
* test_basic_openai_responses_streaming_delete_endpoint
* Add _transform_input_image_item_to_image_item() to handle image transformations
* Update content transformation to process input_image items from Responses API format to Chat Completions format
Add reasoning content support when Responses API falls
back to completions API, enabling reasoning content for
all LLM providers (Anthropic, Vertex AI, etc.) since
OpenAI is currently the only native Responses API
provider.
* Add ReasoningSummaryTextDeltaEvent for streaming
reasoning deltas
* Update streaming iterator to detect and transform
reasoning content
* Extract reasoning content as separate output items in
transformations
* Support reasoning content alongside regular message
content
Closes https://github.com/BerriAI/litellm/issues/11302
* Handle file content type transformation in responses api (#11310)
* Handle file content type transformation in responses api
* change to use input_file
* -
* TestLiteLLMCompletionResponsesConfig
* test: TestLiteLLMCompletionResponsesConfig
* fix: fix linting
---------
Co-authored-by: Jayme Gordon <jayme_gordon@icloud.com>
