* fix(text_completion): support token IDs (list of integers) as prompt
Add support for passing token IDs (list of integers) to the text_completion
endpoint for OpenAI-compatible providers (openai, azure, vllm, etc.).
Fixes#17118
* test(text_completion): replace live test with mock test for token IDs
Move token IDs test from local_testing to test_litellm with mocks
per PR review feedback.
* feat(bedrock): add OpenAI-compatible service_tier parameter translation
Translates OpenAI's service_tier parameter (string) to Bedrock's
serviceTier format (object with type field).
* docs(bedrock): add OpenAI-compatible service_tier parameter documentation
Document the automatic translation from OpenAI-style service_tier
parameter to Bedrock's native serviceTier format.
* feat(bedrock): add service_tier to response when present
According to OpenAI's API documentation, when service_tier is sent in the
request, it should be returned in the response. This commit implements
this behavior for Bedrock Converse API to maintain compatibility with
OpenAI's API.
Changes:
- Added serviceTier field to ConverseResponseBlock type definition
- Moved ServiceTierBlock definition before ConverseResponseBlock to fix
type reference order
- Added response transformation to map Bedrock serviceTier (object) to
OpenAI service_tier (string format)
- Added 4 new tests for response transformation with service_tier
The service_tier is only added to the response when present in Bedrock's
response, maintaining backward compatibility.
Fixes#18137
Similar to the fix for web_search_tool_result (#17746, #17798), this PR
preserves web_fetch_tool_result blocks in multi-turn conversations.
Changes:
- Add handling for web_fetch_tool_result in transformation.py (non-streaming)
- Add capture of web_fetch_tool_result in handler.py (streaming)
- Fix streaming tool arguments bug where empty input {} was prepended to
actual arguments by using empty string instead of str({})
- Add unit tests for web_fetch_tool_result handling
* fix(gemini): prevent negative text_tokens with explicit caching (#18750)
## Problem
When using Gemini with explicit caching (especially with images),
text_tokens would become negative (e.g., -3327) due to incorrectly
subtracting total cached_tokens from modality-specific text_tokens.
## Root Cause
The old code did:
```python
text_tokens = text_tokens - cached_tokens # 737 - 4064 = -3327
```
This was wrong because:
- cached_tokens includes ALL modalities (text + image + audio + video)
- text_tokens only contains text
- Subtracting total from specific caused negative values
## Solution
Parse cacheTokensDetails to get per-modality cached token breakdown:
```python
if "cacheTokensDetails" in usage_metadata:
cached_text_tokens = parse from cacheTokensDetails["TEXT"]
text_tokens = text_tokens - cached_text_tokens # Correct!
```
Now we subtract cached tokens per modality, preventing negatives.
## Changes
- Parse cacheTokensDetails field from Gemini response
- Calculate non-cached tokens per modality (text, image, audio)
- Remove incorrect global cached_tokens subtraction
- Add tests for explicit caching and implicit/no caching scenarios
## Testing
- Added test_gemini_cache_tokens_details_no_negative_values
- Added test_gemini_without_cache_tokens_details
- All existing Gemini caching tests pass
Fixes#18750
* feat: add cache_read_input_tokens to Usage object
Addresses reviewer feedback to include cached tokens at the top level
of the Usage object. This aligns with how Anthropic provider handles
cached tokens and ensures they are visible in the final usage response.
* fix: add cacheTokensDetails field to UsageMetadata TypedDict
Fixes mypy error where cacheTokensDetails was being accessed but not defined
in the UsageMetadata TypedDict type definition.
* fix(anthropic): prevent dropping thinking when any message has thinking_blocks
When Claude returns multiple assistant messages in a conversation, some may
have thinking_blocks while others may not (Claude's behavior varies). The
previous logic only checked the LAST assistant message with tool_calls,
dropping the thinking param if it had no thinking_blocks.
This caused errors when earlier messages still contained thinking_blocks:
"When thinking is disabled, an assistant message cannot contain thinking"
The fix adds a new check: only drop thinking if NO assistant messages
have thinking_blocks. If any message has thinking_blocks, we keep
thinking enabled.
Fixes#18926
* chore: re-trigger CI
* fix(generic-guardrail-api): fix SerializationIterator error on multimodal requests
When sending multimodal messages (with images) through the Generic Guardrail API,
the `model_dump()` call fails with "Object of type SerializationIterator is not
JSON serializable" error.
Root cause: The `ChatCompletionAssistantMessage` type defines `content` as an
`Iterable` (not just `List`), and Pydantic's `model_dump()` creates a
`SerializationIterator` for iterables which is not JSON serializable.
Fix: Use `model_dump(mode="json")` which properly converts all iterables to
lists and ensures all complex objects are JSON serializable.
* fix(guardrails): pass tools (function definitions) to guardrail inputs
The unified guardrail handler was not passing the `tools` parameter
(function definitions) from the request to the guardrail inputs.
This meant guardrails could not inspect or validate tool definitions.
Added extraction of `data.get("tools")` and inclusion in the
GenericGuardrailAPIInputs passed to `apply_guardrail()`.
* test(guardrails): add tests for tools passed to guardrail
Added tests verifying that tools (function definitions) are correctly
passed to guardrails in the unified guardrail handler:
- test_tools_passed_to_guardrail
- test_multiple_tools_passed_to_guardrail
- test_no_tools_in_request
- test_tools_and_tool_calls_both_passed
When using the generateContent endpoint with non-Google providers like
github_copilot, the extra_headers from model config were not being
forwarded to the underlying litellm.completion/acompletion calls.
This caused providers requiring custom headers (e.g., Editor-Version
for GitHub Copilot authentication) to reject requests with errors like
"missing Editor-Version header for IDE auth".
Changes:
- Forward extra_headers in _prepare_completion_kwargs() handler
- Pass extra_headers explicitly to adapter in generate_content()
- Pass extra_headers explicitly to adapter in agenerate_content_stream()
- Pass extra_headers explicitly to adapter in generate_content_stream()
- Add tests for extra_headers forwarding behavior
- Update existing test to expect extra_headers in passed fields
Co-authored-by: Claude <noreply@anthropic.com>
* fix: align max_tokens with max_output_tokens for consistency
Fixed inconsistent max_tokens definitions in model_prices_and_context_window.json.
According to LiteLLM convention, max_tokens should equal max_output_tokens when available.
Models fixed:
- deepseek-chat: 131072 → 8192 (now equals max_output_tokens)
- dashscope/qwen-flash: 1000000 → 32768 (now equals max_output_tokens)
- databricks/databricks-gemma-3-12b: 128000 → 32000 (now equals max_output_tokens)
This ensures consistency across all providers where max_tokens represents
the maximum number of tokens that can be generated in the output.
* fix: align max_tokens with max_output_tokens for 244 models
- Fix 244 models where max_tokens != max_output_tokens
- Add test to validate max_tokens consistency and prevent regressions
According to model_prices_and_context_window.json spec:
- max_tokens is a LEGACY parameter
- Should always equal max_output_tokens when both are present
This ensures consistency across all model definitions.
* fix(azure): add logprobs support for Azure OpenAI GPT-5 models
Azure OpenAI GPT-5 models (including gpt-5.2) support logprobs
parameters, unlike OpenAI's GPT-5 reasoning models. This fix
overrides the parent class restriction to enable logprobs for Azure.
Changes:
- Override get_supported_openai_params() in AzureOpenAIGPT5Config
- Add "logprobs" and "top_logprobs" to supported params
- Add comprehensive tests for logprobs functionality
Testing:
- Verified with direct Azure API calls to gpt-5.2
- API version: 2025-01-01-preview
- Successfully returns logprobs data
Related: #7974, #4022
* refactor: restrict logprobs support to gpt-5.2 only
Only gpt-5.2 has been verified to support logprobs on Azure.
Other gpt-5 variants (gpt-5, gpt-5.1) have not been tested.
Changes:
- Add conditional check for is_model_gpt_5_2_model()
- Update tests to be specific to gpt-5.2
- Add negative tests for gpt-5 and gpt-5.1
- Update documentation to reflect gpt-5.2 specificity
