* Prompt Management API - new API to interact with Prompt Management integrations (no PR required) (#17800)
* feat: initial commit adding prompt management api
* feat: initial commit adding prompt management api
* fix: refactoring to make sure get prompt is async
* fix: additional fixes
* fix: partially working generic api prompt management
* feat(langfuse): Add support for custom masking function
Allow users to pass a custom masking function via metadata to selectively
redact sensitive data (credit cards, emails, PII) before sending to Langfuse.
Usage:
```python
def mask_pii(data):
if isinstance(data, str):
data = re.sub(r'\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b', '[CARD]', data)
return data
litellm.completion(
model="gpt-4",
messages=[...],
metadata={"langfuse_masking_function": mask_pii}
)
```
* fix(langfuse): Isolate masking function from other logging integrations
Extract langfuse_masking_function from metadata early in the flow and store
it in a dedicated key (_langfuse_masking_function) that only the Langfuse
logger knows to look for. This prevents the callable from leaking to other
logging integrations (Datadog, S3, etc.) which would serialize it as
"<function at 0x...>".
Changes:
- scrub_sensitive_keys_in_metadata() now extracts and stores the function
- Langfuse logger looks in the dedicated key first, falls back to metadata
- Added tests to verify isolation works correctly
Replace `str | List[str]` with `Union[str, List[str]]` in
EmbeddingInput model to support Python 3.9.
The pipe union syntax (PEP 604) is only available in Python 3.10+,
but LiteLLM supports Python >=3.9.
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* feat(deepseek): add native support for thinking and reasoning_effort params
Add proper parameter mapping for DeepSeek thinking mode, allowing users
to use the unified LiteLLM interface instead of extra_body workarounds.
Supported formats:
- thinking={"type": "enabled"}
- thinking={"type": "enabled", "budget_tokens": X} (budget_tokens ignored)
- reasoning_effort="low|medium|high" (maps to thinking enabled)
DeepSeek only supports {"type": "enabled"} without budget_tokens,
so any budget_tokens are stripped and all reasoning_effort values
(except "none") map to enabled.
Reference: https://api-docs.deepseek.com/guides/thinking_mode
* docs(deepseek): add thinking and reasoning_effort parameter documentation
* fix(openai): use optimized async http client for text completions
OpenAITextCompletion.acompletion was using litellm.aclient_session directly
instead of the optimized http client with aiohttp transport that
OpenAIChatCompletion uses. This fixes inconsistent behavior where custom
SSL configs and the faster aiohttp transport were not applied to async
text completion requests.
Fixes#17676
* test(openai): add test for text completion async http client
Verify that OpenAITextCompletion.acompletion uses the optimized
BaseOpenAILLM._get_async_http_client() instead of litellm.aclient_session.
Related to #17676
* test: move http client test to existing test file
Move test_acompletion_uses_optimized_http_client to
test_text_completion_unit_tests.py instead of separate file.
* fix: resolve mypy type errors in hiddenlayer guardrail and transformation
- Fix return type of apply_guardrail from str to GenericGuardrailAPIInputs
- Add None checks for logging_obj before accessing attributes
- Convert AllMessageValues to dict format for HiddenLayer API compatibility
- Fix payload type annotation in _call_hiddenlayer
- Ensure transformed_output always returns list[dict[str, Any]] in transformation.py
* fix: use litellm_call_id as trace_id fallback in langfuse logging
- Only use standard_logging_object.trace_id if explicitly set via litellm_session_id or litellm_trace_id params
- Fallback to litellm_call_id when no explicit trace_id is provided (matches test expectation)
- Return the trace_id we set instead of generation_client.trace_id for consistency
- Add warning if langfuse modifies the trace_id to help debug potential issues
Fixes test_logging_trace_id test failure where auto-generated UUID was used instead of litellm_call_id
* fix: document envs
* fix: handle None response in /spend/logs endpoint when no records found
- Return empty list [] instead of [None] when spend_log is None
- Prevents 500 errors when querying by request_id, api_key, or user_id with no matching records
- Fixes test_chat_completion_bad_model_with_spend_logs test failure
* fix: use standard_logging_object trace_id when available in langfuse logger
- Fix trace_id selection logic to use standard_logging_object.trace_id when available
- Previously only used standard_logging_object.trace_id if explicitly set via params
- Now uses standard_logging_object.trace_id whenever it's present, matching test expectations
- Falls back to litellm_call_id if no trace_id is found
- Fixes test_log_langfuse_v2_uses_standard_trace_id_when_available test failure
When using litellm.completion() with model="openai/responses/...", images
in tool message content were not being transformed from Chat Completion
format to Responses API format.
Chat Completion format: {"type": "image_url", "image_url": {"url": "..."}}
Responses API format: {"type": "input_image", "image_url": "..."}
This caused OpenAI to reject the request with error 400 since "image_url"
is not a valid type for function_call_output content.
This fix addresses two issues with Anthropic web search streaming:
1. Fix trailing {} in tool call arguments
- web_search_tool_result blocks have input_json_delta events that were
incorrectly emitted as tool calls
- Added current_content_block_type tracking to only emit tool calls for
tool_use and server_tool_use blocks
2. Capture web_search_tool_result for multi-turn
- The web_search_tool_result content comes ALL AT ONCE in content_block_start
- Now captured in provider_specific_fields.web_search_results
- stream_chunk_builder combines these for final message
- Allows multi-turn conversations to work with streaming web search
Add support for the Bedrock Converse API serviceTier parameter to allow
specifying processing tier (priority, default, or flex).
Changes:
- Add ServiceTierBlock type in litellm/types/llms/bedrock.py
- Add serviceTier to CommonRequestObject
- Add serviceTier to get_config_blocks() in AmazonConverseConfig
- Add comprehensive tests for serviceTier functionality
- Add documentation for serviceTier usage
This allows users to configure service tier via:
- litellm_params in proxy config
- optional_params in SDK calls
* fix(azure_ai): Remove unsupported params from Azure AI Anthropic requests
Azure AI Anthropic endpoint rejects max_retries and stream_options parameters
with "Extra inputs are not permitted" error. These are LiteLLM-internal
parameters that should not be sent to the API.
Fixes 400 Bad Request error when using azure_ai/claude-sonnet-4-5 and other
Azure AI Anthropic models.
* test(azure_ai): Add test for unsupported params removal in Azure AI Anthropic
Verifies that max_retries, stream_options, and extra_body are properly
removed from the request before sending to Azure AI Anthropic endpoint.
Sameer Kankute