- Add time_to_first_token_histogram using api_call_start_time for accurate measurement
- Add time_per_output_token_histogram for average time per output token
- Add response_duration_histogram for total LLM API generation time
- Extract latency metric recording into dedicated helper methods
- Fix parent span double-ending bug when reused as primary span
- Use api_call_start_time for TTFT to exclude LiteLLM overhead (matches Prometheus)
- Support both streaming and non-streaming requests
- Handle both datetime and float timestamp formats
- Add database and Redis setup to litellm_mapped_tests_proxy job in CircleCI
- Create shared test helpers in tests/test_litellm/proxy/conftest.py for proxy test setup
- Refactor health endpoint tests to use shared helpers from conftest
- Support automatic Redis cache configuration when REDIS_HOST is set
- Ensure minimal config is created when Redis/database is needed
Fixes#17821
The `is_cached_message` function crashed with TypeError when message
content was a string instead of a list of content blocks.
Changes:
- Add explicit `isinstance(content, list)` check before iteration
- Add `isinstance(content_item, dict)` check inside loop to skip non-dict items
- Use `.get()` for safer nested dict access
- Follow same pattern as `extract_ttl_from_cached_messages` (same module)
Tests:
- Add TestIsCachedMessage class with 9 test cases covering:
- String content (the reported bug)
- None content
- Missing content key
- Empty list content
- List with/without cache_control
- Mixed content types (strings + dicts)
- Wrong cache_control type
Moved speechConfig from RequestBody to GenerationConfig TypedDict so that
TTS configuration survives the filtering in _transform_request_body().
This fixes the 400 INVALID_ARGUMENT error when using Gemini TTS models
(gemini-2.5-flash-tts, gemini-2.5-flash-preview-tts, etc.) with both
vertex_ai and gemini providers.
Fixes: speechConfig was being created correctly in map_openai_params()
but then filtered out because GenerationConfig.__annotations__.keys()
didn't include it.
Tested with both preview and non-preview TTS model names and both
vertex_ai and gemini providers.
Helm chart versions now include a prerelease suffix based on release_type:
- stable: 0.1.830 (no suffix)
- rc: 0.1.830-rc
- latest: 0.1.830-latest
This allows users to easily identify stable vs non-stable chart versions.
Helm also hides prerelease versions by default, preventing accidental
upgrades to non-stable versions.
Also fixes version extraction to strip existing suffixes before bumping.
* Prompt Management API - new API to interact with Prompt Management integrations (no PR required) (#17800)
* feat: initial commit adding prompt management api
* feat: initial commit adding prompt management api
* fix: refactoring to make sure get prompt is async
* fix: additional fixes
* fix: partially working generic api prompt management
