mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-13 11:07:39 +00:00
CSteigstra
98974771fd
* fix: add sync streaming mid-stream fallback + fix 429 for all streaming paths Some LiteLLM providers (Vertex AI, Bedrock, Predibase, Codestral) use a deferred HTTP pattern where the streaming HTTP request is made lazily on the first iteration, not during completion()/acompletion(). This means errors surface during __next__/__anext__, outside the Router's retry/fallback machinery. Two gaps existed: 1. __anext__ had a blanket 4xx filter (PR #18698) that blocked 429 from MidStreamFallbackError — fixed here by exempting 429. 2. __next__ had NO MidStreamFallbackError support at all, and the Router had no sync streaming fallback wrapper — both added here. Changes: - streaming_handler.py: Extract shared _handle_stream_fallback_error() used by both __next__ and __anext__. Maps exceptions, filters non-retriable 4xx (excluding 429), wraps everything else in MidStreamFallbackError. - router.py: Add _completion_streaming_iterator() (sync mirror of _acompletion_streaming_iterator). Modify _completion() to wrap streaming responses. Add is_pre_first_chunk check to both async and sync iterators to skip continuation prompt on pre-call errors. Fixes #22296 Relates to #20870, #8648, #6532 * fix: no-op assertion in sync streaming fallback test The assertion `... is None or True` always evaluated to True, meaning it never actually verified anything. Replace with a proper check that messages match the original (no continuation prompt on pre-first-chunk errors). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>