Ishaan Jaff f1c9cb7e71 feat(vertex_ai): Vertex AI Gemini Live via unified /realtime endpoint (#22153) ...

* feat(vertex_ai): add Vertex AI Gemini Live support via unified /realtime endpoint

Adds VertexAIRealtimeConfig which translates the OpenAI Realtime WebSocket
protocol to Vertex AI BidiGenerateContent. Supports voice in/voice out
(16 kHz mic → 24 kHz speaker) and text in/text out through the proxy's
/realtime endpoint.

Key changes:
- New litellm/llms/vertex_ai/realtime/transformation.py with VertexAIRealtimeConfig
  - Builds correct wss:// URL (regional + global)
  - OAuth2 Bearer token auth (not API key)
  - Full model path (projects/.../publishers/google/models/...)
  - Ignores session.update (Vertex AI only accepts one setup message)
- realtime_api/main.py: vertex_ai branch resolves OAuth token + constructs config
- llm_http_handler.py: auto-sends session setup before bidirectional_forward
- gemini/realtime/transformation.py: fix crashes on empty turnComplete events
- realtime_streaming.py: try/except guard so bad messages don't kill the loop
- proxy_server.py: add missing websockets.exceptions import

* docs: add vertex_realtime to sidebars

* fix: drop unknown event types in Gemini transform; add vertex_ai health check

* fix: propagate UUID fallback IDs from transform_content_done_event to return_additional_content_done_events

* fix: route guardrail backend sends through provider transform; fix str.strip misuse for model prefix

* fix: handle Vertex AI full resource path in session.created; route guardrail block sends through _send_to_backend

* fix: remove unused VertexBase in transformation.py; apply UUID fallback in return_additional_content_done_events

2026-02-25 22:11:06 -08:00

..

my-website

feat(vertex_ai): Vertex AI Gemini Live via unified /realtime endpoint (#22153)

2026-02-25 22:11:06 -08:00