mirror of
https://github.com/tiennm99/litellm.git
synced 2026-08-05 08:23:17 +00:00
[Feat] Add xAI /realtime API Support - works with LiveKitSDK (#20381)
* init: _realtime_health_check + routing * refactor: OpenAIRealtime * refactor: XAI_API_BASE * feat: XAIRealtime * init feat: XAIRealtime * OpenAIRealtime * TestXAIRealtime * test fixes * test OAI * TEST xAI, OAI * clean realtime jobs * refactor * test XAI * docs xAI * fix xAI * fix lint errors * test_async_realtime_url_contains_model * test fix * document test changes * _realtime_health_check * docs xai realtime * fix handlers * add additional_headers * fix
This commit is contained in:
Ishaan Jaff
+66
-2
@@ -1255,7 +1255,15 @@ jobs:
|
||||
ls
|
||||
# Add --timeout to kill hanging tests after 120s (2 min)
|
||||
# Add --durations=20 to show 20 slowest tests for debugging
|
||||
python -m pytest -vv tests/llm_translation --cov=litellm --cov-report=xml -v --junitxml=test-results/junit.xml --durations=20 -n 4 --timeout=120 --timeout_method=thread
|
||||
# Subdirectories with dedicated jobs (maintain this list as new jobs are added)
|
||||
IGNORE_DIRS=(
|
||||
"tests/llm_translation/realtime"
|
||||
)
|
||||
IGNORE_ARGS=""
|
||||
for dir in "${IGNORE_DIRS[@]}"; do
|
||||
IGNORE_ARGS="$IGNORE_ARGS --ignore=$dir"
|
||||
done
|
||||
python -m pytest -vv tests/llm_translation $IGNORE_ARGS --cov=litellm --cov-report=xml -v --junitxml=test-results/junit.xml --durations=20 -n 4 --timeout=120 --timeout_method=thread
|
||||
no_output_timeout: 120m
|
||||
- run:
|
||||
name: Rename the coverage files
|
||||
@@ -1271,6 +1279,54 @@ jobs:
|
||||
paths:
|
||||
- llm_translation_coverage.xml
|
||||
- llm_translation_coverage
|
||||
realtime_translation_testing:
|
||||
docker:
|
||||
- image: cimg/python:3.11
|
||||
auth:
|
||||
username: ${DOCKERHUB_USERNAME}
|
||||
password: ${DOCKERHUB_PASSWORD}
|
||||
working_directory: ~/project
|
||||
|
||||
steps:
|
||||
- checkout
|
||||
- setup_google_dns
|
||||
- run:
|
||||
name: Install Dependencies
|
||||
command: |
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install -r requirements.txt
|
||||
pip install "pytest==7.3.1"
|
||||
pip install "pytest-retry==1.6.3"
|
||||
pip install "pytest-cov==5.0.0"
|
||||
pip install "pytest-asyncio==0.21.1"
|
||||
pip install "respx==0.22.0"
|
||||
pip install "pytest-xdist==3.6.1"
|
||||
pip install "pytest-timeout==2.2.0"
|
||||
pip install "websockets"
|
||||
# Run pytest and generate JUnit XML report
|
||||
- run:
|
||||
name: Run realtime tests
|
||||
command: |
|
||||
pwd
|
||||
ls
|
||||
# Add --timeout to kill hanging tests after 120s (2 min)
|
||||
# Add --durations=20 to show 20 slowest tests for debugging
|
||||
python -m pytest -vv tests/llm_translation/realtime --cov=litellm --cov-report=xml -v --junitxml=test-results/junit.xml --durations=20 -n 4 --timeout=120 --timeout_method=thread
|
||||
no_output_timeout: 120m
|
||||
- run:
|
||||
name: Rename the coverage files
|
||||
command: |
|
||||
mv coverage.xml realtime_translation_coverage.xml
|
||||
mv .coverage realtime_translation_coverage
|
||||
|
||||
# Store test results
|
||||
- store_test_results:
|
||||
path: test-results
|
||||
- persist_to_workspace:
|
||||
root: .
|
||||
paths:
|
||||
- realtime_translation_coverage.xml
|
||||
- realtime_translation_coverage
|
||||
mcp_testing:
|
||||
docker:
|
||||
- image: cimg/python:3.11
|
||||
@@ -3532,7 +3588,7 @@ jobs:
|
||||
python -m venv venv
|
||||
. venv/bin/activate
|
||||
pip install coverage
|
||||
coverage combine llm_translation_coverage llm_responses_api_coverage ocr_coverage search_coverage mcp_coverage logging_coverage audio_coverage litellm_router_coverage litellm_router_unit_coverage local_testing_part1_coverage local_testing_part2_coverage litellm_assistants_api_coverage auth_ui_unit_tests_coverage langfuse_coverage caching_coverage litellm_proxy_unit_tests_part1_coverage litellm_proxy_unit_tests_part2_coverage image_gen_coverage pass_through_unit_tests_coverage batches_coverage litellm_security_tests_coverage guardrails_coverage litellm_mapped_tests_coverage
|
||||
coverage combine llm_translation_coverage realtime_translation_coverage llm_responses_api_coverage ocr_coverage search_coverage mcp_coverage logging_coverage audio_coverage litellm_router_coverage litellm_router_unit_coverage local_testing_part1_coverage local_testing_part2_coverage litellm_assistants_api_coverage auth_ui_unit_tests_coverage langfuse_coverage caching_coverage litellm_proxy_unit_tests_part1_coverage litellm_proxy_unit_tests_part2_coverage image_gen_coverage pass_through_unit_tests_coverage batches_coverage litellm_security_tests_coverage guardrails_coverage litellm_mapped_tests_coverage
|
||||
coverage xml
|
||||
- codecov/upload:
|
||||
file: ./coverage.xml
|
||||
@@ -4196,6 +4252,12 @@ workflows:
|
||||
only:
|
||||
- main
|
||||
- /litellm_.*/
|
||||
- realtime_translation_testing:
|
||||
filters:
|
||||
branches:
|
||||
only:
|
||||
- main
|
||||
- /litellm_.*/
|
||||
- mcp_testing:
|
||||
filters:
|
||||
branches:
|
||||
@@ -4307,6 +4369,7 @@ workflows:
|
||||
- upload-coverage:
|
||||
requires:
|
||||
- llm_translation_testing
|
||||
- realtime_translation_testing
|
||||
- mcp_testing
|
||||
- google_generate_content_endpoint_testing
|
||||
- guardrails_testing
|
||||
@@ -4384,6 +4447,7 @@ workflows:
|
||||
- e2e_openai_endpoints
|
||||
- test_bad_database_url
|
||||
- llm_translation_testing
|
||||
- realtime_translation_testing
|
||||
- mcp_testing
|
||||
- google_generate_content_endpoint_testing
|
||||
- llm_responses_api_testing
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
# LiveKit Voice Agent with LiteLLM Gateway
|
||||
|
||||
Simple example showing how to use LiveKit's xAI realtime plugin with LiteLLM as a proxy. This lets you switch between xAI, OpenAI, and Azure realtime APIs without changing your code.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Install dependencies
|
||||
|
||||
```bash
|
||||
pip install livekit-agents[xai] websockets
|
||||
```
|
||||
|
||||
### 2. Start LiteLLM proxy
|
||||
|
||||
```bash
|
||||
# With xAI
|
||||
export XAI_API_KEY="your-xai-key"
|
||||
litellm --config config.yaml --port 4000
|
||||
```
|
||||
|
||||
### 3. Run the voice agent
|
||||
|
||||
```bash
|
||||
python main.py
|
||||
```
|
||||
|
||||
Type your message and get a voice response from Grok!
|
||||
|
||||
## Configuration
|
||||
|
||||
Set these environment variables if needed:
|
||||
|
||||
```bash
|
||||
export LITELLM_PROXY_URL="http://localhost:4000"
|
||||
export LITELLM_API_KEY="sk-1234"
|
||||
export LITELLM_MODEL="grok-voice-agent"
|
||||
```
|
||||
|
||||
Or use the defaults - connects to `http://localhost:4000` by default.
|
||||
|
||||
## Example Config File
|
||||
|
||||
Create a `config.yaml` with your realtime models:
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: grok-voice-agent
|
||||
litellm_params:
|
||||
model: xai/grok-2-vision-1212
|
||||
api_key: os.environ/XAI_API_KEY
|
||||
model_info:
|
||||
mode: realtime
|
||||
|
||||
- model_name: openai-voice-agent
|
||||
litellm_params:
|
||||
model: gpt-4o-realtime-preview
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
model_info:
|
||||
mode: realtime
|
||||
|
||||
general_settings:
|
||||
master_key: sk-1234
|
||||
```
|
||||
|
||||
Then start: `litellm --config config.yaml --port 4000`
|
||||
|
||||
## How It Works
|
||||
|
||||
LiveKit's xAI plugin connects through LiteLLM proxy by setting `base_url`:
|
||||
|
||||
```python
|
||||
from livekit.plugins import xai
|
||||
|
||||
model = xai.realtime.RealtimeModel(
|
||||
voice="ara",
|
||||
api_key="sk-1234", # LiteLLM proxy key
|
||||
base_url="http://localhost:4000", # Point to LiteLLM
|
||||
)
|
||||
```
|
||||
|
||||
## Switching Providers
|
||||
|
||||
Just change the model in your config - no code changes needed:
|
||||
|
||||
**xAI Grok:**
|
||||
```yaml
|
||||
model: xai/grok-2-vision-1212
|
||||
```
|
||||
|
||||
**OpenAI:**
|
||||
```yaml
|
||||
model: gpt-4o-realtime-preview
|
||||
```
|
||||
|
||||
**Azure OpenAI:**
|
||||
```yaml
|
||||
model: azure/gpt-4o-realtime-preview
|
||||
api_base: https://your-endpoint.openai.azure.com/
|
||||
```
|
||||
|
||||
## Why Use LiteLLM?
|
||||
|
||||
- ✅ **Switch providers** without changing agent code
|
||||
- ✅ **Cost tracking** across all voice sessions
|
||||
- ✅ **Rate limiting** and budgets
|
||||
- ✅ **Load balancing** across multiple API keys
|
||||
- ✅ **Fallbacks** to backup models
|
||||
|
||||
## Learn More
|
||||
|
||||
- [LiveKit xAI Realtime Tutorial](/docs/tutorials/livekit_xai_realtime)
|
||||
- [xAI Realtime Docs](/docs/providers/xai_realtime)
|
||||
- [LiveKit Agents Documentation](https://docs.livekit.io/agents/)
|
||||
- [LiteLLM Realtime API](/docs/realtime)
|
||||
@@ -0,0 +1,21 @@
|
||||
model_list:
|
||||
- model_name: grok-voice-agent
|
||||
litellm_params:
|
||||
model: xai/grok-2-vision-1212
|
||||
api_key: os.environ/XAI_API_KEY
|
||||
model_info:
|
||||
mode: realtime
|
||||
|
||||
- model_name: openai-voice-agent
|
||||
litellm_params:
|
||||
model: gpt-4o-realtime-preview
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
model_info:
|
||||
mode: realtime
|
||||
|
||||
litellm_settings:
|
||||
drop_params: True
|
||||
telemetry: False
|
||||
|
||||
general_settings:
|
||||
master_key: sk-1234 # Change this to a secure key
|
||||
@@ -0,0 +1,112 @@
|
||||
"""
|
||||
Simple xAI Voice Agent using LiveKit SDK with LiteLLM Gateway
|
||||
|
||||
This example shows how to use LiveKit's xAI realtime plugin through LiteLLM proxy.
|
||||
LiteLLM acts as a unified interface, allowing you to switch between xAI, OpenAI,
|
||||
and Azure realtime APIs without changing your agent code.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import websockets
|
||||
|
||||
# Configuration
|
||||
PROXY_URL = os.getenv("LITELLM_PROXY_URL", "http://localhost:4000")
|
||||
API_KEY = os.getenv("LITELLM_API_KEY", "sk-1234")
|
||||
MODEL = os.getenv("LITELLM_MODEL", "grok-voice-agent")
|
||||
|
||||
|
||||
async def run_voice_agent():
|
||||
"""
|
||||
Simple voice agent that:
|
||||
1. Connects to xAI realtime API through LiteLLM proxy
|
||||
2. Sends a user message
|
||||
3. Streams back the response
|
||||
"""
|
||||
|
||||
url = f"ws://{PROXY_URL.replace('http://', '').replace('https://', '')}/v1/realtime?model={MODEL}"
|
||||
headers = {"Authorization": f"Bearer {API_KEY}"}
|
||||
|
||||
print(f"🎙️ Connecting to voice agent...")
|
||||
print(f" Model: {MODEL}")
|
||||
print(f" Proxy: {PROXY_URL}")
|
||||
print()
|
||||
|
||||
async with websockets.connect(url, additional_headers=headers) as ws:
|
||||
# Receive initial connection event
|
||||
initial = json.loads(await ws.recv())
|
||||
print(f"✅ Connected! Event: {initial['type']}\n")
|
||||
|
||||
# Get user input
|
||||
user_message = input("💬 Your message: ").strip()
|
||||
if not user_message:
|
||||
user_message = "Tell me a fun fact about AI!"
|
||||
|
||||
print(f"\n🤖 Sending to {MODEL}...\n")
|
||||
|
||||
# Send user message
|
||||
await ws.send(json.dumps({
|
||||
"type": "conversation.item.create",
|
||||
"item": {
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": [{"type": "input_text", "text": user_message}]
|
||||
}
|
||||
}))
|
||||
|
||||
# Request response
|
||||
await ws.send(json.dumps({
|
||||
"type": "response.create",
|
||||
"response": {"modalities": ["text", "audio"]}
|
||||
}))
|
||||
|
||||
# Stream response
|
||||
print("🎤 Response: ", end='', flush=True)
|
||||
transcript = []
|
||||
|
||||
try:
|
||||
while True:
|
||||
msg = await asyncio.wait_for(ws.recv(), timeout=15.0)
|
||||
event = json.loads(msg)
|
||||
|
||||
# Capture transcript deltas
|
||||
if event['type'] == 'response.output_audio_transcript.delta':
|
||||
delta = event.get('delta', '')
|
||||
if delta:
|
||||
print(delta, end='', flush=True)
|
||||
transcript.append(delta)
|
||||
|
||||
# Done when response completes
|
||||
elif event['type'] == 'response.done':
|
||||
break
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
|
||||
print("\n")
|
||||
|
||||
if transcript:
|
||||
print(f"✅ Complete response: {''.join(transcript)}")
|
||||
|
||||
await ws.close()
|
||||
|
||||
|
||||
def main():
|
||||
"""Run the voice agent"""
|
||||
print("=" * 70)
|
||||
print("LiveKit xAI Voice Agent via LiteLLM Proxy")
|
||||
print("=" * 70)
|
||||
print()
|
||||
|
||||
try:
|
||||
asyncio.run(run_voice_agent())
|
||||
except KeyboardInterrupt:
|
||||
print("\n\n👋 Goodbye!")
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error: {e}")
|
||||
print("\nMake sure LiteLLM proxy is running:")
|
||||
print(f" litellm --config config.yaml --port 4000")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,2 @@
|
||||
livekit-agents[xai]>=1.3.12
|
||||
websockets>=15.0.1
|
||||
@@ -0,0 +1,308 @@
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# xAI Voice Agent (Realtime API)
|
||||
|
||||
xAI's Grok Voice Agent provides real-time voice conversation capabilities through WebSocket connections, enabling natural bidirectional audio interactions.
|
||||
|
||||
| Feature | Description | Comments |
|
||||
| --- | --- | --- |
|
||||
| LiteLLM AI Gateway | ✅ | |
|
||||
| LiteLLM Python SDK | ✅ | Full support via `litellm.realtime()` |
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Supported Model
|
||||
|
||||
| Model | Context | Features |
|
||||
|-------|---------|----------|
|
||||
| `xai/grok-4-1-fast-non-reasoning` | 2M tokens | Voice conversation, Function calling, Vision, Audio, Web search, Caching |
|
||||
|
||||
**Note:** xAI Realtime API uses the non-reasoning variant for optimal real-time performance.
|
||||
|
||||
## Python SDK Usage
|
||||
|
||||
### Basic Realtime Connection
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from litellm import realtime
|
||||
|
||||
async def test_xai_realtime():
|
||||
"""
|
||||
Test xAI Grok Voice Agent via LiteLLM SDK
|
||||
"""
|
||||
# Initialize realtime connection
|
||||
ws = await realtime(
|
||||
model="xai/grok-4-1-fast-non-reasoning",
|
||||
api_key="your-xai-api-key", # or set XAI_API_KEY env var
|
||||
)
|
||||
|
||||
# Connection established, xAI sends "conversation.created" event
|
||||
print("Connected to xAI Grok Voice Agent")
|
||||
|
||||
# Send a message
|
||||
await ws.send_text(json.dumps({
|
||||
"type": "conversation.item.create",
|
||||
"item": {
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": [{
|
||||
"type": "input_text",
|
||||
"text": "Hello! How are you?"
|
||||
}]
|
||||
}
|
||||
}))
|
||||
|
||||
# Request a response
|
||||
await ws.send_text(json.dumps({
|
||||
"type": "response.create"
|
||||
}))
|
||||
|
||||
# Listen for responses
|
||||
async for message in ws:
|
||||
data = json.loads(message)
|
||||
print(f"Received: {data['type']}")
|
||||
|
||||
if data['type'] == 'response.done':
|
||||
break
|
||||
|
||||
await ws.close()
|
||||
|
||||
# Run the async function
|
||||
asyncio.run(test_xai_realtime())
|
||||
```
|
||||
|
||||
### With Audio Input/Output
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import json
|
||||
from litellm import realtime
|
||||
|
||||
async def xai_voice_conversation():
|
||||
"""
|
||||
Voice conversation with xAI Grok Voice Agent
|
||||
"""
|
||||
ws = await realtime(
|
||||
model="xai/grok-4-1-fast-non-reasoning",
|
||||
api_key="your-xai-api-key",
|
||||
)
|
||||
|
||||
# Send audio data (base64 encoded PCM16 24kHz)
|
||||
await ws.send_text(json.dumps({
|
||||
"type": "conversation.item.create",
|
||||
"item": {
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": [{
|
||||
"type": "input_audio",
|
||||
"audio": "base64_encoded_audio_data_here"
|
||||
}]
|
||||
}
|
||||
}))
|
||||
|
||||
# Request response with audio
|
||||
await ws.send_text(json.dumps({
|
||||
"type": "response.create",
|
||||
"response": {
|
||||
"modalities": ["text", "audio"],
|
||||
"instructions": "Please respond in a friendly tone."
|
||||
}
|
||||
}))
|
||||
|
||||
# Process streaming audio response
|
||||
async for message in ws:
|
||||
data = json.loads(message)
|
||||
|
||||
if data['type'] == 'response.audio.delta':
|
||||
# Handle audio chunks
|
||||
audio_chunk = data['delta']
|
||||
# Process audio_chunk (play it, save it, etc.)
|
||||
|
||||
elif data['type'] == 'response.done':
|
||||
break
|
||||
|
||||
await ws.close()
|
||||
|
||||
asyncio.run(xai_voice_conversation())
|
||||
```
|
||||
|
||||
## LiteLLM Proxy (AI Gateway) Usage
|
||||
|
||||
Load balance across multiple xAI deployments or combine with other providers.
|
||||
|
||||
### 1. Add Model to Config
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: grok-voice-agent
|
||||
litellm_params:
|
||||
model: xai/grok-4-1-fast-non-reasoning
|
||||
api_key: os.environ/XAI_API_KEY
|
||||
model_info:
|
||||
mode: realtime
|
||||
|
||||
# Optional: Add fallback to OpenAI
|
||||
- model_name: grok-voice-agent
|
||||
litellm_params:
|
||||
model: openai/gpt-4o-realtime-preview-2024-10-01
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
model_info:
|
||||
mode: realtime
|
||||
```
|
||||
|
||||
### 2. Start Proxy
|
||||
|
||||
```bash
|
||||
litellm --config /path/to/config.yaml
|
||||
|
||||
# RUNNING on http://0.0.0.0:4000
|
||||
```
|
||||
|
||||
### 3. Test Connection
|
||||
|
||||
#### Python Client
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import websockets
|
||||
import json
|
||||
|
||||
async def test_proxy():
|
||||
url = "ws://0.0.0.0:4000/v1/realtime?model=grok-voice-agent"
|
||||
|
||||
async with websockets.connect(
|
||||
url,
|
||||
extra_headers={
|
||||
"Authorization": "Bearer sk-1234", # Your LiteLLM proxy key
|
||||
"OpenAI-Beta": "realtime=v1"
|
||||
}
|
||||
) as ws:
|
||||
# Wait for conversation.created event from xAI
|
||||
message = await ws.recv()
|
||||
print(f"Connected: {message}")
|
||||
|
||||
# Send a message
|
||||
await ws.send(json.dumps({
|
||||
"type": "conversation.item.create",
|
||||
"item": {
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": [{
|
||||
"type": "input_text",
|
||||
"text": "Hello from LiteLLM proxy!"
|
||||
}]
|
||||
}
|
||||
}))
|
||||
|
||||
# Request response
|
||||
await ws.send(json.dumps({
|
||||
"type": "response.create"
|
||||
}))
|
||||
|
||||
# Listen for response
|
||||
async for message in ws:
|
||||
data = json.loads(message)
|
||||
print(f"Event: {data['type']}")
|
||||
|
||||
if data['type'] == 'response.done':
|
||||
break
|
||||
|
||||
asyncio.run(test_proxy())
|
||||
```
|
||||
|
||||
#### Node.js Client
|
||||
|
||||
```javascript
|
||||
// test.js - Run with: node test.js
|
||||
const WebSocket = require("ws");
|
||||
|
||||
const url = "ws://0.0.0.0:4000/v1/realtime?model=grok-voice-agent";
|
||||
|
||||
const ws = new WebSocket(url, {
|
||||
headers: {
|
||||
"Authorization": "Bearer sk-1234",
|
||||
"OpenAI-Beta": "realtime=v1",
|
||||
},
|
||||
});
|
||||
|
||||
ws.on("open", function open() {
|
||||
console.log("Connected to xAI via LiteLLM proxy");
|
||||
|
||||
// Send a message
|
||||
ws.send(JSON.stringify({
|
||||
type: "conversation.item.create",
|
||||
item: {
|
||||
type: "message",
|
||||
role: "user",
|
||||
content: [{
|
||||
type: "input_text",
|
||||
text: "What's the weather like?"
|
||||
}]
|
||||
}
|
||||
}));
|
||||
|
||||
// Request response
|
||||
ws.send(JSON.stringify({
|
||||
type: "response.create",
|
||||
response: {
|
||||
modalities: ["text"],
|
||||
instructions: "Please assist the user."
|
||||
}
|
||||
}));
|
||||
});
|
||||
|
||||
ws.on("message", function incoming(message) {
|
||||
const data = JSON.parse(message.toString());
|
||||
console.log(`Event: ${data.type}`);
|
||||
|
||||
if (data.type === 'response.done') {
|
||||
ws.close();
|
||||
}
|
||||
});
|
||||
|
||||
ws.on("error", function handleError(error) {
|
||||
console.error("Error: ", error);
|
||||
});
|
||||
```
|
||||
|
||||
## Key Differences from OpenAI
|
||||
|
||||
xAI's Grok Voice Agent has some differences from OpenAI's Realtime API:
|
||||
|
||||
| Feature | xAI | OpenAI | LiteLLM Handling |
|
||||
|---------|-----|--------|------------------|
|
||||
| Initial Event | `conversation.created` | `session.created` | ⚠️ Passed through as-is |
|
||||
| WebSocket URL | `wss://api.x.ai/v1/realtime` | `wss://api.openai.com/v1/realtime` | ✅ Auto-configured |
|
||||
| Model | `grok-4-1-fast-non-reasoning` | `gpt-4o-realtime-preview` | ✅ Via model prefix |
|
||||
| Audio Format | PCM16 24kHz mono | PCM16 24kHz mono | ✅ Compatible |
|
||||
| Context Window | 2M tokens | 128K tokens | N/A |
|
||||
|
||||
**What LiteLLM Handles:**
|
||||
- ✅ Automatic URL routing to correct provider
|
||||
- ✅ Authentication headers (no `OpenAI-Beta` header for xAI)
|
||||
- ✅ WebSocket connection management
|
||||
- ✅ All other event types are compatible
|
||||
|
||||
**What You Need to Handle:**
|
||||
- ⚠️ Initial event type difference (`conversation.created` vs `session.created`)
|
||||
|
||||
**Tip:** Make your client compatible with both event types:
|
||||
```python
|
||||
# Handle both providers
|
||||
if event['type'] in ['session.created', 'conversation.created']:
|
||||
print("Connection established")
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [xAI Chat/Text Models](/docs/providers/xai)
|
||||
- [LiteLLM Realtime API Overview](/docs/realtime)
|
||||
- [xAI Official Documentation](https://docs.x.ai/docs)
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
- [LiteLLM GitHub Issues](https://github.com/BerriAI/litellm/issues)
|
||||
- [xAI Documentation](https://docs.x.ai/docs)
|
||||
@@ -3,11 +3,12 @@ import TabItem from '@theme/TabItem';
|
||||
|
||||
# /realtime
|
||||
|
||||
Use this to loadbalance across Azure + OpenAI.
|
||||
Use this to loadbalance across Azure + OpenAI + xAI and more.
|
||||
|
||||
Supported Providers:
|
||||
- OpenAI
|
||||
- Azure
|
||||
- xAI ([see full docs](/docs/providers/xai_realtime))
|
||||
- Google AI Studio (Gemini)
|
||||
- Vertex AI
|
||||
- Bedrock
|
||||
@@ -46,6 +47,21 @@ model_list:
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="xai" label="xAI Grok Voice Agent">
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: grok-voice-agent
|
||||
litellm_params:
|
||||
model: xai/grok-4-1-fast-non-reasoning
|
||||
api_key: os.environ/XAI_API_KEY
|
||||
model_info:
|
||||
mode: realtime
|
||||
```
|
||||
|
||||
**[See full xAI Realtime documentation →](/docs/providers/xai_realtime)**
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# LiveKit xAI Realtime Voice Agent
|
||||
|
||||
Use LiveKit's xAI Grok Voice Agent plugin with LiteLLM Proxy to build low-latency voice AI agents.
|
||||
|
||||
The LiveKit Agents framework provides tools for building real-time voice and video AI applications. By routing through LiteLLM Proxy, you get unified access to multiple realtime voice providers, cost tracking, rate limiting, and more.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Install Dependencies
|
||||
|
||||
```bash
|
||||
pip install livekit-agents[xai]
|
||||
```
|
||||
|
||||
### 2. Start LiteLLM Proxy
|
||||
|
||||
Create a config file with your xAI realtime model:
|
||||
|
||||
```yaml title="config.yaml" showLineNumbers
|
||||
model_list:
|
||||
- model_name: grok-voice-agent
|
||||
litellm_params:
|
||||
model: xai/grok-2-vision-1212
|
||||
api_key: os.environ/XAI_API_KEY
|
||||
model_info:
|
||||
mode: realtime
|
||||
|
||||
litellm_settings:
|
||||
drop_params: True
|
||||
|
||||
general_settings:
|
||||
master_key: sk-1234 # Change this to a secure key
|
||||
```
|
||||
|
||||
Start the proxy:
|
||||
|
||||
```bash
|
||||
litellm --config config.yaml --port 4000
|
||||
```
|
||||
|
||||
### 3. Configure LiveKit xAI Plugin
|
||||
|
||||
Point LiveKit's xAI plugin to your LiteLLM proxy:
|
||||
|
||||
```python
|
||||
from livekit.plugins import xai
|
||||
|
||||
# Configure xAI to use LiteLLM proxy
|
||||
model = xai.realtime.RealtimeModel(
|
||||
voice="ara", # Voice option
|
||||
api_key="sk-1234", # Your LiteLLM proxy master key
|
||||
base_url="http://localhost:4000", # LiteLLM proxy URL
|
||||
)
|
||||
```
|
||||
|
||||
## Complete Example
|
||||
|
||||
Here's a complete working example:
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="python" label="Python Client">
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Simple xAI realtime voice agent through LiteLLM proxy.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import websockets
|
||||
|
||||
PROXY_URL = "ws://localhost:4000/v1/realtime"
|
||||
API_KEY = "sk-1234"
|
||||
MODEL = "grok-voice-agent"
|
||||
|
||||
async def run_voice_agent():
|
||||
"""Connect to xAI realtime API through LiteLLM proxy"""
|
||||
url = f"{PROXY_URL}?model={MODEL}"
|
||||
headers = {"Authorization": f"Bearer {API_KEY}"}
|
||||
|
||||
async with websockets.connect(url, extra_headers=headers) as ws:
|
||||
# Wait for initial connection event
|
||||
initial = json.loads(await ws.recv())
|
||||
print(f"✅ Connected: {initial['type']}")
|
||||
|
||||
# Send user message
|
||||
await ws.send(json.dumps({
|
||||
"type": "conversation.item.create",
|
||||
"item": {
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": [{
|
||||
"type": "input_text",
|
||||
"text": "Hello! Tell me a joke."
|
||||
}]
|
||||
}
|
||||
}))
|
||||
|
||||
# Request response
|
||||
await ws.send(json.dumps({
|
||||
"type": "response.create",
|
||||
"response": {"modalities": ["text", "audio"]}
|
||||
}))
|
||||
|
||||
# Collect response
|
||||
transcript = []
|
||||
async for message in ws:
|
||||
event = json.loads(message)
|
||||
|
||||
# Capture text response
|
||||
if event['type'] == 'response.output_audio_transcript.delta':
|
||||
transcript.append(event['delta'])
|
||||
print(event['delta'], end='', flush=True)
|
||||
|
||||
# Done when response completes
|
||||
elif event['type'] == 'response.done':
|
||||
break
|
||||
|
||||
print(f"\n\n✅ Full response: {''.join(transcript)}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(run_voice_agent())
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="livekit" label="LiveKit Agent">
|
||||
|
||||
```python
|
||||
from livekit.agents import Agent, AgentSession, WorkerOptions, cli
|
||||
from livekit.plugins import xai
|
||||
|
||||
class VoiceAgent(Agent):
|
||||
def __init__(self):
|
||||
super().__init__(
|
||||
instructions="You are a helpful voice assistant.",
|
||||
llm=xai.realtime.RealtimeModel(
|
||||
voice="ara",
|
||||
api_key="sk-1234",
|
||||
base_url="http://localhost:4000",
|
||||
),
|
||||
)
|
||||
|
||||
if __name__ == "__main__":
|
||||
cli.run_app(
|
||||
WorkerOptions(
|
||||
agent_factory=VoiceAgent,
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Running the Example
|
||||
|
||||
1. **Start LiteLLM Proxy** (if not already running):
|
||||
```bash
|
||||
litellm --config config.yaml --port 4000
|
||||
```
|
||||
|
||||
2. **Run the example**:
|
||||
```bash
|
||||
python your_script.py
|
||||
```
|
||||
|
||||
## Expected Output
|
||||
|
||||
```
|
||||
✅ Connected: conversation.created
|
||||
Hello! Here's a joke for you: Why don't scientists trust atoms?
|
||||
Because they make up everything!
|
||||
|
||||
✅ Full response: Hello! Here's a joke for you: Why don't scientists trust atoms? Because they make up everything!
|
||||
```
|
||||
|
||||
|
||||
## Complete Working Example
|
||||
|
||||
**[LiveKit Agent SDK Cookbook](https://github.com/BerriAI/litellm/tree/main/cookbook/livekit_agent_sdk)**
|
||||
|
||||
Includes:
|
||||
- ✅ Simple voice agent (`main.py`)
|
||||
- ✅ Config example (`config.example.yaml`)
|
||||
- ✅ How to run guide (`HOW_TO_RUN.md`)
|
||||
|
||||
### Quick Test
|
||||
|
||||
```bash
|
||||
# 1. Start proxy
|
||||
poetry run litellm --config cookbook/livekit_agent_sdk/config.example.yaml --port 4000
|
||||
|
||||
# 2. Run agent (in new terminal)
|
||||
cd cookbook/livekit_agent_sdk
|
||||
poetry run python main.py
|
||||
```
|
||||
|
||||
## Learn More
|
||||
|
||||
- [xAI Realtime API](/docs/providers/xai_realtime)
|
||||
- [LiveKit xAI Plugin](https://docs.livekit.io/agents/models/realtime/plugins/xai/)
|
||||
- [LiteLLM Realtime API](/docs/realtime)
|
||||
@@ -151,6 +151,7 @@ const sidebars = {
|
||||
items: [
|
||||
"tutorials/claude_agent_sdk",
|
||||
"tutorials/google_adk",
|
||||
"tutorials/livekit_xai_realtime",
|
||||
]
|
||||
},
|
||||
|
||||
@@ -851,7 +852,14 @@ const sidebars = {
|
||||
"providers/watsonx/audio_transcription",
|
||||
]
|
||||
},
|
||||
"providers/xai",
|
||||
{
|
||||
type: "category",
|
||||
label: "xAI",
|
||||
items: [
|
||||
"providers/xai",
|
||||
"providers/xai_realtime",
|
||||
]
|
||||
},
|
||||
"providers/xiaomi_mimo",
|
||||
"providers/xinference",
|
||||
"providers/zai",
|
||||
|
||||
@@ -99,6 +99,9 @@ DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET = int(
|
||||
os.getenv("DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET", 128)
|
||||
)
|
||||
|
||||
# Provider-specific API base URLs
|
||||
XAI_API_BASE = "https://api.x.ai/v1"
|
||||
|
||||
DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET = int(
|
||||
os.getenv("DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET", 1024)
|
||||
)
|
||||
|
||||
@@ -16,6 +16,62 @@ from ..openai import OpenAIChatCompletion
|
||||
|
||||
|
||||
class OpenAIRealtime(OpenAIChatCompletion):
|
||||
"""
|
||||
Base handler for OpenAI-compatible realtime WebSocket connections.
|
||||
|
||||
Subclasses can override template methods to customize:
|
||||
- _get_default_api_base(): Default API base URL
|
||||
- _get_additional_headers(): Extra headers beyond Authorization
|
||||
- _get_ssl_config(): SSL configuration for WebSocket connection
|
||||
"""
|
||||
|
||||
def _get_default_api_base(self) -> str:
|
||||
"""
|
||||
Get the default API base URL for this provider.
|
||||
Override this in subclasses to set provider-specific defaults.
|
||||
"""
|
||||
return "https://api.openai.com/"
|
||||
|
||||
def _get_additional_headers(self, api_key: str) -> dict:
|
||||
"""
|
||||
Get additional headers beyond Authorization.
|
||||
Override this in subclasses to customize headers (e.g., remove OpenAI-Beta).
|
||||
|
||||
Args:
|
||||
api_key: API key for authentication
|
||||
|
||||
Returns:
|
||||
Dictionary of additional headers
|
||||
"""
|
||||
return {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"OpenAI-Beta": "realtime=v1",
|
||||
}
|
||||
|
||||
def _get_ssl_config(self, url: str) -> Any:
|
||||
"""
|
||||
Get SSL configuration for WebSocket connection.
|
||||
Override this in subclasses to customize SSL behavior.
|
||||
|
||||
Args:
|
||||
url: WebSocket URL (ws:// or wss://)
|
||||
|
||||
Returns:
|
||||
SSL configuration (None, True, or SSLContext)
|
||||
"""
|
||||
if url.startswith("ws://"):
|
||||
return None
|
||||
|
||||
# Use the shared SSL context which respects custom CA certs and SSL settings
|
||||
ssl_config = get_shared_realtime_ssl_context()
|
||||
|
||||
# If ssl_config is False (ssl_verify=False), websockets library needs True instead
|
||||
# to establish connection without verification (False would fail)
|
||||
if ssl_config is False:
|
||||
return True
|
||||
|
||||
return ssl_config
|
||||
|
||||
def _construct_url(self, api_base: str, query_params: RealtimeQueryParams) -> str:
|
||||
"""
|
||||
Construct the backend websocket URL with all query parameters (including 'model').
|
||||
@@ -45,8 +101,9 @@ class OpenAIRealtime(OpenAIChatCompletion):
|
||||
):
|
||||
import websockets
|
||||
from websockets.asyncio.client import ClientConnection
|
||||
|
||||
if api_base is None:
|
||||
api_base = "https://api.openai.com/"
|
||||
api_base = self._get_default_api_base()
|
||||
if api_key is None:
|
||||
raise ValueError("api_key is required for OpenAI realtime calls")
|
||||
|
||||
@@ -56,30 +113,27 @@ class OpenAIRealtime(OpenAIChatCompletion):
|
||||
url = self._construct_url(api_base, query_params)
|
||||
|
||||
try:
|
||||
# Only use SSL context for secure websocket connections (wss://)
|
||||
# websockets library doesn't accept ssl argument for ws:// URIs
|
||||
ssl_context = None if url.startswith("ws://") else get_shared_realtime_ssl_context()
|
||||
# Get provider-specific SSL configuration
|
||||
ssl_config = self._get_ssl_config(url)
|
||||
|
||||
# Get provider-specific headers
|
||||
headers = self._get_additional_headers(api_key)
|
||||
|
||||
# Log a masked request preview consistent with other endpoints.
|
||||
logging_obj.pre_call(
|
||||
input=None,
|
||||
api_key=api_key,
|
||||
additional_args={
|
||||
"api_base": url,
|
||||
"headers": {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"OpenAI-Beta": "realtime=v1",
|
||||
},
|
||||
"headers": headers,
|
||||
"complete_input_dict": {"query_params": query_params},
|
||||
},
|
||||
)
|
||||
async with websockets.connect( # type: ignore
|
||||
url,
|
||||
additional_headers={
|
||||
"Authorization": f"Bearer {api_key}", # type: ignore
|
||||
"OpenAI-Beta": "realtime=v1",
|
||||
},
|
||||
additional_headers=headers, # type: ignore
|
||||
max_size=REALTIME_WEBSOCKET_MAX_MESSAGE_SIZE_BYTES,
|
||||
ssl=ssl_context,
|
||||
ssl=ssl_config,
|
||||
) as backend_ws:
|
||||
realtime_streaming = RealTimeStreaming(
|
||||
websocket, cast(ClientConnection, backend_ws), logging_obj
|
||||
|
||||
@@ -4,6 +4,7 @@ import httpx
|
||||
|
||||
import litellm
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.constants import XAI_API_BASE
|
||||
from litellm.litellm_core_utils.prompt_templates.common_utils import (
|
||||
filter_value_from_dict,
|
||||
strip_name_from_messages,
|
||||
@@ -14,8 +15,6 @@ from litellm.types.utils import Choices, ModelResponse, Usage, PromptTokensDetai
|
||||
|
||||
from ...openai.chat.gpt_transformation import OpenAIGPTConfig
|
||||
|
||||
XAI_API_BASE = "https://api.x.ai/v1"
|
||||
|
||||
|
||||
class XAIChatConfig(OpenAIGPTConfig):
|
||||
@property
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
"""xAI Realtime API handler."""
|
||||
|
||||
from .handler import XAIRealtime
|
||||
|
||||
__all__ = ["XAIRealtime"]
|
||||
@@ -0,0 +1,38 @@
|
||||
"""
|
||||
This file contains the handler for xAI's Grok Voice Agent API `/v1/realtime` endpoint.
|
||||
|
||||
xAI's Realtime API is fully OpenAI-compatible, so we inherit from OpenAIRealtime
|
||||
and only override the configuration differences.
|
||||
|
||||
This requires websockets, and is currently only supported on LiteLLM Proxy.
|
||||
"""
|
||||
|
||||
from litellm.constants import XAI_API_BASE
|
||||
|
||||
from ...openai.realtime.handler import OpenAIRealtime
|
||||
|
||||
|
||||
class XAIRealtime(OpenAIRealtime):
|
||||
"""
|
||||
Handler for xAI Grok Voice Agent API.
|
||||
|
||||
xAI's Realtime API uses the same WebSocket protocol as OpenAI but with:
|
||||
- Different endpoint: wss://api.x.ai/v1/realtime (via _get_default_api_base)
|
||||
- No OpenAI-Beta header required (via _get_additional_headers)
|
||||
- Model: grok-4-1-fast-non-reasoning
|
||||
|
||||
All WebSocket logic is inherited from OpenAIRealtime.
|
||||
"""
|
||||
|
||||
def _get_default_api_base(self) -> str:
|
||||
"""xAI uses a different API base URL."""
|
||||
return XAI_API_BASE
|
||||
|
||||
def _get_additional_headers(self, api_key: str) -> dict:
|
||||
"""
|
||||
xAI does NOT require the OpenAI-Beta header.
|
||||
Only send Authorization header.
|
||||
"""
|
||||
return {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
}
|
||||
@@ -2,6 +2,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union
|
||||
|
||||
import litellm
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.constants import XAI_API_BASE
|
||||
from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
|
||||
from litellm.secret_managers.main import get_secret_str
|
||||
from litellm.types.llms.openai import ResponsesAPIOptionalRequestParams
|
||||
@@ -16,8 +17,6 @@ if TYPE_CHECKING:
|
||||
else:
|
||||
LiteLLMLoggingObj = Any
|
||||
|
||||
XAI_API_BASE = "https://api.x.ai/v1"
|
||||
|
||||
|
||||
class XAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
|
||||
"""
|
||||
|
||||
@@ -19,11 +19,13 @@ from ..llms.azure.realtime.handler import AzureOpenAIRealtime
|
||||
from ..llms.bedrock.realtime.handler import BedrockRealtime
|
||||
from ..llms.custom_httpx.http_handler import get_shared_realtime_ssl_context
|
||||
from ..llms.openai.realtime.handler import OpenAIRealtime
|
||||
from ..llms.xai.realtime.handler import XAIRealtime
|
||||
from ..utils import client as wrapper_client
|
||||
|
||||
azure_realtime = AzureOpenAIRealtime()
|
||||
openai_realtime = OpenAIRealtime()
|
||||
bedrock_realtime = BedrockRealtime()
|
||||
xai_realtime = XAIRealtime()
|
||||
base_llm_http_handler = BaseLLMHTTPHandler()
|
||||
|
||||
|
||||
@@ -188,6 +190,30 @@ async def _arealtime(
|
||||
aws_bedrock_runtime_endpoint=aws_bedrock_runtime_endpoint,
|
||||
aws_external_id=aws_external_id,
|
||||
)
|
||||
elif _custom_llm_provider == "xai":
|
||||
api_base = (
|
||||
dynamic_api_base
|
||||
or litellm_params.api_base
|
||||
or get_secret_str("XAI_API_BASE")
|
||||
or "https://api.x.ai/v1"
|
||||
)
|
||||
# set API KEY
|
||||
api_key = (
|
||||
dynamic_api_key
|
||||
or litellm.api_key
|
||||
or get_secret_str("XAI_API_KEY")
|
||||
)
|
||||
|
||||
await xai_realtime.async_realtime(
|
||||
model=model,
|
||||
websocket=websocket,
|
||||
logging_obj=litellm_logging_obj,
|
||||
api_base=api_base,
|
||||
api_key=api_key,
|
||||
client=None,
|
||||
timeout=timeout,
|
||||
query_params=query_params,
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unsupported model: {model}")
|
||||
|
||||
@@ -230,6 +256,10 @@ async def _realtime_health_check(
|
||||
url = openai_realtime._construct_url(
|
||||
api_base=api_base or "https://api.openai.com/", query_params={"model": model}
|
||||
)
|
||||
elif custom_llm_provider == "xai":
|
||||
url = xai_realtime._construct_url(
|
||||
api_base=api_base or "https://api.x.ai/v1", query_params={"model": model}
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unsupported model: {model}")
|
||||
ssl_context = get_shared_realtime_ssl_context()
|
||||
|
||||
@@ -2183,7 +2183,8 @@
|
||||
"batches": false,
|
||||
"rerank": false,
|
||||
"a2a": true,
|
||||
"interactions": true
|
||||
"interactions": true,
|
||||
"realtime": true
|
||||
}
|
||||
},
|
||||
"xinference": {
|
||||
|
||||
@@ -0,0 +1,426 @@
|
||||
"""
|
||||
Base test class for LiteLLM Realtime API E2E tests.
|
||||
|
||||
Provides common test infrastructure for testing realtime WebSocket connections
|
||||
across different providers (OpenAI, xAI, etc.)
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Optional
|
||||
|
||||
import pytest
|
||||
import websockets
|
||||
|
||||
sys.path.insert(0, os.path.abspath("../../.."))
|
||||
|
||||
import litellm
|
||||
|
||||
|
||||
class RealTimeWebSocketClient:
|
||||
"""
|
||||
Mock WebSocket client for testing realtime connections.
|
||||
Captures messages sent from the backend and provides a simple interface
|
||||
for testing connection success.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.messages_sent = []
|
||||
self.messages_received = []
|
||||
self.received_initial_event = False
|
||||
self.connection_successful = False
|
||||
self.close_code = None
|
||||
self.close_reason = None
|
||||
# Required by realtime_streaming.py - import exceptions module
|
||||
from websockets import exceptions as websockets_exceptions
|
||||
self.exceptions = websockets_exceptions
|
||||
|
||||
async def accept(self):
|
||||
"""Accept the WebSocket connection"""
|
||||
pass
|
||||
|
||||
async def send_text(self, message):
|
||||
"""Receive message from backend and store it"""
|
||||
self.messages_sent.append(message)
|
||||
try:
|
||||
if isinstance(message, bytes):
|
||||
message_str = message.decode('utf-8')
|
||||
else:
|
||||
message_str = message
|
||||
|
||||
msg_data = json.loads(message_str)
|
||||
msg_type = msg_data.get('type', 'unknown')
|
||||
|
||||
# Pretty print API response
|
||||
print(f"\n{'='*80}")
|
||||
print(f"API RESPONSE #{len(self.messages_received) + 1} - Event: {msg_type}")
|
||||
print(f"{'='*80}")
|
||||
print(json.dumps(msg_data, indent=2, sort_keys=False))
|
||||
print(f"{'='*80}\n")
|
||||
|
||||
self.messages_received.append(msg_data)
|
||||
|
||||
# Check for initial connection event
|
||||
if not self.received_initial_event and self._is_initial_event(msg_type):
|
||||
self.received_initial_event = True
|
||||
self.connection_successful = True
|
||||
|
||||
except (json.JSONDecodeError, UnicodeDecodeError) as e:
|
||||
# Non-JSON messages are acceptable
|
||||
print(f"\n[Non-JSON message: {e}]")
|
||||
print(f"Raw content: {str(message)[:200]}\n")
|
||||
pass
|
||||
|
||||
def _is_initial_event(self, msg_type: str) -> bool:
|
||||
"""Check if message type is an initial connection event"""
|
||||
# OpenAI sends "session.created", xAI sends "conversation.created"
|
||||
return msg_type in ["session.created", "conversation.created"]
|
||||
|
||||
async def receive_text(self):
|
||||
"""
|
||||
Wait briefly for messages, then close connection.
|
||||
This allows the backend forwarding task to send messages.
|
||||
"""
|
||||
print(f"\nWaiting for connection to establish...")
|
||||
max_wait = 5.0
|
||||
check_interval = 0.1
|
||||
waited = 0.0
|
||||
|
||||
while waited < max_wait:
|
||||
if self.connection_successful:
|
||||
print(f"Connection successful after {waited:.1f}s\n")
|
||||
break
|
||||
await asyncio.sleep(check_interval)
|
||||
waited += check_interval
|
||||
|
||||
if not self.connection_successful:
|
||||
print(f"Warning: No initial event received after {max_wait}s\n")
|
||||
|
||||
# If we have a pending message to send, send it now
|
||||
if hasattr(self, '_pending_client_message') and self._pending_client_message:
|
||||
print(f"Sending client message to backend...\n")
|
||||
# This simulates receiving a message from the client that needs to be forwarded to backend
|
||||
# We return it as if it came from the client
|
||||
msg = self._pending_client_message
|
||||
self._pending_client_message = None
|
||||
return msg
|
||||
|
||||
# Close connection to end the test
|
||||
print(f"\n{'='*80}")
|
||||
print(f"TEST COMPLETE - Closing connection")
|
||||
print(f"Total messages received from API: {len(self.messages_received)}")
|
||||
print(f"{'='*80}\n")
|
||||
raise websockets.exceptions.ConnectionClosed(None, None)
|
||||
|
||||
def queue_client_message(self, message: str):
|
||||
"""Queue a message to be sent from 'client' to backend"""
|
||||
self._pending_client_message = message
|
||||
|
||||
async def close(self, code=1000, reason=""):
|
||||
"""Close the WebSocket"""
|
||||
self.close_code = code
|
||||
self.close_reason = reason
|
||||
|
||||
@property
|
||||
def headers(self):
|
||||
return {}
|
||||
|
||||
|
||||
class BaseRealtimeTest(ABC):
|
||||
"""
|
||||
Abstract base test class for realtime API tests.
|
||||
|
||||
Child classes must implement:
|
||||
- get_model(): Return the model name to test
|
||||
- get_api_key_env_var(): Return the environment variable name for the API key
|
||||
- get_initial_event_type(): Return the expected initial event type (e.g., "session.created")
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def get_model(self) -> str:
|
||||
"""Return the model name to test (e.g., 'gpt-4o-realtime-preview-2024-10-01')"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_api_key_env_var(self) -> str:
|
||||
"""Return the environment variable name for the API key (e.g., 'OPENAI_API_KEY')"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_initial_event_type(self) -> str:
|
||||
"""Return the expected initial event type (e.g., 'session.created' or 'conversation.created')"""
|
||||
pass
|
||||
|
||||
def get_skip_reason(self) -> str:
|
||||
"""Return the skip reason when API key is missing"""
|
||||
return f"No {self.get_api_key_env_var()} provided"
|
||||
|
||||
def should_skip(self) -> bool:
|
||||
"""Check if tests should be skipped due to missing API key"""
|
||||
return os.environ.get(self.get_api_key_env_var()) is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_realtime_connection(self):
|
||||
"""
|
||||
Test basic realtime WebSocket connection.
|
||||
Verifies that:
|
||||
1. Connection is established successfully
|
||||
2. Initial event is received
|
||||
3. Messages are properly forwarded
|
||||
"""
|
||||
litellm._turn_on_debug()
|
||||
if self.should_skip():
|
||||
pytest.skip(self.get_skip_reason())
|
||||
|
||||
websocket_client = RealTimeWebSocketClient()
|
||||
caught_exception = None
|
||||
|
||||
print(f"\n{'='*80}")
|
||||
print(f"STARTING REALTIME CONNECTION TEST")
|
||||
print(f"Model: {self.get_model()}")
|
||||
print(f"API Key Env Var: {self.get_api_key_env_var()}")
|
||||
print(f"{'='*80}\n")
|
||||
|
||||
try:
|
||||
await litellm._arealtime(
|
||||
model=self.get_model(),
|
||||
websocket=websocket_client,
|
||||
api_key=os.environ.get(self.get_api_key_env_var()),
|
||||
timeout=60
|
||||
)
|
||||
except websockets.exceptions.ConnectionClosed:
|
||||
pass
|
||||
except Exception as e:
|
||||
print(f"\nException: {type(e).__name__}: {e}\n")
|
||||
caught_exception = e
|
||||
|
||||
# Build debug info
|
||||
error_details = []
|
||||
error_details.append(f"messages_sent: {len(websocket_client.messages_sent)}")
|
||||
error_details.append(f"messages_received: {len(websocket_client.messages_received)}")
|
||||
error_details.append(f"close_code: {websocket_client.close_code}")
|
||||
error_details.append(f"close_reason: {websocket_client.close_reason}")
|
||||
if caught_exception:
|
||||
error_details.append(f"exception: {type(caught_exception).__name__}: {caught_exception}")
|
||||
|
||||
# Skip on transient connection failures
|
||||
if not websocket_client.connection_successful and websocket_client.close_code is not None:
|
||||
pytest.skip(f"Transient connection failure: {'; '.join(error_details)}")
|
||||
|
||||
# Assertions
|
||||
assert websocket_client.connection_successful, f"Failed to connect. Debug: {'; '.join(error_details)}"
|
||||
assert websocket_client.received_initial_event, f"Did not receive initial event"
|
||||
assert len(websocket_client.messages_received) > 0, "No messages received"
|
||||
|
||||
# Verify initial event
|
||||
initial_event = websocket_client.messages_received[0]
|
||||
assert initial_event["type"] == self.get_initial_event_type(), \
|
||||
f"Expected {self.get_initial_event_type()}, got {initial_event.get('type')}"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_realtime_with_query_params(self):
|
||||
"""
|
||||
Test realtime connection with explicit query parameters.
|
||||
Verifies that query params are properly passed to the backend.
|
||||
"""
|
||||
litellm._turn_on_debug()
|
||||
if self.should_skip():
|
||||
pytest.skip(self.get_skip_reason())
|
||||
|
||||
from litellm.types.realtime import RealtimeQueryParams
|
||||
|
||||
websocket_client = RealTimeWebSocketClient()
|
||||
caught_exception = None
|
||||
|
||||
# Strip provider prefix from model name for query params
|
||||
model_name = self.get_model()
|
||||
if "/" in model_name:
|
||||
model_name = model_name.split("/", 1)[1]
|
||||
|
||||
query_params: RealtimeQueryParams = {"model": model_name}
|
||||
|
||||
try:
|
||||
await litellm._arealtime(
|
||||
model=self.get_model(),
|
||||
websocket=websocket_client,
|
||||
api_key=os.environ.get(self.get_api_key_env_var()),
|
||||
query_params=query_params,
|
||||
timeout=60
|
||||
)
|
||||
except websockets.exceptions.ConnectionClosed:
|
||||
pass
|
||||
except Exception as e:
|
||||
caught_exception = e
|
||||
|
||||
# Build debug info
|
||||
error_details = []
|
||||
error_details.append(f"messages_sent: {len(websocket_client.messages_sent)}")
|
||||
error_details.append(f"messages_received: {len(websocket_client.messages_received)}")
|
||||
if caught_exception:
|
||||
error_details.append(f"exception: {type(caught_exception).__name__}: {caught_exception}")
|
||||
|
||||
# Skip on transient failures
|
||||
if not websocket_client.connection_successful and websocket_client.close_code is not None:
|
||||
pytest.skip(f"Transient connection failure: {'; '.join(error_details)}")
|
||||
|
||||
# Assertions
|
||||
assert websocket_client.connection_successful, f"Failed to connect. Debug: {'; '.join(error_details)}"
|
||||
assert len(websocket_client.messages_received) > 0, "No messages received"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_user_message(self):
|
||||
"""
|
||||
Test sending an actual user message and receiving responses.
|
||||
This creates a more realistic conversation flow.
|
||||
"""
|
||||
if self.should_skip():
|
||||
pytest.skip(self.get_skip_reason())
|
||||
|
||||
litellm._turn_on_debug()
|
||||
|
||||
# Create a custom websocket client that sends a message
|
||||
class InteractiveWebSocketClient(RealTimeWebSocketClient):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.sent_user_message = False
|
||||
self.response_messages = []
|
||||
self.wait_for_responses = True
|
||||
|
||||
async def receive_text(self):
|
||||
"""Enhanced receive that sends a user message after connection"""
|
||||
print(f"\n{'='*80}")
|
||||
print(f"CLIENT-SIDE RECEIVE HANDLER")
|
||||
print(f"{'='*80}\n")
|
||||
|
||||
# Wait for initial connection
|
||||
max_wait = 5.0
|
||||
check_interval = 0.1
|
||||
waited = 0.0
|
||||
|
||||
while waited < max_wait:
|
||||
if self.connection_successful:
|
||||
print(f"Connection established after {waited:.1f}s\n")
|
||||
break
|
||||
await asyncio.sleep(check_interval)
|
||||
waited += check_interval
|
||||
|
||||
# Step 1: Send a user message after connection is established
|
||||
if self.connection_successful and not self.sent_user_message:
|
||||
self.sent_user_message = True
|
||||
user_msg_data = {
|
||||
"type": "conversation.item.create",
|
||||
"item": {
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": [{"type": "input_text", "text": "Say hi back to me!"}]
|
||||
}
|
||||
}
|
||||
user_msg = json.dumps(user_msg_data)
|
||||
|
||||
print(f"\n{'='*80}")
|
||||
print(f"STEP 1: SENDING USER MESSAGE TO BACKEND")
|
||||
print(f"{'='*80}")
|
||||
print(json.dumps(user_msg_data, indent=2))
|
||||
print(f"{'='*80}\n")
|
||||
|
||||
return user_msg
|
||||
|
||||
# Step 2: Trigger the response after user message is acknowledged
|
||||
if not hasattr(self, 'triggered_response'):
|
||||
self.triggered_response = True
|
||||
# Wait a bit for the user message to be processed
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
response_create_data = {
|
||||
"type": "response.create"
|
||||
}
|
||||
response_create = json.dumps(response_create_data)
|
||||
|
||||
print(f"\n{'='*80}")
|
||||
print(f"STEP 2: TRIGGERING LLM RESPONSE")
|
||||
print(f"{'='*80}")
|
||||
print(json.dumps(response_create_data, indent=2))
|
||||
print(f"{'='*80}\n")
|
||||
|
||||
return response_create
|
||||
|
||||
# Step 3: Wait for LLM responses
|
||||
if self.wait_for_responses:
|
||||
print(f"\nSTEP 3: Waiting 5 seconds for LLM to respond...\n")
|
||||
await asyncio.sleep(5.0)
|
||||
self.wait_for_responses = False
|
||||
|
||||
# Collect response info
|
||||
for msg in self.messages_received:
|
||||
msg_type = msg.get('type', 'unknown')
|
||||
if msg_type not in ['conversation.created', 'ping']:
|
||||
self.response_messages.append(msg)
|
||||
|
||||
print(f"\nReceived {len(self.response_messages)} response messages (excluding init/ping)\n")
|
||||
|
||||
print(f"\n{'='*80}")
|
||||
print(f"CLOSING CONNECTION")
|
||||
print(f"Total messages received: {len(self.messages_received)}")
|
||||
print(f"{'='*80}\n")
|
||||
raise websockets.exceptions.ConnectionClosed(None, None)
|
||||
|
||||
websocket_client = InteractiveWebSocketClient()
|
||||
caught_exception = None
|
||||
|
||||
print(f"\n{'='*80}")
|
||||
print(f"STARTING INTERACTIVE MESSAGE TEST")
|
||||
print(f"Model: {self.get_model()}")
|
||||
print(f"Message: 'Say hi back to me!'")
|
||||
print(f"{'='*80}\n")
|
||||
|
||||
try:
|
||||
await litellm._arealtime(
|
||||
model=self.get_model(),
|
||||
websocket=websocket_client,
|
||||
api_key=os.environ.get(self.get_api_key_env_var()),
|
||||
timeout=60
|
||||
)
|
||||
except websockets.exceptions.ConnectionClosed:
|
||||
pass
|
||||
except Exception as e:
|
||||
print(f"\nException: {type(e).__name__}: {e}\n")
|
||||
caught_exception = e
|
||||
|
||||
# Print results
|
||||
print(f"\n{'='*80}")
|
||||
print(f"TEST RESULTS SUMMARY")
|
||||
print(f"{'='*80}")
|
||||
print(f"Connection successful: {websocket_client.connection_successful}")
|
||||
print(f"User message sent: {websocket_client.sent_user_message}")
|
||||
print(f"Total messages received: {len(websocket_client.messages_received)}")
|
||||
print(f"Response messages (excluding init/ping): {len(websocket_client.response_messages)}")
|
||||
|
||||
if websocket_client.response_messages:
|
||||
print(f"\nResponse Event Types:")
|
||||
for i, msg in enumerate(websocket_client.response_messages, 1):
|
||||
print(f" {i}. {msg.get('type', 'unknown')}")
|
||||
|
||||
print(f"{'='*80}\n")
|
||||
|
||||
# Skip if no responses (might be timing issue)
|
||||
if not websocket_client.response_messages:
|
||||
pytest.skip("No response messages received (might be timing/network issue)")
|
||||
|
||||
assert websocket_client.connection_successful, "Failed to establish connection"
|
||||
assert websocket_client.sent_user_message, "Failed to send user message"
|
||||
|
||||
def test_query_params_construction(self):
|
||||
"""Test that query params are constructed correctly"""
|
||||
from litellm.types.realtime import RealtimeQueryParams
|
||||
|
||||
# Strip provider prefix from model name
|
||||
model_name = self.get_model()
|
||||
if "/" in model_name:
|
||||
model_name = model_name.split("/", 1)[1]
|
||||
|
||||
query_params: RealtimeQueryParams = {"model": model_name}
|
||||
|
||||
assert "model" in query_params
|
||||
assert query_params["model"] == model_name
|
||||
@@ -0,0 +1,29 @@
|
||||
"""
|
||||
OpenAI Realtime API E2E Tests (using base class)
|
||||
|
||||
Tests OpenAI's Realtime API through LiteLLM's realtime interface.
|
||||
Uses the base test class to ensure consistent behavior across providers.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, os.path.abspath("../../.."))
|
||||
|
||||
from tests.llm_translation.realtime.base_realtime_tests import BaseRealtimeTest
|
||||
|
||||
|
||||
class TestOpenAIRealtime(BaseRealtimeTest):
|
||||
"""
|
||||
E2E tests for OpenAI Realtime API using base test class.
|
||||
"""
|
||||
|
||||
def get_model(self) -> str:
|
||||
return "gpt-4o-realtime-preview"
|
||||
|
||||
def get_api_key_env_var(self) -> str:
|
||||
return "OPENAI_API_KEY"
|
||||
|
||||
def get_initial_event_type(self) -> str:
|
||||
return "session.created"
|
||||
@@ -0,0 +1,34 @@
|
||||
"""
|
||||
xAI Realtime API E2E Tests
|
||||
|
||||
Tests xAI's Grok Voice Agent API through LiteLLM's realtime interface.
|
||||
Uses the base test class to ensure consistent behavior across providers.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, os.path.abspath("../../.."))
|
||||
|
||||
from tests.llm_translation.realtime.base_realtime_tests import BaseRealtimeTest
|
||||
|
||||
|
||||
class TestXAIRealtime(BaseRealtimeTest):
|
||||
"""
|
||||
E2E tests for xAI Realtime API.
|
||||
|
||||
xAI's Grok Voice Agent API is OpenAI-compatible but uses:
|
||||
- Different initial event: "conversation.created" instead of "session.created"
|
||||
- Different endpoint: wss://api.x.ai/v1/realtime
|
||||
- Model: grok-4-1-fast-non-reasoning
|
||||
"""
|
||||
|
||||
def get_model(self) -> str:
|
||||
return "xai/grok-4-1-fast-non-reasoning"
|
||||
|
||||
def get_api_key_env_var(self) -> str:
|
||||
return "XAI_API_KEY"
|
||||
|
||||
def get_initial_event_type(self) -> str:
|
||||
return "conversation.created"
|
||||
@@ -0,0 +1,82 @@
|
||||
# OpenAI Realtime Handler Tests
|
||||
|
||||
## Important Context: `additional_headers` vs `extra_headers`
|
||||
|
||||
### Background
|
||||
|
||||
There was confusion about the correct parameter name for passing headers to `websockets.connect()`. This README documents the resolution for future maintainers.
|
||||
|
||||
### Timeline of Changes
|
||||
|
||||
1. **Dec 5, 2025** - Changed `extra_headers` → `additional_headers` (commit `8db7f1b8e4`)
|
||||
2. **Dec 18, 2025** - Changed `extra_headers` → `additional_headers` again (PR #17950, commit `9f88d61d10`)
|
||||
3. **Jan 15, 2026** - Upgraded `websockets` from 13.1.0 → 15.0.1 (commit `a3cf178e24`, Issue #19089)
|
||||
|
||||
### The Issue & Resolution
|
||||
|
||||
**The `websockets` library changed its API between versions:**
|
||||
|
||||
- **websockets < 14.0**: Used `extra_headers` parameter ✅
|
||||
- **websockets >= 14.0**: Uses `additional_headers` parameter ✅
|
||||
|
||||
**LiteLLM uses websockets 15.0.1** (per requirements.txt), which requires `additional_headers`.
|
||||
|
||||
### Verification
|
||||
|
||||
You can verify the correct parameter name:
|
||||
|
||||
```bash
|
||||
poetry run python -c "import websockets; import inspect; print(inspect.signature(websockets.connect))"
|
||||
```
|
||||
|
||||
This shows: `additional_headers: 'HeadersLike | None' = None` for websockets 15.0.1.
|
||||
|
||||
### Current Implementation (Correct)
|
||||
|
||||
```python
|
||||
# ✅ Correct for websockets 15.0.1+
|
||||
await websockets.connect(url, additional_headers={
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"OpenAI-Beta": "realtime=v1"
|
||||
})
|
||||
```
|
||||
|
||||
### Impact
|
||||
|
||||
This is NOT just a test fix - this was a **critical bug** that affected all realtime APIs:
|
||||
- OpenAI realtime
|
||||
- Azure realtime
|
||||
- xAI realtime
|
||||
- Any pass-through realtime connections
|
||||
|
||||
Using `extra_headers` with websockets 15.0.1 resulted in:
|
||||
```
|
||||
TypeError: connect() got an unexpected keyword argument 'extra_headers'
|
||||
```
|
||||
|
||||
### For Future Maintainers
|
||||
|
||||
If you see test failures related to header parameters:
|
||||
|
||||
1. **Check installed websockets version:**
|
||||
```bash
|
||||
poetry run python -c "import websockets; print(websockets.__version__)"
|
||||
```
|
||||
|
||||
2. **Check requirements.txt** for the specified version
|
||||
|
||||
3. **Verify the correct parameter:**
|
||||
- websockets >= 14.0: use `additional_headers`
|
||||
- websockets < 14.0: use `extra_headers`
|
||||
|
||||
4. **Ensure consistency** across all files:
|
||||
- `litellm/llms/openai/realtime/handler.py`
|
||||
- `litellm/llms/azure/realtime/handler.py`
|
||||
- `litellm/llms/custom_httpx/llm_http_handler.py`
|
||||
- `litellm/realtime_api/main.py`
|
||||
- `litellm/proxy/pass_through_endpoints/pass_through_endpoints.py`
|
||||
|
||||
**Current Status (Feb 2026):**
|
||||
- ✅ websockets version: 15.0.1
|
||||
- ✅ Correct parameter: `additional_headers`
|
||||
- ✅ All handlers updated and working
|
||||
@@ -199,7 +199,9 @@ async def test_async_realtime_url_contains_model():
|
||||
additional_headers = called_kwargs["additional_headers"]
|
||||
assert additional_headers["Authorization"] == f"Bearer {api_key}"
|
||||
assert additional_headers["OpenAI-Beta"] == "realtime=v1"
|
||||
assert called_kwargs["ssl"] is shared_context
|
||||
# Verify SSL is configured (should be an SSLContext or True, not None or False)
|
||||
assert called_kwargs["ssl"] is not None
|
||||
assert called_kwargs["ssl"] is not False
|
||||
|
||||
mock_realtime_streaming.assert_called_once()
|
||||
mock_streaming_instance.bidirectional_forward.assert_awaited_once()
|
||||
@@ -259,7 +261,9 @@ async def test_async_realtime_uses_max_size_parameter():
|
||||
# Verify max_size is set (default None for unlimited, matching OpenAI's SDK)
|
||||
assert "max_size" in called_kwargs
|
||||
assert called_kwargs["max_size"] is None
|
||||
assert called_kwargs["ssl"] is shared_context
|
||||
# Verify SSL is configured (should be an SSLContext or True, not None or False)
|
||||
assert called_kwargs["ssl"] is not None
|
||||
assert called_kwargs["ssl"] is not False
|
||||
# Default should be None (unlimited) to match OpenAI's official agents SDK
|
||||
# https://github.com/openai/openai-agents-python/blob/cf1b933660e44fd37b4350c41febab8221801409/src/agents/realtime/openai_realtime.py#L235
|
||||
|
||||
|
||||
Reference in New Issue
Block a user