Files
litellm/tests/logging_callback_tests/test_assemble_streaming_responses.py
T
Mateo Wang 2c733c00f5 chore(ci): modernize model references in tests and configs (#27856)
* test: modernize models used in CircleCI e2e test suites

Replaces obsolete models (gpt-4o, gpt-4o-mini, gpt-3.5-turbo,
claude-3-5-sonnet-20240620, claude-sonnet-4-20250514) with current
equivalents across the e2e_openai_endpoints and
proxy_e2e_anthropic_messages_tests CircleCI jobs.

- gpt-4o -> gpt-5.5 (responses API e2e tests)
- gpt-4o-mini -> gpt-5-mini (websocket responses, oai_misc_config)
- gpt-4o-mini-2024-07-18 -> gpt-4.1-mini-2025-04-14 (fine-tuning,
  still actively fine-tunable)
- gpt-4 / gpt-3.5-turbo target_model_names example -> gpt-5.5 /
  gpt-5-mini
- bedrock claude-3-5-sonnet-20240620 batch entry -> haiku-4-5-20251001
  (also aligning oai_misc_config model_name with what
  test_bedrock_batches_api.py actually requests)
- bedrock claude-sonnet-4-20250514 (deprecated, retires 2026-06-15)
  -> claude-sonnet-4-5-20250929

* test: point bedrock-claude-sonnet-4 alias at Sonnet 4.6, not 4.5

Greptile/Cursor flagged that after the previous commit, the
bedrock-claude-sonnet-4 alias collided with bedrock-claude-sonnet-4.5
(both pointed to claude-sonnet-4-5-20250929). Rename to
bedrock-claude-sonnet-4.6 and point it at the Sonnet 4.6 Bedrock ID
(us.anthropic.claude-sonnet-4-6, already in the litellm model
registry) so the alias name matches the underlying model version.

* test: modernize models across remaining CI-mounted configs & tests

Expands the modernization sweep to all CircleCI-mounted proxy configs
and to test directories where the model literal is a fixture/route key
(not the test's subject).

Config changes:
- proxy_server_config.yaml: bump gpt-3.5-turbo / gpt-3.5-turbo-1106 /
  gpt-4o / gemini-1.5-flash / dall-e-3 underlying models; rename
  gpt-3.5-turbo-end-user-test alias to gpt-5-mini-end-user-test; bump
  text-embedding-ada-002 underlying to text-embedding-3-small. User-
  facing aliases (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, etc.)
  preserved for backward compatibility with tests.
- simple_config.yaml, otel_test_config.yaml, spend_tracking_config.yaml:
  bump gpt-3.5-turbo underlying to gpt-5-mini.
- pass_through_config.yaml: claude-3-5-sonnet / claude-3-7-sonnet /
  claude-3-haiku entries replaced with claude-sonnet-4-5 / claude-
  haiku-4-5 / claude-opus-4-7.
- oai_misc_config.yaml: align alias name with the gpt-5-mini rename.

Test changes (proactive: claude-sonnet-4-20250514 / claude-opus-4-
20250514 retire 2026-06-15):
- tests/llm_translation/test_anthropic_completion.py: bump 3 references
  + paired Vertex AI ID to claude-sonnet-4-5.
- tests/llm_translation/test_optional_params.py: bump 2 references.
- tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py
  and test_bedrock_anthropic_messages_test.py: bump router fixtures
  using the deprecated model IDs.
- tests/pass_through_unit_tests/base_anthropic_messages_tool_search_test.py:
  modernize docstring examples.
- tests/test_end_users.py: update references to renamed alias.

* test: modernize placeholder model literals in router_unit_tests

Mass replace_all on fixture/placeholder model literals across the
router_unit_tests/ suite (model name is a routing key / label, not the
test subject). Sub-agent sweep so far — additional commits will follow
for logging_callback_tests/, enterprise/, top-level tests/test_*.py,
and other CI-mounted dirs.

Mappings applied:
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 / claude-3-opus-20240229 /
  claude-3-haiku-20240307 / claude-3-5-sonnet-20240620 ->
  claude-sonnet-4-5-20250929 / claude-opus-4-7 /
  claude-haiku-4-5-20251001 as appropriate

Explicitly preserved:
- gpt-4o-mini-* variants (transcribe, tts, etc.) where they're current
- gpt-4-turbo / gpt-4-vision-preview / gpt-4-0613 (subject literals)
- JSONL batch body literals
- Mock LLM response model fields (must match upstream)
- Fake/mock identifiers

* test: modernize placeholder model literals across remaining CI suites

Sub-agent sweep across logging_callback_tests/, guardrails_tests/,
enterprise/, pass_through_unit_tests/, otel_tests/,
llm_responses_api_testing/, batches_tests/, spend_tracking_tests/,
litellm_utils_tests/, unified_google_tests/, and a few top-level
tests/test_*.py files where the model literal is a fixture or
placeholder (router model_list, mock standard logging payload, mock
callback data) rather than the test's subject.

Mappings applied (see scope notes below):
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5.5 (corrected from initial gpt-5 — bare gpt-5
  is not a valid OpenAI alias; only gpt-5.5 / gpt-5.4 / gpt-5.2-codex
  / gpt-5-mini exist)
- gpt-4o-mini (bare) -> gpt-5-mini
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 -> claude-sonnet-4-5-20250929
- claude-3-opus-20240229 -> claude-opus-4-7
- claude-3-haiku-20240307 -> claude-haiku-4-5-20251001
- claude-3-5-sonnet-20240620/20241022 -> claude-sonnet-4-5-20250929
- claude-3-7-sonnet-20250219 -> claude-sonnet-4-6
- gemini-1.5-flash -> gemini-2.5-flash
- gemini-1.5-pro -> gemini-2.5-pro

Explicitly preserved (not modernized):
- llm_translation/ tests where model is the SUBJECT (provider-specific
  translation/transformation logic). Only the deprecated 20250514
  references were already bumped in a prior commit.
- Cost-calc / tokenizer subject tests in test_utils.py (skip-ranges
  documented by the sub-agent).
- Bedrock model IDs in test_health_check.py path-stripping tests.
- JSONL batch request bodies and mock LLM response bodies (must match
  upstream literal).
- Langfuse expected-request-body JSON fixtures (cost values are exact-
  match-asserted; changing the model would shift response_cost).
- gpt-3.5-turbo-instruct (text-completion endpoint; no modern OpenAI
  equivalent).
- Top-level tests calling the proxy through user-facing aliases
  (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, dall-e-3) — aliases
  in proxy_server_config.yaml stay; only the underlying model was
  bumped.
- tests/test_gpt5_azure_temperature_support.py (the test's whole point
  is model-name handling).
- Fake / mock / openai/fake identifiers.

Notable side fixes:
- test_spend_accuracy_tests.py: UPSTREAM_MODEL now matches what
  spend_tracking_config.yaml's proxy actually routes to (gpt-5-mini),
  resolving a latent inconsistency.
- proxy_server_config.yaml: bare `gpt-5` alias renamed to `gpt-5.5`
  (bare gpt-5 is not a valid OpenAI alias).
- test_batches_logging_unit_tests.py: explicit_models list entries
  kept distinct (gpt-5-mini + gpt-5.5) after bulk rename.

* test: fix CI failures from model modernization sweep

CI surfaced 4 categories of regression from the bulk modernization:

1. Azure deployment names are customer-specific. Reverted:
   - tests/litellm_utils_tests/test_health_check.py: azure/text-
     embedding-3-small -> azure/text-embedding-ada-002 (the CI Azure
     account does not have a text-embedding-3-small deployment).
   - tests/logging_callback_tests/test_custom_callback_router.py:
     same revert for two router fixtures driving aembedding.

2. gpt-5 family does not accept temperature != 1. Tests that pass a
   custom temperature swapped from gpt-5-mini to gpt-4.1-mini (modern
   non-reasoning OpenAI mini that still accepts temperature/logprobs):
   - tests/logging_callback_tests/test_datadog.py
   - tests/logging_callback_tests/test_langsmith_unit_test.py
   - tests/logging_callback_tests/test_otel_logging.py

3. proxy_server_config.yaml's gpt-3.5-turbo-large alias was routing to
   gpt-5.5 (a reasoning model that rejects logprobs). The proxy test
   tests/test_openai_endpoints.py::test_chat_completion_streaming
   exercises logprobs/top_logprobs through that alias. Bumped the
   underlying model to gpt-4.1 (non-reasoning, still modern).

4. tests/logging_callback_tests/test_gcs_pub_sub.py asserts against a
   pinned JSON fixture (gcs_pub_sub_body/spend_logs_payload.json) with
   hardcoded model="gpt-4o" and a model-specific spend value. Reverted
   the litellm.acompletion calls in the test to model="gpt-4o" so the
   fixture's exact-match assertions still hold.

5. tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py:
   anthropic.messages.create routing to openai/gpt-5-mini returned an
   empty content[0] with max_tokens=100 (reasoning-token consumption).
   Swapped to openai/gpt-4.1-mini.

* test: fix Assistants API model + 2 cursor[bot] review nits

1. pass_through_unit_tests/test_custom_logger_passthrough.py: gpt-5.5
   isn't accepted by the /v1/assistants endpoint
   ("unsupported_model"). Switch to gpt-4.1-mini (modern, Assistants-
   API-supported, non-reasoning).

2. example_config_yaml/pass_through_config.yaml: the previous sweep
   bumped the claude-3-7-sonnet alias to claude-opus-4-7, which is a
   tier change (Sonnet -> Opus). Map to claude-sonnet-4-6 to keep the
   Sonnet tier intact. (Cursor bugbot review.)

3. example_config_yaml/simple_config.yaml: model_name was left as
   gpt-3.5-turbo while the underlying was bumped to gpt-5-mini, which
   muddles the "simple" example. Make both sides gpt-5-mini so the
   most basic example is a straight 1:1 mapping again. (Cursor bugbot
   review.)

* fix: revert gpt-4/gpt-3.5-turbo alias underlying to non-reasoning models

tests/test_openai_endpoints.py::test_completion calls the proxy alias
"gpt-4" with temperature=0, and other tests call gpt-3.5-turbo with
custom temperature / logprobs / the legacy /v1/completions endpoint.
The earlier modernization mapped both aliases to gpt-5.5 / gpt-5-mini,
which are reasoning models that reject temperature != 1 and don't
expose /v1/completions. Map the aliases to gpt-4.1 / gpt-4.1-mini
(modern non-reasoning OpenAI models) instead — keeps user-facing
aliases preserved while picking a current underlying that still
supports the parameters/endpoints the tests exercise.
2026-05-15 15:44:28 -07:00

368 lines
12 KiB
Python

"""
Testing for _assemble_complete_response_from_streaming_chunks
- Test 1 - ModelResponse with 1 list of streaming chunks. Assert chunks are added to the streaming_chunks, after final chunk sent assert complete_streaming_response is not None
- Test 2 - TextCompletionResponse with 1 list of streaming chunks. Assert chunks are added to the streaming_chunks, after final chunk sent assert complete_streaming_response is not None
- Test 3 - Have multiple lists of streaming chunks, Assert that chunks are added to the correct list and that complete_streaming_response is None. After final chunk sent assert complete_streaming_response is not None
- Test 4 - build a complete response when 1 chunk is poorly formatted
"""
import json
import os
import sys
from datetime import datetime
from unittest.mock import AsyncMock
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
import httpx
import pytest
from respx import MockRouter
import litellm
from litellm import (
Choices,
Message,
ModelResponse,
ModelResponseStream,
TextCompletionResponse,
TextChoices,
)
from litellm.litellm_core_utils.logging_utils import (
_assemble_complete_response_from_streaming_chunks,
)
@pytest.mark.parametrize("is_async", [True, False])
def test_assemble_complete_response_from_streaming_chunks_1(is_async):
"""
Test 1 - ModelResponse with 1 list of streaming chunks. Assert chunks are added to the streaming_chunks, after final chunk sent assert complete_streaming_response is not None
"""
request_kwargs = {
"model": "test_model",
"messages": [{"role": "user", "content": "Hello, world!"}],
}
list_streaming_chunks = []
chunk = {
"id": "chatcmpl-9mWtyDnikZZoB75DyfUzWUxiiE2Pi",
"choices": [
litellm.utils.StreamingChoices(
delta=litellm.utils.Delta(
content="hello in response",
function_call=None,
role=None,
tool_calls=None,
),
index=0,
logprobs=None,
)
],
"created": 1721353246,
"model": "gpt-5-mini",
"object": "chat.completion.chunk",
"system_fingerprint": None,
"usage": None,
}
chunk = ModelResponseStream(**chunk)
complete_streaming_response = _assemble_complete_response_from_streaming_chunks(
result=chunk,
start_time=datetime.now(),
end_time=datetime.now(),
request_kwargs=request_kwargs,
streaming_chunks=list_streaming_chunks,
is_async=is_async,
)
# this is the 1st chunk - complete_streaming_response should be None
print("list_streaming_chunks", list_streaming_chunks)
print("complete_streaming_response", complete_streaming_response)
assert complete_streaming_response is None
assert len(list_streaming_chunks) == 1
assert list_streaming_chunks[0] == chunk
# Add final chunk
chunk = {
"id": "chatcmpl-9mWtyDnikZZoB75DyfUzWUxiiE2Pi",
"choices": [
litellm.utils.StreamingChoices(
finish_reason="stop",
delta=litellm.utils.Delta(
content="end of response",
function_call=None,
role=None,
tool_calls=None,
),
index=0,
logprobs=None,
)
],
"created": 1721353246,
"model": "gpt-5-mini",
"object": "chat.completion.chunk",
"system_fingerprint": None,
"usage": None,
}
chunk = ModelResponseStream(**chunk)
complete_streaming_response = _assemble_complete_response_from_streaming_chunks(
result=chunk,
start_time=datetime.now(),
end_time=datetime.now(),
request_kwargs=request_kwargs,
streaming_chunks=list_streaming_chunks,
is_async=is_async,
)
print("list_streaming_chunks", list_streaming_chunks)
print("complete_streaming_response", complete_streaming_response)
# this is the 2nd chunk - complete_streaming_response should not be None
assert complete_streaming_response is not None
assert len(list_streaming_chunks) == 2
assert isinstance(complete_streaming_response, ModelResponse)
assert isinstance(complete_streaming_response.choices[0], Choices)
pass
@pytest.mark.parametrize("is_async", [True, False])
def test_assemble_complete_response_from_streaming_chunks_2(is_async):
"""
Test 2 - TextCompletionResponse with 1 list of streaming chunks. Assert chunks are added to the streaming_chunks, after final chunk sent assert complete_streaming_response is not None
"""
from litellm.utils import TextCompletionStreamWrapper
_text_completion_stream_wrapper = TextCompletionStreamWrapper(
completion_stream=None, model="test_model"
)
request_kwargs = {
"model": "test_model",
"messages": [{"role": "user", "content": "Hello, world!"}],
}
list_streaming_chunks = []
chunk = {
"id": "chatcmpl-9mWtyDnikZZoB75DyfUzWUxiiE2Pi",
"choices": [
litellm.utils.StreamingChoices(
delta=litellm.utils.Delta(
content="hello in response",
function_call=None,
role=None,
tool_calls=None,
),
index=0,
logprobs=None,
)
],
"created": 1721353246,
"model": "gpt-5-mini",
"object": "chat.completion.chunk",
"system_fingerprint": None,
"usage": None,
}
chunk = ModelResponseStream(**chunk)
chunk = _text_completion_stream_wrapper.convert_to_text_completion_object(chunk)
complete_streaming_response = _assemble_complete_response_from_streaming_chunks(
result=chunk,
start_time=datetime.now(),
end_time=datetime.now(),
request_kwargs=request_kwargs,
streaming_chunks=list_streaming_chunks,
is_async=is_async,
)
# this is the 1st chunk - complete_streaming_response should be None
print("list_streaming_chunks", list_streaming_chunks)
print("complete_streaming_response", complete_streaming_response)
assert complete_streaming_response is None
assert len(list_streaming_chunks) == 1
assert list_streaming_chunks[0] == chunk
# Add final chunk
chunk = {
"id": "chatcmpl-9mWtyDnikZZoB75DyfUzWUxiiE2Pi",
"choices": [
litellm.utils.StreamingChoices(
finish_reason="stop",
delta=litellm.utils.Delta(
content="end of response",
function_call=None,
role=None,
tool_calls=None,
),
index=0,
logprobs=None,
)
],
"created": 1721353246,
"model": "gpt-5-mini",
"object": "chat.completion.chunk",
"system_fingerprint": None,
"usage": None,
}
chunk = ModelResponseStream(**chunk)
chunk = _text_completion_stream_wrapper.convert_to_text_completion_object(chunk)
complete_streaming_response = _assemble_complete_response_from_streaming_chunks(
result=chunk,
start_time=datetime.now(),
end_time=datetime.now(),
request_kwargs=request_kwargs,
streaming_chunks=list_streaming_chunks,
is_async=is_async,
)
print("list_streaming_chunks", list_streaming_chunks)
print("complete_streaming_response", complete_streaming_response)
# this is the 2nd chunk - complete_streaming_response should not be None
assert complete_streaming_response is not None
assert len(list_streaming_chunks) == 2
assert isinstance(complete_streaming_response, TextCompletionResponse)
assert isinstance(complete_streaming_response.choices[0], TextChoices)
pass
@pytest.mark.parametrize("is_async", [True, False])
def test_assemble_complete_response_from_streaming_chunks_3(is_async):
request_kwargs = {
"model": "test_model",
"messages": [{"role": "user", "content": "Hello, world!"}],
}
list_streaming_chunks_1 = []
list_streaming_chunks_2 = []
chunk = {
"id": "chatcmpl-9mWtyDnikZZoB75DyfUzWUxiiE2Pi",
"choices": [
litellm.utils.StreamingChoices(
delta=litellm.utils.Delta(
content="hello in response",
function_call=None,
role=None,
tool_calls=None,
),
index=0,
logprobs=None,
)
],
"created": 1721353246,
"model": "gpt-5-mini",
"object": "chat.completion.chunk",
"system_fingerprint": None,
"usage": None,
}
chunk = ModelResponseStream(**chunk)
complete_streaming_response = _assemble_complete_response_from_streaming_chunks(
result=chunk,
start_time=datetime.now(),
end_time=datetime.now(),
request_kwargs=request_kwargs,
streaming_chunks=list_streaming_chunks_1,
is_async=is_async,
)
# this is the 1st chunk - complete_streaming_response should be None
print("list_streaming_chunks_1", list_streaming_chunks_1)
print("complete_streaming_response", complete_streaming_response)
assert complete_streaming_response is None
assert len(list_streaming_chunks_1) == 1
assert list_streaming_chunks_1[0] == chunk
assert len(list_streaming_chunks_2) == 0
# now add a chunk to the 2nd list
complete_streaming_response = _assemble_complete_response_from_streaming_chunks(
result=chunk,
start_time=datetime.now(),
end_time=datetime.now(),
request_kwargs=request_kwargs,
streaming_chunks=list_streaming_chunks_2,
is_async=is_async,
)
print("list_streaming_chunks_2", list_streaming_chunks_2)
print("complete_streaming_response", complete_streaming_response)
assert complete_streaming_response is None
assert len(list_streaming_chunks_2) == 1
assert list_streaming_chunks_2[0] == chunk
assert len(list_streaming_chunks_1) == 1
# now add a chunk to the 1st list
@pytest.mark.parametrize("is_async", [True, False])
def test_assemble_complete_response_from_streaming_chunks_4(is_async):
"""
Test 4 - build a complete response when 1 chunk is poorly formatted
- Assert complete_streaming_response is None
- Assert list_streaming_chunks is not empty
"""
request_kwargs = {
"model": "test_model",
"messages": [{"role": "user", "content": "Hello, world!"}],
}
list_streaming_chunks = []
chunk = {
"id": "chatcmpl-9mWtyDnikZZoB75DyfUzWUxiiE2Pi",
"choices": [
litellm.utils.StreamingChoices(
finish_reason="stop",
delta=litellm.utils.Delta(
content="end of response",
function_call=None,
role=None,
tool_calls=None,
),
index=0,
logprobs=None,
)
],
"created": 1721353246,
"model": "gpt-5-mini",
"object": "chat.completion.chunk",
"system_fingerprint": None,
"usage": None,
}
chunk = ModelResponseStream(**chunk)
# remove attribute id from chunk
del chunk.object
complete_streaming_response = _assemble_complete_response_from_streaming_chunks(
result=chunk,
start_time=datetime.now(),
end_time=datetime.now(),
request_kwargs=request_kwargs,
streaming_chunks=list_streaming_chunks,
is_async=is_async,
)
print("complete_streaming_response", complete_streaming_response)
assert complete_streaming_response is None
print("list_streaming_chunks", list_streaming_chunks)
assert len(list_streaming_chunks) == 1