perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths (#28289)

* perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths - Introduce pure-text fast-path in `_build_complete_streaming_response` that collapses O(N) `content_block_delta` events into a single equivalent SSE event before conversion, eliminating per-output-token Pydantic `ModelResponseStream` construction; non-text streams (tool_use, thinking, citations) fall back to the unchanged legacy path - Skip agentic streaming wrapper entirely when no callback overrides `async_should_run_agentic_loop`; the wrapper buffered every chunk and rebuilt the SSE response only to call hooks that all return `(False, {})` — a pure no-op for the default config - Serialize request body once (`json.dumps`) for both the pre-call log input and the wire, instead of twice; avoids a full O(payload) scan per request, significant for long-context Claude Code histories - Add fast path in `async_streaming_data_generator` that bypasses the per-chunk `async_post_call_streaming_hook` coroutine await, response-string materialization, and cost-injection call when no callback/guardrail/cost-injection is active (the default config) - Resolve `_DD_STREAMING_TRACE_ENABLED` once at import time; eliminate per-chunk `NullSpan` context manager allocation when Datadog tracing is disabled (the default) - Memoize `get_type_hints(AnthropicMessagesRequestOptionalParams)` with `@lru_cache(maxsize=1)` — resolves once per process instead of once per `/v1/messages` request (~80µs each) - Hoist `cost_injection_active` out of the per-chunk loop in `chunk_processor`; eliminates repeated `getattr` + endpoint-type checks on every streamed byte chunk - Extract `_build_passthrough_logging_result` from `_route_streaming_logging_to_handler` as a standalone static method to facilitate future off-loop dispatch - Convert `async_sse_data_generator` from an `async for: yield` trampoline to a direct return of the underlying generator, removing one async-generator layer per streamed chunk - Skip redundant `strip_empty_text_blocks_from_anthropic_messages` scan in `anthropic_messages_handler` when the async wrapper already sanitized (signalled via `_litellm_messages_presanitized` sentinel, popped before reaching provider params) - Gate debug log `f-string` evaluation behind `isEnabledFor(DEBUG)` in both the streaming generator and the transformation layer to avoid serializing entire message payloads on every request at non-debug log levels - Add benchmark script (`scripts/benchmark_anthropic_messages_perf.py`) with a local mock Anthropic SSE provider for reproducible TTFT and TPM measurement across commits/branches - Add parity tests asserting fast-path and legacy-path produce byte-identical logged/billed payloads, plus unit tests for agentic hook detection, pre-serialized body reuse, and memoized key resolution * perf: address greptile review for anthropic streaming hot path - Bail to legacy in `_collapse_pure_text_chunks` when content_block_delta events from different block indexes are observed without an intervening flush. Anthropic sends blocks strictly sequentially, but defensive bail prevents silent text-merging if the protocol ever interleaves. - Replace leaf-class `__dict__` check for `async_post_call_streaming_hook` in `_callback_capabilities` with a function-identity comparison that walks the MRO. A vendor base class can carry the override and the registered class can add nothing else; before this PR the hook was unconditionally invoked, so an inherited-override miss would silently drop the hook on the streaming path. - Add unit tests for both behaviors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(mypy): narrow model_name to str in cost-injection branch The hoisted cost_injection_active flag in chunk_processor encodes the `bool(model_name)` requirement but mypy can't track that invariant through the local, so the per-chunk `_process_chunk_with_cost_injection( chunk, model_name)` calls flagged Optional[str] vs str. Pin a typed non-None local inside the cost-injection branch so mypy narrows correctly without changing runtime behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-08-03 20:23:46 +00:00 · 2026-05-23 12:15:59 -07:00
parent 3b2ce201d8
commit 2eab9ee2c0
15 changed files with 1978 additions and 91 deletions
@@ -10,6 +10,7 @@ from fastapi.responses import JSONResponse, StreamingResponse

 import litellm
 from litellm._uuid import uuid
+from litellm.integrations.custom_logger import CustomLogger
 from litellm.integrations.opentelemetry import UserAPIKeyAuth
 from litellm.proxy.common_request_processing import (
    ProxyBaseLLMRequestProcessing,
@@ -1316,8 +1317,17 @@ class TestCommonRequestProcessingHelpers:
            yield 'data: {"content": "chunk 3"}\n\n'
            yield "data: [DONE]\n\n"

-        # Patch the tracer in the common_request_processing module
-        with patch("litellm.proxy.common_request_processing.tracer", mock_tracer):
+        # Patch the tracer in the common_request_processing module. The
+        # per-chunk span is gated on _DD_STREAMING_TRACE_ENABLED (resolved at
+        # import from the real tracer, a NullTracer by default), so enable it
+        # explicitly to exercise the tracing path.
+        with (
+            patch("litellm.proxy.common_request_processing.tracer", mock_tracer),
+            patch(
+                "litellm.proxy.common_request_processing._DD_STREAMING_TRACE_ENABLED",
+                True,
+            ),
+        ):
            response = await create_response(mock_generator(), "text/event-stream", {})

            assert response.status_code == 200
@@ -1345,6 +1355,40 @@ class TestCommonRequestProcessingHelpers:
                    args[0] == "streaming.chunk.yield"
                ), f"Call {i} should have operation name 'streaming.chunk.yield', got {args[0]}"

+    async def test_create_streaming_response_skips_dd_trace_when_disabled(self):
+        """When DD tracing is disabled (the default), the per-chunk span
+        context manager is skipped entirely but all chunks still stream."""
+        from unittest.mock import patch
+
+        mock_tracer = MagicMock()
+
+        async def mock_generator():
+            yield 'data: {"content": "chunk 1"}\n\n'
+            yield 'data: {"content": "chunk 2"}\n\n'
+            yield "data: [DONE]\n\n"
+
+        with (
+            patch("litellm.proxy.common_request_processing.tracer", mock_tracer),
+            patch(
+                "litellm.proxy.common_request_processing._DD_STREAMING_TRACE_ENABLED",
+                False,
+            ),
+        ):
+            response = await create_response(mock_generator(), "text/event-stream", {})
+
+            assert response.status_code == 200
+
+            content = await self.consume_stream(response)
+
+            # All chunks stream through unchanged ...
+            assert content == [
+                'data: {"content": "chunk 1"}\n\n',
+                'data: {"content": "chunk 2"}\n\n',
+                "data: [DONE]\n\n",
+            ]
+            # ... but no per-chunk span was created.
+            assert mock_tracer.trace.call_count == 0
+
    async def test_create_streaming_response_dd_trace_with_error_chunk(self):
        """
        Test that when the first chunk contains an error, JSONResponse is returned
@@ -2199,3 +2243,77 @@ class TestHandleLLMApiExceptionDictDetail:
        proxy_exc = await self._invoke(exc)
        assert proxy_exc.message == "Content blocked by guardrail"
        assert proxy_exc.provider_specific_fields is None
+
+
+class TestAsyncStreamingDataGeneratorFastPath:
+    """Fast/slow path branching in async_streaming_data_generator."""
+
+    @staticmethod
+    async def _aiter(items):
+        for item in items:
+            yield item
+
+    @pytest.mark.asyncio
+    async def test_fast_path_skips_per_chunk_hook(self, monkeypatch):
+        """With no callbacks/guardrails/cost-injection, chunks pass through
+        unchanged and the per-chunk hook is NOT awaited."""
+        monkeypatch.setattr(litellm, "callbacks", [])
+        ProxyLogging._callback_capabilities_cache.clear()
+
+        proxy_logging_obj = ProxyLogging(user_api_key_cache=MagicMock())
+        hook_spy = AsyncMock(side_effect=lambda **kw: kw["response"])
+        monkeypatch.setattr(
+            proxy_logging_obj, "async_post_call_streaming_hook", hook_spy
+        )
+
+        chunks = [b"event: a\ndata: {}\n\n", b"event: b\ndata: {}\n\n"]
+        out = [
+            c
+            async for c in ProxyBaseLLMRequestProcessing.async_streaming_data_generator(
+                response=self._aiter(chunks),
+                user_api_key_dict=MagicMock(spec=UserAPIKeyAuth),
+                request_data={"model": "claude-x"},
+                proxy_logging_obj=proxy_logging_obj,
+                serialize_chunk=ProxyBaseLLMRequestProcessing.return_sse_chunk,
+                serialize_error=lambda e: "data: error\n\n",
+            )
+        ]
+
+        assert out == chunks  # bytes pass through return_sse_chunk untouched
+        hook_spy.assert_not_awaited()
+
+    @pytest.mark.asyncio
+    async def test_slow_path_runs_per_chunk_hook(self, monkeypatch):
+        """A callback that overrides async_post_call_streaming_hook forces the
+        slow path and the per-chunk hook is invoked."""
+
+        class _StreamingCb(CustomLogger):
+            async def async_post_call_streaming_hook(self, user_api_key_dict, response):
+                return response
+
+        cb = _StreamingCb()
+        monkeypatch.setattr(litellm, "callbacks", [cb])
+        ProxyLogging._callback_capabilities_cache.clear()
+
+        proxy_logging_obj = ProxyLogging(user_api_key_cache=MagicMock())
+        hook_spy = AsyncMock(side_effect=lambda **kw: kw["response"])
+        monkeypatch.setattr(
+            proxy_logging_obj, "async_post_call_streaming_hook", hook_spy
+        )
+
+        out = [
+            c
+            async for c in ProxyBaseLLMRequestProcessing.async_streaming_data_generator(
+                response=self._aiter([{"type": "message_stop"}]),
+                user_api_key_dict=MagicMock(spec=UserAPIKeyAuth),
+                request_data={"model": "claude-x"},
+                proxy_logging_obj=proxy_logging_obj,
+                serialize_chunk=ProxyBaseLLMRequestProcessing.return_sse_chunk,
+                serialize_error=lambda e: "data: error\n\n",
+            )
+        ]
+
+        assert len(out) == 1
+        hook_spy.assert_awaited_once()
+
+        ProxyLogging._callback_capabilities_cache.clear()