Files
litellm/tests/test_litellm
fangkangandGitHub dd34b09893 fix(bedrock): stop buffering streamed tool-call argument deltas (#30231)
* fix(bedrock): stop buffering streamed tool-call argument deltas

Two issues made Bedrock tool-use streaming arrive as a single end-of-stream
burst through LiteLLM while plain text streamed fine.

First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14
to null for bedrock and bedrock_converse, so the header was silently stripped.
Without that beta, Anthropic models on Bedrock buffer tool input server-side and
emit all toolUse.input deltas at once (verified against converse-stream and
invoke-with-response-stream directly). Bedrock accepts the beta via
additionalModelRequestFields.anthropic_beta, so it is now forwarded.

Second, the streaming reads re-chunked the AWS event stream with
iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte
blocks, so the small early events (messageStart, contentBlockStart, first
deltas) sat in the buffer until enough bytes accumulated, pushing
time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The
default is now no re-chunking; an explicit stream_chunk_size is still honored.

* test(bedrock): cover explicit stream_chunk_size on sync invoke path

* test(bedrock): cover stream_chunk_size plumbing through converse completion

* test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming

* test(bedrock): merge converse handler tests into existing mapped test file

pytest imports test modules by basename in non-package test dirs, so the new
tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with
the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and
broke collection in CI. Move the new tests into the existing file
2026-06-12 16:14:44 +05:30
..
2026-06-11 22:30:26 -07:00
2026-06-10 10:34:07 -07:00
2026-06-08 13:49:52 -07:00
2026-03-27 21:21:43 +05:30
2026-02-27 13:33:34 +05:30
2026-05-18 16:27:44 -07:00
2026-03-28 19:17:38 -07:00

Testing for litellm/

This directory 1:1 maps the the litellm/ directory, and can only contain mocked tests.

The point of this is to:

  1. Increase test coverage of litellm/
  2. Make it easy for contributors to add tests for the litellm/ package and easily run tests without needing LLM API keys.

File name conventions

  • litellm/proxy/test_caching_routes.py maps to litellm/proxy/caching_routes.py
  • test_<filename>.py maps to litellm/<filename>.py