Commit Graph
2 Commits
Author SHA1 Message Date
michelligabrieleandGitHub 0dd64baa66 fix(caching): preserve prompt_tokens_details through embedding cache round-trip (#26653)
* fix(caching): preserve prompt_tokens_details through embedding cache round-trip

The embedding caching layer was dropping prompt_tokens_details (including
image_count) because CachedEmbedding had no field for usage metadata and
the cache retrieval code reconstructed Usage without it. This caused
inconsistent responses where the first call returned image_count but
cached responses did not, breaking cost tracking for multimodal embeddings.

Add prompt_tokens_details to CachedEmbedding, persist per-item details
during cache storage, aggregate them on retrieval, and merge them in
combine_usage() for partial cache hits.

* style: apply Black formatting to caching files

* fix(caching): address Greptile review — cyclic import, guarded construction, nested dict merge

Move PromptTokensDetailsWrapper to inline import to resolve CodeQL cyclic
import warning. Guard PromptTokensDetailsWrapper construction with
try/except to handle unexpected cached keys. Add recursive dict merging
in _merge_prompt_tokens_details for nested fields like
cache_creation_token_details.
2026-04-28 08:25:11 -07:00
Krish DholakiaandGitHub ba39f9e360 Helicone base url support + fix for embedding cache hits on str input (#11211)
* fix(helicone.py): add helicone api base support

Fixes https://github.com/BerriAI/litellm/issues/10825

* test: add unit test for cache hit response on embedding calls

* fix(caching_handler.py): fix handling cache hit on embedding when input is string

Fixes LIT-197

* docs(helicone_integration.md): document new helicone api base param
2025-05-28 22:02:55 -07:00