mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-18 14:17:29 +00:00
Julio Quinteros ProandClaude Sonnet 4.6
fd1237e9af
Two independent fixes for test_token_counter.py failures in CI: 1. test_disable_hf_tokenizer_download leaked litellm.disable_hf_tokenizer_download=True because pytest.MonkeyPatch() was never undone. The setting persisted into the alphabetically-subsequent test_llama2/3_tokenizer_api_failure tests, causing _select_tokenizer_helper to short-circuit before calling from_pretrained. Fix: wrap the test body in try/finally and call monkeypatch.undo(). 2. encode() returns a HuggingFace Encoding object when the HF tokenizer loads, but falls back to returning a plain List[int] (tiktoken) when the model hub is unreachable. test_encoding_and_decoding called .ids on the result, which raises AttributeError when the list-based fallback is active. Fix: normalize encode() to always return List[int] by extracting .ids when present, and remove the now-unnecessary .ids access in the test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>