mirror of
https://github.com/tiennm99/litellm.git
synced 2026-07-15 00:22:34 +00:00
Jugal D. Bhatt
7c49197f29
* Vllm rerank (#12737) * Add Hosted VLLM rerank provider integration This commit implements the Hosted VLLM rerank provider integration for LiteLLM. The integration includes: Adding Hosted VLLM as a supported rerank provider in the main rerank function Implementing the HostedVLLMRerank handler class for making API requests Creating a transformation class to convert Hosted VLLM responses to LiteLLM's standardized format The integration supports both synchronous and asynchronous rerank operations. API credentials can be provided directly or through environment variables (HOSTED_VLLM_API_KEY and HOSTED_VLLM_API_BASE). Notable features: Proper error handling for missing credentials Standard response transformation Support for common rerank parameters (top_n, return_documents, etc.) Proper token usage tracking This expands LiteLLM's rerank provider ecosystem to include Hosted VLLM alongside existing providers like Cohere, Together AI, Azure AI, and Bedrock. * refactor(rerank): use base_llm_http_handler for hosted_vllm rerank - Replace custom HostedVLLMRerank handler with base_llm_http_handler - Implement proper HostedVLLMRerankConfig inheriting from BaseRerankConfig - Follow Cohere-compatible implementation pattern - Clean up unnecessary comments * Fix lint errors in hosted_vllm rerank transformer: remove unused imports * Fix linting errors in rerank transformation modules * fix: resolve type errors in Hosted VLLM rerank module --------- Co-authored-by: Philip D'Souza <philip.dsouza@macro4.com> Co-authored-by: Philip D'Souza <philip.a.dsouza@gmail.com> * added a few tests --------- Co-authored-by: Philip D'Souza <philip.dsouza@macro4.com> Co-authored-by: Philip D'Souza <philip.a.dsouza@gmail.com>
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/litellm
This folder can only run mock tests.