- Vertex AI batch cost tests: replace removed gemini-1.5-flash-001 model
with gemini-2.0-flash-001 in pricing lookups
- MCP test_executes_tool_when_allowed: add server_id and auth_type attrs
to StubServer to match new _resolve_allowed_mcp_servers_with_ip_filter
- MCP M2M tests: infer oauth2_flow='client_credentials' in
_execute_with_mcp_client when client_id/client_secret/token_url present
(NewMCPServerRequest lacks oauth2_flow field)
- Team list test: update mock find_many to filter by team_id per the
current per-team query pattern in list_team
- Azure DALL-E 3 health check: skip test due to 410 ModelDeprecated
Co-authored-by: yuneng-jiang <yuneng-jiang@users.noreply.github.com>
Keep both sets of tests: upstream's OAuth2 token injection test and
our case-insensitive tool matching tests. Use upstream's version of
the bedrock output_config test (more comprehensive).
- Mock litellm.completion_cost in test_pass_through_success_handler_gemini_routing
to decouple it from model_prices_and_context_window.json; prevents the same
breakage if gemini-2.0-flash is ever removed from the pricing map
- Revert _create_passthrough_logging_payload URL back to gemini-1.5-flash to
eliminate inconsistency with the other tests that use gemini-1.5-flash explicitly
Fixes issue where multipart file uploads through passthrough endpoints failed with RequestValidationError. The proxy was consuming the request body stream and FastAPI was trying to parse multipart bodies as JSON dicts.
Changes:
- Try JSON parsing first for multipart content-type (handles misconfigured clients)
- Skip multipart parsing if JSON succeeds to avoid stream consumption
- Remove custom_body parameter from endpoint_func to prevent FastAPI auto-parsing
- Check for parsed body before using multipart handler
- Add regression test for multipart boundary preservation
Handles both actual multipart uploads and JSON bodies with incorrect multipart content-type headers.
Made-with: Cursor
mapped passthrough routes (vertex_ai, bedrock, etc) were compared
against the raw request path without prepending SERVER_ROOT_PATH.
db-registered routes already used _build_full_path_with_root for this
but the mapped routes branch was missed.
fixes#22272
test_vertex_passthrough_with_default_credentials and
test_view_spend_logs_with_date_range_summarized fail intermittently when a
prior xdist worker sets master_key — auth then rejects the unauthenticated
test requests before the code under test is reached.
- mock user_api_key_auth in test_vertex_passthrough_with_default_credentials
(same pattern used for test_vertex_passthrough_with_no_default_credentials
in #21810)
- wrap test_view_spend_logs_with_date_range_summarized in
app.dependency_overrides[ps.user_api_key_auth] with try/finally cleanup
(same pattern used for the other spend log tests in #21810)
* fix(tests): add app.dependency_overrides for auth in spend logs tests
test_ui_view_spend_logs_with_status, test_ui_view_spend_logs_with_model,
test_ui_view_spend_logs_with_model_id, and test_view_spend_logs_summarize_parameter
all send Bearer sk-test without mocking user_api_key_auth. When a prior test
in the same xdist worker sets master_key, the auth check fails for sk-test
and the test fails intermittently.
Fix: use app.dependency_overrides[ps.user_api_key_auth] to bypass auth,
same pattern as other tests in the same file.
* fix(tests): mock user_api_key_auth in test_vertex_passthrough_with_no_default_credentials
vertex_proxy_route calls user_api_key_auth internally. When a prior test in the
same xdist worker sets master_key, the auth check fails for the test request
and create_pass_through_route is never called, causing assert_called_once_with to fail.
Fix: patch user_api_key_auth as an AsyncMock in the with mock.patch() block.
* fix: add custom_body parameter to endpoint_func in create_pass_through_route
The bedrock_proxy_route calls `endpoint_func(custom_body=data)` to
pass a pre-parsed, SigV4-signed request body. However, the
`endpoint_func` closure created by `create_pass_through_route` does
not accept a `custom_body` keyword argument, causing:
TypeError: endpoint_func() got an unexpected keyword argument 'custom_body'
Add `custom_body: Optional[dict] = None` to both `endpoint_func`
definitions (adapter-based and URL-based). In the URL-based path,
when `custom_body` is provided by the caller, use it instead of
re-parsing the body from the raw request.
Fixes#16999
* Add tests for custom_body handling in create_pass_through_route
Address reviewer feedback on PR #20849:
- Document why the adapter-based endpoint_func accepts custom_body
for signature compatibility but does not forward it (the underlying
chat_completion_pass_through_endpoint does not support it).
- Add test_create_pass_through_route_custom_body_url_target: verifies
that when a caller (e.g. bedrock_proxy_route) supplies custom_body,
it takes precedence over the body parsed from the raw request.
- Add test_create_pass_through_route_no_custom_body_falls_back:
verifies that the default path (no custom_body) correctly uses the
request-parsed body, preserving existing behavior.
Both tests are fully mocked following the project's CONTRIBUTING.md
guidelines and the patterns established in the existing test file.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: themavik <themavik@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(vertex_ai): replace custom model names with actual Vertex AI model names in passthrough URLs (#19948)
When the passthrough URL already contains project and location, the code
was skipping the deployment lookup and forwarding the URL as-is to Vertex AI.
For custom model names like gcp/google/gemini-2.5-flash, Vertex AI returned
404 because it only knows the actual model name (gemini-2.5-flash).
The fix makes the deployment lookup always run, so the custom model name
gets replaced with the actual Vertex AI model name before forwarding.
* add _resolve_vertex_model_from_router
* fix: get_llm_provider
* Potential fix for code scanning alert no. 4020: Clear-text logging of sensitive information
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
---------
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix(vertex_ai): improve passthrough endpoint url parsing and construction (#17402)
* test(proxy): add test for vertex passthrough load balancing
Add a test that verifies _base_vertex_proxy_route uses
get_available_deployment for proper load balancing instead of
get_model_list. This ensures the correct deployment is selected
from the router and vertex credentials are properly fetched.
Also refactor the implementation to:
- Use get_available_deployment instead of get_model_list
- Add error handling for deployment retrieval
- Improve code structure with try-except block
* feat(proxy): add pass-through deployment filtering methods
Add dedicated methods to filter and select deployments for pass-through endpoints:
- Implement get_available_deployment_for_pass_through() to ensure only deployments with use_in_pass_through=True are considered
- Implement async_get_available_deployment_for_pass_through() for async operations
- Add _filter_pass_through_deployments() helper method to filter by use_in_pass_through flag
- Update vertex pass-through route to use the new dedicated method
This ensures pass-through endpoints respect the use_in_pass_through configuration and apply proper load balancing strategy only to configured deployments.
Add comprehensive tests to verify filtering and load balancing behavior.
Fixes#17477
Guardrails couldn't access request headers (like User-Agent) on Bedrock
pass-through endpoints because headers were only stored in
data["proxy_server_request"]["headers"] but not in data["metadata"]["headers"]
where guardrails typically look for them.
This fix adds headers to metadata in add_litellm_data_to_request() so
guardrails can access User-Agent, API keys, and other header-based checks
on all endpoints including Bedrock pass-through.
Test added to verify headers are available in metadata for guardrails.
extract model id from vertex ai passthrough routes that follow the pattern:
/vertex_ai/*/models/{model_id}:*
the model extraction now handles vertex ai routes by regex matching the model
segment from the url path, which allows proper model identification for
authentication and authorization in proxy pass-through endpoints.
adds comprehensive test coverage for vertex ai model extraction including:
- various vertex api versions (v1, v1beta1)
- different locations (us-central1, asia-southeast1)
- model names with special suffixes (gemini-1.5-pro, gemini-2.0-flash)
- precedence verification (request body model over url)
- non-vertex route isolation
* fix(unified_guardrails.py): send all chunks on completion of final stream
* feat(generic_guardrail_api.py): handle tool call response on streaming LLM responses
* fix(anthropic/chat/guardrail_translation): initial commit adding anthropic tool response streaming guardrails
enables guardrail checks on tool response from llm's to work via `/v1/messages`
* feat(anthropic/): working guardrail checks on tool response from LLMs
ensures guardrail checks on anthropic /v1/messages works as expected
* feat(responses/guardrail_translation): support tool call response guardrails on streaming for /v1/responses
ensures complete coverage of tool call responses
* refactor(openai.py): refactor to use consistent pydantic model for responses api tool response on streaming
enables non-openai model tool call response to work correctly with guardrail checks on /v1/responses
* test: update tests
* fix: fix linting error
* fix: fix failing tests
* fix: fix import errors
* fix(openai/chat/guardrail_transformation): fix final chunk returned on streaming
* refactor: remove api-key conversion logic for Azure Anthropic
Co-authored-by: Erdem Halil <erdemhalil@users.noreply.github.com>
* fix(passthrough): pass custom_llm_provider to completion_cost for Azure AI Anthropic
The passthrough logging for Anthropic was failing when using Azure AI Anthropic
because the completion_cost function was not receiving the custom_llm_provider
parameter, causing it to fail with "LLM Provider NOT provided" error.
This fix:
- Retrieves custom_llm_provider from logging_obj.model_call_details
- Prepends provider prefix to model name for cost calculation
- Passes both formatted model and custom_llm_provider to completion_cost
- Centralizes provider prefix logic in _create_anthropic_response_logging_payload
This ensures cost calculation works correctly for Azure AI Anthropic requests
with models like azure_ai/claude-sonnet-4-5_gb_20250929.
Co-authored-by: Erdem Halil <erdemhalil@users.noreply.github.com>
* test: add unit tests for Azure AI Anthropic fixes
- Add tests for custom_llm_provider cost calculation in passthrough logging
- Add tests for ProviderConfigManager returning AzureAnthropicMessagesConfig
- Update existing tests to reflect removal of api-key to x-api-key conversion
Co-authored-by: Erdem Halil <erdemhalil@users.noreply.github.com>
---------
Co-authored-by: Erdem Halil <erdemhalil@users.noreply.github.com>
* feat(llm_passthrough_endpoints.py): support milvus passthrough api
* fix(llm_passthrough_endpoints.py): move streaming request value to the top of the function
* docs: document new milvus vector store passthrough flow
* feat(milvus/): initial commit adding milvus vector store support to LiteLLM
allows querying milvus vector store through litellm
* feat(bedrock/vector_stores): support translating openai filters param to aws kb
adds filtering to aws kb
* feat(milvus/): add milvus vector store unified search support
allows calling milvus vector store in through chat completions
* docs(milvus_vector_stores.md): document new milvus vector search integration
* feat(pass_through_endpoints.py): support passing form data through to a passthrough endpoint
Closes LIT-1147
* fix: fix linting errors
Cursor Agent