* [Fix] Containers API - Allow routing to regional endpoints (#19118)
* fix get_complete_url
* fix url resolution containers API
* TestContainerRegionalApiBase
* feat(proxy): add keepalive_timeout support for Gunicorn server
Add configurable keepalive timeout parameter for Gunicorn workers to
match existing Uvicorn functionality. This allows users to tune the
keep-alive connection timeout based on their deployment requirements.
Changes:
- Add keepalive_timeout parameter to _run_gunicorn_server method
- Configure Gunicorn's keepalive setting (defaults to 90s if not specified)
- Update --keepalive_timeout CLI help text to document both Uvicorn and Gunicorn behavior
- Pass keepalive_timeout from run_server to _run_gunicorn_server
Tests:
- Add test to verify keepalive_timeout flag is properly passed to Gunicorn
- Add test to verify default 90s timeout when flag is not specified
Co-Authored-By: lizhen921 <294474470@qq.com>
Signed-off-by: Kris Xia <xiajiayi0506@gmail.com>
---------
Signed-off-by: Kris Xia <xiajiayi0506@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: lizhen921 <294474470@qq.com>
* fix(vertex_ai): improve passthrough endpoint url parsing and construction (#17402)
* test(proxy): add test for vertex passthrough load balancing
Add a test that verifies _base_vertex_proxy_route uses
get_available_deployment for proper load balancing instead of
get_model_list. This ensures the correct deployment is selected
from the router and vertex credentials are properly fetched.
Also refactor the implementation to:
- Use get_available_deployment instead of get_model_list
- Add error handling for deployment retrieval
- Improve code structure with try-except block
* feat(proxy): add pass-through deployment filtering methods
Add dedicated methods to filter and select deployments for pass-through endpoints:
- Implement get_available_deployment_for_pass_through() to ensure only deployments with use_in_pass_through=True are considered
- Implement async_get_available_deployment_for_pass_through() for async operations
- Add _filter_pass_through_deployments() helper method to filter by use_in_pass_through flag
- Update vertex pass-through route to use the new dedicated method
This ensures pass-through endpoints respect the use_in_pass_through configuration and apply proper load balancing strategy only to configured deployments.
Add comprehensive tests to verify filtering and load balancing behavior.
