- Guard logging_obj for None when skip_pre_call_logic=True: raise ValueError
if litellm_logging_obj not in data, preventing AttributeError downstream
- Add model=None to common_processing_pre_call_logic call in endpoints.py
to match style of other call sites
- Add test verifying rate-limited request never receives polling ID
Move pre-call checks (rate limits, guardrails, budget) to run BEFORE
polling ID creation in the background streaming flow. This prevents the
edge case where a rate-limited request receives a polling ID that
immediately fails.
Changes:
- Add skip_pre_call_logic parameter to base_process_llm_request to allow
skipping pre-call checks (avoiding double-counting of RPM/parallel requests)
- Run common_processing_pre_call_logic before generating polling ID in the
responses API endpoint. If rate limits/guardrails fail, return error
immediately without creating a polling ID
- Background streaming task passes skip_pre_call_logic=True to avoid re-running
pre-call checks that were already done before polling ID creation
- Add tests verifying skip_pre_call_logic parameter works correctly
Fixes the edge case where polling_via_cache would return a polling ID
for a request that immediately fails due to rate limiting.
This follow-up to PR #16862 allows users to specify models that should use
the native provider's background mode instead of polling via cache.
Config example:
litellm_settings:
responses:
background_mode:
polling_via_cache: ["openai"]
native_background_mode: ["o4-mini-deep-research"]
ttl: 3600
When a model is in native_background_mode list, should_use_polling_for_request
returns False, allowing the request to fall through to native provider handling.
Committed-By-Agent: cursor
- Fix bug where model names without slash (e.g., 'gpt-5') couldn't
match providers in polling_via_cache list
- Look up model in llm_router.model_name_to_deployment_indices
- Check ALL deployments for matching provider (supports load balancing)
- Check custom_llm_provider first, then extract from model string
- Add comprehensive tests for provider resolution logic
Committed-By-Agent: cursor
- Create new background_streaming.py in response_polling/
- Update endpoints.py to import from new location
- Update __init__.py to export background_streaming_task
- Add tests for module imports and structure
Committed-By-Agent: cursor
* add GET responses endpoints on router
* add GET responses endpoints on router
* add GET responses endpoints on router
* add DELETE responses endpoints on proxy
* fixes for testing GET, DELETE endpoints
* test_basic_responses api e2e
Sameer Kankute