- Extract helper methods in langsmith._prepare_log_data to reduce from 51 to <50 statements
- Extract helper methods in anthropic.transform_parsed_response to reduce from 57 to <50 statements
- Fixes PLR0915 linter errors
- All existing tests pass (10 langsmith tests, 126 anthropic tests)
Made-with: Cursor
Aligns proxy default with litellm.AZURE_DEFAULT_API_VERSION (2025-02-01-preview)
so Azure response_format + json_schema works without tools fallback.
Made-with: Cursor
Enable deployment_affinity, responses_api_deployment_check, and session_affinity to be configured per model group via router_settings.model_group_affinity_config, falling back to global settings for unconfigured groups.
- Add model_group_affinity_config parameter to Router and DeploymentAffinityCheck
- Add _get_effective_flags helper to resolve flags per model group
- Update async_filter_deployments and async_pre_call_deployment_hook to use per-group config
- Add 4 comprehensive tests covering per-group config, fallback, and override scenarios
This allows fine-grained control of affinity behavior across model groups, e.g., enabling stickiness only for cross-provider deployments while leaving other groups free to load-balance.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Previously the test called common_processing_pre_call_logic in isolation,
making generate_polling_id.assert_not_called() vacuously true. Now the test
calls responses_api() end-to-end so it actually verifies that a rate-limited
request never receives a polling ID.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Guard logging_obj for None when skip_pre_call_logic=True: raise ValueError
if litellm_logging_obj not in data, preventing AttributeError downstream
- Add model=None to common_processing_pre_call_logic call in endpoints.py
to match style of other call sites
- Add test verifying rate-limited request never receives polling ID
Move pre-call checks (rate limits, guardrails, budget) to run BEFORE
polling ID creation in the background streaming flow. This prevents the
edge case where a rate-limited request receives a polling ID that
immediately fails.
Changes:
- Add skip_pre_call_logic parameter to base_process_llm_request to allow
skipping pre-call checks (avoiding double-counting of RPM/parallel requests)
- Run common_processing_pre_call_logic before generating polling ID in the
responses API endpoint. If rate limits/guardrails fail, return error
immediately without creating a polling ID
- Background streaming task passes skip_pre_call_logic=True to avoid re-running
pre-call checks that were already done before polling ID creation
- Add tests verifying skip_pre_call_logic parameter works correctly
Fixes the edge case where polling_via_cache would return a polling ID
for a request that immediately fails due to rate limiting.
- Change chunk["id"] to chunk.get("id") for compatibility with MiniMax
- ModelResponseStream auto-generates id when None is passed
- Add regression test test_chunk_parser_without_id_field
Replace bare _get_deployment_default_tpm/rpm_limit calls in the
async_log_success_event condition with get_key_model_tpm/rpm_limit
(model_name=model_group). The higher-level getters short-circuit on
key/team metadata hits before ever reaching the router, so requests
that don't use deployment defaults incur no extra router lookup. Remove
the now-unused bare helper imports.
Also fix invalid `int = None` type hints in test helper signatures
to `Optional[int] = None`.
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
async_log_success_event only updated the per-model cache counter when
model_rpm_limit / model_tpm_limit were present in key metadata or
model_max_budget was set. For the new deployment-default path
(default_api_key_tpm_limit / default_api_key_rpm_limit), none of those
conditions held, so current_tpm stayed at zero and tpm enforcement was
never applied across multiple requests.
Extend the guard condition to also trigger when the model group has a
deployment-default tpm or rpm limit, and import the two helpers at
module level.
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
Compute get_key_model_tpm/rpm_limit once before the guard condition
instead of calling each function twice (once to check non-None, once to
retrieve). Removes 2 extra llm_router.get_model_list() calls per request
when deployment defaults are active.
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
- Use min() across all matching deployments instead of first-wins when
resolving default_api_key_tpm/rpm_limit for a model group, so
load-balanced setups with different per-deployment limits always apply
the most conservative value
- Replace the global SensitiveDataMasker non_sensitive_overrides change
with a targeted excluded_keys set at the remove_sensitive_info_from_deployment
call site, avoiding unintended suppression of other fields
- Update the v1 parallel request limiter to pass model_name to
get_key_model_tpm/rpm_limit so deployment defaults apply there too
- Add 4 tests covering multi-deployment min semantics
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
Helicone (PRs #19288, #22603) and Langfuse (#22390) were present in the
v1.82.0-stable...v1.82.3-stable diff but omitted from the AI Integrations
logging section. Also updates the AI Integrations diff summary count from 2 to 4.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds `default_api_key_tpm_limit` and `default_api_key_rpm_limit` to
`GenericLiteLLMParams` so operators can set per-deployment rate limit
defaults in config.yaml. When a key has no model-specific tpm/rpm limit
configured, the proxy falls back to these deployment defaults (Case 2 in
spec). Key-level limits always take priority (Case 1).
- Extends `get_key_model_tpm_limit` / `get_key_model_rpm_limit` with a
`model_name` param and a priority-4 deployment-default fallback
- Passes `model_name=requested_model` in the parallel request limiter so
the fallback is triggered at enforcement time
- Adds `"limit"` to `SensitiveDataMasker` non-sensitive overrides so
`*_limit` fields are not masked in `/model/info` responses
- Adds 17 unit tests covering both spec cases and the `/model/info` path
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
- Add 'Contributing to Guardrails' category with links to:
- Generic Guardrail API (integrate without PR)
- Adding a New Guardrail Integration tutorial
- Adding Guardrail Support to Endpoints
- Add 'Team Bring-Your-Own Guardrails' link for team BYOG workflow
These docs existed but were only accessible from the 'LiteLLM AI Gateway'
sidebar. Now they're also accessible when browsing the 'Guardrail Providers'
section.
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
- Only remove wildcard path from openai_routes when the route entry has
type="subpath", avoiding accidental removal when two endpoints share
the same base path but differ in include_subpath
- Clean up _registered_pass_through_routes in the test finally block to
prevent stale entries from polluting subsequent tests on failure
- Add dedup guard for base path registration (prevents unbounded list
growth on config reload)
- Clean up base path and wildcard path from openai_routes when an
endpoint is removed via remove_endpoint_routes
- Rewrite test to exercise initialize_pass_through_endpoints directly,
covering registration, dedup on reload, and cleanup on removal
When a pass-through endpoint has both auth=true and include_subpath=true,
non-admin users got 401 errors on subpath requests because only the base
path was registered in openai_routes. Now the wildcard path is also
registered so the auth check recognizes subpath requests as LLM API routes.
Also fixes pre-existing pyright error where logging_obj was possibly
unbound in the except block.
Add support for {"location": "tool_config"} in cache_control_injection_points,
which appends a cachePoint block to the Bedrock Converse toolConfig.tools array.
This enables prompt caching of tool definitions on Bedrock Claude models.
Also update the cache control hook to pass through non-message injection points
to provider-specific handling instead of silently dropping them.
Fixes#21969
OpenAI strict mode requires both additionalProperties:false AND all
property keys in required. Without required, OpenAI rejects the schema
even with additionalProperties:false set.
Enables Gemini 3+ models to combine built-in tools (Google Search, etc.)
with custom functions via `include_server_side_tool_invocations=True`.
Server-side invocations are surfaced in provider_specific_fields and
automatically re-injected on subsequent turns for multi-turn coherence.
Closes#24047
When translating Anthropic output_format to OpenAI response_format,
the adapter sets strict: true but didn't add additionalProperties: false,
which OpenAI requires at every object nesting level. This caused
BadRequestError for structured output requests routed to OpenAI models.
Fixes#20997
The check `content.get("thinking", None) is not None` incorrectly
drops thinking blocks when the `thinking` key is explicitly null or
absent. Changed to `content.get("type") == "thinking"` to match
the fix already applied in the experimental pass-through path (PR #15501).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add ExportOutlined icon next to nav items that link to external pages,
making it clear to users when a link opens in a new tab.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sameer Kankute