* test_should_not_fail_with_forwarded_headers_bedrock_invoke_messages
* use common get_request_headers for BaseAWS
* fix get_request_headers
* test_should_not_fail_with_forwarded_headers_bedrock_invoke_messages
Tools with no parameters (like EnterPlanMode from Anthropic Agents SDK)
send schemas with only $schema and no type field. Gemini rejects these
with "functionDeclaration parameters schema should be of type OBJECT".
Adds type: object when schema has no type and no anyOf/oneOf/allOf.
Extract and preserve provider-specific headers (llm_provider-*) when
handling error responses from LLM providers. This ensures that useful
debugging information from providers is available even when requests
fail with BadRequestError or similar exceptions.
* fix(vertex_ai): improve passthrough endpoint url parsing and construction (#17402)
* test(proxy): add test for vertex passthrough load balancing
Add a test that verifies _base_vertex_proxy_route uses
get_available_deployment for proper load balancing instead of
get_model_list. This ensures the correct deployment is selected
from the router and vertex credentials are properly fetched.
Also refactor the implementation to:
- Use get_available_deployment instead of get_model_list
- Add error handling for deployment retrieval
- Improve code structure with try-except block
* feat(proxy): add pass-through deployment filtering methods
Add dedicated methods to filter and select deployments for pass-through endpoints:
- Implement get_available_deployment_for_pass_through() to ensure only deployments with use_in_pass_through=True are considered
- Implement async_get_available_deployment_for_pass_through() for async operations
- Add _filter_pass_through_deployments() helper method to filter by use_in_pass_through flag
- Update vertex pass-through route to use the new dedicated method
This ensures pass-through endpoints respect the use_in_pass_through configuration and apply proper load balancing strategy only to configured deployments.
Add comprehensive tests to verify filtering and load balancing behavior.
* Optimize _get_model_cost_key to avoid expensive scans
- Remove expensive O(n) scan fallback that was causing 42.87% CPU overhead
- Only scan when size mismatch detected (O(1) check)
- Add warning in docstring: Only O(1) lookup operations are acceptable
- Clean up comments to be more concise
- Keep stale entry rebuild for pop() case (only triggers when stale entry found)
This fixes the performance issue where the scan was being triggered on every
failed lookup, causing severe CPU overhead during router operations.
* Add code quality check to enforce O(1) operations in _get_model_cost_key
- Add check_get_model_cost_key_performance.py to statically analyze _get_model_cost_key
- Detects O(n) operations (loops, comprehensions, problematic function calls)
- Recursively checks called functions to find nested O(n) operations
- Allows conditional O(n) rebuilds in helper functions (_rebuild_model_cost_lowercase_map, _handle_stale_map_entry_rebuild, _handle_new_key_with_scan)
* Integrate _get_model_cost_key performance check into CI pipeline
- Add check_get_model_cost_key_performance.py to check_code_and_doc_quality job
- Ensures O(1) requirement is enforced in CI to prevent performance regressions
* Remove unused performance test and clean up utils.py
- Remove test_get_model_info_performance.py (no longer needed)
- Remove extra blank line in utils.py
* Document allowed helper functions and exception process in _get_model_cost_key
- Add documentation listing allowed helper functions with O(n) operations
- Explain why these are acceptable (conditionally called)
- Add instructions for adding new exceptions to check_get_model_cost_key_performance.py
* Fix docstring detection and type checker error in performance check
- Add proper docstring tracking to skip docstring content (fixes false positive for 'map' in docstring)
- Add None check for docstring_quote to fix type checker error
- Restore _handle_new_key_with_scan to allowed_helpers list
* Remove check_get_model_cost_key_performance from CI pipeline
- Temporarily remove the performance check from CI to avoid blocking builds
* Restore performance check and remove memory leak tests from CI
- Add back check_get_model_cost_key_performance.py to CI pipeline
- Remove memory_leak_tests job that was causing port conflicts
* Remove extra blank line in CI config
* do not fallback to token counter if disable_token_counter is enabled, and return errors instead
* add exceptions and exception utils to map the same as /v1/chat/completions
* use safe_json_loads
* feat(gemini): add opt-in support for responseJsonSchema
Add support for Gemini's native responseJsonSchema parameter which uses
standard JSON Schema format instead of OpenAPI-style responseSchema.
Benefits of responseJsonSchema (Gemini 2.0+ only):
- Standard JSON Schema format (lowercase types)
- Supports additionalProperties for stricter validation
- Better compatibility with Pydantic's model_json_schema()
- No propertyOrdering required
Usage:
```python
response_format={
"type": "json_schema",
"json_schema": {"schema": {...}},
"use_json_schema": True # opt-in
}
```
This is backwards compatible - existing code continues to use
responseSchema by default.
Closes#16340
* docs: add documentation for use_json_schema parameter
Document the new use_json_schema option for Gemini 2.0+ models
in the JSON Mode documentation.
* refactor(gemini): use responseJsonSchema by default for Gemini 2.0+
Remove opt-in flag `use_json_schema` and automatically detect model version:
- Gemini 2.0+: uses responseJsonSchema (standard JSON Schema, supports additionalProperties)
- Gemini 1.5: uses responseSchema (OpenAPI format, legacy)
This follows LiteLLM's philosophy of abstracting provider differences -
users write the same code regardless of model version.
When extract_and_raise_litellm_exception tries to raise a LiteLLM exception
from an error string, it was always passing the response parameter. However,
some exceptions like APIConnectionError don't accept this parameter, causing
a TypeError.
This fix tries to raise the exception with the response parameter first,
and falls back to raising without it if a TypeError occurs.
This fixes the error:
TypeError: APIConnectionError.__init__() got an unexpected keyword argument 'response'
Which was occurring when Gemini returned UNEXPECTED_TOOL_CALL finish reason
and LiteLLM tried to convert the error to an APIConnectionError.
Fixes: cascading error when Gemini uses thinking feature (__thought__ tool calls)
