* feat(gemini): add opt-in support for responseJsonSchema
Add support for Gemini's native responseJsonSchema parameter which uses
standard JSON Schema format instead of OpenAPI-style responseSchema.
Benefits of responseJsonSchema (Gemini 2.0+ only):
- Standard JSON Schema format (lowercase types)
- Supports additionalProperties for stricter validation
- Better compatibility with Pydantic's model_json_schema()
- No propertyOrdering required
Usage:
```python
response_format={
"type": "json_schema",
"json_schema": {"schema": {...}},
"use_json_schema": True # opt-in
}
```
This is backwards compatible - existing code continues to use
responseSchema by default.
Closes#16340
* docs: add documentation for use_json_schema parameter
Document the new use_json_schema option for Gemini 2.0+ models
in the JSON Mode documentation.
* refactor(gemini): use responseJsonSchema by default for Gemini 2.0+
Remove opt-in flag `use_json_schema` and automatically detect model version:
- Gemini 2.0+: uses responseJsonSchema (standard JSON Schema, supports additionalProperties)
- Gemini 1.5: uses responseSchema (OpenAPI format, legacy)
This follows LiteLLM's philosophy of abstracting provider differences -
users write the same code regardless of model version.
* test(vertex): update json_schema tests to accept both responseSchema formats
Gemini 2.x+ uses responseJsonSchema while Gemini 1.x uses responseSchema.
Update tests to accept both formats since litellm now auto-selects based
on model version.
* fix: use fastuuid helper across the codebase
First batch of changes, simple drop in replacement.
* second batch of changes
* fixed: script mistake on helper file
* add support for vertex AI QWEN API
* streaming QWEN API support
* test_partner_models_httpx
* test_partner_models_httpx_streaming
* add cost tracking for vertex_ai/qwen/qwen3-235b-a22b-instruct-2507-maa
* docs qwen models vertexAI
* feat(proxy/utils.py): track pre-call hooks in OTEL
some pre call hooks can cause latency in high traffic - make sure this is tracked
* fix(router.py): move redis call on deployment_callback_on_success to pipeline operation
reduces p99 latency by half when redis is enabled
* fix(parallel_request_limiter_v3.py): only run check if any item has rate limits set
Prevents unnecessary latency added by rate limit checks
* test: add unit tests
* Latency Improvements: only track tpm/rpm usage when set on deployment+ LLM Caching - use an in-memory cache to reduce redis calls + OTEL - track time spent on LLM caching (#13472)
* fix(router.py): only track usage for deployments with tpm/rpm set
ensures additional latency avoided for non-tpm/rpm models
* fix(caching_handler.py): log time spent on request get cache to OTEL
enables easy debugging of call latency
* fix(caching_handler.py): use dual cache object for in-memory caching + trace redis call within caching handler
* fix(caching_handler.py): working in-memory cache for redis calls
ensures dual cache works when redis cache setup for llm calls
makes calls quicker by only checking redis when in-memory cache missed for llm api call
* test: remove redundant test
* test: add unit tests
* fix(prompt_templates/factory.py): handle anthropic cache control on individual tool results
Fixes issue where cache control on individual tool result was being ignored
* test(test_vertex_And_google_ai_studio_gemini.py): initial unit test covering translation for grounding metadata on streaming chunk
* fix(vertex_and_google_ai_studio.py): ensure grounding metadata is preserved on streaming
Closes https://github.com/BerriAI/litellm/issues/10237
* fix(core_helpers.py): include usage in expected openai keys
* docs(deploy.md): move docker recommendation to `main-stable`
* feat(enterprise/internal_user_endpoints.py): expose endpoint for checking available premium users
* feat(usage_indictor.tsx): add new element to help track remaining premium users
* feat(usage_indicator.tsx): show premium user remaining usage
allows users with user caps to know how much is left
* fix(vertex_and_google_ai_studio_gemini.py): bubble up stream is not finished, even if stop reason is given
prevents early completion of stream
Closes https://github.com/BerriAI/litellm/issues/11549
* fix(streaming_handler.py): respect is_finished = False in hidden params
internal logic for preventing ending stream early
* fix(litellm_license.py): add function to check if user is over limit
* fix(internal_user_endpoints.py): add function to check if user is over limit
* refactor: move test
* docs(customer_endpoints.py): document new param
* fix(vertex_and_google_ai_studio_gemini.py): handle both camel case and underscores in the tool for vertex ai code execution
support vertex ai code execution
* docs(vertex.md): add code execution example to vertex ai
* fix(vertex_ai/common_utils.py): when anyof in field, just select anyof - don't include other k,v pairs - vertex throws error
Fixes https://github.com/BerriAI/litellm/issues/11164
* fix(common_utils.py): add title field inside anyof - to retain some description
Addresses https://github.com/BerriAI/litellm/issues/11164#issuecomment-2914728385
* fix: add flag for disabling use_aiohttp_transport
* feat: add _create_async_transport
* feat: fixes for transport
* add httpx-aiohttp
* feat: fixes for transport
* refactor: fixes for transport
* build: fix deps
* fixes: test fixes
* fix: ensure aiohttp does not auto set content type
* test: test fixes
* feat: add LiteLLMAiohttpTransport
* fix: fixes for responses API handling
* test: fixes for responses API handling
* test: fixes for responses API handling
* feat: fixes for transport
* fix: base embedding handler
* test: test_async_http_handler_force_ipv4
* test: fix failing deepeval test
* fix: add YARL for bedrock urls
* fix: issues with transport
* fix: comment out linting issues
* test fix
* test: XAI is unstable
* test: fixes for using respx
* test: XAI fixes
* test: XAI fixes
* test: infinity testing fixes
* docs(config_settings.md): document param
* test: test_openai_image_edit_litellm_sdk
* test: remove deprecated test
* bump respx==0.22.0
* test: test_xai_message_name_filtering
* test: fix anthropic test after bumping httpx
* use n 4 for mapped tests (#11109)
* fix: use 1 session per event loop
* test: test_client_session_helper
* fix: linting error
* fix: resolving GET requests on httpx 0.28.1
* test fixes proxy unit tests
* fix: add ssl verify settings
* fix: proxy unit tests
* fix: refactor
* tests: basic unit tests for aiohttp transports
* tests: fixes xai
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Sameer Kankute