* Add capability to override default GitHub Copilot authentication endpoints
This feature adds support for GitHub Enterprise subsriptions with custom domain/data ownership (which use a different URL compared to standard accounts)
* Update documentation with new parameters
* Move access token URL and Client ID retrieval outside for loop
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Fix spurious comment from Greptile review
* Align api_base retrieval behavior across chat and embedding transformations
* Add missing GitHub Copilot client ID parameter in docs
* Update website documentation with newer options for GitHub Enterprise Copilot
* Fix default value for Copilot client ID in docs
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Extends the prior moonshot mocking to cover every inherited
BaseLLMChatTest test that still made a live AWS Bedrock call. Adds
request-body assertions for each override.
New overrides:
- test_content_list_handling: verifies the outgoing body round-trips
user content in list-of-text form; asserts response.choices[0].
message.content parses back from the canned response.
- test_pydantic_model_input: verifies a pydantic Message input does
not raise and produces a parseable response.
- test_response_format_type_text_with_tool_calls_no_tool_choice:
verifies tools are forwarded and response_format + drop_params do
not break the call.
- test_streaming: verifies stream=True routes to the
invoke-with-response-stream endpoint. Bedrock invoke streaming is
intercepted at the make_sync_call import site rather than via the
caller-supplied client, because CustomStreamWrapper.fetch_sync_stream
invokes the stored make_call partial with
client=litellm.module_level_client, overriding any client passed by
the caller.
Extracts a shared _make_moonshot_response helper and a
_invoke_with_mocked_post harness so all the sync mocks share one
canned response body.
After this change TestBedrockMoonshotInvoke runs 23 passed, 29
skipped, 0 live-callers, all in under 1s locally.
TogetherAIConfig.get_supported_openai_params called get_model_info(),
whose first line calls litellm.get_supported_openai_params() — which for
together_ai routes straight back into this method. The recursion only
terminated when Python's recursion limit was hit or when
_get_model_info_helper raised "not mapped" at the deepest level. Either
way the try/except caught it, so the bug stayed silent — but the cycle
ran ~332 deep every time, emitting hundreds of DEBUG log lines per
call. Surfaced as "infinite loop" in CI when the success_handler thread
emitted that log spam against an already-closed stderr during test
teardown.
Replace the get_model_info() call with supports_function_calling(),
which uses _get_model_info_helper directly and does not call
get_supported_openai_params. Measured drop from 332 to 2
_get_model_info_helper calls per first uncached lookup.
Also swap the test model from Qwen/Qwen3.5-9B (not in model_cost map)
back to a mapped serverless model, Qwen/Qwen2.5-7B-Instruct-Turbo. The
mapping gap is what made the recursion's tail end raise up into the
success handler during teardown in the first place.
Three tests inherited by TestBedrockMoonshotInvoke from BaseLLMChatTest
make live AWS Bedrock completion calls: test_developer_role_translation,
test_message_with_name, and test_completion_cost. These have been
crashing llm_translation_testing CI workers (reported as "failed on
setup with worker 'gwN' crashed").
Replace each with a mocked override that intercepts the outgoing
request via HTTPHandler.post / AsyncHTTPHandler.post patching:
- test_developer_role_translation asserts the outgoing body maps the
developer role to system (LiteLLM's translation for non-OpenAI
providers).
- test_message_with_name asserts the outgoing body preserves the user
message.
- test_completion_cost returns a canned moonshot-shaped response body
with usage and asserts response_cost > 0 against the local model
cost map.
Follows the existing HTTPHandler + patch.object(client, "post") pattern
used in test_bedrock_gpt_oss.py and test_bedrock_completion.py. No
network traffic; the three tests now complete in ~0.3s.
Drop test_bedrock_invoke_messages_injects_thinking_for_clear_thinking_context_management.
Its assertion 'interleaved-thinking-2025-05-14' in betas cannot hold because
anthropic_beta_headers_config.json maps that header to null for the bedrock
provider, so filter_and_transform_beta_headers drops it from the auto-added
beta set before anthropic_beta is written to the request.
The adjacent test_bedrock_invoke_messages_skips_thinking_injection_when_already_enabled
already covers the inverse behavior for the same model, so no coverage is lost.
langgraph-prebuilt was previously pulled in as a transitive of langgraph
so PyPI license metadata was reported as unknown. Now that it is
explicitly pinned (==1.0.8) to avoid the broken 1.0.9 release, the
license checker flags it. It is published under MIT by the same
langchain-ai/langgraph repository as langgraph itself.
Bedrock rejects clear_thinking_20251015 unless thinking is enabled or adaptive.
Inject minimal extended thinking and interleaved-thinking beta when Claude Code
sends context_management without thinking. Adds unit tests.
Made-with: Cursor
* feat(proxy): add NO_OPENAPI env var to disable /openapi.json endpoint (#25696)
* feat(proxy): add NO_OPENAPI env var to disable /openapi.json endpoint - Fixes#25538
* test(proxy): add tests for _get_openapi_url
---------
Co-authored-by: Progressive-engg <lov.kumari55@gmail.com>
* feat(prometheus): add api_provider label to spend metric (#25693)
* feat(prometheus): add api_provider label to spend metric
Add `api_provider` to `litellm_spend_metric` labels so users can
build Grafana dashboards that break down spend by cloud provider
(e.g. bedrock, anthropic, openai, azure, vertex_ai).
The `api_provider` label already exists in UserAPIKeyLabelValues and
is populated from `standard_logging_payload["custom_llm_provider"]`,
but was not included in the spend metric's label list.
* add api_provider to requests metric + add test
Address review feedback:
- Add api_provider to litellm_requests_metric too (same call-site as
spend metric, keeps label sets in sync)
- Add test_api_provider_in_spend_and_requests_metrics following the
existing pattern in test_prometheus_labels.py
* fix: ensure `litellm_metadata` is attached to `pre_call` guardrail to align with `post_call` guardrail (#25641)
* fix: ensure `litellm_metadata` is attached to pre_call to align with post_call
* refactor: remove unused BaseTranslation._ensure_litellm_metadata
* refactor: module level imports for ensure_litellm_metadata and CodeQL
* fix: update based off of Codex comment
* revert: undo usage of `_guardrail_litellm_metadata`
* feat: add pricing entry for openrouter/google/gemini-3.1-flash-lite-preview (#25610)
* fix(bedrock): skip synthetic tool injection for json_object with no schema (#25740)
When response_format={"type": "json_object"} is sent without a JSON
schema, _create_json_tool_call_for_response_format builds a tool with an
empty schema (properties: {}). The model follows the empty schema and
returns {} instead of the actual JSON the caller asked for.
This patch:
- Skips synthetic json_tool_call injection when no schema is provided.
The model already returns JSON when the prompt asks for it.
- Fixes finish_reason: after _filter_json_mode_tools strips all
synthetic tool calls, finish_reason stays "tool_calls" instead of
"stop". Callers (like the OpenAI SDK) misinterpret this as a pending
tool invocation.
json_schema requests with an explicit schema are unchanged.
Co-authored-by: Claude <noreply@anthropic.com>
* fix(utils): allowed_openai_params must not forward unset params as None
`_apply_openai_param_overrides` iterated `allowed_openai_params` and
unconditionally wrote `optional_params[param] = non_default_params.pop(param, None)`
for each entry. If the caller listed a param name but did not actually
send that param in the request, the pop returned `None` and `None` was
still written to `optional_params`. The openai SDK then rejected it as
a top-level kwarg:
AsyncCompletions.create() got an unexpected keyword argument 'enable_thinking'
Reproducer (from #25697):
allowed_openai_params = ["chat_template_kwargs", "enable_thinking"]
body = {"chat_template_kwargs": {"enable_thinking": False}}
Here `enable_thinking` is only present nested inside
`chat_template_kwargs`, so the helper should forward
`chat_template_kwargs` and leave `enable_thinking` alone. Instead it
wrote `optional_params["enable_thinking"] = None`.
Fix: only forward a param if it was actually present in
`non_default_params`. Behavior is unchanged for the happy path (param
sent → still forwarded), and the explicit `None` leakage is gone.
Adds a regression test exercising the helper in isolation so the test
does not depend on any provider-specific `map_openai_params` plumbing.
Fixes#25697
---------
Co-authored-by: lovek629 <59618812+lovek629@users.noreply.github.com>
Co-authored-by: Progressive-engg <lov.kumari55@gmail.com>
Co-authored-by: Ori Kotek <ori.k@codium.ai>
Co-authored-by: Alexander Grattan <51346343+agrattan0820@users.noreply.github.com>
Co-authored-by: Mohana Siddhartha Chivukula <103447836+iamsiddhu3007@users.noreply.github.com>
Co-authored-by: Amiram Mizne <amiramm@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Noma v1 resolved application_id from user_api_key_alias when no explicit
value was set (PR #16832). Noma v2 (PR #21400) was rewritten from scratch
and this fallback was not ported, causing all requests from shared LiteLLM
instances to appear as a single generic "litellm" application in the Noma
dashboard — breaking per-user traceability.
Fix: after checking dynamic_params and self.application_id, fall back to
user_api_key_alias from litellm_metadata or metadata. This matches the
pattern used by PromptSecurityGuardrail._resolve_key_alias_from_request_data()
and restores the v1 behavior where each API key gets its own application
entry in the Noma dashboard.
Fixes#25794
Co-authored-by: Brendan Smith-Elion <brendan.smith-elion@arcadia.io>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ollama): propagate done_reason='length' as finish_reason for max_tokens truncation
Ollama returns done_reason='length' when a response is cut off by num_predict
(the max_tokens limit). Previously, non-streaming responses hardcoded
finish_reason='stop', and streaming used chunk.get('done_reason', 'stop')
which also defaulted to 'stop' when done_reason was absent.
This meant callers (e.g. the Anthropic pass-through adapter, which maps
OpenAI 'length' -> Anthropic 'max_tokens') could never detect truncation,
making stop_reason always appear as 'end_turn' even for cut-off responses.
Fix: read done_reason from the response JSON in the non-streaming path and
use `chunk.get('done_reason') or 'stop'` in the streaming path, so Ollama's
actual done_reason passes through to the caller unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Update test_ollama_chat_transformation.py
* Update litellm/llms/ollama/chat/transformation.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
The Vertex AI count-tokens endpoint rejects model names that include
version suffixes (@default, @20251001, etc.) with:
"claude-sonnet-4-6@default is not supported for token counting"
The same model without the suffix ("claude-sonnet-4-6") works correctly.
Strip @suffix from both the model parameter and request_data["model"]
in handle_count_tokens_request before sending to the API.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MaskedHTTPStatusError constructs a new httpx.Response from the original
error. Two bugs surfaced under real HTTP error responses:
1. The new Response was created without request=, so response.request
raised RuntimeError("The .request property has not been set.") for
any downstream caller (e.g. exception_mapping_utils) that inspected it.
2. The decoded response bytes were passed together with the original
Content-Encoding header. On construction httpx tried to decompress
the already-decoded bytes and raised httpx.DecodingError
("Error -3 while decompressing data: incorrect header check").
Set response.request to the masked Request and strip Content-Encoding
(and the now-stale Content-Length) before rebuilding the Response.
URL/message masking is unchanged; the new request carries the already
masked URL.
Also update test_logging_key_masking_gemini: the security commit
25f93bed91 moved Gemini API keys from ?key=... URL params to the
x-goog-api-key header, so api_base no longer contains the key.
RestrictedPython (ZPL-2.1, a BSD-style permissive license) was added as
a dependency for the custom_code guardrail sandbox, but the license
checker didn't recognize it. Add to authorized packages list.
* [Test] Add Azure async chat completion timeout test. WIP
* Capture TTFT for /v1/messages streaming responses
The pass-through streaming path for /v1/messages (Anthropic, Bedrock,
Vertex AI, Azure AI, Minimax) logged completion_start_time only after
the entire stream finished. async_success_handler then fell back to
end_time, making TTFT equal to total duration or null in the UI and
Prometheus.
Record the timestamp of the first chunk in async_sse_wrapper and
propagate it to model_call_details before the logging handler runs,
so gen_ai.response.time_to_first_token reflects the real first-chunk
latency.
Fixes#25598
* [Refactor] Implement timeout resolution logic in completion function
add fetch ``request_timeout`` from litellm_settings
* remove stale test case
* remove extra print statement
* default request timeout value in constants to 600s to match timeout defaults handled in the proxy
* fix request timeout if using default value from constants.py
* update code structure, test cases
* only override if the global timeout sets timeout to 6000s
* update code structure, move hard coded values to const and make the reslve function readable by moving fallback logic to a seperate function
* modify default timeout values, replacing hard coded ones with default values defined
---------
Co-authored-by: harish876 <harishgokul01@gmail.com>
Co-authored-by: Joaquin Hui Gomez <joaquinhuigomez@users.noreply.github.com>
Tighten validation of request body parameters in the proxy routing
layer. Use context variables for internal call state management
instead of passing flags through request kwargs. Clean up metadata
handling at the proxy boundary.
* fix: make PodLockManager.release_lock atomic compare-and-delete
Re-lands #21226 (reverted in #21469).
release_lock() previously did GET + compare + DEL in separate calls,
leaving a window where another pod could reacquire the lock between
the GET and DEL, causing a stale owner to delete a live lock.
Fix: use a Redis Lua script for atomic compare-and-delete. Script
registration is cached per PodLockManager instance. Falls back to
the old GET+DEL path for cache backends that don't expose
async_register_script.
Original revert was due to e2e tests running in CI without Redis.
Those tests now carry @pytest.mark.skip(reason="Requires Redis connection.")
so this re-land is safe.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: add Lua fallback on execution error + test coverage gaps
Address Greptile review feedback on #24466:
1. Wrap Lua script execution in try/except — if Redis clears loaded
scripts (restart) or scripting is disabled, fall back to GET+DEL
rather than letting the exception propagate and leave the lock held
until TTL. Reset cached script handle so the next call re-registers.
2. Add test_release_lock_lua_path_emits_released_event — verifies
_emit_released_lock_event is called when Lua path returns 1.
3. Add test_release_lock_falls_back_to_get_del_when_lua_execution_fails
— verifies the fallback path is taken and script handle is reset.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Complements the stubbed-out live integration test by verifying the
outgoing Bedrock Converse request body for GPT-OSS is well-formed when
the caller supplies a tool schema with OpenAI-style metadata
($id, $schema, additionalProperties, strict):
- correct converse URL for bedrock/converse/openai.gpt-oss-20b-1:0
- toolConfig.tools[0].toolSpec has the expected name/description
- inputSchema.json keeps type/properties/required and strips fields
Bedrock does not accept
