Vertex AI rejects requests containing both search tools (googleSearch,
enterpriseWebSearch, urlContext) and function declarations with error:
'Multiple tools are supported only when they are all search tools.'
When _merge_tools_from_deployment() combines deployment-level search
tools with user-request function tools (e.g. via MCP), the mixed tool
list causes a 400 error. This fix detects the conflict in _map_function()
and drops search tools, keeping function declarations.
Non-search tools like code_execution and computerUse are preserved.
Fixes#23337
* fix#25506
* address greptile review feedback
* [Test] UI - Models: Add E2E tests for Add Model flow
Add E2E tests covering:
- Test connection with bad credentials shows failure modal
- Adding a specific model and verifying it appears in All Models table
- Adding a wildcard route and verifying it appears in All Models table
- Verifying model dropdown shows provider-specific models (existing test updated)
Added data-testid attributes to UI components to support stable test selectors.
Tests verified passing 3/3 consecutive runs with zero flakiness.
* address greptile review feedback (greploop iteration 1)
Add cleanup helper to delete models created during tests, preventing
stale data accumulation across repeated test runs.
* fix CI: replace data-testid selectors with text/role-based selectors
The data-testid attributes added to React components are not present
in the CI-built UI output. Switch to using getByRole and getByText
selectors which work with the rendered DOM regardless of build cache.
* remove unnecessary cleanup helper
The database is freshly seeded on every test run via seed.sql,
so per-test cleanup is not needed.
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
* fix: emit input_json_delta for tool args bundled in first streaming chunk
Some providers (xAI, Gemini) include tool_call function arguments in the
same streaming chunk as the function name/id. The AnthropicStreamWrapper
was discarding the trigger chunk entirely when starting a new content
block, which silently dropped the input_json_delta carrying tool
arguments. This caused tool_use blocks to arrive with empty input {}.
Now queue the processed_chunk after content_block_start when it carries
non-empty input_json_delta data. Backward compatible: providers that send
empty arguments in the first chunk (OpenAI-style) are unaffected since
the condition checks for truthy partial_json.
* test: add tests for input_json_delta emission on bundled tool args
Covers the fix for providers (xAI, Gemini) that bundle tool_call
arguments in the same streaming chunk as the function name/id.
Verifies the AnthropicStreamWrapper emits input_json_delta after
content_block_start, and that empty-arg chunks (OpenAI-style) are
unaffected.
* style: apply Black formatting to streaming_iterator.py
* fix: mirror input_json_delta fix to sync __next__ and add sync tests
* test: make no_extra_delta tests assert explicitly instead of passing silently
* feat: add litellm.compress() for BM25-based context compression
Adds a compress() utility that reduces context size for LLM calls using
BM25 relevance scoring (with optional semantic embeddings via
litellm.embedding()). Messages below a token threshold pass through
unchanged; messages above are scored, ranked, and the lowest-relevance
ones replaced with stubs. Originals are cached and a retrieval tool is
injected so the model can recover dropped content on demand.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(compress): truncate high-scoring messages instead of fully stubbing them
When a relevant message was too large to fit in the token budget it was
replaced with a stub, leaving the LLM with no real content to work with.
Now the highest-scoring overflow message is truncated (first 70% + last 30%
of words) to fill the remaining budget, so the LLM always receives actual
content rather than just a retrieval pointer.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(bm25): add prefix expansion so query terms match inflected doc tokens
"cook" now matches "cooking", "auth" matches "authentication", etc.
Without this, short query terms scored 0 against longer inflected forms
in documents, causing the wrong message to be kept.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add routing correctness test and eval harness for litellm.compress()
- test_simple_compression: parametrized test verifying BM25 routes the
right message based on query ("How to cook?" keeps cooking, "Fix auth"
keeps auth content)
- eval_compression.py: end-to-end eval harness comparing baseline vs
compressed model performance on HumanEval-style coding problems
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(eval): add SWE-bench Lite compression eval harness
Uses princeton-nlp/SWE-bench_Lite_bm25_27K which bundles ~27k tokens of
BM25-retrieved repo context per problem — large enough to meaningfully
stress litellm.compress() without Docker or GitHub API calls.
Proxy eval metrics (no test runner needed):
- has_diff: model produced a valid unified diff
- file_overlap: fraction of gold-patch files in generated patch
- exact_file_match: generated patch touches exactly the right files
Run: python tests/eval_swe_bench.py --model gpt-4o --problems 10
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(eval): robust dataset loading + sys.path fix for worktree imports
- Add HuggingFace API fallback so the SWE-bench loader doesn't need
the `datasets` library (avoids pyarrow/numpy binary compat issues)
- Insert repo root into sys.path so compression module resolves
from worktrees
- Use direct import of litellm_compress to avoid __getattr__ issues
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* improve compression quality: line-based truncation, multi-message budget, 70% default target
- Switch truncate_message from word-based to line-based splitting to
preserve code structure (function boundaries, indentation)
- Allow multiple messages to be truncated instead of burning entire
budget on one overflow message
- Raise default compression target from 50% to 70% of trigger for
better quality/cost tradeoff
- Add --compression-target CLI arg to SWE-bench eval harness
- Move tests to canonical locations (tests/test_litellm/, scripts/)
- Add docs page and sidebar entries for compress()
Eval results (5 problems, Opus, trigger=10k):
Hunk overlap delta improved from -0.417 to -0.221
Content similarity now matches baseline (+0.006)
Cost savings: 72%
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add SWE-bench performance results to compress() docs
Include benchmark table from Opus eval (5 problems, trigger=10k)
showing 72% cost savings with file-level quality fully preserved.
Add metric explanations and eval runner examples.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(eval): use tolerance-based hunk overlap metric
The exact line-number matching was too brittle — LLM-generated patches
often target the right code region but with slightly offset line numbers.
Switch to hunk-level overlap with a 10-line tolerance window so nearby
edits count as matches. This better reflects actual patch quality.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add compression_interception callback for LiteLLM Proxy
Add a proxy callback that automatically compresses incoming /v1/messages
payloads above a configurable token threshold, runs the retrieval tool
loop server-side, and returns the final response. This brings compress()
support to proxy deployments (e.g. Claude Code via /v1/messages).
- New callback: litellm/integrations/compression_interception/
- Proxy config: compression_interception_params in litellm_settings
- Support for input_type param in compress() (openai vs anthropic)
- Docs: proxy setup instructions with YAML config example
- Tests: 139-line unit test suite for the interception handler
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Revert "feat: add compression_interception callback for LiteLLM Proxy"
This reverts commit 72bd5cb152ca1df07f14a14e14a2816e188874a8.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Previously these were silently dropped with a verbose warning, which
could break observability integrations without surfacing a clear error.
Now raises ValueError with remediation steps (configure server-side
or pass the resolved value) so callers get immediate, actionable feedback.
Brings the date-range branch in line with the non-date-range branch which
already hashes sk- prefixed tokens before querying. Adds coverage for
filter-combination behavior in view_spend_logs.
- Log a warning when dropping callback params that carry os.environ/
references so operators notice the misconfiguration.
- Require absolute paths in oidc/file/ and correct the documented
example to use the leading-slash form.
- Drop the unused return value from _reject_os_environ_references.
- Reject os.environ/ references supplied via /health/test_connection
request params instead of resolving them; config-sourced values are
already resolved before reaching the endpoint.
- Skip os.environ/ references in dynamic callback params loaded from
per-request metadata.
- Constrain oidc/file/ to an allowed credential directory allowlist
(defaults to /var/run/secrets and /run/secrets, overridable via
LITELLM_OIDC_ALLOWED_CREDENTIAL_DIRS).
Budget table entries (team members, end-users) used duration_in_seconds()
for a sliding-window reset, while keys/users/teams used calendar-aligned
get_budget_reset_time(). This made "30d" and "1mo" mean different things
depending on entity type. Now both paths use get_budget_reset_time() for
consistent calendar-aligned resets (e.g. "30d" → 1st of next month).
Fixes#25432
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two code paths in key_management_endpoints.py call hash_token()
unconditionally when invalidating the user_api_key_cache after a key
update. When the caller passes a pre-hashed token ID (not an sk-
prefixed key), hash_token() double-hashes it, producing a cache key
that does not match the actual cached entry. Cache invalidation
silently fails.
This is compounded by update_cache() which writes the stale cached key
object back with a fresh 60s TTL after every successful request,
preventing natural TTL expiry. The stale entry (with outdated fields
like max_budget=None) persists indefinitely under load.
PR #24969 fixed this in update_key_fn but missed two other call sites:
- _process_single_key_update (bulk update path)
- _execute_virtual_key_regeneration (key rotation path)
Fix: replace hash_token() with _hash_token_if_needed() in both
locations, matching the pattern already used elsewhere in the file.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Renames the new per-guardrail opt-out field from `disabled_global_guardrails`
to `opted_out_global_guardrails` to eliminate the one-character collision with
the legacy `disable_global_guardrails` boolean kill switch. Adds a type guard
on the new gate so a misnamed bool can't crash the guardrail check. Filters
duplicates out of the team-edit guardrail display for legacy teams that have a
global name persisted in `metadata.guardrails` from before this PR. Drops the
unused `isGuardrailsLoading` and `guardrailsError` destructures left in
AddModelForm after the hook refactor.
Adds Python tests for the new gate behavior (root, litellm_metadata, metadata,
non-matching name, empty list, malformed bool value, opt-in coexistence) and
extends useGuardrails.test.ts to exercise the global / optional partition
logic that the rebuilt hook performs in its `select` transform.
Wires the legacy kill switch and the new opt-out list together in the team
edit form so they can never fall out of sync:
- Toggling the kill switch reactively updates the Guardrails Select via
`onValuesChange` — switch on strips all globals from the selection, switch
off re-adds them. Existing opt-in extras are preserved either way.
- When the switch is on, global options in the Select are individually
disabled (greyed out) so the user can still manage opt-in guardrails but
cannot accidentally re-enable a global the kill switch is bypassing.
- The save handler writes both fields together: `disable_global_guardrails`
reflects the switch, and `opted_out_global_guardrails` is set to either
every global (when the switch is on) or the user's explicit opt-outs.
- `effectiveGuardrails` for the form's initialValues honors the kill switch
on legacy teams so the form opens in a state that matches what the runtime
gate is actually doing — fixes the visual lie where chips appeared active
while the switch was bypassing them.
The backend gate already reads the list as the primary path with the bool
as a fallback, so untouched legacy teams keep working until they get edited,
at which point they migrate naturally.
Rename disable_global_guardrail → disable_global_guardrails to match
the key name used by litellm_pre_call_utils.py, the API endpoints,
and the UI when propagating key/team metadata.
The singular form was introduced in PR #16983 and has never matched
the plural form written by the rest of the codebase, so the feature
silently did nothing.
Re-applies fix originally from #25488. Original commit could not be
merged due to missing signature.
Co-Authored-By: Remi Mabon <remi.mabon@redcare-pharmacy.com>
- Add _verify_org_access to deprecated POST /organization/info endpoint
- Move get_user_object to module-level import in organization_endpoints.py
- Add tests for _verify_team_access 403 denial path
