* fix(tests): add app.dependency_overrides for auth in spend logs tests
test_ui_view_spend_logs_with_status, test_ui_view_spend_logs_with_model,
test_ui_view_spend_logs_with_model_id, and test_view_spend_logs_summarize_parameter
all send Bearer sk-test without mocking user_api_key_auth. When a prior test
in the same xdist worker sets master_key, the auth check fails for sk-test
and the test fails intermittently.
Fix: use app.dependency_overrides[ps.user_api_key_auth] to bypass auth,
same pattern as other tests in the same file.
* fix(tests): mock user_api_key_auth in test_vertex_passthrough_with_no_default_credentials
vertex_proxy_route calls user_api_key_auth internally. When a prior test in the
same xdist worker sets master_key, the auth check fails for the test request
and create_pass_through_route is never called, causing assert_called_once_with to fail.
Fix: patch user_api_key_auth as an AsyncMock in the with mock.patch() block.
* fix(test): remove deprecated Click mix_stderr param in test_use_prisma_db_push_flag_behavior
Click 8.2+ removed the mix_stderr parameter from CliRunner. Use CliRunner() without it.
* fix(test): use app.dependency_overrides for auth mock in test_role_mappings_stored_and_retrieved
monkeypatch.setattr doesn't affect FastAPI's Depends() resolution in parallel
test execution. Use app.dependency_overrides which is the proper FastAPI pattern.
Any PrismaError should be treated as a DB connection error for the
allow_requests_on_db_unavailable feature and 503 responses. The narrow
keyword-based check is now in is_database_transport_error, which is
what the reconnect logic in auth_checks.py should use.
Fixes test_delete_access_group_503_on_db_connection_error and
test_handle_authentication_error_db_unavailable failures caused by
PR #21706 narrowing is_database_connection_error.
* fix(tests): isolate flaky files endpoint tests from global proxy state
* test(secret_managers): add mocked unit test for write/read JSON secret cycle
* fix(tests): restore litellm.callbacks in TestSpendLogsPayload setup/teardown
* fix(tests): clear app.openapi_schema in TestSwaggerChatCompletions setup/teardown
* fix(tests): add flaky marker to test_async_increment_tokens_with_ttl_preservation
* fix(tests): isolate flaky files endpoint tests from global proxy state
* test(secret_managers): add mocked unit test for write/read JSON secret cycle
- Restore preserve_fields param for backward compatibility (deprecated)
- Use zip() instead of index-based iteration to prevent IndexError
- Add backward compatibility test
* fix: reset db_health_cache in source module to prevent stale cache hits
The test was reassigning db_health_cache via `global` in the test module,
which doesn't affect the _health_endpoints module's variable. When a prior
test set the cache to "connected" within 2 minutes, _db_health_readiness_check
returned early without calling health_check(), causing assert_called_once to fail.
Also use PrismaError with a connection message so it's properly recognized
as a connection error by PrismaDBExceptionHandler.is_database_connection_error.
* fix: replace asyncio.sleep with polling loop in spend logs tests
The GLOBAL_LOGGING_WORKER processes callbacks via an async queue, so
asyncio.sleep(1) is a race condition - under CI load the worker may not
have processed the queued task within 1 second. Replace with a polling
helper that waits up to 10 seconds for the mock to be called.
Also add metadata.attempted_retries and metadata.max_retries to
ignored_keys since these are new fields.
* fix: isolate test_skip_server_startup from CI environment
Remove mix_stderr=False (unsupported in some Click versions). Strip
DATABASE_URL/DIRECT_URL from environment during the test to prevent
real prisma operations when these are set in CI.
Add user_email field to LiteLLM_OrganizationMembershipTable with a model_validator
that populates it from the nested user object, and update the /organization/info
Prisma query to select user_email from the related user record.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: add competitor name blocker guardrail
* fix: fix batch test endpoint for compliance playground
* fix(airline.py): add list of all known airlines to airline competitor name detector
prevent competitor discussion on company chatbot
* feat: ui tweaks for prod
* Add keyword-based topic blocker implementation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add embedding-based topic blocker using MiniLM
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add topic blocker package init with exports
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add synthetic engine eval set (34 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment questions eval set (207 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine eval synthetic policy config
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MiniLM eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MPNet eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment TF-IDF eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add unified eval runner with confusion matrix reporting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add benchmarks comparison table in markdown
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Clean up topic blocker: remove unused blockers, add phrase_patterns to content filter
- Remove embedding_blocker.py, api_embedding_blocker.py, nli_blocker.py,
tfidf_blocker.py, onnx_blocker.py (heavy deps not in Docker, inferior accuracy)
- Remove airline_off_topic_restriction policy template and its test
- Fix __init__.py to only export DeniedTopic and TopicBlocker (no eager import crash)
- Add phrase_patterns support to ContentFilterGuardrail for regex-based paraphrase detection
- Rewrite denied_financial_advice.yaml with conditional matching (identifier + block word),
always-block keywords, phrase patterns, and exception phrases
- Clean up test_eval.py: only keyword blocker + content filter tests remain (no network calls)
- All 207 eval cases pass at 100% F1, 0 FP, 0 FN, <0.1ms latency
Addresses all Greptile review comments:
- Eager import crash (embedding deps) → fixed
- Undeclared dependencies → fixed (files deleted)
- lru_cache memory leak → fixed (file deleted)
- Real network calls in tests → fixed (embedding tests removed)
- Unused Dict import → already fixed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add LLM-as-judge eval and update BENCHMARKS.md
- Add TestInvestmentLlmJudgeGpt4oMini and TestInvestmentLlmJudgeClaude
test classes that use litellm.completion() to classify messages
- System prompt instructs LLM to act as airline chatbot content moderator
- Tests skip gracefully when API keys aren't set
- Update BENCHMARKS.md with production results table, historical comparison,
and instructions for running LLM judge evals
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move evals and benchmarks to guardrail_benchmarks folder
Move eval runner, eval data (JSONL), and results from
tests/test_litellm/.../topic_blocker/ into the guardrail implementation
folder at litellm/.../litellm_content_filter/guardrail_benchmarks/.
This keeps benchmarks co-located with the guardrail code they test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove standalone topic_blocker package, consolidate into content_filter
The standalone keyword_blocker.py was redundant with content_filter.py +
denied_financial_advice.yaml. Removed the entire topic_blocker/ package,
engine eval files, and old keyword blocker results. Simplified test_eval.py
to only test ContentFilter + LLM judge baselines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix compliance playground batch scoring bug, add display_name support
The compliance playground was sending all texts in a single batch API call,
but the content filter raises HTTPException on the first blocked text. This
caused a single blocked/allowed result to be applied to all rows, producing
incorrect scores (e.g. 41% instead of 100%). Fix by sending each text
individually to get per-text results with progressive UI updates.
Also add display_name field support for category YAML files so
denied_financial_advice shows as "Denied Financial / Investment Advice"
in the UI dropdown.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add block_investment CSV eval set and update benchmark result JSON
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* address greptile review feedback (greploop iteration 1)
Fix stale test path in denied_financial_advice.yaml comment.
Other comments were on files already deleted in prior commits.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Add new SpendLogsMetadata keys to ignored_keys in spend logs tests
(regression from ccecc10c82 which intentionally includes all keys)
- Mock PrismaManager.setup_database and should_update_prisma_schema in
proxy CLI tests to prevent real DB migrations from running in CI
- Use CliRunner(mix_stderr=False) to fix Click stream lifecycle issues
- Use unique UUID suffix for Redis TTL test keys to avoid stale state
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add attempted_retries and max_retries fields to SpendLogsMetadata so the
Logs page can display how many retries occurred for each request. The
router now injects retry tracking metadata before each make_call, which
flows through the logging pipeline into the spend logs metadata JSON.
The UI shows "Not Retried" when the first attempt succeeded, and
"N / M" (attempted / max) when retries occurred. The field is hidden
for requests that did not go through the router.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reset logo_path to default_logo when custom UI_LOGO_PATH file doesn't
exist, so the else branch at the bottom of get_image serves the default
logo instead of the non-existent custom path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add os.path.exists check before serving custom local logo so that a
non-existent UI_LOGO_PATH gracefully falls through to the cache/default
instead of causing a FileResponse error.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The /get_image endpoint checked for cached_logo.jpg before reading the
UI_LOGO_PATH env var, so a pre-existing cache (e.g. baked into the base
Docker image) would always be served, ignoring the user's custom logo.
Move the UI_LOGO_PATH read before the cache check and serve local file
paths directly, bypassing the cache. The cache optimization is preserved
for HTTP URLs and the default logo where it is actually needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Regular team members could not see service accounts belonging to their
team. Additionally, former team members could still see service accounts
they created via the created_by filter after leaving the team.
- Add get_member_team_ids() to retrieve all teams a user belongs to
- Regular members now see team service accounts (user_id=NULL) but not
other members' personal keys
- Scope created_by filter to the user's current team memberships
- Add 7 unit tests covering the new visibility rules
Replace find_many + Python-side aggregation with a single SQL GROUP BY
query via query_raw in get_daily_activity_aggregated. This collapses
rows across entities (users/teams/orgs) in the database, reducing ~150k
rows to ~2-3k grouped rows before transfer to Python.
Also adds composite indexes (entity_id, date) to all 6 daily spend
tables for faster filtered queries.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The admin bulk-delete path (prisma_client.delete_data) was never comparing
the DB return value against the requested tokens, so failed_tokens was
always empty for admins even when the DB silently skipped some tokens.
Adds the same mismatch check as the non-admin path, and a new test that
exercises admin bulk-delete returning fewer tokens than requested.
Fixes gap identified by Greptile review on #21609.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Instead of raising an exception when some requested tokens are not found
in the DB, populate `failed_tokens` in the return dict so callers can
inspect which keys were not deleted without catching a generic Exception.
Supersedes #12577.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix aviation safety topic filter: remove overly broad exceptions, add cockpit access block words
* fix airline brand protection filter: add identifier words, competitor/ops block words, tighten exceptions
* add example_sentences to all policy templates + topic-filtering and prompt-injection templates
* add policy_endpoints package with AI policy suggester
* update test patch targets for policy_endpoints package move
* add unit tests for AI policy suggester
* add suggestPolicyTemplates networking function
* add AI suggestion modal component
* add Use AI button and template loading callback to PolicyTemplates
* wire up AI suggestion modal in policies page
* fix policy_templates_backup.json path after package move
* add estimated_latency field to all policy templates
* use llm_router and accept model parameter in ai_policy_suggester
* add model param to suggest templates endpoint
* pass model param in suggestPolicyTemplates
* polish ai suggestion modal: model selector, auto-growing textareas, latency badges
* add template queue for processing multiple AI-suggested templates
* show template progress badge in guardrail selection modal
* fix(scim): handle deprovisioning operations without path field
When SCIM providers send deprovisioning requests without a path field
(e.g., {"op": "replace", "value": {"active": false}}), the code was
storing the value under an empty string key in metadata.
This fix:
- Detects operations with no path where value is a dict
- Extracts and handles known fields like 'active' correctly
- Sets metadata["scim_active"] = false instead of metadata[""] = {"active": false}
Fixes: SCIM deprovisioning creating empty string keys in user metadata
* fix(scim): handle all known fields in operations without path
Extended the fix to handle all SCIM fields (not just active) when
operations have no path field:
- active -> scim_active
- displayName -> user_alias
- externalId -> user_id
- name.givenName/familyName -> scim_metadata
Added comprehensive test for multiple fields without path.
Addresses Greptile review feedback on RFC 7644 compliance.
* trigger PR update
Virtual keys only track created_at and updated_at, which don't indicate
when a key was last used. This adds a last_active field that gets updated
during the async batch spend update, giving admins visibility into which
keys are actively being used.
Changes:
- Add last_active DateTime? to VerificationToken and
DeletedVerificationToken in all 3 schema files and Python types
- Set last_active in the batch key spend update alongside spend increment
- Add Last Active column to virtual keys UI table with info popover
and hover tooltip showing full date/time with timezone
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix aviation safety topic filter: remove overly broad exceptions, add cockpit access block words
* fix airline brand protection filter: add identifier words, competitor/ops block words, tighten exceptions
* update policy templates with competitor pre/post guardrails and streaming enrichment
* sync policy_templates_backup.json with policy_templates.json
* add streaming enrichment endpoint, competitor variations, and model selection
* add streaming enrichPolicyTemplate networking function
* rewrite template parameter modal with streaming tags, AI/manual toggle, model selector
* update policies panel to pass enrichment options and show discovered competitors
* show discovered competitors as tags in guardrail selection modal
* use llm_router instead of litellm.acompletion, extract helpers, move constants
* validate competitors list size, cap variation prompt input
* add refinement instruction support for competitor discovery
* add instruction and existingCompetitors params to streaming enrichment
* add refinement input for iterating on competitor list with AI
* emit status events during variation generation so UI shows progress
* add onStatus callback to streaming enrichment
* show status spinner during variation generation, widen modal to 700px
* add tests for competitor enrichment helper functions
Add a new team member permission `/team/daily/activity` that allows
non-admin team members to see all team usage data, not just their own.
## Changes
- Add `TEAM_DAILY_ACTIVITY` to `KeyManagementRoutes` enum and available
team member permissions
- Add `_team_member_has_permission` helper in `common_utils.py`
- Modify `/team/daily/activity` endpoint to skip API key filtering when
the member has this permission
- Add permission description and method detection in the UI
- Add backend and frontend tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Four tests were broken by commit e00c181f0c (Mcp user permissions #21462):
1. test_list_tools_single_server_unprefixed_names: The commit changed
_get_tools_from_mcp_servers to always add server prefixes (add_prefix=True),
removing the conditional that skipped prefixing for single servers.
Updated assertion from "toolA" → "zapier-toolA".
2. test_mcp_get_prompt_success: mcp_get_prompt now extracts the server name
from a prefixed prompt name via split_server_prefix_from_name(). Passing
unprefixed "hello" returns server_name="" which matches no server → 403.
Updated call to use "server_a-hello" so the server lookup succeeds.
3. test_e2e_jwt_team_mcp_permissions_enforced &
4. test_e2e_jwt_team_mcp_key_intersection:
The commit replaced `from typing import List` with
`from litellm.proxy.proxy_server import general_settings` in
MCPRequestHandler.get_allowed_mcp_servers(). Both tests mock
litellm.proxy.proxy_server with a types.ModuleType that lacked
general_settings, causing ImportError. Added general_settings={} to
both mock modules.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
test_complete_reload_flow and test_distributed_reload_check_function both
trigger code paths that assign a minimal stub dict to litellm.model_cost
(via the /reload/model_cost_map endpoint and _check_and_reload_model_cost_map).
Without restoring, subsequent tests in the same worker can't find gpt-4o
pricing and calculate spend=0.0 instead of the expected value.
Added try/finally save-and-restore of litellm.model_cost in both tests,
matching the pattern used in test_reload_model_cost_map_admin_access.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(schema.prisma): add object permissions for end users
allows controlling if end user can call specific mcp servers
* feat: cleanup for customer_endpoints support of object permission id
* fix: cleanup str
* feat(customers/): enforce end user can only call allowed mcps - if configured
* docs: document customer/end user object permission usage
* feat: enforce end user permissions on MCP tool calls
This commit implements end user permission enforcement for MCP servers:
1. Always add server prefixes to MCP tool names
- Removed conditional logic that only added prefixes when multiple servers existed
- Now always adds server prefix for consistent tool naming across all scenarios
- Updated 5 locations in server.py (list_tools, get_prompts, get_resources,
get_resource_templates, get_prompt)
2. Created MCP End User Permission Guardrail Hook
- New guardrail hook: litellm/proxy/guardrails/guardrail_hooks/mcp_end_user_permission.py
- Runs on post_call to validate tool calls in LLM responses
- Extracts MCP server name from tool names (splits on first '-')
- Checks if end_user_id has permissions for the MCP server
- Raises GuardrailRaisedException if end user lacks permission
- Supports both streaming and non-streaming responses
3. Added comprehensive tests
- Test file: tests/test_litellm/proxy/guardrails/guardrail_hooks/test_mcp_end_user_permission.py
- Tests cover: authorized/unauthorized tools, non-MCP tools, no end_user scenarios
- Tests permission checking logic and exception raising
The hook integrates with the existing MCPRequestHandler._get_allowed_mcp_servers_for_end_user
to fetch end user permissions and enforce access control at the response level.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* refactor: remove redundant add_prefix variable assignments
Simplified the code by removing intermediate `add_prefix` variable
assignments and passing `True` directly to function calls since
we now always add server prefixes.
Changes:
- Removed `add_prefix = True` variable assignments in 5 locations
- Changed `add_prefix=add_prefix` to `add_prefix=True` in function calls
- Added inline comments to clarify the behavior
This makes the code more concise and clearer in intent.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* feat(auth_utils.py): support safety_identifier as a valid way of passing the end user id for responses api
* feat(llms): ensure 'tools' is correctly updated for responses api
* fix: fix greptile feedback
* feat: transformation.py
proper responses api tool handling for guardrail translation layer
---------
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
