* feat(ui): add forward_client_headers_to_llm_api toggle to general settings
* feat(ui): add forward_client_headers_to_llm_api toggle to UI Settings tab
- Add toggle to UISettings.tsx frontend (switch + label + description)
- Add field to UISettings model and ALLOWED_UI_SETTINGS_FIELDS
- Sync setting to general_settings on get/update so proxy picks it up at runtime
---------
Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com>
* fix: reset db_health_cache in source module to prevent stale cache hits
The test was reassigning db_health_cache via `global` in the test module,
which doesn't affect the _health_endpoints module's variable. When a prior
test set the cache to "connected" within 2 minutes, _db_health_readiness_check
returned early without calling health_check(), causing assert_called_once to fail.
Also use PrismaError with a connection message so it's properly recognized
as a connection error by PrismaDBExceptionHandler.is_database_connection_error.
* fix: replace asyncio.sleep with polling loop in spend logs tests
The GLOBAL_LOGGING_WORKER processes callbacks via an async queue, so
asyncio.sleep(1) is a race condition - under CI load the worker may not
have processed the queued task within 1 second. Replace with a polling
helper that waits up to 10 seconds for the mock to be called.
Also add metadata.attempted_retries and metadata.max_retries to
ignored_keys since these are new fields.
* fix: isolate test_skip_server_startup from CI environment
Remove mix_stderr=False (unsupported in some Click versions). Strip
DATABASE_URL/DIRECT_URL from environment during the test to prevent
real prisma operations when these are set in CI.
Prisma Python client does not support nested select within include.
Use include user:True to fetch the full user object; model_validator
extracts user_email from it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add user_email field to LiteLLM_OrganizationMembershipTable with a model_validator
that populates it from the nested user object, and update the /organization/info
Prisma query to select user_email from the related user record.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Prefix credential name tags with "Credential: " to distinguish them
from user-defined tags when litellm_credential_name is injected
- Remove stale "new feature" banners from Organization, Customer, and
A2A usage views
- Add closable info banner to Tag usage view noting that reusable
credentials are automatically tracked and appear as "Credential: <name>"
* feat: add competitor name blocker guardrail
* fix: fix batch test endpoint for compliance playground
* fix(airline.py): add list of all known airlines to airline competitor name detector
prevent competitor discussion on company chatbot
* feat: ui tweaks for prod
* ui: redesign guardrail creation form with inline vertical stepper
Replace horizontal Ant Design Steps with an inline vertical stepper.
Completed steps collapse to a single line, active step expands.
Switch to Tremor buttons, rename steps for clarity.
* ui: rename Content Categories to Blocked topics and fix overflow
Update heading and description text, add flexWrap to prevent
text from going off-screen, fix YAML preview overflow with
pre-wrap and word-break.
* feat: support explicit display_name in content filter category YAML
Check for a display_name field before auto-generating from
category_name. Lets categories have human-friendly names
without changing their API identifier.
* fix: update denied_financial_advice display name
Add display_name field so it shows as
"Denied Financial / Investment Advice" in the UI.
Fields like input_cost_per_token contain "token" as a key segment,
which incorrectly matched the sensitive patterns list and caused values
like 3.6e-06 to be displayed as "3.60*******e-06" in the model info UI.
Add a non_sensitive_overrides set (defaulting to {"cost"}) so that any
key containing "cost" as a segment is never masked, regardless of other
matching patterns.
* Add keyword-based topic blocker implementation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add embedding-based topic blocker using MiniLM
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add topic blocker package init with exports
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add synthetic engine eval set (34 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment questions eval set (207 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine eval synthetic policy config
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MiniLM eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MPNet eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment TF-IDF eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add unified eval runner with confusion matrix reporting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add benchmarks comparison table in markdown
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Clean up topic blocker: remove unused blockers, add phrase_patterns to content filter
- Remove embedding_blocker.py, api_embedding_blocker.py, nli_blocker.py,
tfidf_blocker.py, onnx_blocker.py (heavy deps not in Docker, inferior accuracy)
- Remove airline_off_topic_restriction policy template and its test
- Fix __init__.py to only export DeniedTopic and TopicBlocker (no eager import crash)
- Add phrase_patterns support to ContentFilterGuardrail for regex-based paraphrase detection
- Rewrite denied_financial_advice.yaml with conditional matching (identifier + block word),
always-block keywords, phrase patterns, and exception phrases
- Clean up test_eval.py: only keyword blocker + content filter tests remain (no network calls)
- All 207 eval cases pass at 100% F1, 0 FP, 0 FN, <0.1ms latency
Addresses all Greptile review comments:
- Eager import crash (embedding deps) → fixed
- Undeclared dependencies → fixed (files deleted)
- lru_cache memory leak → fixed (file deleted)
- Real network calls in tests → fixed (embedding tests removed)
- Unused Dict import → already fixed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add LLM-as-judge eval and update BENCHMARKS.md
- Add TestInvestmentLlmJudgeGpt4oMini and TestInvestmentLlmJudgeClaude
test classes that use litellm.completion() to classify messages
- System prompt instructs LLM to act as airline chatbot content moderator
- Tests skip gracefully when API keys aren't set
- Update BENCHMARKS.md with production results table, historical comparison,
and instructions for running LLM judge evals
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move evals and benchmarks to guardrail_benchmarks folder
Move eval runner, eval data (JSONL), and results from
tests/test_litellm/.../topic_blocker/ into the guardrail implementation
folder at litellm/.../litellm_content_filter/guardrail_benchmarks/.
This keeps benchmarks co-located with the guardrail code they test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove standalone topic_blocker package, consolidate into content_filter
The standalone keyword_blocker.py was redundant with content_filter.py +
denied_financial_advice.yaml. Removed the entire topic_blocker/ package,
engine eval files, and old keyword blocker results. Simplified test_eval.py
to only test ContentFilter + LLM judge baselines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix compliance playground batch scoring bug, add display_name support
The compliance playground was sending all texts in a single batch API call,
but the content filter raises HTTPException on the first blocked text. This
caused a single blocked/allowed result to be applied to all rows, producing
incorrect scores (e.g. 41% instead of 100%). Fix by sending each text
individually to get per-text results with progressive UI updates.
Also add display_name field support for category YAML files so
denied_financial_advice shows as "Denied Financial / Investment Advice"
in the UI dropdown.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add block_investment CSV eval set and update benchmark result JSON
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* address greptile review feedback (greploop iteration 1)
Fix stale test path in denied_financial_advice.yaml comment.
Other comments were on files already deleted in prior commits.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
litellm-dashboard is the default UI team and will never have agents registered.
Skip the warning log when get_team_object raises a 404 for this team to avoid
log noise.
Route through LiteLLM proxy using LITELLM_VIRTUAL_KEY and LITELLM_BASE_URL
secrets. Also adds --repo flag to all gh commands to fix missing repo context.
