- Prefix credential name tags with "Credential: " to distinguish them
from user-defined tags when litellm_credential_name is injected
- Remove stale "new feature" banners from Organization, Customer, and
A2A usage views
- Add closable info banner to Tag usage view noting that reusable
credentials are automatically tracked and appear as "Credential: <name>"
* feat: add competitor name blocker guardrail
* fix: fix batch test endpoint for compliance playground
* fix(airline.py): add list of all known airlines to airline competitor name detector
prevent competitor discussion on company chatbot
* feat: ui tweaks for prod
* ui: redesign guardrail creation form with inline vertical stepper
Replace horizontal Ant Design Steps with an inline vertical stepper.
Completed steps collapse to a single line, active step expands.
Switch to Tremor buttons, rename steps for clarity.
* ui: rename Content Categories to Blocked topics and fix overflow
Update heading and description text, add flexWrap to prevent
text from going off-screen, fix YAML preview overflow with
pre-wrap and word-break.
* feat: support explicit display_name in content filter category YAML
Check for a display_name field before auto-generating from
category_name. Lets categories have human-friendly names
without changing their API identifier.
* fix: update denied_financial_advice display name
Add display_name field so it shows as
"Denied Financial / Investment Advice" in the UI.
Fields like input_cost_per_token contain "token" as a key segment,
which incorrectly matched the sensitive patterns list and caused values
like 3.6e-06 to be displayed as "3.60*******e-06" in the model info UI.
Add a non_sensitive_overrides set (defaulting to {"cost"}) so that any
key containing "cost" as a segment is never masked, regardless of other
matching patterns.
* Add keyword-based topic blocker implementation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add embedding-based topic blocker using MiniLM
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add topic blocker package init with exports
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add synthetic engine eval set (34 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment questions eval set (207 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine eval synthetic policy config
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MiniLM eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MPNet eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment TF-IDF eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add unified eval runner with confusion matrix reporting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add benchmarks comparison table in markdown
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Clean up topic blocker: remove unused blockers, add phrase_patterns to content filter
- Remove embedding_blocker.py, api_embedding_blocker.py, nli_blocker.py,
tfidf_blocker.py, onnx_blocker.py (heavy deps not in Docker, inferior accuracy)
- Remove airline_off_topic_restriction policy template and its test
- Fix __init__.py to only export DeniedTopic and TopicBlocker (no eager import crash)
- Add phrase_patterns support to ContentFilterGuardrail for regex-based paraphrase detection
- Rewrite denied_financial_advice.yaml with conditional matching (identifier + block word),
always-block keywords, phrase patterns, and exception phrases
- Clean up test_eval.py: only keyword blocker + content filter tests remain (no network calls)
- All 207 eval cases pass at 100% F1, 0 FP, 0 FN, <0.1ms latency
Addresses all Greptile review comments:
- Eager import crash (embedding deps) → fixed
- Undeclared dependencies → fixed (files deleted)
- lru_cache memory leak → fixed (file deleted)
- Real network calls in tests → fixed (embedding tests removed)
- Unused Dict import → already fixed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add LLM-as-judge eval and update BENCHMARKS.md
- Add TestInvestmentLlmJudgeGpt4oMini and TestInvestmentLlmJudgeClaude
test classes that use litellm.completion() to classify messages
- System prompt instructs LLM to act as airline chatbot content moderator
- Tests skip gracefully when API keys aren't set
- Update BENCHMARKS.md with production results table, historical comparison,
and instructions for running LLM judge evals
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move evals and benchmarks to guardrail_benchmarks folder
Move eval runner, eval data (JSONL), and results from
tests/test_litellm/.../topic_blocker/ into the guardrail implementation
folder at litellm/.../litellm_content_filter/guardrail_benchmarks/.
This keeps benchmarks co-located with the guardrail code they test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove standalone topic_blocker package, consolidate into content_filter
The standalone keyword_blocker.py was redundant with content_filter.py +
denied_financial_advice.yaml. Removed the entire topic_blocker/ package,
engine eval files, and old keyword blocker results. Simplified test_eval.py
to only test ContentFilter + LLM judge baselines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix compliance playground batch scoring bug, add display_name support
The compliance playground was sending all texts in a single batch API call,
but the content filter raises HTTPException on the first blocked text. This
caused a single blocked/allowed result to be applied to all rows, producing
incorrect scores (e.g. 41% instead of 100%). Fix by sending each text
individually to get per-text results with progressive UI updates.
Also add display_name field support for category YAML files so
denied_financial_advice shows as "Denied Financial / Investment Advice"
in the UI dropdown.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add block_investment CSV eval set and update benchmark result JSON
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* address greptile review feedback (greploop iteration 1)
Fix stale test path in denied_financial_advice.yaml comment.
Other comments were on files already deleted in prior commits.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
litellm-dashboard is the default UI team and will never have agents registered.
Skip the warning log when get_team_object raises a 404 for this team to avoid
log noise.
Route through LiteLLM proxy using LITELLM_VIRTUAL_KEY and LITELLM_BASE_URL
secrets. Also adds --repo flag to all gh commands to fix missing repo context.
- Revert test_anthropic_web_search_in_model_info to use claude-3-5-haiku-latest
(model info test doesn't make API calls, so the -latest alias is fine here)
- Replace claude-3-7-sonnet-20250219 with claude-sonnet-4-5-20250929 in
test_anthropic_prompt_caching.py (10 instances)
- Include pending doc updates for COMPETITOR_LLM_TEMPERATURE and
MAX_COMPETITOR_NAMES env vars in config_settings.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add new SpendLogsMetadata keys to ignored_keys in spend logs tests
(regression from ccecc10c82 which intentionally includes all keys)
- Mock PrismaManager.setup_database and should_update_prisma_schema in
proxy CLI tests to prevent real DB migrations from running in CI
- Use CliRunner(mix_stderr=False) to fix Click stream lifecycle issues
- Use unique UUID suffix for Redis TTL test keys to avoid stale state
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace claude-3-7-sonnet-20250219 with claude-sonnet-4-5-20250929 in
test_anthropic_completion.py (9 instances). Add missing "store" param
to OPENAI_CHAT_COMPLETION_PARAMS to fix test_store_in_openai_chat_completion_params.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GuardrailViewer was rewritten from ant-design Collapse to a custom card
layout. Tests now match the new component: updated header text, ms-based
duration, expand-to-reveal provider details, and removed ant-collapse
references.
AllModelsTab tests failed because ModelSettingsModal now uses useMutation
via useStoreModelInDB. Switched from bare render() to renderWithProviders()
which wraps in QueryClientProvider.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The dict comprehension in _get_spend_logs_metadata was only including
metadata keys that existed in the input dict. After user_api_key_project_id
was added to SpendLogsMetadata, payloads missing that key in input would
not include it in output, causing test_spend_logs_payload to fail.
Use metadata.get(key) instead of filtering with `if key in metadata`
to ensure all SpendLogsMetadata keys are always present (defaulting to
None), consistent with the metadata-is-None branch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The previous approach tried to inject litellm_credential_name as a tag
in get_standard_logging_object_payload, but the credential name was never
available in litellm_params because the Logging object is created by the
proxy BEFORE the Router selects a deployment. The credential name only
exists in the deployment's litellm_params, which is resolved later.
This fix injects the credential name as a tag in
Router._update_kwargs_with_deployment(), right alongside the existing
deployment-level tags mechanism. This ensures the credential name flows
through the normal metadata.tags pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- RC1: Override _remove_key() in LLMClientCache to schedule aclose() on
evicted async clients instead of relying on GC
- RC2: Use passed connection_pool for URL configs instead of creating an
orphaned pool via from_url()
- RC3: Pass max_connections through to BlockingConnectionPool.from_url()
for URL configs, with input validation for invalid values
- RC5: Close sync redis_client in disconnect() with try/except guard
When a model has a litellm_credential_name, append it to request_tags during
logging so it flows into DailyTagSpend and becomes filterable in the Usage page.
- Add litellm_credential_name to _OPTIONAL_KWARGS_KEYS so it survives into
litellm_params during get_litellm_params() filtering
- Read credential name from litellm_params in get_standard_logging_object_payload()
and append to request_tags if not already present
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(responses): fix O(n²) CPU overhead in reasoning streaming path
stream_chunk_builder was called on every reasoning chunk, rebuilding the
entire response from all collected chunks each time. Replace with
incremental accumulation of reasoning_content parts, only joining at
reasoning end.
* fix(responses): eliminate per-chunk thread spawning in async streaming path
_process_chunk() called run_async_function() on every SSE chunk, which
when invoked from an async context spawns a thread + event loop per call.
Move the hook call out of _process_chunk into the callers: async __anext__
directly awaits it, sync __next__ uses run_async_function.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* perf: reduce responses streaming CPU for text-only streams
* fix(test): replace deprecated claude-3-7-sonnet-latest in responses API test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(test): replace deprecated claude-3-7-sonnet-latest in tool result fix test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(test): replace deprecated claude-3-7-sonnet-latest in tool result empty call_id test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When backend filters (Key Alias, Key Hash, etc.) were active, the main
logs query still refetched whenever startTime/endTime/sort/page changed,
firing a redundant unfiltered server request whose result was discarded.
Expose hasBackendFilters from useLogFilterLogic and use it to gate the
main query's enabled condition.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove fetchKeyHashForAlias: Key Alias filtering is handled server-side
by performSearch via key_alias; translating the alias to api_key hash
caused a duplicate main-query request alongside performSearch's request.
The effect now sets selectedKeyHash = filters["Key Hash"] || "" directly.
- Add setCurrentPage(1) to quick select time range handler so the page
resets to 1 when the user picks a preset time window (was keeping
the previous page number, e.g. page=4, in the API request).
- Add comments explaining the intentionally omitted react-hooks/exhaustive-deps
in the performSearch effect per Greptile review feedback.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Fix two bugs in the logs table with backend filters (e.g., Key Alias):
1. Bug 1 - Table doesn't update with custom time range: When Key Alias filter was active and user selected a custom time range, the main query would refetch (network request visible) but backendFilteredLogs would stay stale because the performSearch effect only watched [sortBy, sortOrder, currentPage]. Added startTime, endTime, isCustomDate to the effect deps.
2. Bug 2 - Pagination shows wrong results: fetchKeyHashForAlias incorrectly had currentPage (log page) in its deps, causing it to search the wrong page of the key list and trigger unnecessary effect re-runs. Removed currentPage from deps and always pass page 1 for key alias lookup.
Also added debouncedSearch.cancel() in the effect to prevent race conditions when pagination happens within 300ms of filter application.
Added tests verifying that time range changes trigger refetch when backend filters are active.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
