1 Commits

Author	SHA1	Message	Date
ryan-crabbe-berri	be84d5cd7d	ci: add manually-triggered mutation testing workflow (#27576) ... * ci: add manually-triggered mutation testing smoke workflow Adds a workflow_dispatch-only GitHub Actions workflow that runs mutmut against a single source/test pair (router_settings_endpoints) to validate the tooling end-to-end before scaling. The workflow reinstalls litellm non-editable so the mutants/ sandbox is not shadowed by the editable .pth on sys.path, and sets PYTHONPATH so the trampolined sandbox copy wins over site-packages. mutmut itself is pulled in via uv run --with so it does not appear in uv.lock or affect the shared dev environment. Includes a temporary push: trigger scoped to this branch so we can iterate before the workflow file lands on the default branch — to be removed before merging (workflow_dispatch only requires the file on the default branch to surface the manual trigger button). * ci(mutation): disable rerun and xdist plugins for mutmut runs mutmut's in-process pytest.main() call hits `INTERNALERROR: no option named 'filtered_exceptions'` from pytest-retry's pytest_configure hook. Reruns are also wrong for mutation testing — a "failed" mutant test that gets retried would mask which mutants are killed vs. survive. Disable retry, rerunfailures, and xdist via pytest_add_cli_args in [tool.mutmut]. * ci(mutation): uninstall pytest-retry before mutmut runs `-p no:retry` (and similar names) didn't match pytest-retry's entry-point name, so the plugin still loaded and crashed during mutmut's "Running clean tests" phase. Uninstalling the package is surgical and doesn't depend on guessing the entry-point name. * ci(mutation): emit per-survivor diffs to run-page summary + artifact The previous artifact only contained `mutmut results` text (which in mutmut 3.x lists survivor names but not the actual mutations). Adds: - `mutmut export-cicd-stats` to produce mutmut-cicd-stats.json with the killed/survived/total scoreboard. - `mutmut show <name>` per surviving mutant to capture each mutation as a unified diff. - A `mutmut-report.md` that combines summary + run-progress tail + per-survivor diffs, written to both the artifact and $GITHUB_STEP_SUMMARY (visible on the run page, no download needed). - Corrected artifact paths: stats files live under mutants/, not the project root. - The trampolined source file from the sandbox so survivors can be inspected even outside `mutmut show`. * ci(mutation): document intended manual weekly cadence in trigger comment * ci(mutation): generate ACH-style report with embedded function bodies Replaces the inline bash markdown generation with a Python script that: - Groups survivors by function (one section per function, function body shown once per section, surviving mutants nested as subsections) - Embeds each enclosing function's source via Python AST (so the agent has full context, not just a 3-line `mutmut show` diff) - Inlines the existing test file(s) listed in [tool.mutmut].tests_dir - Writes an ACH-style task description at the bottom following the prompt template from arXiv 2501.12862 Output goes to mutation-report.md (artifact) and the head of the file is appended to $GITHUB_STEP_SUMMARY for at-a-glance visibility. * fix(mutation report): correctly parse function names with leading underscores mutmut's mutant-name prefix is x_ (single underscore), so a function named _foo produces mutants x__foo__mutmut_N. The previous regex \.x__(.+)__mutmut_ ate the function's leading underscore as part of the prefix. Changed to \.x_(.+)__mutmut_ so leading underscores are preserved in the captured function name; verified for normal, leading- underscore, and dunder-method names. * feat(mutation report): full Meta ACH-style rendering with MUTANT delimiters For each surviving mutant, parse the mutmut sandbox trampoline file and render the mutated function as it appears in the source — with the differing lines wrapped in `# MUTANT START` / `# MUTANT END` comments, matching the format from Meta's ACH paper (arXiv 2501.12862, Table 1). Renames the function header back to its original name so the agent sees the function as it would appear in the file. Falls back to the unified diff if the trampoline lookup fails. Handles replace, insert, and delete diff ops; uses difflib's SequenceMatcher to find the differing line ranges. The unified diff is preserved in a collapsible <details> block as secondary context. * ci(mutation): scope to whole management_endpoints folder, drop temp push trigger Final scope before merge: - paths_to_mutate / tests_dir broadened from one file to the entire management_endpoints source/test folders - Trigger is now `workflow_dispatch` only — the temporary push: block used during workflow iteration is removed - timeout-minutes bumped from 60 to 350 (just under the GH-hosted job cap of 360); whole-folder mutation against ~15 files / ~7.5k LOC can take a few hours - Artifact path for the trampoline files glob-expanded to cover all files under mutants/litellm/proxy/management_endpoints/ * fix(mutation report): warn when multiple functions in a file share a name Addresses the Greptile review concern: ast.walk's first-match-wins behavior could embed the wrong function body when a file defines the same name in multiple places (e.g., a module-level helper and a class method). mutmut's mutant identifier does not carry class context, so we can't always determine which definition was mutated. find_function_in_file now returns the start line of every matching definition; render() surfaces a "Note: N functions named X" warning in the report when there is more than one match. The first match is still embedded as the body — the warning tells the reader to verify manually instead of silently using the wrong context. Smoke-tested against the existing artifact: single-match files render unchanged. * Fix mutation report anchors * Fix mutation report TOC anchors --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-05-11 15:19:57 -07:00