Scan Regression Candidates

Months of manual QA leave hundreds of throwaway scripts, evidence folders and logs in the scratch root. Some of them describe flows the team re-tests by hand every release, or bugs that came back after being fixed — those are worth a permanent regression test. Most are probes, one-off checks and investigations that would only add noise to a suite.

This skill reads the scratch root (never writes to it), groups what it finds into scenarios, scores each with the same criteria promote-to-regression uses, checks the committed suite for existing coverage, and returns a short ranked backlog. It does not write tests: the user picks candidates, and promote-to-regression promotes them one at a time.

Step 0 — Triage & delegation (mandatory)

This skill is a router. Main conversation: resolve the inputs below (ask only for what you cannot infer), dispatch one scanner subagent (deep tier — it is judgement over many weak signals), then show its report and ask which candidates to promote. Without subagents, run Steps 1–4 yourself.

Inputs to resolve before dispatching:

• Scratch root — default _local/ in the current project. Several are fine (web and mobile).
• Committed suite — where the regression specs live (read the Playwright config's testDir), to check coverage. Without it, every candidate is marked "coverage not checked".
• Ticket pattern — how ticket ids appear in file and folder names (e.g. sc-?\d{5}, PROJ-\d+). Infer it from a listing of the scripts folder; the default catches ABC-123 and sc123. If the prefix is optional in practice (_manual-1042-…), also pass an id range (--id-range 1000-1999) so record ids, timestamps and ports in logs are not read as tickets.
• Window (optional) — only runs after a date (--since).
• Output — default <scratch root>/regression-candidates/<date>.md, a new folder — or any folder outside the scratch root the user prefers. That is the only thing this skill writes; everything already in the scratch root is read-only. Other tools may still be writing into the scratch root while you scan: note the inventory's timestamp in the report.

Scanner brief:

Read <absolute path>/scan-regression-candidates/SKILL.md and run Steps 1–4. Read-only on the scratch
root and the suite. Scratch root: <path>. Suite: <path>. Ticket pattern: <regex>. Window: <…>.
Write the report to <output path>. Do not ask anything; return the report's path and its summary.

Step 1 — Inventory (mechanical)

Run the bundled script from the project root:

node <this skill's dir>/scripts/inventory.mjs <scratch root> --ticket "<regex>" [--id-range <min>-<max>] [--since <date>] --top 120 --out <scratch root>/regression-candidates/inventory.json

Groups are keyed by the ticket number, so two projects whose keys share a number (ABC-12, XYZ-12) merge — pass a stricter --ticket when a scratch root mixes projects. Free text it keeps (test titles, step names) is redacted (emails, phones, tokens, long numbers) and paths are collapsed to routes; review the inventory before sharing it anyway.

It groups scripts, evidence folders, logs and reports by ticket — taken from any path component, so scripts/ABC-1042-retest/recon.spec.ts belongs to #1042 — or by name stem, and records per group: dates and active days, number of specs vs. probe scripts, evidence folders, verdicts found in manual-test's saved results (_local/reports/), reports, comment drafts and ledgers — counted once per file, with the group's latest_verdict (value, date, file, reviewed) and latest_reviewed_verdict — a saved result's reviewer_verdict is the reviewed one — so "a later PASS after the FAIL" can be read off directly — test titles, test.step names, visited paths (origin and query string removed, ids collapsed), expect count and red flags (hard waits, force clicks). It never reads folders named secrets, state, .auth, worktrees, node_modules, test-results, playwright-report*, allure-*, nor files whose name suggests credentials or browser state — and it extracts no values from scripts beyond titles, step names and paths.

Dates. A date in the path wins over the file's modification time; dates_from_names says how many of a group's dates came from names. File times are unreliable after bulk copies or touches — when most dates are mtimes and many groups share the same day, do not trust active_days as recurrence; count distinct re-test runs (retest / pass-2 folders, dated evidence) instead.

Look for a tracker export in the scratch root (a JSON / CSV of stories with titles, types and workflow states). With no tracker access, it is the best answer to "what is this ticket" and "was the bug fixed": a bug in a done/completed state plus a later PASS re-test is a fixed bug.

Privacy rule for the rest of the scan: you may open specs, reports and run notes to understand a group, but never open storageState/auth files, .env*, token files, or anything under the skipped folders, and never copy credentials, tokens, emails or personal data into the report.

Step 2 — Understand the strong groups

Work from the inventory, not from a full read of the folder. For each group that is not obviously noise (see below), read just enough to answer:

• What user-meaningful outcome was being checked? (from titles, steps, paths; open the main spec and the report/notes if any — not every script.)
• What happened? Accepted PASS, a bug found (and was it fixed and re-tested?), or only probes.
• How often did this flow come back — other tickets touching the same paths, re-tests, passes 2/3?
• Is it covered? Search the committed suite for the route and the outcome (grep the paths, feature words, page-object names). Partial coverage → "extend <spec>", not a new test.

Fold groups that test the same flow (same paths and outcome across tickets) into one candidate.

Obvious noise — count by category, do not analyse one by one:

• groups with only probe/recon/diag/annotate/crop scripts;
• login / session / seeding helpers, configs, shared libs (lib, config, global-setup);
• data or API investigations with no UI outcome (unless the suite also covers APIs);
• runs with no accepted result (BLOCKED / INCONCLUSIVE / PO-DECISION only);
• groups seen once, on one day, with a narrow, ticket-specific assertion (copy of a banner, one pixel layout at one width);
• a different surface than the suite — mobile app, an admin tool, infrastructure or monitoring config — unless the suite has a harness for it;
• suite-authoring sweeps — the runs that produced specs already committed (folders named after a suite area or a "regression"/"recon" sweep): they are covered by definition.

Runs older than saved reviews. Scratch roots that predate <scratch root>/reports/ have no reviewer sign-off. Treat a run as reviewed when there is a reviewer-style artefact (a review file, a second-pass verdict) or the ticket moved past QA in the tracker or its export — and mark it review: inferred in the report. A PASS with neither stays MAYBE.

Recurrence vs. bug re-tests. Re-tests of a bug count toward bug history, not recurrence; recurrence is feature passes, release / regression sweeps and other tickets exercising the same flow.

An open question on one step only. Drop that step from the candidate (which can stay PROMOTE) and list it as its own MAYBE with the question.

--top cuts. The inventory lists dropped_ticket_groups: if any look relevant, re-run with a higher --top instead of ignoring them.

Step 3 — Score

Score each remaining candidate with ../promote-to-regression/references/promotion-criteria.md (installed next to this skill; if it is missing, stop and say so — do not improvise a rubric): value V/7 · cost C/7, one line per non-zero criterion, then PROMOTE / MAYBE / SKIP.

Be strict. The point of the report is a short list the user can trust. A backlog of 40 "PROMOTE" items is a failed scan: if more than ~10 survive, raise the bar (higher value first, then lower cost) and move the rest to MAYBE with a reason.

Step 4 — Report

Write the report (English, Markdown) with:

1. Summary — scratch root(s), window, groups found, how many were noise (by category, with counts), how many scored, how many PROMOTE / MAYBE.
2. PROMOTE — ranked. For each: a one-line title of the outcome; tickets; the evidence it rests on (spec paths, report verdicts, dates); the score with its per-criterion lines; existing coverage ("none found in <suite>" / "extend <spec>"); the shape of the test (one paragraph: entry, the steps, the key assertions — from the ACs or product rules, never from the screen); what it needs (fixture user, seeding helper, page object) and open questions.
3. MAYBE — one line each: what is missing for it to become PROMOTE.
4. SKIP highlights — only the non-obvious ones (a group that looks important but is covered, or too flaky), one line each.
5. Limits — what could not be checked (no tracker access to confirm a bug was fixed, suite not searchable, scripts unreadable).

Return to the orchestrator: the report path and a ≤ 10-line summary.

After the report (main conversation)

Show the summary and the PROMOTE list, and ask which candidates (if any) to promote. For each one the user picks, run promote-to-regression with that candidate — one at a time; it has its own gates before anything is written to the suite or committed.

Rules

• Read-only. Never modify, move or delete anything in the scratch root or the suite; the only write is the new regression-candidates/ report folder (or the output path the user gave).
• No secrets, no personal data — not read, not copied, not summarised.
• Strong over many. Few, well-argued candidates beat a long list. "Tested once" is not a reason.
• Expected behaviour comes from tickets and product rules. If a candidate's expected values can only be taken from what a manual run saw, it is at most MAYBE ("needs the AC").
• Say what you could not verify — especially "bug fixed?" and "already covered?".