regen-eval-baseline

Captures a promptfoo run for one sumo-qa skill and stores its JSON output in docs/qa/runs/eval-baselines/ (gitignored). The deterministic work lives in scripts/run_baseline.py; this document is the guide for picking inputs and reading the output.

When to use

Trigger this skill when the user wants a per-skill eval snapshot. Common phrasings: "baseline this skill", "snapshot the eval", "run the eval for skill X", "capture before/after for the rewrite I just made". The user invokes it explicitly with /regen-eval-baseline; it doesn't auto-trigger.

This is single-skill on purpose. Full-sweep regeneration belongs on npm run eval:all, which runs the same tests/evals/promptfoo/skill-*.yaml files on the Claude pair sequentially without snapshotting.

Inputs

Pass exactly one config selector — either --skill (the base config) or --config (a suffixed scenario / .ab.yaml control / explicit path) — plus an optional --label.

1. --skill <name> — the base config tests/evals/promptfoo/skill-<name>.yaml. Resolves that file exactly; it never cross-matches a longer suffixed sibling (e.g. --skill reviewing-before-merge drives skill-reviewing-before-merge.yaml, NOT skill-reviewing-before-merge-adversarial.yaml). If the named config is absent the script lists the available configs.
2. --config <selector> — for a suffixed scenario config or an .ab.yaml A/B control. Accepts a bare stem (skill-reviewing-before-merge-adversarial), a filename including the double suffix (skill-reviewing-before-merge-adversarial.ab.yaml), or a full/relative path. Resolved exactly — a stem composes one filename and that file must exist; the resolver never falls back to a near-named neighbour, so a base name can't accidentally snapshot a suffixed sibling and vice-versa.
3. --label — short kebab-case tag for the snapshot, defaults to baseline. Past conventions in the dir: baseline, postcut, greenfix, or describe-the-change like removability-gate. The label vocabulary is open-ended and may itself be multi-hyphen, so the snapshot filename separates the slug from the label with a literal __ (double underscore) — <date>-skill-<slug>__<label>.json. Because a validated kebab slug/label can never contain __, the slug/label boundary stays unambiguous even for a multi-hyphen label. The label is what makes two snapshots taken on the same day distinguishable.

The snapshot filename derives from the resolved config, not just --skill: a .ab infix becomes a hyphen in the slug (skill-x-adversarial.ab.yaml → …-skill-x-adversarial-ab__<label>.json), so the base config, its suffixed scenario, and its .ab control each snapshot to a distinct, non-colliding path and diff only against their own prior snapshots.

Wrapper vs raw promptfoo eval -c

Use this wrapper (--skill / --config) whenever you want the repeatable before/after snapshot it exists to provide: it writes the dated JSON to docs/qa/runs/eval-baselines/, prints pass/fail, and diffs against the prior snapshot for the same config. That covers base configs, suffixed scenario configs, and .ab.yaml controls — drive all three through the wrapper so the baseline/postcut delta is preserved (don't hand-roll npx promptfoo eval -c … and lose the snapshot). Reach for raw promptfoo eval -c <path> only for flags the wrapper does not expose (e.g. --repeat N variance runs, -j 1 legible logs, generate dataset) — see tests/evals/promptfoo/README.md. For a normal baseline/postcut capture, the wrapper is the supported path.

Prerequisites the script will check

• The claude CLI must be on PATH and signed in. Every config pins the Claude eval pair (providers/claude-candidate.yaml + providers/claude-judge.yaml), which runs through claude -p on the user's Claude subscription, so no API key is involved. If the script reports the CLI missing, tell the user to install or sign in to Claude Code.
• The selected config must exist: tests/evals/promptfoo/skill-<name>.yaml for --skill, or the exact suffixed / .ab.yaml config for --config. If it doesn't, the script lists every available config so the user can pick again — it does not guess a near-named sibling.
• A snapshot at the target path already existing will block the run unless --force is passed. Don't pass --force reflexively — snapshots are evidence of past runs and silently clobbering them loses history.

Run the script

# Base config:
python3 .claude/skills/regen-eval-baseline/scripts/run_baseline.py \
  --skill <skill-name> \
  --label <label>

# Suffixed scenario config (no rename to the base skill needed):
python3 .claude/skills/regen-eval-baseline/scripts/run_baseline.py \
  --config skill-reviewing-before-merge-adversarial \
  --label baseline

# .ab.yaml A/B control:
python3 .claude/skills/regen-eval-baseline/scripts/run_baseline.py \
  --config skill-reviewing-before-merge-adversarial.ab.yaml \
  --label postcut

The script:

1. Computes the snapshot path: docs/qa/runs/eval-baselines/<today>-skill-<slug>__<label>.json (the slug comes from the resolved config — see Inputs; __ separates slug from label so a multi-hyphen label stays unambiguous).
2. Runs npx promptfoo eval with --no-cache (so the snapshot reflects fresh judge calls, not stale cache hits), writing the JSON output to a scratch file of the same name in a .partial/ subdirectory of the baselines folder (promptfoo only accepts an output path with a known format extension such as .json). If the run is interrupted (Ctrl-C), the scratch file is removed and nothing is captured.
3. Checks the report records a verdict: results.stats present and results.results a non-empty list. A report without them (for example {}) means no test case ran: a harness or config error, not a skill verdict and not a provider error, the same shape run-eval.sh exits 4 on. The script moves it aside to <snapshot>.rejected, prints no pass/fail summary and no delta, leaves every earlier snapshot untouched, and exits 4. A run where promptfoo wrote no report, or a report that is not JSON, is the same harness or config error: nothing is captured and the script exits 4, not promptfoo's own exit code. Check the config and its YAML in the promptfoo output, then re-run.
4. Checks the report for provider or judge errors (stats.errors > 0, or a judge result tagged metadata.graderError). A run with any is not a skill verdict: the script moves the report aside to <snapshot>.rejected, prints no pass/fail summary and no delta, leaves every earlier snapshot untouched, and exits 3. Resolve the error the report records (a Claude usage limit or a CLI failure) and re-run; do not hand a rejected report to the eval-failure-diagnoser.
5. Otherwise moves the report onto the snapshot path and prints pass/fail counts.
6. If a prior snapshot for the same config exists, prints a delta: passed and failed counts vs the previous run.

Reading the output

The pass/fail summary tells you the state of the snapshot. The delta tells you whether the most recent SKILL.md edit moved the needle. Three patterns to watch for:

• Passes increased, failures decreased — the edit helped. Keep it.
• Passes decreased, failures increased — the edit hurt. Investigate before reverting; the failure may be informative.
• No change — either the edit was outside the assertions' coverage, or the judge gave the same verdict on different reasoning. Read the JSON's per-test reasons before drawing conclusions.

When FAILs appear

Don't propose loosening the rubric. The standing repo policy (feedback_eval_fixes_target_skill_not_rubric) is to strengthen the SKILL.md so the candidate naturally satisfies the existing rubric. Hand the failing snapshot to the eval-failure-diagnoser subagent — it reads the JSON, locates the failing assertion, and recommends a concrete SKILL.md edit. Re-run this skill after the edit to confirm the failure flipped.

Why the snapshots are gitignored

docs/qa/ is excluded from the repo (.gitignore line 48). Eval snapshots are local evidence — useful for the contributor doing the optimisation work, but not artefact that ships with the package. Per feedback_no_process_artifacts_in_public_repo, results live alongside the QA work, not in the public source tree. Don't propose committing snapshots or removing the gitignore entry.