agentleFS
dawn840705dawn840705/codex-code-studios/skills/regression-suite/SKILL.md

regression-suite

Skill17 starsChanged 2 months agoMITView on GitHub

Map test coverage to GDD critical paths, identify fixed bugs without regression tests, flag coverage drift from new features, and maintain tests/regression-suite.md. Run after implementing a bug fix or before a release gate.

We split this skill into the rules it gives an agent

11 rules. 6 are common, 0 are distinctive, and 5 are very specific to this project.

  • 6

    Common

    Most of the 9 edited versions we read keep these.

  • 0

    Distinctive

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Common:

    After fixing a bug (confirm a regression test was written or identify gap)

    line 30
  2. Very specific:

    Before a release gate ($gate-check polish requires regression suite exists)

    line 31
  3. Very specific:

    No argument — run update if a sprint is active, else audit

    line 45
  4. Very specific:

    Read tests/regression-suite.md if it exists.

    line 53
  5. Common:

    If it does not exist: note "No regression suite found — will create one."

    line 58
  6. Common:

    Do not read test file contents unless needed for name-to-test mapping.

    line 70
  7. Common:

    Check for drift indicators:

    line 136
  8. Very specific:

    Ask: "May I write/update tests/regression-suite.md with the current regression suite manifest?"

    line 226
  9. Common:

    After writing (if approved):

    line 234
  10. Common:

    If bug regression gaps > 0: "These bugs can silently return without regression tests.

    line 238
  11. Very specific:

    If coverage drift detected: "Regression suite may be drifting.

    line 240

“Common” is measured: kept by at least 6 in 10 of the 9 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Track check — this skill is game-framed. Resolve production/track.txt (or the session's PROJECT_TYPE line) before Phase 0.

  • game — run as written.
  • product (web / mobile / service) — substitute as you read: player becomes user, game becomes product, GDD becomes PRD (design/gdd/ → product/prd/), engine becomes the stack pinned in .codex/studio/technical-preferences.md. Map coverage to the PRD's critical user journeys, not GDD critical paths.
  • Unresolved — ask which track this is before doing anything. A greenfield project has no signal either way; do not infer one from the repository contents.

Regression Suite

This skill ensures that every bug fix is backed by a test that would have caught the original bug — and that the regression suite stays current as the game evolves. It also detects when new features have been added without corresponding regression coverage.

A regression suite is not a new test category — it is a curated list of tests already in tests/ that collectively cover the game's critical paths and known failure points. This skill maintains that list.

Output: tests/regression-suite.md

When to run:

  • After fixing a bug (confirm a regression test was written or identify gap)
  • Before a release gate ($gate-check polish requires regression suite exists)
  • As part of sprint close to detect coverage drift

1. Parse Arguments

Modes:

  • $regression-suite update — scan new bug fixes this sprint and check for regression test presence; add new tests to the suite manifest
  • $regression-suite audit — full audit of all GDD critical paths vs. existing test coverage; flag paths with no regression test
  • $regression-suite report — read-only status report (no writes); suitable for sprint reviews
  • No argument — run update if a sprint is active, else audit

2. Load Context

Step 2a — Load existing regression suite

Read tests/regression-suite.md if it exists. Extract:

  • Total registered regression tests
  • Last updated date
  • Any tests flagged as STALE or QUARANTINED

If it does not exist: note "No regression suite found — will create one."

Step 2b — Load test inventory

Glob all test files:

tests/unit/**/*_test.*
tests/integration/**/*_test.*
tests/regression/**/*

For each file, note the system (from directory path) and file name. Do not read test file contents unless needed for name-to-test mapping.

Step 2c — Load GDD critical paths

For audit mode: read design/gdd/systems-index.md to get all systems. For each MVP-tier system, read its GDD and extract:

  • Acceptance Criteria (these define the critical paths)
  • Formulas section (formulas must have regression tests)
  • Edge Cases section (known edge cases should have regression tests)

For update mode: skip full GDD scan. Instead read the current sprint plan and story files to find stories with Status: Complete this sprint.

Step 2d — Load closed bugs

Glob production/qa/bugs/*.md and filter for bugs with a Status: Closed or Status: Fixed field. Note:

  • Which story or system the bug was in
  • Whether a regression test was mentioned in the fix description

3. Map Coverage — Critical Paths

For audit mode only:

For each GDD acceptance criterion, determine whether a test exists:

  1. Grep tests/unit/[system]/ and tests/integration/[system]/ for file names and function names related to the criterion's key noun/verb
  2. Assign coverage:
StatusMeaning
COVEREDA test file exists that targets this criterion's logic
PARTIALA test exists but doesn't cover all cases (e.g. happy path only)
MISSINGNo test found for this critical path
EXEMPTVisual/Feel or UI criterion — not automatable by design
  1. Elevate MISSING items that correspond to formulas or state machines to HIGH PRIORITY gap — these are the most likely regression sources.

4. Map Coverage — Fixed Bugs

For each closed bug:

  1. Extract the system slug from the bug's metadata
  2. Grep tests/unit/[system]/ and tests/integration/[system]/ for a test that references the bug ID or the specific failure scenario
  3. Assign:
    • HAS REGRESSION TEST — a test was found that would catch this bug
    • MISSING REGRESSION TEST — bug was fixed but no test guards against recurrence

For MISSING REGRESSION TEST items:

  • Flag them as regression gaps
  • Suggest the test file path: tests/unit/[system]/[bug-slug]_regression_test.[ext]
  • Note: "Without this test, this bug can silently return in a future sprint."

5. Detect Coverage Drift

Coverage drift occurs when the game grows but the regression suite doesn't.

Check for drift indicators:

  • Stories completed this sprint with no corresponding test files in tests/
  • New systems added to systems-index.md since the last regression-suite update
  • GDD sections added or revised since the regression suite was last updated (use Grep on GDD file modification hints if available, or ask the user)
  • tests/regression-suite.md last-updated date vs. current date — if gap > 2 sprints, flag as likely stale

6. Generate Report and Suite Manifest

Report format (in conversation)

## Regression Suite Status

**Mode**: [update | audit | report]
**Existing registered tests**: [N]
**Test files scanned**: [N]

### Critical Path Coverage (audit mode only)
| System | Total ACs | Covered | Partial | Missing | Exempt |
|--------|-----------|---------|---------|---------|--------|
| [name] | [N] | [N] | [N] | [N] | [N] |

**Coverage rate (non-exempt)**: [N]%

### Bug Regression Coverage
| Bug ID | System | Severity | Has Regression Test? |
|--------|--------|----------|----------------------|
| BUG-NNN | [system] | S[N] | YES / NO ⚠ |

**Bugs without regression tests**: [N]

### Coverage Drift Indicators
[List new systems or stories with no test coverage, or "None detected."]

### Recommended New Regression Tests
| Priority | System | Suggested Test File | Covers |
|----------|--------|---------------------|--------|
| HIGH | [system] | `tests/unit/[system]/[slug]_regression_test.[ext]` | BUG-NNN / AC-[N] |
| MEDIUM | [system] | `tests/unit/[system]/[slug]_test.[ext]` | [criterion] |

Suite manifest format (tests/regression-suite.md)

The manifest is a curated index — not the tests themselves, but a registry of which tests should always pass before a release:

# Regression Suite Manifest

> Last Updated: [date]
> Total registered tests: [N]
> Coverage: [N]% of GDD critical paths

## How to run

[Engine-specific command to run all regression tests]

## Registered Regression Tests

### [System Name]

| Test File | Test Function (if known) | Covers | Added |
|-----------|--------------------------|--------|-------|
| `tests/unit/[system]/[file]_test.[ext]` | `test_[scenario]` | AC-N / BUG-NNN | [date] |

## Known Gaps

Tests that should exist but don't yet:

| Priority | System | Suggested Path | Covers | Reason Not Yet Written |
|----------|--------|----------------|--------|------------------------|
| HIGH | [system] | `tests/unit/[system]/[path]` | BUG-NNN | Bug fixed without test |

## Quarantined Tests

Tests that are flaky or disabled (do not run in CI):

| Test File | Function | Reason | Quarantined Since |
|-----------|----------|--------|-------------------|
| (none) | | | |

7. Write Output

Ask: "May I write/update tests/regression-suite.md with the current regression suite manifest?"

For update mode: append new entries; never remove existing entries (use Edit with targeted insertions). For audit mode: rewrite the full manifest with updated coverage data. For report mode: do not write anything.

After writing (if approved):

  • For each HIGH priority gap: "Consider creating the missing regression test before the next sprint. Run $test-helpers to scaffold the test file."
  • If bug regression gaps > 0: "These bugs can silently return without regression tests. The next sprint should include a story to write the missing tests."
  • If coverage drift detected: "Regression suite may be drifting. Consider running $regression-suite audit at the next sprint boundary."

Verdict: COMPLETE — regression suite updated. (If user declined write: Verdict: BLOCKED.)


Collaborative Protocol

  • Never remove existing regression tests from the manifest without explicit user approval — removing a test that was deliberately written is a regression risk itself
  • Gaps are advisory, not blocking — surface them clearly but do not prevent other work from proceeding (except at release gate where regression suite is required)
  • Quarantine is not deletion — tests with intermittent failures should be quarantined (noted in manifest) but not removed; they should be fixed by $test-flakiness
  • Ask before writing — always confirm before creating or updating the manifest

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.