agentleFS
alfredoperezalfredoperez/speckit-companion/.claude/skills/eval-speckit-extension/SKILL.md

eval-speckit-extension

Skill105 starsChanged 29 days agoMITView on GitHub

Evaluate the speckit-extension end-to-end against a real spec — verify lifecycle hooks fired, .spec-context.json captured each step into canonical history[], timing is real (not backfilled), and per-task execution was journaled. Umbrella eval for the spec-kit-side extension; grows as features land (auto-mode, etc.). Use when the user says "/eval-speckit-extension", "verify the extension worked", "did the hooks fire", "check the capture", or just ran a /speckit.* spec and wants it validated.

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 1 is distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If the user named a spec (NNN or a dir), use specs/<that>/.

    line 20
  2. Very specific:

    Read the PASS/FAIL/INFO rows and the timing breakdown.

    line 29
  3. Distinctive:

    When a feature ships:

    line 66
  4. Very specific:

    add a deterministic assertion to check_capture.py (run_checks) and, if it's a new area, a new eval section + assumption block above,

    line 67
  5. Very specific:

    keep VALID_BY / CANONICAL_STEPS / CANONICAL_STATUSES in check_capture.py in sync with apps/vscode/src/core/types/spec-context.schema.json …

    line 68

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Eval: speckit-extension

The regression net for the speckit-extension (the spec-kit-side half of SpecKit Companion). It verifies the assumptions behind each feature against a real spec the user created, combining a deterministic file checker with a conversational cross-check of what the user actually did this session. Re-run it whenever a lifecycle/auto-mode/future feature ships.

Eval areas (add a section per feature as the extension grows):

  1. Lifecycle capture (shipped) — hooks → write-context.py → canonical history[]; per-task journaling; derive-from-files fallback.
  2. Command quality (shipped) — check_quality.py: verbosity budgets on the spec artifacts, time-waste checks on history[] (untrusted spans, bursts, outliers), static prompting checks on the command-body sources.
  3. Auto-mode (future) — add checks here when it lands.
  4. …

Steps

1. Resolve the target spec

  • If the user named a spec (NNN or a dir), use specs/<that>/.
  • Else pick the spec the user most recently created/ran in this conversation (whose /speckit.* or /sdd:* commands appear above). Fall back to the most recently modified specs/*/ containing .spec-context.json.

2. Run the deterministic checker

python3 apps/speckit-extension/scripts/check_capture.py specs/<NNN>-<slug>/

Read the PASS/FAIL/INFO rows and the timing breakdown. (--json for machine output; --strict to exit non-zero on any FAIL.)

2b. Run the command-quality checker

python3 apps/speckit-extension/scripts/check_quality.py --feature-dir specs/<NNN>-<slug> --commands-dir apps/speckit-extension/commands

Same report shape plus a WARN tier (judgment calls — budgets, duration outliers, untrusted spans; never affects --strict, which fails only on FAIL). Three dimensions: verbosity (spec/plan/tasks against line/char WARN/FAIL bands calibrated on the completed specs 484/509/510 — only oversize flags; a missing or lean artifact is fine), time-waste (each reached step's span trusted only with ordered extension-stamped boundaries; ≥3 by:ai task finishes inside one second FAIL as the pre-#509 burst shape; a step >8× the median of the others and >5m WARNs), and prompting (never-halts commands — the after-* hooks, living-drift/sync/coverage, mark-complete, status, resume, classify — must contain no ask-the-user instruction; the clarify carrier must ask; negated mentions and fenced templates don't count; a missing roster file fails loudly). CI runs all three strict over specs/509-timing-capture, specs/510-living-sync, and the shipped command sources.

3. Conversational cross-check

The checker can't know what the user did. Reconstruct it from this conversation and reconcile against the capture:

  • List every /speckit.specify|plan|tasks|implement (or /sdd:*) the user ran this session, with rough timestamps.
  • For each, confirm a matching history[] entry exists with the expected step/status and by: "extension" (hook fired) — or by: "ai" if the AI appended it, by: "derive" if reconstructed. (The canonical by vocabulary is extension/user/cli/ai/derive.)
  • Timing reality: does each capture's at land just after the command ran (live hook), or are timestamps clustered/round (backfilled)? The timestamps-real check flags round-ms; you add the "fired at the right moment" judgement.
  • Task execution: if /speckit.implement ran, confirm per-task substep entries appear. Read task-cadence's source: live (by:ai) means the AI journaled each task as it finished — non-zero gaps are the real-cadence signal; hook burst (by:extension) means the single end-of-step hook synced them — 0ms gaps are expected there and not a defect (the AI didn't journal live, so the hook backstopped). Only a mixed or duplicated picture is a smell.

4. Score the assumptions

Render a verdict table (PASS / PARTIAL / FAIL + one-line evidence):

#Assumption (lifecycle capture)
A1Each lifecycle step the user ran produced a capture (after_specify/plan/tasks/implement → a history[] entry).
A2Writes use canonical history[] with explicit kind — no legacy transitions[]/stepHistory.
A3Timing is real for deterministic writes (by:extension/derive/cli/user): ms-precision and monotonic (timestamps-real/timestamps-monotonic check these only). by:ai entries carry second precision (date -u +%SZ) and may burst — that's graded by task-cadence, not failed.
A4/speckit.implement journaled per-task progress as implement substeps (substep == task id), matching tasks.md completed markers.
A5No-backward-clobber held — no step regressed; an advanced/terminal spec was never dragged back.
A6(On demand) derive-from-files.py reconstructs the same state from artifacts when a hook didn't fire. Test: back up .spec-context.json, delete it, run python3 apps/speckit-extension/scripts/derive-from-files.py --feature-dir specs/<NNN>-<slug>, diff, restore.
A7Per-task journaling is deduped — each task carries one start + one complete, so dedup is checked per (task, kind) (per-task-no-duplicates); a repeated (task, kind) means the after_implement hook re-added an already-journaled entry. Live AI entries (by: ai, real date -u timing) are the cadence source. Real per-task cadence requires task-cadence source live (by:ai) with non-zero gaps; a hook burst source means the AI didn't journal live and the hook backstopped (correct final state, coarse timing — not a defect).

End with a one-paragraph plain verdict: did the extension work, and where reality diverged from the assumptions.

Extending

When a feature ships:

  • add a deterministic assertion to check_capture.py (run_checks) and, if it's a new area, a new eval section + assumption block above,
  • keep VALID_BY / CANONICAL_STEPS / CANONICAL_STATUSES in check_capture.py in sync with apps/vscode/src/core/types/spec-context.schema.json (the jest drift guard locks schema↔TS; this is the eval side).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.