Session Verify

Produce evidence-based proof that a completed implementation matches its design and plan, or a falsification artifact identifying gaps. A full run is heavy, but existing applicable evidence shortens it.

Open with one sentence saying what you are about to do and what it will produce.

Core Principle

Never assume. Always cite. Every claim in the output must cite a file path + line number, command output, test name, or log excerpt. Statements like "should work", "appears correct", or "likely functions as intended" are forbidden.

The skill treats absence of evidence as failure, not as "probably fine". It runs tests, writes probes, inspects real state, and produces a structured evidence artifact. Fixing defects is out of scope — document them, don't patch.

Non-Negotiables

1. Verify the accepted outcome, not the newest document. The behaviour under verification is the work item's accepted scope, identified by its acceptance fingerprint and by the work-root commit of each document that scope references. A plan file that happens to be newest is not the specification.
2. Evidence-first. Run the probe before writing the verdict. No conclusion without cited evidence.
3. Falsify, don't confirm. For every spec claim, design a test intended to disprove it. A claim only survives if the falsification attempt fails.
4. Run the code. Passing unit tests are necessary but insufficient. Execute real flows, inspect real state, validate real round-trips.
5. Never edit source to make tests pass. New verification scripts belong under _verification/ (or verification/ if the project prefers). Do not modify production code or existing tests to paper over defects.
6. Report defects, don't silently fix. Document each defect with a failing probe or reproducible trace. Escalate to the user; do not patch without explicit authorization.
7. Every task passing is not a delivered outcome. Tasks can pass while the parent still needs integration or delivery. Completion is judged in Completion and Evidence below, against the accepted outcome, and never inferred from task results.
8. Cite on every line. Format: path/to/file:LINE for code, $ command + output for execution, MEMORY #id for memory citations.

Workflow

Step 0: Resolve the Runtime and the Accepted Outcome

Resolve the helper by the one rule for this host (references/runtime-integration.md) and confirm it answers with doctor. Then read the work item under verification:

python3 -B "$ENTRYPOINT" --project-root "$PROJECT_ROOT" show --seq SEQ-042

Take from the record:

• The accepted scope region — the behaviour being verified, and the text the acceptance fingerprint covers.
• acceptance.fingerprint against the record's current scope_fingerprint — when they differ, the accepted scope was edited after acceptance. Stop and report it: verifying against text nobody accepted proves nothing. Re-acceptance or a recorded same-meaning decision comes first.

• references — each behaviour or criteria document as path@commit. Read each one at that work-root commit, not at whatever the file says today:

  git -C "$WORK_ROOT" show <commit>:<path>

• delivery — the delivery target the accepted outcome requires, if any.
• tasks/ — each task's result and lifecycle, which are inputs to the verdict and not the verdict.

When the item has no accepted scope, or there is no work item at all, say so and stop: capture and acceptance come first, through /session-add-task or /session-gatekeeper. There is nothing here to verify against.

Which artifact prevails. The accepted scope region is the specification. spec.md elaborates the behaviour it references and is read at the referenced commit; plan.md carries approach and order and never supplies acceptance criteria. When spec.md and plan.md disagree, the accepted scope decides, and the disagreement is itself a finding.

Step 1: Scope

Collect the remaining inputs through a brief collaborative dialogue. Ask one question at a time. Prefer multiple choice.

Required inputs:

• Work item identity — SEQ-NNN, resolved in Step 0. Everything else hangs off it.
• Scope label — kebab-case, used for artifact naming (e.g. banner-redesign, hybrid-search).
• Memory source (optional) — claude-mem MCP, session transcripts, or PR list that captures implementation history and any flagged defects.

Confirm these inputs before proceeding. Do not guess.

Assess before running. Verification follows one procedure, whatever the size of the item:

1. List the accepted criteria from the scope region (Step 0).
2. Map existing applicable evidence to each criterion: the record's metadata.evidence, delegation's recorded suite run, and earlier logs under _verification/. Evidence applies only when it names the current fingerprint and its code revision still matches.
3. Run only the checks that are missing, failing or stale. Steps 4 to 8 are the checks to choose from. Skip any that existing evidence already covers, and say which.
4. Run the full suite only when no applicable passing full-suite result exists, recorded or reused (see Reusing a result).
5. Record the verdict, with one evidence entry per criterion.

An explicit request for exhaustive verification overrides this procedure: run everything and ignore existing evidence.

Step 2: Scope Matrix

Produce and confirm with the user a 6-row matrix that defines what "working" means for this verification. Present as a table. A row that existing evidence already covers cites that evidence instead of a new run.

The matrix is a contract. You cannot change it mid-verification. If a row does not apply (e.g. backend-only change), mark it N/A with a one-line justification.

Step 3: Required Reading

Before running anything, read:

1. The accepted scope region and each referenced document at its recorded commit (Step 0)
2. The item's plan.md and task records, for what was attempted and in which order
3. The project's architecture index if present (architecture_index.md, ARCHITECTURE.md, or equivalent per .session-flow.json.paths.architecture)
4. The project's CLAUDE.md / AGENTS.md / GEMINI.md for conventions
5. Any testing guide (TESTING.md, _devdocs/guides/TESTING.md) — note if it is stale; outdated testing docs are themselves a finding
6. If memory source provided: query for relevant observations and extract any pre-flagged defects

An item that carries provenance.imported_from is assessed from that import provenance and its breakdown link, both read as history. Rerun whatever that history does not show. A historical completion is never recorded as a fresh pass.

Produce a written hypothesis list — one falsifiable hypothesis per non-trivial claim in the plan. Example: "H1: The SQL wildcard injection flagged in review is still present because commit X did not touch history.py." Hypotheses should be specific enough that you can design a single probe to kill each.

Phase exit criterion: hypothesis list exists, written down, shared with the user.

Step 4: Structural Audit

Walk the claimed module surface with Grep and Read. For each claimed symbol, confirm it exists with the documented signature. Build an inventory table:

| Claimed symbol | File:line | Signature matches? | Notes |
|----------------|-----------|--------------------|-------|

Phase exit criterion: every symbol in the plan has a verdict row. No "I think it's there" entries.

Step 5: Functional Test Execution

Skip the full suite when step 4 of the procedure finds an applicable passing result: cite it and do not rerun. Otherwise detect the project's test runner:

• Check CLAUDE.md / AGENTS.md for test instructions
• Look for scripts/run_tests.sh, scripts/run_tests_wsl.sh, Makefile targets
• Fall back to: pytest / npm test / cargo test / go test / mvn test as appropriate

Run:

1. The full suite, with output captured to _verification/{date}-{label}-tests.log
2. Narrower runs targeting the new modules/packages (list them explicitly)
3. Test-count verification: if the plan claims N tests, run the collector and assert the count. If the delta is non-zero, report it as a finding.
4. Migration replay (if applicable): run any new migration against both a fresh database and a database with prior migrations. Confirm idempotency.

If a test fails, do not fix it. Record the failure, its symptom, and your root-cause hypothesis. Fixing is the user's call, not yours.

Then compare the failing test ids with the project's known-failure baseline, {paths.testing}/known-failures.md, a table of test id, reason, owner and review date (format in references/work-item-contract.md):

• A failure not in the baseline file is a fail entry and blocks completion.
• A baseline test that now passes is reported for removal, and an entry past its review date is reported.
• The evidence entry cites the file as baseline: {path, commit}. A baseline is never recorded as waived, and it never authorizes shipping a blocker: a listed test that fails against the accepted outcome stays fail.

Environment reality: If the project has a WSL/Windows or Linux/macOS split, and a test cannot run in your current environment, mark it DEFERRED-TO-{target} and prepare an executable command for the user to run themselves. Do not silently skip.

Phase exit criterion: test log captured or applicable result cited, test-count delta reported, all failures enumerated with root-cause hypotheses and compared with the baseline.

Step 6: Defect Regression Probes

For each defect flagged in prior reviews (from memory source, PR comments, or the plan itself), write a targeted probe — a short script or pytest test under _verification/probes/ that would fail if the defect were present.

Each probe has exactly one job: falsify one fix claim. Keep probes small (<40 lines each). Run each probe. Record:

• VERIFIED-FIXED — probe runs green, defect is provably absent
• CLAIMED-BUT-NOT-VERIFIED — fix exists in code but probe cannot reach the relevant path (explain why)
• NOT-FIXED — probe reproduces the defect
• REGRESSED — fix existed and has been removed or bypassed

Phase exit criterion: one probe per flagged defect, all executed, all classified.

Step 7: Integration Probes

Live end-to-end checks. Examples (adapt to the project):

• RPC round-trip: spawn the sidecar/server, issue a real RPC call, assert the response shape
• Error paths: call with invalid input, verify well-formed errors (not crashes or silent success)
• Data persistence: write → read → verify round-trip through the real storage backend
• Cross-component flow: emit an event from one module, confirm it lands in the subscriber
• Idempotency: run migrations/initializers twice, verify no duplicate rows or errors
• Frontend compile: tsc --noEmit, ESLint, pnpm build (or project equivalent) on new files only
• Frontend reachability: grep for the new component's mount site; if none, it is dead code → finding

Probes live under _verification/probes/. Each probe captures stdout/stderr and writes a one-line summary to _verification/{date}-{label}-integration.log.

Phase exit criterion: at least one integration probe executed, or each skipped probe has a cited reason + a user-executable alternative.

Step 8: Spec-vs-Reality Gap

For every feature in the design spec, classify:

• IMPLEMENTED — present and verified in prior phases
• PARTIAL — some sub-features work, others don't; list exactly which
• STUB — code exists but does nothing meaningful (e.g. pass, return None, raises NotImplementedError)
• NOT-IMPLEMENTED — no code for it

If the plan has a "Not covered in this plan" or "Out of scope" section, do not flag those as missing. Verify they are genuinely not half-done.

Phase exit criterion: every spec feature has a status + citation.

Step 9: Write Evidence Artifact

Write the final report to _verification/{date}-{label}-verification.md, following the structured output contract below. Fixed sections, fixed headings. No substitutions.

Step 10: Present Verdict

Summarize to the user with:

• Overall verdict: PASS | PASS-WITH-CAVEATS | FAIL
• The completion decision from Completion and Evidence, stated separately from the verdict
• Top 3 risks (1 line each)
• Path to the artifact
• Any defects that require user decision (ship / fix / defer)

Do not recommend fixes unless the user asks.

Completion and Evidence

A verdict is a judgement about probes. Completion is a lifecycle transition on the work item, and it is allowed only when applicable evidence covers the accepted outcome — not when the probes were green, and not when every task passed.

Record the evidence

Each check that survived falsification becomes one evidence entry on the record, named against the fingerprint it was gathered at:

{"expect_revision": 7,
 "metadata": {"evidence": [
   "<each stored entry, first, unchanged and in the same order>",
   {"criteria": "<the accepted criterion this covers>",
    "fingerprint": "<the record's current scope_fingerprint>",
    "revision": "<code or artifact revision the check ran against>",
    "environment": "<where it ran>", "result": "pass | fail | waived | unverified",
    "limitations": ["<what this check does not cover>"]}]}}

python3 -B "$ENTRYPOINT" --project-root "$PROJECT_ROOT" revise --seq SEQ-042 --input "$PAYLOAD"

For the fields of this and every other payload, run python3 -B "$ENTRYPOINT" --project-root "$PROJECT_ROOT" schema <command>; the example above is the only one shown here. The entry fields, including the optional reused_from and baseline, are tabulated in references/work-item-contract.md.

Reusing a result. An entry that cites an earlier result instead of rerunning it carries reused_from: the commit the result ran at, a statement that the tested paths were clean (git status --porcelain -- <paths> printed nothing), the command, the interpreter and lockfile hash, and the environment variables that matter. Only passing results are reused. A failing or unverified result is rerun. The gate is unchanged and checks only that the entry applies, so the reuse decision is stated in the entry.

Evidence entries with source: "record-result" are the implementing agent's own report. Reproduce or independently check them; do not count them as independent evidence. The entries you append must not carry source: "record-result", because the runtime reserves that marker for itself and refuses the patch. Moving the item to done is judged by the gate on the record as written, with the new evidence and any scope change in the same payload, not on the stored record.

The response reports each entry's applies. An entry that does not apply is not evidence about this outcome, whatever its result says. metadata.evidence is append-only: send the stored entries first, unchanged and in the same order, followed by the new one. The runtime refuses a list that removes, edits or reorders a stored entry, or that sends null. For each criteria value only the latest entry in list order counts, so re-running a check and appending its result supersedes the earlier entry for that criterion; the older entries stay as history. Appending does not clear a stale entry for a criterion you did not re-run.

The completion gate

Re-resolve the acceptance and its authority immediately before this decision, and check every row before calling for a transition. The runtime records a transition; it does not judge whether the evidence covers the accepted outcome. That judgement is this step's, which is why the gate is checked first.

When every row passes, transition the item:

python3 -B "$ENTRYPOINT" --project-root "$PROJECT_ROOT" revise --seq SEQ-042 --input "$PAYLOAD"

with {"metadata": {"lifecycle": "done"}, "expect_revision": <current>, "actor": "<claim holder>"}. The acting identity must hold the claim; take or reassign the claim before changing lifecycle. revise also refuses an illegal transition and, for a reopening, refuses without a correction. Use transition instead when one operation must change several records together.

Reopening and successor work

• Premature closure: reopen under the same identity. Send {"expect_revision": <current>, "actor": "<claim holder>", "metadata": {"lifecycle": "active"}, "correction": {"reason": "<why the closure was premature>", "actor": "<who decided>"}}. The correction appends to the record's corrections; nothing already recorded is rewritten.
• An incident after valid delivery: capture linked successor work under a new SEQ, referencing the delivered item. Do not reopen a delivered outcome to hold new work — the delivery happened, and the record has to keep saying so.

Evidence Artifact Contract

The artifact at _verification/{date}-{label}-verification.md must use exactly these H2 sections, in order:

# {Label} Verification Report

**Verifier:** {model or agent id}
**Date:** YYYY-MM-DD
**Work item:** {SEQ-NNN} — accepted fingerprint {scope_fingerprint}
**Scope:** Commits {sha_first}..{sha_last} on branch {branch}
**Verdict:** PASS | PASS-WITH-CAVEATS | FAIL
**Completion:** done | not yet — {the gate row that refused it}

## Executive Summary
<=150 words. Headline finding + top 3 risks.

## Hypotheses and Results
| # | Hypothesis | Predicted | Observed | Evidence | Status |

## Scope Matrix Status
Row 1..6 from the Scope Matrix with PASS/FAIL/N/A + one-line justification.

## Structural Audit
Inventory table from the Structural Audit.

## Test Results
- Full suite: pass/fail counts, duration, command, and whether run now or reused (`reused_from`)
- Targeted subsets: same
- Test-count delta (if applicable)
- Migration replay (if applicable)

## Defect Probe Results
For each prior-flagged defect: status + probe location + raw output excerpt.

## Integration Probe Results
Per-probe: command, outcome, evidence.

## Spec-vs-Reality Gap
Feature -> status -> citation.

## Findings Ledger
### Critical (blocks ship)
### High (must fix before next release)
### Medium (fix when convenient)
### Low / Informational
Each finding: unique ID (`V-001`...), title, evidence, reproduction command, suggested remediation (one sentence).

## Out-of-Scope / Explicitly Deferred
Features the plan marked as future phases and that you confirmed are not half-implemented.

## Cross-Environment Smoke-Test Checklist (if applicable)
Executable commands for environments you couldn't reach (e.g. Windows-only, production-only).

## Documentation Drift
Places where architecture docs, testing guides, or the index are stale. List only — do not fix.

## Appendix A — Commands Executed
Full command transcript with one-line results. Enables reproducibility.

## Appendix B — Probe Source
Paths to all verification probes added under `_verification/probes/`.

Definition of Done

You are done when all are true:

1. Artifact at _verification/{date}-{label}-verification.md exists and follows the contract exactly.
2. Every row in the Hypotheses table has PASS or FAIL with cited evidence.
3. All defect probes have been run; raw output is captured in Appendix A.
4. One applicable passing full-suite result exists: run now with complete output preserved, or reused through reused_from.
5. At least one integration probe has executed, or each skipped probe has a cited reason and a user-executable alternative.
6. The verdict line is one of the three allowed values and is consistent with the Findings Ledger (a FAIL ledger with PASS verdict is not allowed).
7. Each surviving check is recorded as an evidence entry against the current fingerprint, and the response confirms it applies.
8. The completion line states the decision and, when completion was refused, which gate row refused it.
9. No source file outside _verification/ has been modified, and no record outside the item under verification.

Performance Budget

A full run is hours, not minutes; assessing first cuts that when evidence exists. Not appropriate for:

• Bug fixes (just run the tests)
• Single-file refactors
• Small features with no design doc
• Internal polish without user-facing behavior

Appropriate for:

• Plans completed via /session-research-design → /session-task-planning → /session-delegation
• Pre-release gates where ship blockers must be caught
• Large refactors touching multiple architecture layers
• Features where prior code review flagged critical or high defects

Anti-Patterns

Spot-checking instead of exhaustive coverage:

• BAD: Verify 3 of 11 event types exist; assume the other 8 do.
• GOOD: Inventory all 11 in a table with file:line citations.

Treating compilation as correctness:

• BAD: "TypeScript compiles clean — the UI wiring is correct."
• GOOD: "TS compiles. The component is mounted at CaptureFooter.tsx:42 per grep. Runtime path verified by {probe}."

Reading a task board as a delivered outcome:

• BAD: "All six tasks are passed, so the item is done."
• GOOD: "All six tasks are passed. The accepted scope requires a merged pull request; PR #58 is open, so the item stays active."

Rewriting the plan to match broken code:

• BAD: Discover a missing feature; quietly update the plan to say it was out of scope.
• GOOD: Report it as NOT-IMPLEMENTED in the Spec-vs-Reality Gap section.

Hand-waving environment limits:

• BAD: "I couldn't run the integration tests in my environment."
• GOOD: "Integration tests require Windows + Tauri (not available in WSL). User must run pnpm tauri dev on Windows and execute checklist §9.3."

Chain context: see references/workflow-overview.md.