Evaluator

A second, independent layer of validation. After implement-feature finishes a feature, the evaluator looks at it from the outside and confirms — against contract.md — whether the delivery conforms. It finds deviations the implementer's self-check missed. It does not fix code; it evaluates and orchestrates the correction loop, dispatching fix-runner on failure.

Base docs: https://github.com/dayvisonassis/sdd-skills/blob/main/docs/Skill_Evaluator.md (full rationale), https://github.com/dayvisonassis/sdd-skills/blob/main/docs/Contrato_de_Feature.md (contract structure), https://github.com/dayvisonassis/sdd-skills/blob/main/docs/Como_criar_gates.md (gates), https://github.com/dayvisonassis/sdd-skills/blob/main/docs/ARQUITETURA_DAS_SKILLS.md (flow, states, progress.json schema). Report schema: references/evaluation-report-schema.md.

The evaluator complements automated tests — it does not replace them.

INPUT

Free-form. The skill needs:

• The target feature (ID/name) — explicit and required. Abort if absent or ambiguous (list candidates).
• Auto-discovers: progress.json (root of docs/) and the feature's contract.md (in docs/<feature-id>-<kebab>/).
• Optional free-form overrides at the end — e.g. "no screenshots", "gates only", "max 5 attempts" (overrides maxFixAttempts for this run), "no exception: <finding>" (a human refused the gate exception fix-runner asked for; see the schema, "Gate exceptions").

If the feature has no contract.md, abort: "No contract.md for F<ID> — generate it with spec-writer first."

OUTPUT

• Screenshots — visual evidence of observable UI criteria (when applicable).
• Report — consolidated ✓ / ✗ / — per contract criterion and gate.
• Findings in chat — textual summary for the user.
• State in progress.json — CLEAN | FAIL | PENDING | ABORTED for the feature, committed when the evaluation ends (Step 9, which has the one exception).
• On FAIL: a structured evaluation-report.json in the feature folder, with each failure classified by kind (consumed by fix-runner for code, or a test-writer for tests). Removed when the feature ends CLEAN.
• Coverage-table dispute entries — the only thing the evaluator writes in the writer's checklist: it opens a disputed entry when it disagrees with the writer, closes one that became moot and removes one that went stale (Disputes lifecycle, ../e2e-test-writer/references/e2e-rules.md).
• Test commits — the test files a validator confirmed, with their checklists, in commits of their own, and with them a code correction a commit hook held back (Step 8.4). Writers never commit, so this is what keeps the committed tree passing its own gates.
• The evaluator does not edit code or tests itself. Code corrections → fix-runner; test corrections → the matching test-writer (confirmed by the matching test-validator).

EXECUTION STEPS

Step 1: Resolve Input

• Resolve the target feature (ID/name). Abort if absent or ambiguous, listing candidates.
• Locate progress.json (root of docs/) and the feature's contract.md. If contract.md is missing → abort ("generate the contract first with spec-writer"). If progress.json is missing, create it with config.maxFixAttempts default 3.
• Parse any overrides (e.g. max N attempts, gates only, no screenshots, no exception: <finding>) and record them for the report. A no exception override sets exceptionRefused on the matching failure in every report this run writes.

Step 2: Load Context

• Read contract.md: Environment Contract, Quality Gates (with ids), Coverage Manifest, Surfaces & Behaviors, Observable Criteria (with ids), Test-suite hint.
• Read progress.json: this feature's state, attempt, and config.maxFixAttempts (N).
• The contract is the single source of acceptance criteria. Do NOT invent criteria outside it.

Step 3: Verify Environment Contract

• Check each environment prerequisite (runtime up, services reachable, tools available — e.g. dev server serving /, Playwright CLI present).
• If the environment is not satisfied → do NOT proceed to evaluation. Set state: PENDING (human/environment intervention needed), explain that this is an environment problem (not an implementation failure), and stop. Environment-invalid ≠ implementation-wrong.

Step 4: Run Quality Gates

• Execute each gate declared in the contract (typecheck, lint, build, tests, arch...). Use the exact commands from the contract.
• Any gate failing is a contract violation. Collect the command + log as evidence.
• Classify each failure by cause (this drives the correction routing in Step 7):
  • The failure is a code failure (kind: gate or observable-criterion) when production code is wrong — a type error, lint/build/arch violation, or a missing observable behavior.
  • The failure is a test failure (kind: test) when the test itself is broken/non-conforming — e.g. the tests gate fails and the cause is the test file (missing/incorrect mock, wrong pattern, a test that no longer matches correct behavior), not the production code. For a kind: test failure, fill the routing fields (testSuite, testFile, targetFile) per references/evaluation-report-schema.md.
  • When ambiguous (a failing test that might reflect a real code bug), prefer kind: gate/observable-criterion and let fix-runner handle the code; only route to a test-writer when the test is clearly the thing that is wrong.
  • A browser-gate failure (e2e or visual) has a third possible cause — the environment — and it is checked first. A 429, every case failing on the sign-in screen, the app not answering, or a dev server serving a stale bundle (its last build failed — read the dev server's log) is PENDING (environment): neither kind: test nor kind: gate, no corrector is dispatched, and the evaluation stops as in Step 3. Only after the environment is ruled out classify the failure as test or code. For kind: test in an e2e file, also fill targetSurface.
  • A red e2e test is not automatically a product failure. With the environment ruled out, re-observe the behavior yourself (Step 5) against the contract line before classifying. A test with a disputed entry in the coverage table (matched by title and path) follows the Disputes lifecycle of ../e2e-test-writer/references/e2e-rules.md: diverges → code failure as below; matches → PENDING for a human with both readings, never routed. The same lifecycle, applied here — from a run that included that test, made here when the gate's run did not — closes a disputed test that passed (moot, back as unproven), closes one whose contract line was changed, parks a skipped one as PENDING and removes the entry of one that no longer exists (stale). For any other red e2e test: the product diverges from the line → code failure: kind: observable-criterion with the criterion id as ref when the failing test carries a criterion tag, kind: gate with the e2e gate id otherwise — and put the failing test's path and the contract line in location/message, since a gate id alone tells fix-runner nothing. The product matches the line → the test is wrong: kind: test, with the contract line and your observation in message.
  • A failing assertion of the visual suite that is itself wrong (e.g. it measures mid-animation) belongs to no test-writer: record it as PENDING for a human, never as kind: test and never to fix-runner (see the schema).

  • A gate exception is a human decision, even when its gate passes. Treat each kind as the table in references/evaluation-report-schema.md ("Gate exceptions") says; that table is the one definition. In short:

    • every allowlist entry awaiting approval is an exception-approval item, whatever feature it names: the ones the gate's output lists, and the ones in the branch's diff of the allowlist file without the approval (read the file even when the contract does not declare that gate) (the gate's section in GATES.md says how the gate prints them);
    • every other gate exception in the branch's diff against the base goes at the top of Findings.

    Do not judge an exception acceptable yourself, and never route it to a corrector.

  • A red query-growth test whose every repeated statement is a per-item query with an approved allowlist entry is a test to adjust, not code to fix: kind: test, so the writer records the per-item query allowlisted outcome of ../integration-test-writer/references/integration-rules.md.
• (If gates only override is set, skip Step 5 and go to Step 6 with just gate results — except that a red e2e test, which cannot be classified without re-observing it, is then PENDING, never sent to fix-runner by default.)

Step 5: Validate Surfaces & Observable Criteria

• For each surface in the Coverage Manifest, start from its declared initial state and exercise the concrete behaviors (e.g. navigate the route via Playwright CLI, capturing screenshots).
• Exercising a write flow acts on records you create and remove afterwards, never on pre-existing data. This is D1–D7 of ../e2e-test-writer/references/e2e-rules.md, with the project's data policy from GATES.md, with or without the e2e harness — the dev database is shared. Create and remove them through the backend API with your browser session's own token (read from the live page, never from a storage-state file), or through the UI, and remove through the product's delete route, checking each status. When nothing your session can use removes what the flow creates, observe the flow up to the submit and record the rest as PENDING for a human.
• For each Observable Criterion, collect verifiable evidence (e.g. CTA present, login in top nav, redirect behavior, visual identity). A criterion with no observable evidence is a failure (kind: observable-criterion, ref: <crit-id>). One cause that violates several criteria is one entry per criterion.
• When the project has an e2e suite (GATES.md lists an e2e gate proven green), the e2e tests tagged with a criterion's id are part of its evidence. Get them from one run per feature, not one per criterion — every run costs the harness its logins against the auth rate limit. Reuse the run of Step 4 — the e2e gate's, or the feature run made there for a dispute — when it ran the feature's tests and its output can be mapped by tag; otherwise run once with the e2e config from GATES.md and --grep "@<feature-id>(?![\w-])" --reporter=json, and map the results by tag. Criterion ids repeat across features and @F08 is a prefix of @F08-v2, so always pair the anchored feature tag with the criterion. The tests complement the screenshots, never replace the observation.
• When the project has an e2e suite, missing e2e coverage is a failure — per row, not per surface. Every row the contract's Test-suite hint maps to e2e must carry one outcome of the closed list in ../e2e-test-writer/references/e2e-rules.md ("Coverage outcomes") in the writer's coverage table (docs/<feature-id>-<kebab>/e2e-test.md): test titles that exist, carry the feature and surface tags and are not marked unproven, or a not e2e-testable reason worded as that list words it. Check an accepted reason against the product before trusting it (a "no removal path" where a delete route exists is not one); an accepted reason makes that row yours to check by hand, like a runtime-only row, and it goes in Findings (the contract expected automation). "No removal path — harness lacks <fixture>" is an environment gap: PENDING, naming the fixture. disputed entries are arbitrated, closed or removed in Step 4, per the Disputes lifecycle; a row holding only open disputed entries awaits arbitration and is not missing coverage; the row's other tests still count. A row the writer marks out of e2e scope — <suite> (a computed value, say) is a mistake of the contract's mapping: record it as PENDING for a human, with the suite it belongs to — never route it back to the writer. A row with none of the outcomes above — including one marked unproven or carrying any other reason → kind: test, testSuite: e2e, ref = the e2e gate id, targetSurface = its surface, the behavior in message, no testFile — routed to e2e-test-writer, which runs in guard mode — whatever the stack: the writer is generic and reads the harness from GATES.md. When GATES.md does not declare what the writer needs (its e2e section names no profiles, and the harness reveals none), record it as PENDING for a human instead — routing it would only burn attempts.

• When the project's gate documentation describes a query-growth (N+1) check, every row the contract's Test-suite hint marks integration — query growth must have, in the query-growth table of the integration writer's checklist (docs/<feature-id>-<kebab>/integration-test.md), a growth test title or one outcome of the closed list in ../integration-test-writer/references/integration-rules.md ("Query growth — Scope and outcomes"). Check each row like an e2e reason, not by its presence:

  • a growth test title must exist in the suite and have run in this evaluation, green. A red one is a code failure, as in Step 4. The integration gate selects suites by name, so when its run did not include that suite, run the test by title once;
  • a not measurable outcome must hold against the code;
  • a pre-existing N+1 outcome holds only when neither the loop nor the call that repeats is in the branch's diff against the base. Check both locations the outcome names. A loop the feature wrote around a legacy helper is the feature's N+1.

  A missing, unrun or false one → kind: test, testSuite: integration, ref = the integration gate id, targetFile = the endpoint's handler, the endpoint and what is wrong in message, routed to integration-test-writer and confirmed by its validator (Step 8). A pre-existing N+1 outcome that holds goes in Findings, for the legacy migration.

• When the contract's Test-suite hint maps a surface to unit or integration, check its contract coverage — references/evaluation-report-schema.md, "Contract coverage", the one definition. Read the feature's tests of that suite and the writer's checklist, criterion by criterion and value by value. Missing coverage is a failure whether or not the criterion holds when you exercise it: your exercise is evidence, and the contract expected a test. It is a kind: test entry with no testFile, routed to the matching writer and confirmed by its validator (Step 8). When the same report holds a code failure for that criterion, the entry waits for the correction, as the pair of Step 7.3 does: the test is written against the corrected code, in the same round. A value that a test of that source asserts with another outcome is not missing coverage: when the code is what is wrong, list that test with the code failure exactly as the spec mismatch bullet below does, whether or not a note records it. A test written for another source is not yours to look for here: when the correction turns one stale, fix-runner names it (Step 7.4).
• When the feature's folder holds a unit or integration checklist (unit-test.md, integration-test.md), read what its writer left there for whoever evaluates. The writer's signal carried the same lines to whoever dispatched it; the checklist is where they stay.
  • A row whose outcome is not unit-testable — no seam: <dependency>: that behavior has no unit test, and no writer can give it one without a change to production code — never route it to a writer. The criterion it serves is yours to evidence by exercising the behavior, like any criterion no test covers — on records you create and remove, through the helpers the project's GATES.md declares when it has them, and removing whatever the exercise itself created, a database file included; when nothing lets you exercise it, the criterion is PENDING, like a runtime that cannot be exercised — and the row goes in Findings: the contract expected a test, and opening the seam is a design decision. The Finding itself does not change the state.
  • A note spec mismatch: <behavior> — spec: <x>; code: <y>: the test the checklist lists for that behavior asserts what the code does, so it is never evidence for a criterion. Do not take the note on faith either: compare the code with the contract yourself, at the value the note names and for every criterion that value reaches — one observable-criterion entry per criterion violated, even from one cause. Where the code satisfies the criterion, the note is stale — its writer removes it when it rewrites the test, never you — and the test is evidence again. A criterion the code violates is one cause with two entries in the report: kind: observable-criterion, ref = that criterion, for the code — its message names the test that pins the old behavior, which is not fix-runner's to touch; and kind: test for that test (testSuite, testFile, targetFile, ref = the gate that runs it), its message saying what it asserts, what the criterion says, and that it is to be rewritten to the contract once the code is corrected. Step 7.3 dispatches them in that order, in one round. No criterion covers it → Findings, for a human: when spec and behavior disagree outside the contract, which one is wrong is a product decision. It is not a failure and not a PENDING item: the contract is the only source of criteria, and the state does not change.
• Map every PRD-derived acceptance back to a contract criterion/gate (the contract already did this traceability; honor it).

Step 6: Decide State

Determine the feature's state strictly from contract adherence — never from "looks ok":

• CLEAN — no failures and nothing pending: all gates pass, all observable criteria met, contract coverage complete where the hint asks for it, pending[] empty. → record state, proceed to Step 9.
• FAIL — at least one correctable failure (gate/test/observable). Items in pending[] are not failures and do not make a FAIL. → go to Step 7 (loop).
• PENDING — no correctable failure left, and something the evaluator cannot decide or test by itself (needs a human, or an environment it cannot bring up), including any pending[] item. Step 7.4 also stops the loop as PENDING when fix-runner's only answer is exception claims that hold. → record state with a note, proceed to Step 9.
• ABORTED — decided in Step 7 when attempts are exhausted.

List exactly which gates/criteria failed.

Step 7: Correction Loop (when FAIL) — route by failure kind

• Read attempt and maxFixAttempts (N) from progress.json.
• If attempt >= N → set state: ABORTED; stop and report (the loop tried N times without converging). Proceed to Step 9.
• Else:
  1. Write/refresh evaluation-report.json in the feature folder (schema in references/evaluation-report-schema.md) with the current attempt, the classified failures[] and the pending[] items of this evaluation. For a failure you evidenced by exercising the behavior, evidence is the exercise itself, in a form that can be run again: fix-runner revalidates by repeating it.
  2. Set state: FAIL in progress.json and persist the report path in lastEvaluationReport. Every write of the feature's entry — this one, each increment — refreshes its updatedAt.
  3. Route each failure by kind:
     • kind: gate / observable-criterion (code) → dispatch fix-runner, passing the feature ID and the report path. Unchanged behavior — the on-disk report is the source of truth.
     • kind: test → run the test-correction sub-flow (Step 8) for that failure. Never send test failures to fix-runner.
     • A kind: test entry for a test that pins what a code entry of the same report corrects (Step 5, a spec mismatch) waits for that correction, and so does a missing-coverage entry for a criterion the same report lists as a code failure. Dispatch fix-runner first; when it answers "correction applied" — committed, or left uncommitted because a commit hook runs that test — a pinned test is now red by construction — run Step 8 for the waiting entries now, in the same round: the writer rewrites a pinned test to the contract, or writes the test that was missing, and its validator confirms it. When fix-runner does not correct the code, do not dispatch either: a pinned test still passes and still pins, and a new one would be written against the wrong behavior. When the code is corrected and the writer then cannot run (an environment signal), the round still counts for the code correction, and the test stays a kind: test failure for the next evaluation.
  4. When the dispatched correction returns:
     • Correction applied → the round counts. One report is one round, and a round increments once, when everything it dispatched has answered: a code correction; a test correction and its validator's verdict (Step 8); or a code correction, every test entry that waited for it or that its signal added — a pinned test rewritten, a missing one written, a stale one a "test to correct" line names — and each validator's verdict (7.3): there, the increment waits for the last validator. Then increment attempt in progress.json and re-evaluate: go back to Step 3. A writer dispatched again after its validator's FAIL is a new round (Step 8.2), on the same report. A round whose signal comes with a correction applied — committed, or held back by a commit hook (7.3) — always increments, whatever else the signal carries.
     • "test to correct: <file> — <title>" lines on a "correction applied" → fix-runner says its correction made that test stale. Do not take it on faith: read the test against the contract. It asserts what the criterion the correction satisfies contradicts → add a kind: test entry for it to the report on disk, beside the entries already there (the writer reads the file), its routing fields filled as Step 4 fills them, and run Step 8 for it now, in the same round, as for the pair of 7.3. It asserts what the contract supports → the correction broke correct behavior: do not send the test to a writer. The round counts, and the next pass reports that test's failure as a code failure, classified as Step 4 says.

     • "not resolved — needs an allowlist decision: <why>" → check every claim; do not take fix-runner's word for it. A claim holds only when all of these are true:

       • the finding it names is a finding of that gate, and it is not marked exceptionRefused;
       • the gate's section accepts an exception for that kind of finding (a section may rule some kinds out entirely);
       • <why> names one of the categories the section accepts, and the code matches it.

       When the signal carries only claims, every one holds, and no other correction (a test-writer's, say) was applied in this round, the loop stops. Record one exception-decision item per finding, set PENDING, and go to Step 9, with no increment: a human decides next (schema, "Gate exceptions"). When the signal also carries a committed correction, increment and re-evaluate, as above; the claim returns on the next round if the finding is still there. A claim that does not hold turns the whole signal into an ordinary "not resolved", below.

     • "not resolved — <reason>" → still increment attempt; if attempt >= N set ABORTED, else re-evaluate. Do not loop without incrementing.
     • The two writer signals of Step 8 ("environment", and e2e-test-writer's "product diverges") are the exceptions, handled there: no correction was applied, so they do not increment by themselves (a guard the writer adds is judged by its validator, as a new test).

The evaluator owns the counter, the limit N, and the ABORTED decision. Correction skills are stateless and never decide when to stop.

Step 8: Test-correction sub-flow (kind: test)

For a test failure, correcting the code is the wrong move — fix the test, then re-confirm it conforms. Select the suite from testSuite/testFile ("Suite selection" in the schema): unit → unit-test-*, integration → integration-test-*, e2e → e2e-test-*. A test file no declared suite contains, in a repository that reveals none, has no writer: PENDING for a human.

When a round holds several kind: test entries, run the ones that correct an existing test (a testFile) before the ones that write a new one: a writer that adds tests ends by running the whole suite, and a stale test not yet rewritten leaves that run red.

1. Fix the test — dispatch the matching test-writer (autonomous), passing the feature ID, the evaluation-report.json path, testFile, and targetFile (for e2e, targetSurface instead). With a testFile it runs in correction mode and fixes only the flagged test (smallest footprint). With none it writes what is missing and nothing else: e2e-test-writer in guard mode, a unit or integration writer in directed writing, for the tests the entry's message names. It never touches production code. Any writer may instead return "not resolved — environment: <which>", and e2e-test-writer may also return "product diverges":
   • "not resolved — product diverges from <ref>" → do not take the writer's word for it. If it came with a test the writer wrote or changed, first run the validator on that file (step 2) — a test that does not conform is a spent attempt like any other — and on PASS come back here instead of resuming. Then re-observe the criterion yourself (Step 5) and compare with the contract line it quotes.
     • The product diverges → reclassify the failure as code with the same rule as Step 4 (kind: observable-criterion with the criterion id as ref when the test carries a criterion tag, kind: gate with the e2e gate id otherwise, the test's path and the contract line in location/message), refresh the report, and dispatch fix-runner in the same round — the writer's round applied no correction, so it costs no attempt; the fix-runner round increments as in Step 7.4. A failure reclassified this way is never sent back to a test-writer in this evaluation, and a guard the writer kept red turns green when the fix lands.
     • The product does not diverge → you and the writer disagree — about what the contract means, or because the test itself is wrong in a way no static check sees. Record PENDING for a human with both observations and the contract line; do not re-dispatch the writer. Open a dispute on that test — a disputed — <test title> (<path>) — <contract line> entry replacing its title in the coverage table, per the Disputes lifecycle — and name the test first in the PENDING note; the next evaluations arbitrate it as in Step 4.
   • "not resolved — environment: <which>" (environment red or unreachable, a test profile or harness that is missing or not declared, or a stack the writer does not cover) → PENDING (environment), as in Step 3. No increment.
   • A unit or integration writer's done may carry "no seam: …" and "spec mismatch: …" lines. Act on each now as Step 5 acts on the same record in a checklist — a correction may leave no checklist behind: a behavior with a no seam line is never dispatched to a writer again in this evaluation and goes in Findings, and a spec mismatch is compared with the contract when the evaluation resumes. The validator still confirms the file when the writer corrected or wrote a test in it; a test it only removed leaves nothing to confirm. The round counts like any other test correction.
2. Confirm conformance — for any test the writer returned — corrected, new, or newly proven — dispatch the matching test-validator on that file (for e2e, also pass the contract.md and the writer's checklist, so the tags are checked against real ids). An environment signal skips this step: whatever tests it names are unproven — they stay marked unproven in the coverage table and count as missing coverage on the next evaluation (Step 5), where the writer proves them before anything relies on them.
   • Verdict PASS (or PASS WITH WARNINGS) → the test now conforms. A guard that came with "product diverges" goes back to step 1 from here: no correction was applied yet, and nothing is counted. Otherwise commit as step 4 says and go on to the round's next waiting entry. After the last one the round ends and counts once (Step 7.4): increment attempt, then resume the evaluation where it left off (re-run Step 3+ / the failing gate) and continue.
   • Verdict FAIL → the correction did not conform. The round ends here, a spent attempt: increment attempt; if attempt >= N → ABORTED; else dispatch the test-writer again with the validator's findings — a new round — then re-validate. For a test written in this round, first set its path as testFile in the report, so the writer corrects that file instead of writing another. Entries of the round that were not run yet stay waiting, and run in that new round, which counts once, after its last validator answers.
3. When a test was corrected or written, only after the test-validator returns PASS does the evaluator continue its own evaluation.

4. Commit what the validator confirmed. Writers never commit, and a corrected test left in the working tree leaves the committed tree failing its own gate. After a PASS (or PASS WITH WARNINGS), commit the test file and the writer's checklist, and nothing else — the first case below is the one exception — in a commit of their own: test(F<ID>): <what was corrected or written>, or the repository's style when its recent commits show one. Stage those files by name: never git add -A, never skip a hook, never create or switch a branch.

   • A code correction a commit hook held back (the uncommitted form of fix-runner's signal, Steps 7.3 and 7.4) changes when you commit. Hold every commit of the round until each of its test entries is confirmed, then make one commit: the correction, with the paths its signal lists after changed:, and every confirmed test and checklist of the round. Until the stale tests are rewritten the hook refuses any commit, whatever it holds, so no entry of the round is committed ahead of the others. Use fix(F<ID>): <error summary> for it, or the repository's style when its recent commits show one. When a validator's FAIL ends the round first, all of it stays held into the round that dispatches the writer again. When the evaluation ends with it still held (ABORTED), try that one commit all the same — the correction and what was confirmed, under the same message: the hook decides, and what it refuses stays uncommitted, in Findings.
   • A test that is red on purpose waits — an e2e guard of a divergence, a growth test red on the feature's own code. It is committed once a gate run of this evaluation shows it green, after the code correction (Step 9 at the latest); one that stays red is never committed.
   • Never committed: a test whose validator's verdict is FAIL, a test with an open dispute, a test an environment signal left unproven, and a pre-existing N+1 growth test.
   • A hook refuses your commit → do not skip it: leave the files uncommitted.

   Any test or correction still uncommitted when the evaluation ends goes in Findings, with the reason.

The test-writer/validator pair is a sub-loop inside the evaluator's main loop. It still consumes the single attempt/N budget — never iterate the sub-loop without incrementing.

Step 9: Persist State & Report

• Commit any confirmed test that was red on purpose — an e2e guard, a growth test — and that this pass's gates ran green (Step 8.4).
• Write the final state (CLEAN | PENDING | ABORTED) and updatedAt for the feature in progress.json. Preserve config and other features' entries (merge, don't replace).
• The report file. On CLEAN, clear lastEvaluationReport and delete evaluation-report.json: its failures are corrected, and a FAIL report left in the feature folder is found by the next corrector as if it were current. On PENDING and ABORTED it stays, refreshed first with this pass's failures[] and pending[] — the schema asks for it on every outcome, and a PENDING with open decisions writes it even when no round failed: it is what the human reads.
• Commit the state. One commit — chore(F<ID>): evaluation — <state>, or the repository's style when its recent commits show one — holding progress.json and, on PENDING and ABORTED, the report; on CLEAN, the removal of a report an earlier evaluation committed. Stage them by name. A hook that refuses it is not skipped: the files stay uncommitted, and Findings says so. The one exception: an evaluation whose first pass stopped on the environment, before anything was evaluated, commits nothing — the state it wrote is redone by the next run. An environment stop after a correction round commits like any other ending.
• Output the report to chat:

Feature F<ID> — <name>

State: CLEAN | PENDING | ABORTED
Attempts used: <attempt> / <maxFixAttempts>

Quality gates:
✓ <gate-id> passed
✗ <gate-id> failed: <message> (<command>)

Observable criteria:
✓ <crit-id> — <evidence / screenshot path>
✗ <crit-id> — <what was missing>
✗ <crit-id> — no <unit | integration> test asserts <values> (contract coverage)

Surfaces evaluated:
- <surface-id>: <result>

Environment:
✓ contract met  |  ✗ not met → PENDING (<which prerequisite>)

Findings:
- Commits made by this evaluation, for confirmed tests and for a correction a hook had held back: <hash> — <files>
- State committed: <hash> — `progress.json`<, and the report>
- Left uncommitted, for the human: <paths> — <why: its validator's verdict was FAIL, an open dispute, unproven, still red on purpose, or a hook refused the commit>. Until a corrected test is committed, the committed tree alone fails the gate it belongs to
- Logic with no unit test — no seam: <dependency> — <behaviors>
- Spec and code disagree outside the contract: <behavior> — spec: <x>; code: <y>
- <key deviations the evaluation found>

Next:
- CLEAN → feature ready for the next stage
- PENDING → needs human intervention: <what>
- ABORTED → tried <N> fixes without converging; see last evaluation-report.json

RULES

Always:

• Treat contract.md as the single source of acceptance criteria.
• Abort the evaluation if the Environment Contract is not met, and say it is an environment problem (→ PENDING).
• Run the contract's Quality Gates as objective checks before/with functional inspection.
• Derive the state from contract adherence, with evidence per criterion.
• Own the loop: keep attempt/maxFixAttempts in progress.json and decide CLEAN/FAIL/PENDING/ABORTED.
• Classify each failure by kind and route it: gate/observable-criterion → fix-runner; test → the matching test-writer (then confirmed by the matching test-validator).
• Check contract coverage for every surface the hint maps to unit or integration (Step 5).
• Rule out the environment before classifying a browser-gate failure as test or code.
• Get e2e evidence from one run per feature, never one per criterion.
• After a kind: test correction that corrected or wrote a test, require the test-validator PASS before resuming the evaluation.
• Commit the test files a validator confirmed, with their checklists — and with them a code correction a commit hook held back — staged by name (Step 8.4).
• End by committing the state as Step 9 says, and remove the report file on CLEAN.
• Re-evaluate after each correction until CLEAN, PENDING, or ABORTED.
• Increment attempt once per correction round (code or test); never loop without incrementing.
• Treat every gate exception as the schema's "Gate exceptions" table says, never as fixed, failed or acceptable by your own judgement.

Never:

• Alter production code or "fix" the feature yourself — code corrections are the fix-runner's job, test corrections are the test-writers' job.
• Send a kind: test failure to fix-runner, or a kind: gate/observable-criterion failure to a test-writer.
• Commit a test its validator did not pass, or anything else a corrector left — a code correction a commit hook held back is the one exception, in Step 8.4; use git add -A; skip a hook; create or switch a branch.
• Approve based on visual perception without running the objective gates.
• Evaluate a feature without a contract.md.
• Invent criteria not present in the contract.
• Exceed maxFixAttempts without marking ABORTED (the test sub-loop shares the same budget).
• Write PENDING_EVALUATION into progress.json — that is the implementer's pre-state; the evaluator writes CLEAN/FAIL/PENDING/ABORTED.
• Confuse an environment failure (PENDING) with an implementation failure (FAIL).

Edge Cases

No contract.md for the feature: abort and direct the user to spec-writer.

progress.json missing: create it with config.maxFixAttempts = 3 and this feature's entry.

Environment Contract not met: state = PENDING, clearly labeled as environment, no fix-runner dispatch.

Gate command not runnable in this environment (e.g. missing toolchain): treat as PENDING for that gate (needs environment), not FAIL — do not send the fix-runner after an environment gap. Note it in the report.

Browser-gate run (e2e or visual) red for a harness reason (login rate limit 429, sign-in screen on every case, stale bundle, app down): PENDING (environment). Do not re-run it in a loop to "see if it passes" — each run spends logins against the same limit that caused the failure.

maxFixAttempts reached: state = ABORTED; keep the last evaluation-report.json for inspection.

Commit-message style inconsistent in recent history, or only scaffold commits: use the forms of Steps 8.4 and 9.

fix-runner reports "not resolved": increment attempt; abort to ABORTED if the limit is hit, otherwise re-evaluate. The exception is a signal made only of claims that hold: it stops the loop as PENDING, with no increment (Step 7.4).

After a human decided an exception-decision: an added and approved entry makes the gate green on the next evaluation. A refusal comes back as the override no exception: <finding>, and that failure then goes to fix-runner marked exceptionRefused.

Override max N attempts: use N for maxFixAttempts this run (and persist it to config if the user intends it to stick — otherwise apply for the session only and note it).

Observable criterion is runtime-only and the runtime can't be exercised: PENDING for that criterion (human/environment), not FAIL.

Feature already CLEAN in progress.json: re-running is allowed (idempotent re-check); report the result and refresh state.

Cross-feature criteria: if the contract references behavior provided by another feature, evaluate only this feature's surface; note cross-feature dependencies in findings rather than failing on another feature's gap.