AILANG Sprint Evaluator

Independently evaluate a completed sprint implementation against its design doc, acceptance criteria, and quality standards. Based on the generator-evaluator architecture — separating the agent doing the work from the agent judging it.

Current State

• Branch: !'git branch --show-current'
• Active sprints: !'ls .ailang/state/sprints/ 2>/dev/null | head -5 || echo "none"'
• Existing evaluations: !'ls .ailang/state/evaluations/ 2>/dev/null | head -5 || echo "none"'
• Last commit: !'git log -1 --format="%h %s" 2>/dev/null'

Tests and lint are NOT run at skill load: each would be a whole-repo run before the evaluation even starts, repeated by evaluate_sprint.sh (2026-10-02). Phase 2 runs them once.

Use the data above first. Only re-run these commands manually if the injected context is empty or you need to refresh after making changes.

Quick Start

# Triggered automatically after sprint-executor, or manually:
# User says: "Evaluate the sprint M-CACHE"
# This skill will:
# 1. Load design doc, sprint plan, and sprint JSON
# 2. Run automated quality checks (tests, lint, coverage)
# 3. Verify each acceptance criterion from sprint JSON
# 4. Score implementation against concrete rubric (100 points)
# 5. PASS (trigger merge) or FAIL (send feedback to sprint-executor)

When to Use This Skill

Invoke this skill when:

• Sprint-executor completes and sends handoff message
• User says "evaluate sprint", "review implementation", "assess quality"
• Coordinator triggers via trigger_on_complete chain
• User wants independent quality assessment before merge

Coordinator Integration

When invoked by the AILANG Coordinator, output these markers at the end of your response. The LAST line must be the EVALUATION_VERDICT: line — it is the only one the coordinator parses, and it is what lands on the parent's approval card. Without it the approval records UNAVAILABLE however good the review was (2026-10-27: a full FAIL 84 review ended in EVALUATION_RESULT: fail and reached no approval).

EVALUATION_SCORE: 85/100
EVALUATION_ROUND: 1
EVALUATION_REPORT_PATH: .ailang/state/evaluations/eval_<sprint-id>_round_<n>.json
FEEDBACK_SUMMARY: <one-line summary if failed>
EVALUATION_VERDICT: PASS score=85

On fail the last line is EVALUATION_VERDICT: FAIL score=<n> reasons=<short list>. Emit it once, last.

Agent config:

- id: sprint-evaluator
  label: "Sprint Evaluator"
  inbox: sprint-evaluator
  capabilities: [review, test, docs]
  trigger_on_complete: []  # End of chain on pass
  session_continuity: false  # Stateless per round
  approval:
    needs_label: needs-evaluation-approval
    approved_label: evaluation-approved

Core Principles

1. Independent Judge — Never evaluate your own work. This skill judges sprint-executor output
2. Concrete Criteria — Score against measurable rubric, not subjective impressions
3. Skeptical by Default — Tuned toward skepticism; easier to calibrate than making generators self-critical
4. Actionable Feedback — Failed evaluations include specific files, functions, and suggestions
5. Hard Thresholds — Some failures (tests broken, <50% criteria met) cause automatic rejection
6. Bounded Iteration — Max 3 evaluation rounds before escalating to human review

Evaluation Phases

Phase 1: Context Loading

1. Read handoff message — Extract sprint_id, branch, sprint JSON path, design doc path
2. Load sprint JSON from .ailang/state/sprints/sprint_<id>.json
3. Read design doc — The original spec (the "contract")
4. Read sprint plan — The implementation roadmap with milestones
5. Review git diff — git diff dev...<branch> to see all changes
6. Check evaluation round — Is this round 1, 2, or 3? Load prior report if round > 1

Phase 2: Automated Checks

Run automated quality checks using the evaluation script:

.claude/skills/sprint-evaluator/scripts/evaluate_sprint.sh <sprint-id> [branch]

When the work has a pull request (every cloud evaluation), tests and lint come from that PR's CI, not from a local run. The script calls scripts/ci_gate.sh <branch>, which waits at most 4 minutes per call. If it reports CI_GATE=pending, call ci_gate.sh <branch> again until it prints CI_GATE=pass or CI_GATE=fail, and do not run make test yourself: CI is already running the full suite, lint and verify-examples on that head. A single long wait gets killed by the executor's idle timeout or the Bash tool's 10-minute cap, which is how every cloud evaluation on 09-29..10-02 died with no verdict. CI_GATE=fail is a HARD FAIL and names the failed checks. If every check was skipped, the PR conflicts with its base, and that is a fail too. CI_GATE=error means the PR or its checks could not be read: report it as a FAIL with that reason, never as a pass and never by running the gates locally instead. Without an authenticated gh (the cloud executor withholds GITHUB_TOKEN), ci_gate.sh reads GitHub's public REST API itself. The JSON records gate_source: ci|local.

With no PR (CI_GATE=none) or with EVAL_LOCAL_GATES=1, it runs each gate locally once (output captured, never re-run to read the tail):

• make test — All tests must pass (HARD FAIL if not). EVAL_PACKAGES="./a/... ./b" scopes it to the sprint's packages for a fast pre-check; the verdict still needs the full run (CI-equivalent), on a clean tree — another session's uncommitted work in a shared checkout makes a whole-repo run fail for reasons outside the sprint
  • A clean worktree must not live under $TMPDIR. internal/loader TestIsTempPath resolves relative paths against the cwd, so under $TMPDIR every path is "temp" and the test fails for reasons unrelated to the sprint (2026-10-02). Put it under .claude/worktrees/.
  • GitHub CI at a commit that contains the sprint is an independent, free signal. Check gh run list --branch dev --workflow ci.yml before re-running a whole-repo suite locally.
• make lint — Linting must be clean
• make check-file-sizes — No files exceeding 800 lines
• Coverage — opt-in (EVAL_COVERAGE=1): it is a second whole-repo test run

When the sprint touches examples/ or examples/manifest.json, ALSO run make verify-examples (HARD FAIL if red). make test and make check-file-sizes do NOT cover the manifest-drift gate — a sprint that adds an example but omits/leaves-stale its manifest modules field passes local go test yet fails the CI test job's verify-examples step (validate_manifest --ci). Recurrent class ([[project_verify_examples_red_is_usually_manifest_drift]]); 2nd instance of a manifest defect reaching CI through a green evaluator verdict (iter-101, PR #479). If drift is found, fix it surgically (populate the modules field for the new entry — do NOT run backfill_manifest_modules.go, which reserializes the whole file and churns unrelated unicode-escaped entries) or fail the sprint back to the executor.

Phase 3: Acceptance Criteria Verification

.claude/skills/sprint-evaluator/scripts/check_acceptance_criteria.sh <sprint-id>

For each feature in sprint JSON:

1. Read acceptance_criteria array
2. Verify passes: true for each feature
3. For file-related criteria, verify files exist
4. For test-related criteria, verify test functions exist
5. Score: (criteria_met / total_criteria) * 30 points

HARD FAIL if fewer than 50% of acceptance criteria are met.

Phase 3.5: Sprint Artifacts Verification

If a sprint JSON exists (.ailang/state/sprints/sprint_<id>.json), verify:

1. All milestones have passes: true (or false with explanation)
2. All milestones have completed timestamps
3. All milestones have notes with summaries
4. Sprint status is "completed"
5. Velocity metrics are populated

If a sprint plan markdown exists, verify:

1. All milestones marked with ✅ (not pending or in-progress)
2. No placeholder text remaining

Scoring: Deduct from Code Quality (Phase 4) — up to -5 for incomplete sprint artifacts. Sprint artifacts are optional (pre-sprint-system implementations won't have them), so missing artifacts are not penalized — only incomplete/inconsistent ones are.

Phase 4: Design Fidelity Check (AI Judgment)

Read the design doc and compare against the actual implementation:

• Does the implementation match the stated goals?
• Are the architectural decisions consistent with the design?
• Were any design requirements silently dropped?
• Are there unexpected deviations from the spec?

Score 0-10 based on fidelity to design intent.

Phase 5: Documentation Completeness

Check:

• CHANGELOG updated (5 pts) — New entries under appropriate version
• Example files created (5 pts) — examples/runnable/<feature>.ail for new features
• Design doc status (5 pts) — Updated to reflect implementation state

Phase 6: Scoring & Report Generation

Apply scoring rubric (see resources/scoring_rubric.md):

Generated Go is out of scope by default: do not deduct for missing GoCodegenSpecs or emit-go parity unless approved acceptance criteria explicitly scope existing codegen or the live contracts projection (--verify-contracts). Evaluate those criteria when scoped. Engine parity otherwise means evaluator + strict VM (emit-go demoted, emit-go-v2 frozen, Mark 2026-07-14).

Conditional categories:

• Regression Surface Coverage triggers when the sprint touches internal/parser/, internal/lexer/, internal/ast/, internal/types/, internal/elaborate/, internal/iface/, internal/codegen/, internal/eval/, internal/vm/, internal/effects/, or cmd/ailang/exec.go. Hard fail if triggered without a Conflict Surface analysis in the design doc and fixture tests for the named "Programs that MUST still work". Catches M-TAINT-TYPES-style silent regressions.
• Performance Verification triggers on perf sprints (design doc keywords). Hard fail without before/after profiling data.

Generate report:

.claude/skills/sprint-evaluator/scripts/generate_report.sh <sprint-id> <score> <result> <round>

Report saved to .ailang/state/evaluations/eval_<sprint-id>_round_<n>.json

Phase 7: Pass/Fail Decision

Determine execution mode before acting:

• Local mode (running in an interactive chat session): invoke skills directly
• Cloud/coordinator mode (running via coordinator daemon): use inbox messages

On PASS (score >= 70, no hard fails):

1. Move design doc from design_docs/planned/ to design_docs/implemented/<version>/ — and its companion sprint plans in the same move (Mark 2026-07-29): any <doc-stem>-sprint-plan.md / <doc-stem>-m*-sprint-plan.md siblings travel WITH the design doc. A sprint plan left in planned/ after its parent lands is folder drift — the 2026-07-29 attended triage archived two such strays; don't create more. (Multi-phase docs that stay in planned/ until a later milestone — e.g. an M4b — keep their plans beside them until the doc itself moves.)
2. Update design doc status from "Planned" to "Implemented"
3. Output the markers, ending with EVALUATION_VERDICT: PASS score=<n>
4. Post congratulatory summary with score breakdown

On FAIL (score < 70 or hard fail, round < 3):

Local mode — invoke sprint-executor directly:

1. Generate specific, actionable feedback (see resources/feedback_templates.md)
2. Invoke the sprint-executor skill with the evaluation report path and specific issues
3. After executor completes, re-evaluate (increment round)

Cloud/coordinator mode — send inbox message:

1. Generate specific, actionable feedback
2. Send feedback to sprint-executor inbox:

ailang messages send sprint-executor '{
  "type": "evaluation_feedback",
  "correlation_id": "eval_<sprint-id>_round_<n>",
  "sprint_id": "<sprint-id>",
  "evaluation_round": <n>,
  "score": <score>,
  "issues": [
    {"file": "path/to/file.go", "issue": "description", "suggestion": "how to fix"},
    ...
  ],
  "max_rounds": 3
}' --title "Evaluation Failed (Round <n>/<max>)" --from "sprint-evaluator"

3. Output the markers, ending with EVALUATION_VERDICT: FAIL score=<n> reasons=<short list>

On FAIL (round >= 3):

1. Escalate to human review
2. Add needs-human-review label via coordinator
3. Post summary of all 3 rounds with score progression

Available Scripts

scripts/evaluate_sprint.sh <sprint_id> [branch]

Orchestrate automated quality checks. Outputs structured JSON with test/lint/coverage results.

scripts/check_acceptance_criteria.sh <sprint_id>

Verify each acceptance criterion from sprint JSON. Outputs per-criterion pass/fail as JSON.

scripts/generate_report.sh <sprint_id> <score> <result> <round>

Create evaluation report JSON at .ailang/state/evaluations/.

Resources

• Scoring Rubric: resources/scoring_rubric.md — Point allocations and thresholds
• Report Schema: resources/evaluation_report_schema.md — JSON schema for evaluation reports
• Feedback Templates: resources/feedback_templates.md — Actionable feedback examples

Receiving Handoff from sprint-executor

The evaluator receives an implementation_complete message:

{
  "type": "implementation_complete",
  "correlation_id": "eval_M-CACHE_20260326",
  "sprint_id": "M-CACHE",
  "branch_name": "coordinator/task-abc123",
  "sprint_json_path": ".ailang/state/sprints/sprint_M-CACHE.json",
  "design_doc_path": "design_docs/planned/v0_7/M-CACHE.md",
  "files_created": ["internal/cache/store.go"],
  "files_modified": ["internal/server/handler.go"],
  "evaluation_round": 1
}

Execution Modes

The evaluator operates in two modes depending on where it's running:

Local Mode (interactive chat session)

When running in a user's local Claude Code session:

• On fail: Invoke the sprint-executor skill directly to fix issues, then re-evaluate
• On pass: Move design doc to implemented/, update status — no inbox needed
• Detection: Default mode. No coordinator markers in the prompt.

Cloud/Coordinator Mode

When invoked by the coordinator daemon or via agent inbox:

• On fail: Send structured feedback message to sprint-executor inbox
• On pass: Output the markers, ending with EVALUATION_VERDICT: PASS score=<n>, for the coordinator to act on
• Detection: Prompt contains coordinator task reference or implementation_complete message

Feedback Loop Architecture

Local mode:

sprint-evaluator (Round 1)
    ↓
PASS? → move design doc to implemented/, done
FAIL? → invoke sprint-executor skill with issues
            ↓
        sprint-evaluator (Round 2, same session)
            ↓
        PASS? → move design doc, done
        FAIL? → invoke sprint-executor (Round 3)
                    ↓
                PASS? → move design doc, done
                FAIL? → escalate to user for manual review

Cloud/coordinator mode:

sprint-executor completes
        ↓
sprint-evaluator (Round 1)
        ↓
    PASS? → merge approval
    FAIL? → feedback message to sprint-executor
                ↓
        sprint-executor fixes issues
                ↓
        sprint-evaluator (Round 2)
                ↓
            PASS? → merge approval
            FAIL? → feedback message (Round 3)
                        ↓
                    PASS? → merge approval
                    FAIL? → escalate to human (needs-human-review)

Notes

• Evaluation reports are preserved across rounds for audit trail
• Each round's report at .ailang/state/evaluations/eval_<id>_round_<n>.json
• The evaluator does NOT modify the sprint JSON — it creates separate evaluation artifacts
• Hard fails (tests broken) cause immediate rejection regardless of score
• Score progression across rounds is tracked in reports for learning
• Evaluator is stateless per round — no session continuity needed