Test-Specification Alignment Engine

Workflow Steps

1. Load Phase 0 — Verify specification.yaml and tests/ directory exist; BLOCK if missing with guidance to complete brainstorming → delphi-review → specification-generator flow first
2. Phase 1 Alignment (Modify Allowed) — Parse specification.yaml + test files (AST); validate alignment rules; generate alignment report; optionally fix tests to align with requirements; require score ≥80% to proceed
3. Coverage Mapping — Map each REQ-* to at least one test with @test annotation; map each AC-* to at least one assertion; verify @test, @intent, @covers tags
4. Pre-Phase 2 Freeze — Lock test directories via read-only subagent convention (reviewer tools restricted to [Read, Grep, Glob], no Write/Edit/Bash); return confirmation before proceeding
5. Phase 2 Execution (Frozen) — Run all tests; Agent BLOCKED from modifying/deleting/skipping tests; analyze failures into 4 categories
6. Failure Classification — For each failing test, output with Type, Test, Root Cause, Action fields; SPECIFICATION_ERROR requires ESCALATE_TO_HUMAN with user options (A: fix spec → re-Phase 1, B: confirm spec → modify code, C: clarify ambiguity)
7. JSON Report Output — Generate alignment report as valid JSON

核心原则

测试是系统的防护网，也是系统的使用手册。测试必须准确反映原始需求和设计方案。

Output Format (MANDATORY)

MANDATORY FILE OUTPUT: When invoked, you MUST write this report as valid JSON to the file .sprint-state/phase-outputs/test-alignment-report.json in the project root. Stdout output alone is not sufficient — xp-gate phase-transition 4 completed validates the file, not the stdout.

{
  "alignment_status": "PASS|FAIL|BLOCKED",
  "phase": "1|2",
  "score": 85.5,
  "head_commit": "<git rev-parse HEAD output>",
  "spec_hash": "<SHA-256 of specification.yaml if exists, else null>",
  "timestamp": "<ISO 8601>",
  "misaligned_tests": [
    {"test_name": "test_checkout", "spec_requirement": "REQ-003", "gap": "Missing @test annotation"}
  ],
  "anti_pattern_detected": false,
  "errors": []
}

Eval assertions check for: alignment_status, phase, score, head_commit, spec_hash, anti_pattern_detected.

Triggers

Automatic

• BUILD (TDD + review) Round 1 after Driver outputs tests
• Gate 1 verification before proceeding
• Phase 5 SHIP release before deployment (native phase-5 steps)

Manual

• /test-specification-alignment, /verify-tests, "run tests", "verify tests"

Scope

IN Scope

• Validation of test alignment against specification.yaml
• Phase 1: Test modification to improve alignment
• Phase 2: Test execution with frozen test files
• Failure classification (BUSINESS_CODE_ERROR, TEST_DATA_ERROR, SPECIFICATION_ERROR, ENVIRONMENT_ERROR)
• JSON report generation

OUT Scope

• Modifying specification.yaml during Phase 2 (requires ESCALATE_TO_HUMAN)
• Skipping/deleting/skipping tests
• Modifying test assertions to force passes
• Business logic implementation (BUILD phase)
• Environment configuration

Boundaries

• Start: After BUILD (TDD + review) Round 1 outputs tests
• End: Terminal State ✅ ALL_TESTS_PASS or ESCALATE_TO_HUMAN
• Inputs: specification.yaml, tests/ directory
• Outputs: Alignment report (JSON), test execution report, failure classification

核心流程

Phase 0: 准备
  ├─ 验证 specification.yaml 存在
  ├─ 验证 tests/ 目录存在
  └─ ❌ 缺失 → BLOCK + 提示用户

Phase 1: 对齐验证 (可修改测试)
  ├─ 解析 specification.yaml (YAML parser)
  ├─ 解析测试文件 (AST parser)
  ├─ 验证对齐规则
  └─ (可选) 修复测试

Checkpoint: Alignment Score >= 80%?
  ├─ NO → BLOCK
  └─ YES → 继续

⭐ Pre-Phase 2: 调用 freeze skill 锁定测试目录

Phase 2: 执行测试 (禁止修改测试)
  ├─ 运行所有测试
  ├─ IF Agent 尝试修改测试 → freeze 拦截
  ├─ 失败分析: 业务代码/测试数据/Specification/环境
  │   └─ Specification 错误 → ESCALATE_TO_HUMAN
  └─ 全部通过 → 继续

⭐ Post-Phase 2: 调用 unfreeze skill

Terminal State: ✅ ALL_TESTS_PASS

强制对齐规则

Failure Classification

Specification 错误处理选项

• A: 修正 Specification → 重新 Phase 1 (unfreeze tests)
• B: 确认 Specification 正确 → 修改业务代码 (用户明确授权)
• C: 补充 Specification 澄清歧义 (unfreeze tests)

State Machine

Anti-Patterns

Terminal State Checklist

• specification.yaml 存在且可解析
• tests/ 目录存在且有测试文件
• Phase 1 对齐验证完成, score ≥ 80%
• Pre-Phase 2: freeze skill 已调用
• Phase 2: 所有测试已执行, 无修改违规
• Post-Phase 2: unfreeze skill 已调用
• 所有测试通过, 报告已生成

IF Specification 问题 → CANNOT claim complete → MUST ESCALATE_TO_HUMAN

Details → see references/