Validation & Diagnostics

Overview

Structured process for comparing planned vs implemented work, generating diagnostic reports, and identifying gaps. Used by validation agents to provide concrete, evidence-based assessments.

Key Concepts

The Trust Problem

Agents will claim work is complete when it isn't. Validation must be:

• Observable — Based on test output, not agent claims
• Deterministic — Same input produces same verdict
• Evidence-based — Every finding references concrete code or test results
• Structured — Consistent format for easy human review

Patterns & Techniques

Planned vs Implemented Comparison

Step 1: Extract planned items From the spec, issue, or plan, extract a checklist of:

• Acceptance criteria
• Required endpoints/functions
• Expected model fields
• Required test coverage
• UI components specified

Step 2: Verify each item For each planned item, check:

• Does the code exist? (file search, grep)
• Does it match the specification? (read and compare)
• Is it tested? (find corresponding test)
• Does the test pass? (run and capture output)

Step 3: Generate drift report

## Drift Analysis: [Feature]

| Planned Item | Status | Evidence |
|-------------|--------|----------|
| [spec item] | ✅ Implemented | [file:line] |
| [spec item] | ⚠️ Partial | [what's missing] |
| [spec item] | ❌ Not found | [searched in...] |
| [unplanned] | ⚡ Scope creep | [file:line] |

Diagnostic Report Template

## Diagnostic Report: [Component/Feature]

### Summary
[1-2 sentence verdict]

### Test Results

[Actual test output — copy/paste, not paraphrased]

### Code Quality
| Metric | Result |
|--------|--------|
| Type hints present | ✅ / ❌ |
| Pydantic models used | ✅ / ❌ |
| Config class used (no hardcoded secrets) | ✅ / ❌ |
| Error handling present | ✅ / ❌ |
| Logger used (not print) | ✅ / ❌ |

### Findings
| # | Severity | Finding | Location | Recommendation |
|---|----------|---------|----------|----------------|
| 1 | [High/Med/Low] | [what] | [file:line] | [fix] |

### Coverage Gaps
[Code paths without tests]

### Verdict
✅ PASS | ⚠️ CONDITIONAL PASS | ❌ FAIL
[Explanation with evidence]

HASTE-Specific Validation Checks

Feedback Loop

When validation reveals a miss:

1. Document the miss in the diagnostic report
2. Identify the pattern (was it a convention violation? missing test? spec ambiguity?)
3. Propose a skill update or instruction addition to prevent recurrence
4. Flag to the Orchestrator for tracking

Decision Framework

Common Pitfalls

• Accepting "tests passed" without seeing output — Always run and capture
• Validating only happy path — Check error handling and edge cases
• Skipping convention checks — HASTE has specific patterns that must be followed
• Not checking for scope creep — Extra changes can introduce regressions