validation-pipeline

Shared validator-orchestration playbook for the three agency domains. Defines WHICH validator runs WHEN and HOW it is recorded in engagement/validation-log.md. In the agency cascade the engagement-workflow validate phase runs the applicable validators (by agentType), adversarially verifies each finding, and writes the validation-outputs/*.json proof files; this skill is the reference that phase follows. (A lead run standalone, or a focused mid-engagement check, may invoke validators via the Task tool or validator_lg.py — see §"Execution patterns".) This skill is read, not called.

Core principles

1. Validators run before handoff, not after. On M/L tier, adversary (adversary_lg.py --consilium {M|L}) provides the second-opinion check at acceptance time, but it does NOT replace lead's pre-handoff validator pass. Lead's first pass must already be complete and logged.
2. Validation log is append-only. One entry per validator per run, with verdict + findings count + resolution if findings addressed.
3. Findings trigger re-dispatch, not hand-wave. If validator flags ≥1 issue above "minor", dispatch the specialist who produced the artefact for fix, then re-run the validator on the fix ONLY.
4. Scope to what changed. Don't re-validate artefacts untouched by the engagement. Cost-efficiency matters — adversary on M/L is the second-opinion safety net, not a substitute for missing pre-handoff validator runs.

Log format

Two artefacts work together: a per-validator JSON output (proof of run) and a markdown log indexing them.

engagement/validation-outputs/{validator}-iter-{N}-{timestamp}.json (mandatory)

Every validator run produces an output file in this directory. The file is the validator's actual output (verdict, findings, metrics). Without this file, the run is presumed not to have happened — director rejects with validator output missing — likely not run.

Naming: {validator-name}-iter-{N}-{YYYYMMDD-HHMMSS}.json (e.g. code-reviewer-iter-1-20260506-143211.json).

Contents are the validator agent's JSON return value, captured verbatim.

engagement/validation-log.md (index, append-only)

# Validation log — {engagement-name}

## Iteration 1

### {validator-name}
- Ran on: {artefact path(s)}
- Triggered by: {reason — mandatory rule / risk flag / user criteria}
- Verdict: {clean | N findings}
- Findings summary: {short bullet list if any}
- Resolution: {specialist re-dispatched + fix applied + re-run clean | deferred with justification}
- **Output file**: `engagement/validation-outputs/{validator}-iter-1-{timestamp}.json` (REQUIRED — director cross-checks file exists)

### {next validator}
...

## Iteration 2
(appended after director reject + rework)
...

Lead duty: capture validator output to file

Every Task-tool dispatch to a validator agent must end with the lead writing the agent's JSON return value to engagement/validation-outputs/{name}-iter-{N}-{timestamp}.json. This is non-optional. The lead's prompt to the validator should explicitly ask for JSON return value (matching the agent's documented schema).

If a validator returns text instead of JSON (older agent / human-readable mode) — wrap it: {"format": "text", "verdict": "...", "raw": "..."} and save. The wrapper is acceptable; missing file is not.

Per re-run, not just per validator. A validator RE-RUN after rework at iteration N writes its OWN {validator}-iter-N-{ts}.json — a prior iter-1 output does NOT satisfy an iter-2 re-run. Every (re-run) entry in validation-log.md must cite that output: path. handoff-precheck.py's per-validator freshness check WARNs on a (re-run) entry whose {validator}-iter-N output is missing (claimed-without-proof, seen in the field: a logged test-reviewer (re-run) with no test-reviewer-iter-2 slipped past the old any-iter-N check).

Concurrency rules (preventing race corruption)

validation-log.md is written by the lead only. Specialists do NOT append to it directly — the lead aggregates after each Task return. This avoids POSIX O_APPEND non-atomicity for messages > 4 KB.

validation-outputs/{name}-iter-N-{timestamp}.json is per-validator-per-iteration with timestamp suffix → collision-free even under parallel dispatch. Each lead-side write is a single Path.write_text(...), atomic.

executor-reports/{name}.md is per-specialist (one file per specialist). When the lead re-dispatches the same specialist for iter N+1, the specialist appends ## Iteration N+1 section — never overwrites prior. Reads happen between dispatches, so concurrent writes from the same specialist do not occur.

If the lead must dispatch ≥2 specialists in parallel that both touch the same artefact (rare, usually a project-side file like tech-spec.md lock), explicitly serialise — protocol does not currently model file-level locks. For pipeline parallelism, ensure each specialist's outputs land in disjoint paths.

Director duty: cross-check files exist

For every validator listed in validation-log.md:

• File exists at the cited path? If no → REJECT validator output missing — likely not run.
• File parses as JSON? If no → REJECT validator output corrupted.
• File's verdict field matches what the log claims? If diverges → REJECT validator-log discrepancy: log says {X}, output file says {Y}.

Mechanical check is part of handoff-precheck.py (sub-check validator-outputs).

Lead self-recovery from broken validator output (Tier 14)

Retry-on-parse-failure is absorbed into validator_lg.py's retry edge: when a validator's output fails to parse as JSON or is missing required fields (verdict, plus methodology for the numerical validators accessibility-validator | performance-validator | security-auditor | ux-review | anti-pattern-detector), the graph automatically re-dispatches that validator. Up to 2 retries per validator per iteration before escalation. Use --resume to skip already-clean validators on re-run.

python ~/.claude/scripts/validator_lg.py engagement/ --auto --resume

Anti-pattern: editing the JSON file by hand to "fix" it. Re-dispatch is the only valid recovery — hand-edits hide the original validator's actual finding.

Domain matrix — mandatory validators

All domains, before the waves run

This one runs BEFORE dispatch, not after it, and it is the only entry in this file that does. A decomposition defect (umbrella task, missing source anchor or common objective, dropped non-goals, same-shape work split across one specialist seat each) costs one edit here and a whole wave once the specialists are already running. Rubric: ~/.claude/skills/task-decomposition/references/decomposition-integrity.md. The mode=diff and mode=executor-reports runs of the same validator stay dev-only and stay after the waves; see the Dev table below.

Marketing

Dev

Design

Cross-domain (≥2 domains contributing artefacts)

product-context-validator checks coherence ACROSS domains — API↔UI shape, copy↔shipped behavior, brand voice consistency, persona alignment, metaphor collisions. Not a substitute for single-domain validators; it catches misalignment they miss.

Embedding-researcher prerequisites (NEW — Layer 5 of agency)

Before design/brand/marketing specialists begin work on an existing-project engagement, the orchestrating lead SHOULD dispatch the matching researcher to produce {engagement}/design-research.md / brand-research.md. These are READ by specialists during their work, NOT validators — they answer "what already exists in this project" before new work decides to ignore or align with it.

Researchers are SKIPPED for: S-tier single-asset work; new-project bootstrap (nothing to research); engagements explicitly scoped as rebrand / redesign-from-scratch.

Execution patterns

Primary path — the engagement-workflow validate phase

In the agency cascade, validation runs INSIDE the engagement-workflow Workflow. Its validate phase runs every validator in plan.validators (each as its agentType, in parallel), then adversarially verifies each non-info finding (an independent skeptic per finding; refuted findings dropped), and writes the per-validator validation-outputs/{validator}-iter-{N}-{ts}.json proof files with the canonical envelope — byte-identical to what validator_lg.py writes, so handoff-precheck.py and the manager consume them unchanged. This is the path for ALL tiers when the engagement runs through the Workflow; the lead only LISTS the applicable validators in the plan (per the domain matrix above).

Standalone path — validator_lg.py (out-of-Workflow + HITL-in-validation)

validator_lg.py is retained as the standalone tool for two cases: (a) a validation run OUTSIDE the Workflow (a lead/manager doing a focused mid-engagement check), and (b) HITL-in-validation — --interrupt-on-critical pauses on any critical finding for a human directive (the Workflow validate phase has no native mid-run interrupt). Same output contract + retry-on-parse-failure, crash-resume (--resume), and observability hooks.

The output contract is IDENTICAL across paths — the same validation-outputs/{validator}-iter-{N}-{ts}.json files with the canonical block — so downstream (handoff-precheck.py, manager acceptance) consume them the same way regardless of which path produced them.

1. Standalone manual parallel dispatch. A lead run outside the Workflow invokes each validator via the Task tool in a single message — Claude Code runs them in parallel. Use on S-tier or for legitimately custom partial coverage (document the reason in validation-log.md).

2. validator_lg.py LangGraph fan-out (standalone tool) — out-of-Workflow + HITL. Single command runs all applicable validators in parallel via Send fan-out, writes per-validator JSON to validation-outputs/, appends to validation-log.md. Use when:

   • Engagement needs the FULL applicable validator set per validation-pipeline rules (use --auto to plan from criteria.md domain + predicates).
   • Lead wants crash-resume (--resume skips already-completed validators, re-runs missing + failed).
   • Lead wants retry-on-parse-failure (built into the graph as a retry edge).
   • Validator coverage benefits from observability (LangSmith hook auto-instrumented).

   # Auto-plan from criteria.md + run all applicable validators in parallel:
   python ~/.claude/scripts/validator_lg.py engagement/ --auto
   
   # Explicit subset:
   python ~/.claude/scripts/validator_lg.py engagement/ \
       --validators code-reviewer,security-auditor,reality-checker
   
   # Crash-resume after partial failure:
   python ~/.claude/scripts/validator_lg.py engagement/ --auto --resume
   
   # M/L tier — pause for human directive on any critical finding:
   python ~/.claude/scripts/validator_lg.py engagement/ --auto --interrupt-on-critical
   # Graph pauses at critical_check; prints thread_id + resume hint to stderr.
   # Manager / human inspects validation-outputs/ and resumes:
   python ~/.claude/scripts/validator_lg.py engagement/ \
       --resume-interrupt <thread_id> --decision PROCEED|REJECT|DIRECTED [--note ...]
   # Resume writes engagement/human-directive.md via human-directive.py
   # (same format adversary_lg.py uses), so manager phase can pick up cleanly.

   --interrupt-on-critical is silently ignored on tier=S (no manager phase exists for S, so pausing has no consumer). It also requires the SQLite checkpointer; --no-checkpoint together with the flag is auto-promoted with a stderr warning. Defaults are unchanged: without the flag, the graph runs to completion exactly as before.

   Output contract is IDENTICAL to manual dispatch — handoff-precheck.py and director acceptance consume the same validation-outputs/{validator}-iter-{N}-{ts}.json files unchanged.

   Canonical envelope: every validation-outputs/*.json file written by validator_lg.py now carries a canonical block alongside the raw validator output. The canonical schema is the cross-validator stable shape that downstream consumers (manager, Langfuse, analytics) read:

   {
     "...raw fields from the validator (status, findings, summary, ...)": "...",
     "canonical": {
       "schema_version": "1.0",
       "validator": "code-reviewer",
       "validator_type": "judgement",
       "verdict": "approved | approved_with_caveats | changes_required | suspicious | skipped | error",
       "summary": "...",
       "methodology": null,
       "findings": [
         {
           "id": "code-reviewer#0",
           "severity": "critical | high | medium | low | info",
           "category": "...",
           "issue": "normalized message",
           "fix": "...",
           "evidence": null,
           "location": null
         }
       ],
       "metrics": null
     }
   }

   Raw fields remain unchanged — manual-dispatch validators (older agents that predate canonicalization) still write the raw shape. Consumers MAY read canonical when present and fall back to raw fields otherwise. Severity normalization absorbs the 15+ free-form severity strings into 5; verdict normalization collapses LENIENT_STATUSES into 6 canonical values. Numerical validators (accessibility / performance / security-auditor / ux-review / anti-pattern-detector) carry validator_type: "numerical" + a metrics block extracted from top-level numeric extras.

   The canonical envelope is also surfaced in the event ledger (validator_completed events) as canonical_verdict field + output_schema_version: "1.0" so cross-engagement aggregation has a stable key.

Sequential (when validator input depends on prior validator)

Only when output of validator A is explicit input to validator B:

• completeness-validator AFTER userspec-quality-validator (needs validated user-spec)
• post-deploy-qa AFTER pre-deploy-qa (needs deferred criteria list)

Re-dispatch loop

validator → finding → re-dispatch specialist → fix → re-run validator on fix only → record in log

Cap at 2 re-dispatches per finding per iteration. If 3rd re-dispatch needed, escalate: "finding cannot be resolved; defer with scope note OR mark as blocker and surface to director".

When to skip (explicit permission)

You may skip a mandatory validator ONLY if:

• Criteria.md "Explicitly out of scope" covers it.
• User scope-sync waived it in writing.

Never skip silently. Log the waiver in validation-log.md:

### security-auditing (SKIPPED)
- Reason: criteria.md §out-of-scope — "auth is unchanged, skip security audit"
- Waived by: user scope-sync 2026-04-20

Director will verify the waiver exists.

Anti-patterns

• Don't run validators after handoff. Adversary at M/L acceptance provides cross-perspective second opinion, but does NOT substitute for lead's pre-handoff validator run — your first pass is your credibility.
• Don't hide findings. Every finding → resolution or deferral with justification.
• Don't re-run the entire validator on partial fix. Re-run on the fixed artefact only.
• Don't skip reality-checker / skeptic because "the specialist is trustworthy". Trust applies to humans; these validators exist because LLM specialists hallucinate.
• Don't parallelize sequential dependencies. Order matters when validator B needs A's output.
• Don't skip ux-review on ux_heavy: true engagements — that's the validator that keeps Wave-2-style visual regressions from making it past handoff.
• Don't skip anti-pattern-detector on code waves — catches skipped tests / dead code / hidden-tab fakes that other validators miss.
• Don't accept Cross-validation as "all match" without the actual list. The list is the proof; "trust me, they match" = phantom claim.