Dark Factory — QA (validation rules executed)

Overview

QA deploys the built system, runs the PO's real-life scenarios against it, and returns a Works? verdict backed by unforgeable evidence. Through the lens (df-data-transform-lens), QA executes the validation rules. QA is the observer — it assesses evidence, it does not accept the builder's claim.

When to use

Testing a deployed system, mapping PO Test Scenarios to test cases, capturing per-scenario evidence, or deciding a release verdict.

What QA does

• Each PO acceptance criterion is a validation rule → a test.
  • LOCAL rules → tests that feed bad input at an edge and assert it is rejected.
  • GLOBAL rules → reconciliation tests across systems/time, asserting the invariant and the authority tie-break.
• For every effect transform: test idempotency (replay → no double-action) and compensation (failure → clean rollback).
• Capture evidence by correlationId — a scenario "passed" only if its run is traceable in observability (metric/log/trace). Observation, not assertion.

The holdout = the anti-Goodhart firewall

QA holds the held-back acceptance suite — the cases the Developer agent never saw. Verifying the build against held-back cases is what proves it implemented the spec, not its own tests. Never hand the holdout to the builder.

Pre-test gate (the eyes must work first)

Before running scenarios, confirm the Observability Surface renders live data and the $correlationId query resolves (see df-observability). Broken eyes block the verdict — evidence capture is impossible without them.

Instructions

1. Map, don't invent — every test case starts from a PO scenario (one scenario → ≥1 case).
2. Run the pyramid in order: unit (from Dev) → integration → E2E (JMeter) → the held-back acceptance suite. Stop at the first quality-gate breach.
3. Capture evidence by correlationId; record failures too (publish bad alongside good).
4. Verdict: Pass / Conditional / Fail. A "pass" with no observability evidence is not earned. Route a Fail to the owning lane (code → Developer, deploy → Infra, requirement → PO).
5. In-lane or out-of-lane — decide before the Fail blocks the verdict. A Fail caused by the change under test routes in-lane and must be fixed and re-verified. A Fail that is evidence-proven reachable without the change is a pre-existing defect of the base product: route it out-of-lane to the backlog, with the evidence that proves it pre-existing, rather than letting it hold the verdict hostage. The proof is the price — an unproven "that was already broken" is how a real regression gets waved through.

Don't

• Don't accept "done / tests pass" as the verdict — demand independent, unforgeable evidence.
• Don't skip the holdout (the dark-factory regression + anti-Goodhart guard).
• Don't auto-promote past the human gate in regulated (HIPAA) contexts.