verification-oracle-architect
Skill4 starsChanged 31 days agoMITView on GitHub
Enforces high-fidelity, zero-mock verification architecture—replacing banned module mocks (vi.mock, jest.mock, unittest.mock.patch) and weak static happy-path tests with Verified In-Memory Fakes backed by Shared Contract Test Suites, Property-Based & Metamorphic Testing (fast-check, Hypothesis, proptest), and Seeded Deterministic Chaos Fault Injection. Use when writing unit/integration/contract tests, replacing mocks, testing financial/ledger math, parsers, serializers, or state machines, verifying idempotency and rollback under network failures, or auditing test suites for tautological assertions — even if the user doesn't explicitly say "verification oracle". Do NOT use for purely visual CSS adjustments or static markdown copy edits that have no executable behavioral contract.
In this repository47 other files it gives its agents.
- AGENTS.mdAGENTS.md
- anti-slop-engineeringSkill
- anti-slop-engineeringSkill
- api-ergonomics-and-contract-governorSkill
- apple-craft-and-frame-budgetSkill
- apple-craft-and-interaction-physicsSkill
- asset-provenance-guardSkill
- backprop-guardSkill
- chaos-and-degradation-choreographerSkill
- chromatic-drift-guardSkill
- cognitive-load-and-attention-governorSkill
- comment-budget-guardSkill
- context-window-and-token-economics-architectSkill
- cryptographic-agility-and-secret-lifecycle-gateSkill
- decision-guardSkill
- demock-guardSkill
- design-system-drift-guardSkill
- deterministic-simulation-and-property-testingSkill
- distributed-consensus-and-concurrency-gateSkill
- finops-and-unit-economics-governorSkill
- future-scale-gateSkill
- grumpy-senior-staff-engineerSkill
- honeypot-trap-guardSkill
- house-tell-guardSkill
- install-anti-slopSkill
- llm-eval-and-prompt-injection-fortressSkill
- mandela-guardSkill
- mechanical-sympathy-and-perf-gateSkill
- mechanical-sympathy-profilerSkill
- motion-runtime-guardSkill
- observability-and-slo-architectSkill
- observability-and-slo-gateSkill
- paranoid-principal-architectSkill
- pr-boundary-guardSkill
- privacy-and-data-lineage-gateSkill
- re0-handoff-guardSkill
- reduction-guardSkill
- repo-memory-and-invariant-stewardSkill
- root-fix-gateSkill
- safe-evolution-and-rollout-choreographerSkill
- scientific-fault-isolationSkill
- styleseed-guardSkill
- supply-chain-and-sbom-sentinelSkill
- tagore-guardSkill
- type-level-and-ast-slop-gateSkill
- ui-and-copy-disciplineSkill
- zero-downtime-migration-gateSkill
The file
We split this skill into the rules it gives an agent
11 rules. About 0 are common, 2 are distinctive, and 9 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
9
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 16
When strict engineering policies (install-anti-slop / anti-slop/no-module-mocking) ban vi.mock, jest.mock, sinon.stub, and …
- Very specific: line 22
Verified In-Memory Fakes (
100%Contract Parity): Every in-memory fake adapter (InMemoryLedgerRepository) runs against the exact same … - Very specific: line 46
Instead, implement a Stateful In-Memory Fake Adapter that faithfully models constraints: primary/unique key collisions, composite tenant …
- Very specific: line 56
Never assume the network, disk, or downstream RPC succeeds cleanly, and never use non-deterministic Math.random() or wall-clock setTimeout …
- Very specific: line 59
Every chaos run must assert State Invariant Recovery: after retries complete or abort, zero partial/orphaned rows exist, balances reconcile …
- Distinctive: line 62
Verify outputs via Inverse Operations, Algebraic Conservation Laws, Reference Model Comparison (comparing a complex optimized algorithm …
- Very specific: line 70
Contract suites are parameterized by an adapter factory: Write export function runLedgerStorageContractSuite(name: string, createAdapter: …
- Very specific: line 76
Ban assertion-free and tautological tests: Every test case must assert concrete state transitions and domain invariants—never …
- Very specific: line 93
Before writing a single describe(), test(), or def test_() block, answer these five questions:
- Very specific: line 121
Mutation Falsification & Guard Audit: Verify Red-Green falsification against a deliberate production mutation and audit the verification …
- Distinctive: line 123
When a durable verification-oracle record is warranted, save a JSON assessment following the Verification Oracle Record Contract and audit …
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Skill: Verification Oracle Architect (Verified Fakes, Property Invariants & Deterministic Chaos)
A mock proves only what you assumed the dependency would do. An example test proves only that
2 + 2 = 4once. Build verification oracles that cannot be lied to: Contract-Verified In-Memory Fakes, Property-Based Invariants, and Seeded Deterministic Chaos.
Why This Skill Is Needed & Target State
When strict engineering policies (install-anti-slop / anti-slop/no-module-mocking) ban vi.mock, jest.mock, sinon.stub, and unittest.mock.patch, AI coding agents frequently stall or regress into Test-Oracle Slop:
- Unverified Ad-Hoc Stubs: Replacing
vi.mock()with inline fake lambdas (const repo = { save: async () => true }) that ignore unique constraints, concurrency version checks, and transaction rollbacks—drifting silently from the real database. - Toy Happy-Path Examples (
2 + 2 = 4): Testing financial split calculations, parsers, or state machines with two hardcoded inputs while missing penny-rounding drift, integer overflow, Unicode boundary corruption, and illegal state transitions. - Fair-Weather Concurrency & Flaky
sleep()Band-Aids: Testing retry and rollback logic only when the network succeeds, or sprinklingawait new Promise(r => setTimeout(r, 100))and unseededMath.random()into tests, creating non-reproducible CI flakes.
The target state is High-Fidelity Verification Architecture:
- Verified In-Memory Fakes (
100%Contract Parity): Every in-memory fake adapter (InMemoryLedgerRepository) runs against the exact same parameterized Contract Test Suite (runAdapterContractSuite) as the production adapter (PostgresLedgerRepository). - Property-Based & Metamorphic Oracles: Financial arithmetic, parsers, serializers, and state machines are verified across hundreds of generated adversarial inputs (
fast-check,Hypothesis,proptest, Gorapid) checking universal invariants (conservation of cents, roundtrip invertibility, idempotency, metamorphic scaling). - Seeded Deterministic Chaos Testing: Network delays, dropped connections mid-transaction, duplicate webhook deliveries, and clock skew are injected deterministically via a seeded PRNG (
seed: 0x5EED1234) and Virtual Clock to prove atomic rollback and idempotency without a singlesleep().
Activation and Boundaries
| Situation | Behavior |
Writing tests where the agent is tempted to use vi.mock, jest.mock, sinon, or unittest.mock.patch | Ban module mocking; extract a narrow Port interface and build a Stateful In-Memory Fake verified by a Shared Contract Test Suite |
| Testing financial calculations, fee/tax allocations, currency conversions, or billing ledgers | Require Property-Based Testing (fast-check / Hypothesis / proptest) asserting conservation of money ($\sum \text{parts} = \text{total}$) and integer-cent invariants |
| Testing parsers, compilers, serializers, URL/query codecs, or cryptographic envelopes | Require Property-Based Roundtrip (decode(encode(x)) === x) and adversarial boundary fuzzing |
| Testing state machines, workflow engines, order lifecycles, or permission gates | Require Property-Based Model/Transition testing proving no command sequence can reach a forbidden state |
| Testing retries, distributed sagas, webhooks, idempotency keys, or database transaction rollbacks | Require Seeded Deterministic Chaos fault injection (seeded PRNG + Virtual Clock) simulating mid-flight network drops, timeouts, and duplicate delivery |
| Auditing existing tests or PR test diffs for quality and falsifiability | Scan for banned module mocks, unverified inline stubs, toy 1-input assertions, unseeded Math.random(), setTimeout sleeps, and verify Red-Green mutation falsification |
| Static prose edits, README formatting, or pure visual CSS token tweaks without logic | Do not activate; use anti-slop-engineering or apple-craft-and-frame-budget |
The Five Absolute Laws
- The Zero-Mock & Shared Contract Parity Law (Verified In-Memory Fakes):
- Module patching (
vi.mock,jest.mock,mock.module,unittest.mock.patch,monkeypatch) and unverified one-liner return stubs (mockResolvedValue) are strictly prohibited for domain and persistence boundaries. - Instead, implement a Stateful In-Memory Fake Adapter that faithfully models constraints: primary/unique key collisions, composite tenant isolation (
org_id), optimistic locking (versionmismatch), cursor ordering, and atomic state snapshots on transaction rollback. - Mandatory Contract Parity: A Fake is illegal unless paired with a parameterized Shared Contract Test Suite (
describeRepositoryContract(factory)) that executes identical behavioral assertions against both theInMemoryFakeand theRealAdapter.
- Module patching (
- The Property-Based & Metamorphic Invariant Law (Kill
2 + 2 = 4Example Slop):- Hardcoded example tests (
expect(splitInvoice(100, 3)).toEqual([34, 33, 33])) are insufficient for algorithmic, financial, parsing, or state-machine logic. - You must define Universal Invariants and execute them over $\ge 100$ randomized, shrinkable inputs (
fast-check,Hypothesis,proptest):- Conservation Invariant: Splitting, allocating, discounting, or transferring funds must conserve exact integer minor units ($\sum_{i} \text{allocation}_i = \text{principal}$) with zero cent creation or destruction.
- Invertibility (Roundtrip) Invariant: For every valid domain structure $x$, $\text{parse}(\text{format}(x)) \equiv x$ and $\text{deserialize}(\text{serialize}(x)) \equiv x$.
- Idempotency & Commutativity Invariant: Applying an idempotent mutation $N$ times yields the exact same state and response as applying it once: $f(f(s, op)) \equiv f(s, op)$.
- Metamorphic Relation Invariant: When no closed-form oracle exists (e.g., search ranking, fee tiers, graph routing), assert how transforming the input predictably transforms the output (e.g., doubling all line-item quantities with a linear tax rate exactly doubles net tax; permuting arrival order of commutative ledger postings preserves final balance).
- Hardcoded example tests (
- The Seeded Deterministic Chaos Law (Reproducible Fault Injection):
- Never assume the network, disk, or downstream RPC succeeds cleanly, and never use non-deterministic
Math.random()or wall-clocksetTimeoutsleeps in tests. - Inject faults through a Deterministic Chaos Harness controlled by an explicit 32-bit integer seed (
CHAOS_SEED) and a controllable Virtual Clock:- Deterministically inject Pre-Write Failures (connection refused before mutation), Mid-Flight / Post-Write Drops (mutation committed in storage, but socket drops before ACK reaches caller), Transient Serialization Conflicts (
40001/503), Out-of-Order / Duplicate Retries, and Clock Skew / Lease Expiry. - Every chaos run must assert State Invariant Recovery: after retries complete or abort, zero partial/orphaned rows exist, balances reconcile to the cent, and any test failure prints the exact
seed(Reproduce with: CHAOS_SEED=0x9F4A2C01) for instant deterministic replay.
- Deterministically inject Pre-Write Failures (connection refused before mutation), Mid-Flight / Post-Write Drops (mutation committed in storage, but socket drops before ACK reaches caller), Transient Serialization Conflicts (
- Never assume the network, disk, or downstream RPC succeeds cleanly, and never use non-deterministic
- The Independent Oracle Law (Never Mirror Implementation in the Test):
- A test must never copy-paste the production formula to compute its expected value (
expect(calcFee(x)).toBe(Math.floor(x * 0.029 + 30))). That only tests that two identical lines of code agree on the same bug. - Verify outputs via Inverse Operations, Algebraic Conservation Laws, Reference Model Comparison (comparing a complex optimized algorithm against a dead-simple $O(N)$ reference model in the test), or Boundary Exhaustion.
- A test must never copy-paste the production formula to compute its expected value (
- The Red-Green Mutation Falsification Law:
- A test suite is unverified until proven capable of failing. Before declaring any test suite complete, perform a Mutation Falsification Check: deliberately break the core production invariant (e.g., skip the rollback restore, drop the remainder cent, or disable the idempotency check), confirm the new oracle fails immediately with a clear diagnostic, and restore the production fix.
Invariant Rules
- Contract suites are parameterized by an adapter factory. Write
export function runLedgerStorageContractSuite(name: string, createAdapter: () => Promise<LedgerStoragePort>)once, and invoke it forInMemoryLedgerStoragein unit tests andPostgresLedgerStoragein integration tests. - In-Memory Fakes must clone state across boundary reads/writes. Never store or return direct mutable object references inside an In-Memory Fake; use
structuredClone()or immutable snapshots so caller mutations cannot corrupt internal fake storage without calling.save(). - In-Memory Fakes must support transactional rollback snapshots. When simulating a transaction (
withTransaction(tx => ...)), the fake must snapshot its state on entry and restore the exact snapshot if an error or chaos fault is thrown mid-transaction. - Configure Property-Based Tests with explicit run counts and seed logging. Always run at least
numRuns: 200(ormax_examples=200in Hypothesis) and ensure shrinking and seed reporting are enabled so counterexamples reduce to the minimal failing input. - All time-dependent behavior uses an injected
Clockport. BanDate.now()andnew Date()inside testable domain/retry logic; injectClockPort { nowMs(): number; advanceMs(delta: number): void }so TTL expiration, lease timeouts, and exponential backoff jitter execute in0msof real wall-clock time. - Distinguish Pre-Commit vs. Post-Commit Chaos Faults. A chaos harness must test both fail-before-commit (nothing written, safe to retry or abort) and fail-after-commit-before-response (side effect occurred, caller timed out and retries with the same
Idempotency-Key, which must return the original receipt without double-charging). - Ban assertion-free and tautological tests. Every test case must assert concrete state transitions and domain invariants—never
expect(result).toBeDefined(),expect(mockFn).toHaveBeenCalled(), orassert True.
Execution Router
| Task Shape | Primary Workflow | Reference Contract |
Replacing vi.mock/jest.mock or building a fast, high-fidelity test double for DB/RPC/Queue | Build Verified In-Memory Fake | Verified Fakes & Contracts |
| Testing financial math, parsers, codecs, compilers, or state machines beyond static examples | Author Property & Metamorphic Suite | Property-Based & Metamorphic Testing |
| Testing retries, idempotency, transactional rollback, or distributed resilience under network faults | Run Deterministic Chaos Simulation | Deterministic Chaos & Fault Injection |
| Durable verification-oracle assessment record generation & audit | Record Contract | scripts/verification_oracle_guard.py |
Mandatory Pre-Verification Interrogation Matrix
Before writing a single describe(), test(), or def test_() block, answer these five questions:
- The Mock Elimination Check: Am I reaching for
vi.mock,jest.mock,sinon.stub,unittest.mock.patch, or an inline{ method: () => fakeValue }stub? If yes, stop. What is the Port interface, how does the Stateful In-Memory Fake enforce real constraints (uniqueness, tenant isolation, version locking, rollback), and where is the Shared Contract Test Suite that proves parity with the real adapter? - The Invariant vs. Example Check: If I am testing calculations, allocations, parsing, serialization, or state transitions, what universal mathematical or structural invariant holds for all valid inputs $x \in \mathcal{X}$? Why am I writing a single
2 + 2 = 4example when a property generator (fc.assert(fc.property(...))) can test 500 edge cases (0, negative,MAX_SAFE_INTEGER, remainder cents, empty strings, Unicode) automatically? - The Post-Commit Chaos Check: What happens when the database or payment gateway commits the write on attempt #1, the network socket drops before the response arrives, and the caller retries attempt #2 with the same
Idempotency-Key? Does our Seeded Chaos Harness prove the system neither double-executes nor corrupts state? - The Determinism & Zero-Sleep Check: Does this test rely on wall-clock
setTimeout,sleep(), or unseededMath.random()? If this test fails on CI at 3:00 AM, what exact 32-bitseedreproduces the exact fault interleaving on a developer's laptop in1ms? - The Mutation Falsification Check: Which specific line of production code did I temporarily mutate to prove that this test suite turns RED when the invariant is violated?
Prohibited Anti-Patterns (Instant Rejection List)
- Module Mocking (
vi.mock,jest.mock,unittest.mock.patch,sinon.stub): Patching module imports or spying on internal implementation calls (expect(repo.save).toHaveBeenCalledWith(...)) instead of asserting observable state in a Contract-Verified In-Memory Fake. - Unverified Inline Stubs (Divergent Fakes): Creating ad-hoc mock objects inside test files that return hardcoded happy-path payloads while ignoring constraints (e.g., allowing duplicate keys, skipping version increments, or mutating objects in place without rollback).
- Static Happy-Path Example Slop on Rich Domains: Testing financial splits, tax calculations, parsers, or state machines exclusively with 1–3 hand-picked integers (
100,200) instead of Property-Based generators (fast-check/Hypothesis/proptest). - Tautological Implementation-Mirroring Oracles: Computing the
expectedvalue in a test by copying the exact production formula (expect(fn(a, b)).toBe(productionFormula(a, b))) or asserting onlytoBeDefined()/toBeTruthy(). - Unseeded Randomness in Tests (
Math.random(),random.random(),uuid4()without seed): Generating random test inputs or fault schedules without a deterministic, logged PRNG seed, making failures impossible to replay. - Wall-Clock
sleep()/setTimeoutSynchronization in Tests: Waiting real milliseconds/seconds for timeouts, retries, or TTL expirations instead of driving an injected Virtual Clock deterministically.
Required Output & Review Workflow
Structure all test engineering, mock-replacement refactors, and resilience verification tasks through this 4-phase discipline:
- Boundary & Invariant Specification: Define the Port interface, list the storage/protocol invariants the In-Memory Fake must enforce, specify the Universal Domain Properties (conservation, invertibility, idempotency, state-machine safety), and define the Chaos Fault Catalog.
- Verified Fake & Shared Contract Suite Implementation: Implement the Stateful In-Memory Fake (with
structuredCloneisolation, optimistic locking, and transactional rollback snapshots) alongside the parameterized Shared Contract Test Suite (runContractSuite) targeting both Fake and Real adapters. - Property-Based & Seeded Deterministic Chaos Suite: Author the property-based generators (
fast-check/Hypothesis/proptest) and the Seeded Deterministic Chaos Harness (PRNG + Virtual Clock) testing pre-commit drops, post-commit ACK drops, duplicate retries, and rollback recovery. - Mutation Falsification & Guard Audit: Verify Red-Green falsification against a deliberate production mutation and audit the verification record and test source with
scripts/verification_oracle_guard.py.
When a durable verification-oracle record is warranted, save a JSON assessment following the Verification Oracle Record Contract and audit it with:
npx anti-slop-engineering verification-oracle audit <record.json> --json
# or: python3 scripts/verification_oracle_guard.py audit <record.json> [--source <test_file.ts>] --jsonDomain References
- Verification Oracle Architect Playbook Suite: Central orchestration index for zero-mock verification, property testing, and deterministic simulation.
- Verified In-Memory Fakes & Shared Contract Suites Playbook: Port/Adapter boundaries, stateful in-memory fake mechanics (rollback snapshots,
structuredClone, CAS versioning, unique keys), and parameterized contract test suites across Fake and Real DB adapters. - Property-Based, Metamorphic & Deterministic Chaos Playbook: Algebraic invariants, metamorphic testing relations, stateful model-based command testing, seeded PRNG fault injection (FoundationDB/TigerBeetle deterministic simulation style), and Virtual Clocks.
- Verified In-Memory Fakes & Contracts Reference: Production TypeScript, Python, Rust, and Go templates for contract-verified fakes and transactional snapshot rollback.
- Property-Based & Metamorphic Testing Reference: Invariant catalogs and
fast-check/Hypothesis/proptestrecipes for financial math, codecs, and state machines. - Deterministic Chaos & Fault Injection Reference: Seeded PRNG (
Mulberry32), Virtual Clock, Pre-Commit vs. Post-Commit fault injection, and idempotency/rollback verification. - Verification Oracle Record Contract: JSON schema and verification rules for
scripts/verification_oracle_guard.py.
Failure Taxonomy
| Failure Mode | Observable Signal | Remediation |
| Banned module mock regression | Test imports vi.mock, jest.mock, sinon.stub, or unittest.mock.patch | Delete module mock; extract Port interface and inject a Contract-Verified In-Memory Fake |
| Divergent fake drift | Unit tests pass against fake, but production fails on unique constraint or rollback | Run the parameterized Shared Contract Test Suite against both InMemoryFake and RealAdapter |
| Penny-rounding / boundary leak | Static test split(100, 2) passes, but split(100, 3) loses 1 cent in prod | Replace static example with a fast-check / Hypothesis conservation property ($\sum \text{parts} = \text{total}$) over 200+ runs |
| Post-commit retry double-charge | Network drops after DB commit; retry executes a second charge | Inject post_commit_ack_drop fault via Seeded Chaos Harness and enforce atomic Idempotency-Key receipt replay |
| Non-reproducible CI flake | Test uses Math.random() or setTimeout(..., 50) and fails 1 in 40 runs | Replace with explicit seeded PRNG (seed: 0x5EED1234) and deterministic VirtualClock |
| Tautological green suite | Production bug introduced, yet all unit tests stay green | Enforce Red-Green Mutation Falsification; replace implementation-mirroring expectations with independent algebraic oracles |
Delegated Skills
anti-slop-engineering: Enforce Gate 2 (Test Integrity) and Gate 4 (Verify Beyond Green Tests) alongside verification oracle construction.install-anti-slop: Enforceanti-slop/no-module-mockingat the linter level while usingverification-oracle-architectto build the architectural replacement.scientific-fault-isolation: When a property test or deterministic chaos seed uncovers a minimal failing counterexample, use the 4-step root-cause isolation protocol.paranoid-principal-architect: Pair deterministic chaos testing with 10M-concurrent-user idempotency keys, CAS optimistic locking, and zero-trust boundary schemas.
Completion and Limits
The task is complete when all banned module mocks (vi.mock, jest.mock, unittest.mock.patch) and unverified inline stubs are replaced by a Stateful In-Memory Fake passing a Shared Contract Test Suite (100% parity with the real adapter), universal domain invariants are verified via Property-Based / Metamorphic tests (>= 100 runs with shrinking), resilience and rollback are proven under Seeded Deterministic Chaos fault injection with a Virtual Clock (zero sleep() calls), Red-Green mutation falsification is confirmed, and scripts/verification_oracle_guard.py reports zero findings. A review may validly output refuse or pause if a test suite relies on module mocking, unseeded randomness, or lacks contract parity proof.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

