agentleFS
imMamdouhaboammarimMamdouhaboammar/anti-slop-engineering/skills/verification-oracle-architect/SKILL.md

verification-oracle-architect

Skill4 starsChanged 31 days agoMITView on GitHub

Enforces high-fidelity, zero-mock verification architecture—replacing banned module mocks (vi.mock, jest.mock, unittest.mock.patch) and weak static happy-path tests with Verified In-Memory Fakes backed by Shared Contract Test Suites, Property-Based & Metamorphic Testing (fast-check, Hypothesis, proptest), and Seeded Deterministic Chaos Fault Injection. Use when writing unit/integration/contract tests, replacing mocks, testing financial/ledger math, parsers, serializers, or state machines, verifying idempotency and rollback under network failures, or auditing test suites for tautological assertions — even if the user doesn't explicitly say "verification oracle". Do NOT use for purely visual CSS adjustments or static markdown copy edits that have no executable behavioral contract.

The file

We split this skill into the rules it gives an agent

11 rules. About 0 are common, 2 are distinctive, and 9 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 9

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    When strict engineering policies (install-anti-slop / anti-slop/no-module-mocking) ban vi.mock, jest.mock, sinon.stub, and …

    line 16
  2. Very specific:

    Verified In-Memory Fakes (100% Contract Parity): Every in-memory fake adapter (InMemoryLedgerRepository) runs against the exact same …

    line 22
  3. Very specific:

    Instead, implement a Stateful In-Memory Fake Adapter that faithfully models constraints: primary/unique key collisions, composite tenant …

    line 46
  4. Very specific:

    Never assume the network, disk, or downstream RPC succeeds cleanly, and never use non-deterministic Math.random() or wall-clock setTimeout …

    line 56
  5. Very specific:

    Every chaos run must assert State Invariant Recovery: after retries complete or abort, zero partial/orphaned rows exist, balances reconcile …

    line 59
  6. Distinctive:

    Verify outputs via Inverse Operations, Algebraic Conservation Laws, Reference Model Comparison (comparing a complex optimized algorithm …

    line 62
  7. Very specific:

    Contract suites are parameterized by an adapter factory: Write export function runLedgerStorageContractSuite(name: string, createAdapter: …

    line 70
  8. Very specific:

    Ban assertion-free and tautological tests: Every test case must assert concrete state transitions and domain invariants—never …

    line 76
  9. Very specific:

    Before writing a single describe(), test(), or def test_() block, answer these five questions:

    line 93
  10. Very specific:

    Mutation Falsification & Guard Audit: Verify Red-Green falsification against a deliberate production mutation and audit the verification …

    line 121
  11. Distinctive:

    When a durable verification-oracle record is warranted, save a JSON assessment following the Verification Oracle Record Contract and audit …

    line 123

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Skill: Verification Oracle Architect (Verified Fakes, Property Invariants & Deterministic Chaos)

A mock proves only what you assumed the dependency would do. An example test proves only that 2 + 2 = 4 once. Build verification oracles that cannot be lied to: Contract-Verified In-Memory Fakes, Property-Based Invariants, and Seeded Deterministic Chaos.

Why This Skill Is Needed & Target State

When strict engineering policies (install-anti-slop / anti-slop/no-module-mocking) ban vi.mock, jest.mock, sinon.stub, and unittest.mock.patch, AI coding agents frequently stall or regress into Test-Oracle Slop:

  1. Unverified Ad-Hoc Stubs: Replacing vi.mock() with inline fake lambdas (const repo = { save: async () => true }) that ignore unique constraints, concurrency version checks, and transaction rollbacks—drifting silently from the real database.
  2. Toy Happy-Path Examples (2 + 2 = 4): Testing financial split calculations, parsers, or state machines with two hardcoded inputs while missing penny-rounding drift, integer overflow, Unicode boundary corruption, and illegal state transitions.
  3. Fair-Weather Concurrency & Flaky sleep() Band-Aids: Testing retry and rollback logic only when the network succeeds, or sprinkling await new Promise(r => setTimeout(r, 100)) and unseeded Math.random() into tests, creating non-reproducible CI flakes.

The target state is High-Fidelity Verification Architecture:

  1. Verified In-Memory Fakes (100% Contract Parity): Every in-memory fake adapter (InMemoryLedgerRepository) runs against the exact same parameterized Contract Test Suite (runAdapterContractSuite) as the production adapter (PostgresLedgerRepository).
  2. Property-Based & Metamorphic Oracles: Financial arithmetic, parsers, serializers, and state machines are verified across hundreds of generated adversarial inputs (fast-check, Hypothesis, proptest, Go rapid) checking universal invariants (conservation of cents, roundtrip invertibility, idempotency, metamorphic scaling).
  3. Seeded Deterministic Chaos Testing: Network delays, dropped connections mid-transaction, duplicate webhook deliveries, and clock skew are injected deterministically via a seeded PRNG (seed: 0x5EED1234) and Virtual Clock to prove atomic rollback and idempotency without a single sleep().

Activation and Boundaries

SituationBehavior
Writing tests where the agent is tempted to use vi.mock, jest.mock, sinon, or unittest.mock.patchBan module mocking; extract a narrow Port interface and build a Stateful In-Memory Fake verified by a Shared Contract Test Suite
Testing financial calculations, fee/tax allocations, currency conversions, or billing ledgersRequire Property-Based Testing (fast-check / Hypothesis / proptest) asserting conservation of money ($\sum \text{parts} = \text{total}$) and integer-cent invariants
Testing parsers, compilers, serializers, URL/query codecs, or cryptographic envelopesRequire Property-Based Roundtrip (decode(encode(x)) === x) and adversarial boundary fuzzing
Testing state machines, workflow engines, order lifecycles, or permission gatesRequire Property-Based Model/Transition testing proving no command sequence can reach a forbidden state
Testing retries, distributed sagas, webhooks, idempotency keys, or database transaction rollbacksRequire Seeded Deterministic Chaos fault injection (seeded PRNG + Virtual Clock) simulating mid-flight network drops, timeouts, and duplicate delivery
Auditing existing tests or PR test diffs for quality and falsifiabilityScan for banned module mocks, unverified inline stubs, toy 1-input assertions, unseeded Math.random(), setTimeout sleeps, and verify Red-Green mutation falsification
Static prose edits, README formatting, or pure visual CSS token tweaks without logicDo not activate; use anti-slop-engineering or apple-craft-and-frame-budget

The Five Absolute Laws

  1. The Zero-Mock & Shared Contract Parity Law (Verified In-Memory Fakes):
    • Module patching (vi.mock, jest.mock, mock.module, unittest.mock.patch, monkeypatch) and unverified one-liner return stubs (mockResolvedValue) are strictly prohibited for domain and persistence boundaries.
    • Instead, implement a Stateful In-Memory Fake Adapter that faithfully models constraints: primary/unique key collisions, composite tenant isolation (org_id), optimistic locking (version mismatch), cursor ordering, and atomic state snapshots on transaction rollback.
    • Mandatory Contract Parity: A Fake is illegal unless paired with a parameterized Shared Contract Test Suite (describeRepositoryContract(factory)) that executes identical behavioral assertions against both the InMemoryFake and the RealAdapter.
  2. The Property-Based & Metamorphic Invariant Law (Kill 2 + 2 = 4 Example Slop):
    • Hardcoded example tests (expect(splitInvoice(100, 3)).toEqual([34, 33, 33])) are insufficient for algorithmic, financial, parsing, or state-machine logic.
    • You must define Universal Invariants and execute them over $\ge 100$ randomized, shrinkable inputs (fast-check, Hypothesis, proptest):
      • Conservation Invariant: Splitting, allocating, discounting, or transferring funds must conserve exact integer minor units ($\sum_{i} \text{allocation}_i = \text{principal}$) with zero cent creation or destruction.
      • Invertibility (Roundtrip) Invariant: For every valid domain structure $x$, $\text{parse}(\text{format}(x)) \equiv x$ and $\text{deserialize}(\text{serialize}(x)) \equiv x$.
      • Idempotency & Commutativity Invariant: Applying an idempotent mutation $N$ times yields the exact same state and response as applying it once: $f(f(s, op)) \equiv f(s, op)$.
      • Metamorphic Relation Invariant: When no closed-form oracle exists (e.g., search ranking, fee tiers, graph routing), assert how transforming the input predictably transforms the output (e.g., doubling all line-item quantities with a linear tax rate exactly doubles net tax; permuting arrival order of commutative ledger postings preserves final balance).
  3. The Seeded Deterministic Chaos Law (Reproducible Fault Injection):
    • Never assume the network, disk, or downstream RPC succeeds cleanly, and never use non-deterministic Math.random() or wall-clock setTimeout sleeps in tests.
    • Inject faults through a Deterministic Chaos Harness controlled by an explicit 32-bit integer seed (CHAOS_SEED) and a controllable Virtual Clock:
      • Deterministically inject Pre-Write Failures (connection refused before mutation), Mid-Flight / Post-Write Drops (mutation committed in storage, but socket drops before ACK reaches caller), Transient Serialization Conflicts (40001 / 503), Out-of-Order / Duplicate Retries, and Clock Skew / Lease Expiry.
      • Every chaos run must assert State Invariant Recovery: after retries complete or abort, zero partial/orphaned rows exist, balances reconcile to the cent, and any test failure prints the exact seed (Reproduce with: CHAOS_SEED=0x9F4A2C01) for instant deterministic replay.
  4. The Independent Oracle Law (Never Mirror Implementation in the Test):
    • A test must never copy-paste the production formula to compute its expected value (expect(calcFee(x)).toBe(Math.floor(x * 0.029 + 30))). That only tests that two identical lines of code agree on the same bug.
    • Verify outputs via Inverse Operations, Algebraic Conservation Laws, Reference Model Comparison (comparing a complex optimized algorithm against a dead-simple $O(N)$ reference model in the test), or Boundary Exhaustion.
  5. The Red-Green Mutation Falsification Law:
    • A test suite is unverified until proven capable of failing. Before declaring any test suite complete, perform a Mutation Falsification Check: deliberately break the core production invariant (e.g., skip the rollback restore, drop the remainder cent, or disable the idempotency check), confirm the new oracle fails immediately with a clear diagnostic, and restore the production fix.

Invariant Rules

  1. Contract suites are parameterized by an adapter factory. Write export function runLedgerStorageContractSuite(name: string, createAdapter: () => Promise<LedgerStoragePort>) once, and invoke it for InMemoryLedgerStorage in unit tests and PostgresLedgerStorage in integration tests.
  2. In-Memory Fakes must clone state across boundary reads/writes. Never store or return direct mutable object references inside an In-Memory Fake; use structuredClone() or immutable snapshots so caller mutations cannot corrupt internal fake storage without calling .save().
  3. In-Memory Fakes must support transactional rollback snapshots. When simulating a transaction (withTransaction(tx => ...)), the fake must snapshot its state on entry and restore the exact snapshot if an error or chaos fault is thrown mid-transaction.
  4. Configure Property-Based Tests with explicit run counts and seed logging. Always run at least numRuns: 200 (or max_examples=200 in Hypothesis) and ensure shrinking and seed reporting are enabled so counterexamples reduce to the minimal failing input.
  5. All time-dependent behavior uses an injected Clock port. Ban Date.now() and new Date() inside testable domain/retry logic; inject ClockPort { nowMs(): number; advanceMs(delta: number): void } so TTL expiration, lease timeouts, and exponential backoff jitter execute in 0ms of real wall-clock time.
  6. Distinguish Pre-Commit vs. Post-Commit Chaos Faults. A chaos harness must test both fail-before-commit (nothing written, safe to retry or abort) and fail-after-commit-before-response (side effect occurred, caller timed out and retries with the same Idempotency-Key, which must return the original receipt without double-charging).
  7. Ban assertion-free and tautological tests. Every test case must assert concrete state transitions and domain invariants—never expect(result).toBeDefined(), expect(mockFn).toHaveBeenCalled(), or assert True.

Execution Router

Task ShapePrimary WorkflowReference Contract
Replacing vi.mock/jest.mock or building a fast, high-fidelity test double for DB/RPC/QueueBuild Verified In-Memory FakeVerified Fakes & Contracts
Testing financial math, parsers, codecs, compilers, or state machines beyond static examplesAuthor Property & Metamorphic SuiteProperty-Based & Metamorphic Testing
Testing retries, idempotency, transactional rollback, or distributed resilience under network faultsRun Deterministic Chaos SimulationDeterministic Chaos & Fault Injection
Durable verification-oracle assessment record generation & auditRecord Contractscripts/verification_oracle_guard.py

Mandatory Pre-Verification Interrogation Matrix

Before writing a single describe(), test(), or def test_() block, answer these five questions:

  • The Mock Elimination Check: Am I reaching for vi.mock, jest.mock, sinon.stub, unittest.mock.patch, or an inline { method: () => fakeValue } stub? If yes, stop. What is the Port interface, how does the Stateful In-Memory Fake enforce real constraints (uniqueness, tenant isolation, version locking, rollback), and where is the Shared Contract Test Suite that proves parity with the real adapter?
  • The Invariant vs. Example Check: If I am testing calculations, allocations, parsing, serialization, or state transitions, what universal mathematical or structural invariant holds for all valid inputs $x \in \mathcal{X}$? Why am I writing a single 2 + 2 = 4 example when a property generator (fc.assert(fc.property(...))) can test 500 edge cases (0, negative, MAX_SAFE_INTEGER, remainder cents, empty strings, Unicode) automatically?
  • The Post-Commit Chaos Check: What happens when the database or payment gateway commits the write on attempt #1, the network socket drops before the response arrives, and the caller retries attempt #2 with the same Idempotency-Key? Does our Seeded Chaos Harness prove the system neither double-executes nor corrupts state?
  • The Determinism & Zero-Sleep Check: Does this test rely on wall-clock setTimeout, sleep(), or unseeded Math.random()? If this test fails on CI at 3:00 AM, what exact 32-bit seed reproduces the exact fault interleaving on a developer's laptop in 1ms?
  • The Mutation Falsification Check: Which specific line of production code did I temporarily mutate to prove that this test suite turns RED when the invariant is violated?

Prohibited Anti-Patterns (Instant Rejection List)

  1. Module Mocking (vi.mock, jest.mock, unittest.mock.patch, sinon.stub): Patching module imports or spying on internal implementation calls (expect(repo.save).toHaveBeenCalledWith(...)) instead of asserting observable state in a Contract-Verified In-Memory Fake.
  2. Unverified Inline Stubs (Divergent Fakes): Creating ad-hoc mock objects inside test files that return hardcoded happy-path payloads while ignoring constraints (e.g., allowing duplicate keys, skipping version increments, or mutating objects in place without rollback).
  3. Static Happy-Path Example Slop on Rich Domains: Testing financial splits, tax calculations, parsers, or state machines exclusively with 1–3 hand-picked integers (100, 200) instead of Property-Based generators (fast-check / Hypothesis / proptest).
  4. Tautological Implementation-Mirroring Oracles: Computing the expected value in a test by copying the exact production formula (expect(fn(a, b)).toBe(productionFormula(a, b))) or asserting only toBeDefined() / toBeTruthy().
  5. Unseeded Randomness in Tests (Math.random(), random.random(), uuid4() without seed): Generating random test inputs or fault schedules without a deterministic, logged PRNG seed, making failures impossible to replay.
  6. Wall-Clock sleep() / setTimeout Synchronization in Tests: Waiting real milliseconds/seconds for timeouts, retries, or TTL expirations instead of driving an injected Virtual Clock deterministically.

Required Output & Review Workflow

Structure all test engineering, mock-replacement refactors, and resilience verification tasks through this 4-phase discipline:

  1. Boundary & Invariant Specification: Define the Port interface, list the storage/protocol invariants the In-Memory Fake must enforce, specify the Universal Domain Properties (conservation, invertibility, idempotency, state-machine safety), and define the Chaos Fault Catalog.
  2. Verified Fake & Shared Contract Suite Implementation: Implement the Stateful In-Memory Fake (with structuredClone isolation, optimistic locking, and transactional rollback snapshots) alongside the parameterized Shared Contract Test Suite (runContractSuite) targeting both Fake and Real adapters.
  3. Property-Based & Seeded Deterministic Chaos Suite: Author the property-based generators (fast-check / Hypothesis / proptest) and the Seeded Deterministic Chaos Harness (PRNG + Virtual Clock) testing pre-commit drops, post-commit ACK drops, duplicate retries, and rollback recovery.
  4. Mutation Falsification & Guard Audit: Verify Red-Green falsification against a deliberate production mutation and audit the verification record and test source with scripts/verification_oracle_guard.py.

When a durable verification-oracle record is warranted, save a JSON assessment following the Verification Oracle Record Contract and audit it with:

npx anti-slop-engineering verification-oracle audit <record.json> --json
# or: python3 scripts/verification_oracle_guard.py audit <record.json> [--source <test_file.ts>] --json

Domain References

  • Verification Oracle Architect Playbook Suite: Central orchestration index for zero-mock verification, property testing, and deterministic simulation.
  • Verified In-Memory Fakes & Shared Contract Suites Playbook: Port/Adapter boundaries, stateful in-memory fake mechanics (rollback snapshots, structuredClone, CAS versioning, unique keys), and parameterized contract test suites across Fake and Real DB adapters.
  • Property-Based, Metamorphic & Deterministic Chaos Playbook: Algebraic invariants, metamorphic testing relations, stateful model-based command testing, seeded PRNG fault injection (FoundationDB/TigerBeetle deterministic simulation style), and Virtual Clocks.
  • Verified In-Memory Fakes & Contracts Reference: Production TypeScript, Python, Rust, and Go templates for contract-verified fakes and transactional snapshot rollback.
  • Property-Based & Metamorphic Testing Reference: Invariant catalogs and fast-check / Hypothesis / proptest recipes for financial math, codecs, and state machines.
  • Deterministic Chaos & Fault Injection Reference: Seeded PRNG (Mulberry32), Virtual Clock, Pre-Commit vs. Post-Commit fault injection, and idempotency/rollback verification.
  • Verification Oracle Record Contract: JSON schema and verification rules for scripts/verification_oracle_guard.py.

Failure Taxonomy

Failure ModeObservable SignalRemediation
Banned module mock regressionTest imports vi.mock, jest.mock, sinon.stub, or unittest.mock.patchDelete module mock; extract Port interface and inject a Contract-Verified In-Memory Fake
Divergent fake driftUnit tests pass against fake, but production fails on unique constraint or rollbackRun the parameterized Shared Contract Test Suite against both InMemoryFake and RealAdapter
Penny-rounding / boundary leakStatic test split(100, 2) passes, but split(100, 3) loses 1 cent in prodReplace static example with a fast-check / Hypothesis conservation property ($\sum \text{parts} = \text{total}$) over 200+ runs
Post-commit retry double-chargeNetwork drops after DB commit; retry executes a second chargeInject post_commit_ack_drop fault via Seeded Chaos Harness and enforce atomic Idempotency-Key receipt replay
Non-reproducible CI flakeTest uses Math.random() or setTimeout(..., 50) and fails 1 in 40 runsReplace with explicit seeded PRNG (seed: 0x5EED1234) and deterministic VirtualClock
Tautological green suiteProduction bug introduced, yet all unit tests stay greenEnforce Red-Green Mutation Falsification; replace implementation-mirroring expectations with independent algebraic oracles

Delegated Skills

  • anti-slop-engineering: Enforce Gate 2 (Test Integrity) and Gate 4 (Verify Beyond Green Tests) alongside verification oracle construction.
  • install-anti-slop: Enforce anti-slop/no-module-mocking at the linter level while using verification-oracle-architect to build the architectural replacement.
  • scientific-fault-isolation: When a property test or deterministic chaos seed uncovers a minimal failing counterexample, use the 4-step root-cause isolation protocol.
  • paranoid-principal-architect: Pair deterministic chaos testing with 10M-concurrent-user idempotency keys, CAS optimistic locking, and zero-trust boundary schemas.

Completion and Limits

The task is complete when all banned module mocks (vi.mock, jest.mock, unittest.mock.patch) and unverified inline stubs are replaced by a Stateful In-Memory Fake passing a Shared Contract Test Suite (100% parity with the real adapter), universal domain invariants are verified via Property-Based / Metamorphic tests (>= 100 runs with shrinking), resilience and rollback are proven under Seeded Deterministic Chaos fault injection with a Virtual Clock (zero sleep() calls), Red-Green mutation falsification is confirmed, and scripts/verification_oracle_guard.py reports zero findings. A review may validly output refuse or pause if a test suite relies on module mocking, unseeded randomness, or lacks contract parity proof.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.