repro-harness-negative-proof
Skill345 starsChanged 37 days agoApache-2.0View on GitHub
Use when building or running a reproduction harness, a cross-version comparison, or any test asserting that a bug is absent/fixed. Defines the rules for proving a negative result is real rather than a harness failure masquerading as a clean result.
In this repository17 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .github/copilot-instructions.mdCopilot instructions
- clickhouse-sink-reviewSkill
- code-quality-architectureSkill
- code-quality-bashSkill
- code-quality-goSkill
- code-quality-jvmSkill
- code-quality-pythonSkill
- code-quality-sqlSkill
- code-reuse-firstSkill
- code-reuse-reviewSkill
- design-note-debtSkill
- destructive-operation-safetySkill
- mandatory-code-change-validationSkill
- pr-self-reviewSkill
- sink-connector-mysql-source-of-truthSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 34
Positive proof per phase: Every phase of a harness (setup, the actual trigger/action, verification) must emit positive evidence that it ran …
- Distinctive: line 45
If the comparison phase finds zero things to compare (e.g., zero rows matched the condition being tested), that's not evidence the defect …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Repro Harness Negative Proof
Overview
"No error was reported" is not the same as "the thing we were testing for is actually absent." A harness that silently fails to execute its setup phase on one target, then reports no error on the comparison phase, produces exactly the same output as a harness that genuinely found nothing wrong. This skill is the discipline for telling those two apart.
A past investigation into a suspected defect reported one version as "unaffected," when in fact the harness's setup phase had silently failed to run against that version — the absence of errors was mistaken for a clean result, when it was actually a harness failure. This is the single most common way a negative-proof harness lies, and it is exactly what the rules below are designed to catch.
This matters especially for this project because of where cross-version and cross-dialect behavior differences actually live: DDL translation across source database versions, type-mapping edge cases, and ClickHouse version-specific behavior are all places where "it worked" and "the test never actually ran" look identical unless the harness proves otherwise.
The Rules
- Positive proof per phase. Every phase of a harness (setup, the actual trigger/action, verification) must emit positive evidence that it ran and ran correctly — not just the absence of an error. A setup phase should assert the precondition it created actually exists before moving on; a trigger phase should assert the action it took actually happened (e.g., a row was actually inserted, not just that the INSERT statement didn't throw).
- Non-zero exits abort immediately — never continue past them. A harness that logs a warning and keeps going when a setup step fails can end up "testing" a target that was never actually prepared correctly.
- A zero-denominator case is a harness failure, not a 0% defect rate. If the comparison phase finds zero things to compare (e.g., zero rows matched the condition being tested), that's not evidence the defect is absent — it's evidence the harness didn't actually exercise the scenario. Treat it as a failure, investigate, don't report it as a pass.
- Never hardcode version-dependent details into the harness itself. A harness whose assertions assume one specific version's file format, error code, or behavior will silently stop testing anything meaningful the moment that detail changes upstream, while still appearing to run successfully. Derive expected values from the system under test, or make the harness fail loudly when an assumption no longer holds.
Worked Example (illustrative — a real upstream ClickHouse class of bug)
A mutation applied against a table whose data was written in an older
on-disk mark-file format could silently stop progressing on some versions
while completing without error on others, with the difference traceable to
a mark-file format change between versions (e.g., .mrk3 vs .mrk4
variants) rather than to the mutation logic itself. A harness checking "did
the mutation complete" without also checking "did it actually touch the
expected number of parts" would report the broken version as healthy,
because the symptom was a silently-reduced work set, not an error.
The lesson generalizes: a harness must verify the shape of what happened (how much work was done, on what), not just whether an error was raised.
Pre-Publish Checklist (before trusting/reporting a result)
- Every phase (setup, trigger, verify) has positive proof it executed, not just "no exception was thrown"
- Any non-zero exit code anywhere in the harness aborted the run rather than being logged and ignored
- Any ratio/percentage reported has a non-zero, sanity-checked denominator
- No assertion in the harness hardcodes a version-specific detail that could silently go stale
- The harness's own exit code/verdict was checked, not just its stdout
- The test ran in an isolated test environment matching the target configuration, not a convenient but non-representative one
Anti-Patterns
Treating "no error" as proof of correctness · a comparison with a zero-row denominator reported as a clean pass · a harness that logs and continues past a failed setup step · hardcoding a specific version's file format or error code into an assertion meant to generalize across versions · trusting a harness's stdout over its actual exit code.
Cross-References
pr-self-review, sink-connector-mysql-source-of-truth.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

