gate-backtest
Skill0 starsChanged 5 months agoMITView on GitHub
Train and calibrate the gate system against a repo's real git history in an isolated sandbox. MUST be used when the user asks to backtest, replay, calibrate, or train the gates against history, and after installing gates into any repo with meaningful existing history. Uses real historical fixes as ground truth and converts each one into a permanent benchmarks/BM-*.md case that future DDRs are checked against.
In this repository6 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- doubt-driven-reviewSkill
- gate-retrospectiveSkill
- git-disciplineSkill
- solana-security-hardeningSkill
- spec-and-planSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 2 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 12
All work happens in the throwaway clone that scripts/backtest.sh creates — remotes stripped, nothing writable pointing at the real repo.
- Distinctive: line 28
What is the doubt prompt?: Write the one question a fresh-context reviewer would have needed to ask to find it.
- Very specific: line 50
Checklist: every Gate mapping: NEW benchmark adds a line to solana-security-hardening.
- Distinctive: line 57
Every NEW real-world fix on a protected path gets a benchmark authored as part of its own DDR — the dataset grows with the project.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Gate Backtest
The repo's own history is a labeled dataset: every fix commit marks an earlier commit that introduced a real defect. This skill replays that history against the gates and converts each real defect into a permanent benchmark.
Safety invariant
All work happens in the throwaway clone that scripts/backtest.sh creates — remotes stripped, nothing writable pointing at the real repo. Never run analysis inside an active working copy. Outputs (benchmarks, pattern proposals) land only in the xete-agent-skills repo via PR.
Step 1: Mechanical replay
scripts/backtest.sh <path-or-url-to-repo>Produces: recall (fraction of real defect-introducing commits the gate would have caught), friction (gated commits with no known defect), missed culprits with files, and suggested pattern additions.
Step 2: Qualitative pass on every culprit (caught AND missed)
For each culprit commit in the report, answer three questions by reading the actual diff of the culprit and its fix:
- Which gate would have caught it? Map the defect to a specific line of the solana-security-hardening checklist, or to a doubt class in doubt-driven-review. If NO existing gate item maps to it, that is a checklist gap — the fix becomes a new checklist line.
- Was it catchable at commit time? Some defects are only visible with later context (spec changed, dependency behavior). Mark honestly:
catchableorhindsight-only. Hindsight-only cases must not inflate the gate's claimed value. - What is the doubt prompt? Write the one question a fresh-context reviewer would have needed to ask to find it. Concrete, not generic: "what happens when the same commitment is submitted to two tabs?" not "is this secure?"
Step 3: Author benchmark cases
One file per real defect: benchmarks/BM-<short-name>.md
# BM: <defect in one line>
Source: <repo> culprit <hash> fixed by <hash>
Paths: <files/dirs involved>
Class: <overflow | replay | validation-gap | key-leak | logic | ...>
Catchable at commit time: yes | hindsight-only
Gate mapping: <checklist item or doubt class; NEW if it was a gap>
Doubt prompt: <the question that finds this bug>
Real solution: <what the actual fix did, 1-2 lines>Benchmarks are append-only ground truth. They are never deleted when patterns change — they are what patterns are tested against.
Step 4: Close the loop
- Patterns: propose additions from the missed-culprit paths; propose narrowing only where friction is high AND no benchmark lives on those paths. Meta-DDR required, as with all gate changes.
- Checklist: every
Gate mapping: NEWbenchmark adds a line to solana-security-hardening. - Re-score: rerun
backtest.shwith the proposed patterns file (second argument) and report before/after recall in the PR. A pattern change that doesn't improve recall or reduce friction against the benchmarks doesn't ship.
Ongoing use (this is the training part)
The benchmarks are live at every future commit, not just during backtests:
- doubt-driven-review requires checking
benchmarks/for cases whose Paths overlap the current diff; each matching benchmark's doubt prompt MUST be answered in the DDR. - Every NEW real-world fix on a protected path gets a benchmark authored as part of its own DDR — the dataset grows with the project.
- gate-retrospective scores DDR quality against benchmark hit-rate: a DDR that missed a doubt its matching benchmark prescribes is a retrospective trigger.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

