trust-me-bro
Skill1 starsChanged 4 months agoMITView on GitHub
Run controlled, paired evaluations of an Agent Skill against a no-skill baseline or previous revision, using isolated contexts, raw artifacts, deterministic verification, and blind comparison. Use when validating a new or edited skill, testing whether instructions improve pass rate enough to justify tokens or latency, resolving competing skill versions, or producing reproducible efficacy evidence. Do not use for a quick structural lint or an upgrade-only smoke check.
1 other project uses a version of this.
Adopted by
Most recently taken up by
- dip497/onmywatchtheir own version, first seen Oct 20, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- dip497/onmywatch★ 1same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository2 other files it gives its agents.
- skill-issueSkill
- this-is-fineSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 5 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈5
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 25
Read references/trial-format.md for the result record and blind comparator contract.
- Distinctive: line 31
Use a fresh context for each task-condition-trial.
- Distinctive: line 42
Use deterministic acceptance checks first.
- Distinctive: line 57
Keep when the gain or risk reduction justifies instruction, token, latency, and intervention cost.
- Distinctive: line 60
Delete or disable when baseline is equal and cheaper.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Trust Me, Bro
Measure the marginal value of one skill. Keep the task and environment fixed; change only the skill condition.
“Trust me” is not a control condition.
Design
- Name one capability claim. Reject vague claims such as "works better."
- Pin task inputs, repository revision, model, harness, permissions, tools, reasoning setting, verifier, and timeout.
- Select real trigger cases and nearby failures. Start with three to five tasks for a smoke trial. Require repeated runs before publishing a performance claim.
- Define conditions:
baseline: no candidate skill content;candidate: the selected skill revision;previous: optional prior revision.
- Read references/trial-format.md for the result record and blind comparator contract.
Run
- Randomize condition order.
- Use a fresh context for each task-condition-trial. Do not let baseline workers discover the candidate skill or prior conclusions.
- Give every worker the same task-local context, tools, permissions, and budget. Give only candidate workers the candidate skill.
- Capture raw outputs, diffs, logs, timings, token counts when available, and user interventions. Do not grade from worker self-reports.
- Prevent live external writes unless the user explicitly authorizes the exact target. Prefer fixtures, sandboxes, or drafts.
Judge
- Use deterministic acceptance checks first.
- For subjective output, anonymize and randomize artifacts before giving them to a fresh comparator. The comparator receives the original task and rubric, but no condition labels, expected winner, or implementer analysis.
- Record failures as well as successes. Do not repair candidate output before grading it.
Aggregate JSONL results with:
node scripts/summarize-trials.mjs results.jsonl node scripts/summarize-trials.mjs results.jsonl --reference previous --comparison candidate
Decide
- Keep when the gain or risk reduction justifies instruction, token, latency, and intervention cost.
- Trim and rerun when only part of the skill appears causal.
- Delete or disable when baseline is equal and cheaper.
- Block release when the candidate introduces a safety or correctness regression.
One run can catch a bug; it cannot establish a stable uplift. State sample size and uncertainty without manufacturing statistical confidence.
Return
Return the pinned configuration, task matrix, condition-level results, paired improvements/regressions, cost delta, raw artifact locations, decision, and known limits. Never market a smoke test as a benchmark.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.