comment-bench
CLAUDE.md0 starsChanged 7 months agoMITView on GitHub
A benchmark for whether inline comments steer AI coding agents on small surgical edits. Follow comment-policy.md. This repo is the policy - anything here that violates it is a bug. Use AIDEV-NOTE: anchors for rules that must survive future agent edits. Treat stale comments as bugs. scenarios/** files are load-bearing experimental data. Treatment directories deliberately contain bad comments (paraphrases, stale, generic AI-style). That's the experiment. Don't "improve" them. Biome lint is intentionally disabled on scenarios/** via biome.json; don't re-enable. The…
In this repositoryOne other file it gives its agents.
- comment-auditSkill
The file
We split this CLAUDE.md into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 7
Follow comment-policy.md. This repo is the policy - anything here that violates it is a bug.
≈ “Common” is an estimate: at least 1 in 5 of the 26,980 CLAUDE.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Comment Bench
A benchmark for whether inline comments steer AI coding agents on small surgical edits.
Comments
Follow comment-policy.md. This repo is the policy - anything here that violates it is a bug. Use AIDEV-NOTE: anchors for rules that must survive future agent edits. Treat stale comments as bugs.
Scenarios are fixtures
scenarios/** files are load-bearing experimental data. Treatment directories deliberately contain bad comments (paraphrases, stale, generic AI-style). That's the experiment. Don't "improve" them. Biome lint is intentionally disabled on scenarios/** via biome.json; don't re-enable.
The seven comment treatments per scenario must stay aligned: same code shape, only the comment payload varies. If you change one, change them all.
Oracle tests don't run on canonicals
scenarios/<id>/oracle/feature.test.ts and invariant.test.ts are designed to fail against canonical code (that's the trap). They run only via bun benchmark.ts score against a prepped + agent-edited /tmp/work directory. Don't add them to bun test defaults or pre-commit.
tsconfig.json excludes scenarios/ from typecheck for the same reason - the canonical files have intentional gaps.
Conventional commits
semantic-release reads commit messages on main:
feat:-> minor bumpfix:-> patch bumpBREAKING CHANGE:in body -> major bumpchore:,docs:,ci:-> no release
Tooling
Bun + TypeScript strict + Biome for format/lint + Lefthook pre-commit (typecheck + biome check). semantic-release with semantic-release-ai-notes runs on push to main via .github/workflows/release.yml. CI checks run on every PR via .github/workflows/ci.yml.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.