agent-skills-harness
AGENTS.md11 starsChanged 3 months agoMITView on GitHub
Agent Skills Harness is a factory and testing ground for building production-grade agent skills. It provides a structured pipeline for creating skills that are benchmarked against gold standards, autonomously improved via autoresearch loops, and verified through multi-agent consensus. The core output is the create-skill-autoresearch factory skill, which orchestrates the entire skill creation lifecycle. No build step required. Skills are markdown-based and used directly by any AI coding agent that supports the SKILL.md format. The vendored skills are committed, so a…
- Reads credentials
- Deletes or force-pushes
- Installs packages
Where files are headingNew AGENTS.md files have more rules about keeping secrets out of the code, and fewer about code style.
Of the AGENTS.md files we read, the 33,315 first seen in the last six months, against the older ones: the kind of rule that gained most share, and the kind that lost most. About files like this in general, not this one, and an estimate, since a kind is a pattern in the words.
In this repository10 other files it gives its agents.
- autoresearchSkill
- create-skill-autoresearchSkill
- design-taste-frontendSkill
- documentation-writerSkill
- handoffSkill
- llm-councilSkill
- premortemSkill
- production-gradeSkill
- tribunalSkill
- writing-great-skillsSkill
The file
We split this AGENTS.md into the rules it gives an agent
4 rules. About 0 are common, 0 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 64
No build step required. Skills are markdown-based and used directly by any AI coding agent that supports the SKILL.md format.
- Very specific: line 92
Commit in both repos (harness on main.
- Very specific: line 94
Never edit the agent-skills copy directly — edit here and re-sync, or the two diverge (as production-grade once did).
- Very specific: line 116
When running the factory inside a real project repo, gold standards, the rubric, and evaluation scripts should be committed (see …
≈ “Common” is an estimate: at least 1 in 5 of the 42,283 AGENTS.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
AGENTS.md
Project Overview
Agent Skills Harness is a factory and testing ground for building production-grade agent skills. It provides a structured pipeline for creating skills that are benchmarked against gold standards, autonomously improved via autoresearch loops, and verified through multi-agent consensus.
The core output is the create-skill-autoresearch factory skill, which orchestrates the entire skill creation lifecycle.
Repository Structure
.agents/skills/ # Skills the harness USES (factory + vendored companions)
create-skill-autoresearch/ # The factory skill (main deliverable; original)
autoresearch/ # Autonomous experimentation loop (vendored)
production-grade/ # Engineering posture principles (vendored)
premortem/ # Risk analysis before execution (vendored)
handoff/ # Context preservation across sessions (vendored)
documentation-writer/ # Diataxis documentation generation (vendored)
llm-council/ # Multi-agent planning with consensus (vendored)
writing-great-skills/ # Skill-writing craft (vendored)
tribunal/ # Phase 5 verification when delegated (vendored)
design-taste-frontend/ # Anti-slop frontend/UI skill (vendored)
builds/ # Skills the harness PRODUCES (gitignored; one folder per build)
self-test/ # The factory's own regression test + worked example
site/ # Docs + landing site (Nextra → Vercel)
skills-lock.json # Provenance + version pins for vendored skills
docs/
reference/io-contract.md # What goes in / what comes out (start here)
reference/workspace-layout.md # Full file-by-file build layout
reference/ # rubric-format, metric-protocol, ...
thoughts/ # Research notes and design decisions
study/ # Case study materials (gitignored; maintainer-local)
resources/ # External reference implementations (git submodules)
usage-guide.md # How to use the factory
architecture.md # Design overviewSkills
| Skill | Purpose |
create-skill-autoresearch | Factory for creating production-grade skills via autoresearch |
autoresearch | Autonomous iterative experimentation loop with METRIC protocol |
production-grade | Principle-engineering posture for production-grade code |
premortem | Identify failure modes before they occur |
handoff | Compact conversation into handoff document for another agent |
documentation-writer | Diataxis-guided documentation generation |
llm-council | Multi-agent planning with anonymized judging |
writing-great-skills | Skill-writing craft: leading words, information hierarchy, description quality |
tribunal | Doer → verifier-panel → consensus verification; the factory delegates Phase 5 to it when present |
design-taste-frontend | Anti-slop frontend/UI design for landing pages and redesigns |
Setup
git clone <repo-url>
cd agent-skills-harness
npm install
git submodule update --init --recursive # docs/resources/ reference implementationsNo build step required. Skills are markdown-based and used directly by any AI coding agent that supports the SKILL.md format. The vendored skills are committed, so a fresh clone works offline.
To refresh a vendored skill, scope it: npx skills update <skill-name>. npm run skills:refresh-all does every one of them from each upstream's current HEAD and rewrites the lock entries — review that diff before committing, since an upstream edit changes how a companion behaves. skills-lock.json records provenance and a content hash; it is not a pin the CLI restores you to.
npm test runs the regression gate (self-test/evaluation/gate.sh): the structural checks, failing on any failure not on its advisory list — which is currently empty on purpose, so the suite is expected to be fully green. It needs no credentials and no network.
Development Conventions
- Skills live in
.agents/skills/<skill-name>/SKILL.md - Skills follow the official skill-authoring rules (Anthropic best-practices; YAML frontmatter, < 500 lines, progressive disclosure, references one level deep) — see
.agents/skills/create-skill-autoresearch/references/skill-authoring-best-practices.md - A factory run produces
builds/<skill-name>/with three zones:input/(human materials),work/(generated artifacts),output/<skill-name>/(the finished skill).builds/is gitignored. create-skill-autoresearchis developed here (source of truth) and published as a byte-identical release toa-tokyo/agent-skills— never edit the published copy directly. See "Releasing create-skill-autoresearch" below.- Research notes go in
docs/thoughts/with numbered prefixes (00-, 01-, etc.) - Design decisions are logged in
docs/thoughts/07-design-questions.md
Releasing create-skill-autoresearch
The harness is the source of truth for the factory skill (it is developed and benchmarked here against
self-test/). a-tokyo/agent-skills carries a byte-identical published copy for npx skills installs.
The two are cross-linked (each README points to the other). To cut a release after editing the factory:
- Bump
version:in.agents/skills/create-skill-autoresearch/SKILL.md. - Re-sync the published copy as a whole directory (so
references/can't silently drift):rm -rf ../agent-skills/skills/create-skill-autoresearch cp -r .agents/skills/create-skill-autoresearch ../agent-skills/skills/create-skill-autoresearch diff -rq .agents/skills/create-skill-autoresearch ../agent-skills/skills/create-skill-autoresearch # must be empty - Commit in both repos (harness on
main;agent-skillsvia a PR) and push.
Never edit the agent-skills copy directly — edit here and re-sync, or the two diverge (as production-grade once did).
Key Concepts
- Gold Standards: Human-produced exemplars that define "what good looks like"
- METRIC Protocol:
METRIC name=valueline format for deterministic metric extraction - ASI (Actionable Side Information): Structured experiment metadata that survives git reverts
- Panel Consensus: Multi-agent verification with independent scoring, synthesis rounds, and devil's advocate
- Craft-Decisions Ledger: Append-only log of every design decision and iteration
Testing
There is no unit-test suite; the harness verifies itself with a structural self-test plus benchmark/panel verification. Quality is verified through:
npm test(→self-test/evaluation/gate.sh) — this is the gate. It runs the deterministic structural checks inself-test/evaluation/autoresearch-evaluate.sh, then fails on any reported failure that is not on its advisory list. It needs no credentials and runs on a fresh clone (bash+python3, nonpm install). Judge a change by the issues list, not bychecks_passed: adding checks raises the total, so a count comparison hides a regression behind a rising number../self-test/evaluation/evaluate.sh <case-id>— scores a factory-produced skill against a reference. It needs a factory output first, and the tracked cases carryinput_materials: null(their sources are private), so on a public clone you supply your own case. WithJUDGE_MODEL/JUDGE_API_KEYit scores via LLM-as-judge; otherwise structural checks only, and that score is not comparable to a judge-scored target.- Panel verification with multi-agent consensus (Quality + Utility + Devil's Advocate) — the committed worked example is
self-test/benchmarks/premortem-rebuild.md.
Git Workflow
- Feature branches for skill development
- Autoresearch sessions run on
autoresearch/<tag>branches builds/(a run's working area) is gitignored; session artifacts (results.tsv,autoresearch.jsonl,run.log) are never tracked- When running the factory inside a real project repo, gold standards, the rubric, and evaluation scripts should be committed (see
docs/reference/workspace-layout.md) — which is exactly why no credential may ever be written intowork/,judges.yamlincluded. Credentials live in the environment; seedocs/reference/tuning.md. - Scratch material (notes, transcripts, anything pulled in while working) goes in
tmp/, which is gitignored. This repo pushes to public remotes — checkgit status --shortbefore committing.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

