agentleFS
Samjoel3101Samjoel3101/harness-engineering/SKILL.md

harness-engineering

Skill0 starsChanged 2 months agoApache-2.0View on GitHub

Principles and judgment for designing an "agent harness" — the repo-level scaffolding (docs, feedback loops, constraints, memory, review gates) that lets AI coding agents work reliably and autonomously on a codebase. Use this whenever the user asks to make a repo more agent-friendly, set up or audit AGENTS.md/CLAUDE.md, improve how well Claude Code (or another agent) performs on their project, design feedback loops or verification gates for agents, decide what should go in project docs vs skills vs memory, or is generally trying to get better/more reliable results from AI coding agents on a specific codebase. Do not treat this as a rigid setup script — apply the principles to the specific project's language, size, team, and risk profile, and recommend only what that project actually needs.

25 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use self-improvement-loops, bdi-mental-states and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • DVwebBiz11Skill

    self-improvement-loops

    This skill should be used when the harness, scaffold, workflow, or optimizer itself is the optimization target: recursive self-improvement (RSI) loops, meta-harnesses, self-improving harnesses that mine their own failures and propose bounded edits, evolutionary or population-based search over agent scaffolds, acceptance gates for self-modifying systems, and agentic context evolution where the mechanism that produces context is versioned and evolved. Route governance of a single autonomous loop (locked surfaces, durable logs, rollback, novelty gates, approval boundaries) to harness-engineering, measurement and quality-gate design to evaluation, judge design to advanced-evaluation, and remote sandbox infrastructure to hosted-agents.

    Also in 4 projects that use this

  • Downey6689Skill

    bdi-mental-states

    Transforms external RDF context into BDI mental states. Invoke when modeling beliefs, desires, and intentions.

    Also in 5 projects that use this

  • Downey6689Skill

    hosted-agents

    Guidance for hosted/sandboxed agents. Invoke when running background agents in isolated VMs or multi-client setups.

    Also in 5 projects that use this

  • DVwebBiz11Skill

    long-horizon-prompting

    This skill should be used when writing, enhancing, or evaluating the launch prompt for a long-running autonomous agent or a parallel multi-agent orchestration attacking a hard problem: pseudo-formal task briefs that define terms and an exact success predicate linguistically, enumerate non-counting outcomes, set persistence rules with explicit stop and return conditions and effort floors, manage a diverse portfolio of parallel approaches with an approach registry and blocked-route bookkeeping, and gate the return on adversarial audit. Route agent topology and coordination protocols to multi-agent-patterns, runtime control surfaces and loop governance to harness-engineering, evaluator and quality-gate construction to evaluation, judge design to advanced-evaluation, and compaction or memory mechanics to context-compression and memory-systems.

    Also in 4 projects that use this

  • AbdelrhmanAlquranSkill

    latent-briefing

    This skill should be used when the user asks to "share memory between agents", "KV cache compaction for multi-agent", "orchestrator worker context", "latent briefing", "reduce worker tokens", "cross-agent memory without summarization", or discusses Attention Matching compaction, recursive language models with workers, or token explosion in hierarchical agents.

    Also in 4 projects that use this

  • abdullah1854Skill

    context-degradation

    Also in 5 projects that use this

  • +2 more from Downey6689

The file

We split this skill into the rules it gives an agent

1 rule. 0 are common, 1 is distinctive, and 0 are very specific to this project.

  • 0

    Common

    Most of the 22 edited versions we read keep these.

  • 1

    Distinctive

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Always end with a concrete, prioritized recommendation scoped to what this project needs right now — not a full checklist dump.

    line 49

“Common” is measured: kept by at least 6 in 10 of the 22 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Harness Engineering

Agent = Model + Harness. The model can't execute code, remember past sessions, or know the current state of anything — a raw model is not an agent. The harness is everything around the model that makes it one: the docs it reads, the tools it can call, the checks that catch its mistakes, the places it's allowed to write, and the record it leaves behind. If the model is the brain, the harness is the body.

There are two layers of harness. The builder harness — the agent loop, tool schemas, context compaction, sandboxing — is implemented by whoever builds the agent (Anthropic, in Claude Code's case) and isn't yours to redesign. This skill is about the user harness: the repo-level scaffolding any team can build on top of that to get reliable results on their own codebase. Don't reach for a builder-level explanation ("the agent loop should really...") when the actual gap is a missing doc or check you can fix today.

Your job when this skill triggers is not to run a checklist top to bottom. It's to look at the actual project — its size, language, risk profile, team, and where agents are currently failing — and decide which of the principles below are worth applying now, in what order, and at what weight. A 200-line CLI tool and a payments backend need very different harnesses. Recommend the minimum that closes the gap you actually observe; don't scaffold for problems the project doesn't have.

How to use this skill

  1. Diagnose before prescribing — the diagnosis differs by starting point. On an existing repo: read it (or ask), and find where agents have actually gone wrong — hallucinated APIs, broken architecture boundaries, silent regressions, lost context between sessions, unreviewable PRs. Harness work should trace to a real failure mode, not a generic best-practices pass. On a fresh repo there's no failure history to read yet, so don't invent one: stand up only the irreducible core (a short AGENTS.md, a fast test/lint/typecheck loop that fails loudly, the habit of writing a plan before non-trivial work) and let the rest of the pillars earn their way in as real friction shows up. A brand-new repo with all ten pillars pre-built on day one is over-scaffolding, not readiness.
  2. Pick from the pillars below based on that diagnosis. Each pillar section ends with signals for when it matters more or less.
  3. Propose the smallest change that would prevent the failure from recurring — a new check, a doc, a constraint — not a whole new subsystem. Favor additive, reversible changes over big-bang rewrites.
  4. Write it into the repo, not into your own conversational memory. If it isn't in the repo, it doesn't exist for the next agent session.
  5. Expect to be wrong and prune later. Harnesses accrete cruft — stale docs, contradicted rules, unused hooks. Treat "does this still earn its keep" as an ongoing question, not a one-time setup.

See references/pillars.md for the full detail on each pillar — read it when you're actually applying a pillar, not preemptively. See references/anti-patterns.md for failure modes and gaming behaviors to guard against, especially before setting up anything an agent could optimize against (auto-merge, self-graded evals, reward-bearing metrics). See references/long-running-agents.md when the work at hand spans multiple context windows or sessions — it has the concrete patterns (progress files, session-start onboarding, merge-ready checkpoints) rather than just the principle.

The pillars (at a glance)

These roughly split into two families. Pillars 1, 2, 6, and 10 are guides — they shape what the agent does before it acts. Pillars 3, 4, 5, and 8 are sensors — they catch what went wrong after it acts. (7 and 9 are infrastructure that both families depend on: durable state and bounded tool access.) When you're not sure which pillar a given failure calls for, ask "was this a missing steer, or a missing catch?" — the two need different fixes (see references/anti-patterns.md, point 3 of the diagnosis heuristic).

  1. Repository as system of record — everything the agent needs to do the work lives in the repo: architecture, conventions, decisions, taste. Not in Slack, not in a wiki, not in the human's head. If an agent had to ask a question last week, the answer belongs in the repo now.
  2. Map, not manual — docs for agents should be a compact index pointing to where things live and why, not an exhaustive walkthrough. Progressive disclosure: short entry point (AGENTS.md/CLAUDE.md), links out to deeper references only loaded when needed. A bloated CLAUDE.md that tries to explain everything crowds out the context budget and gets skimmed, not read.
  3. Tight feedback loops — the agent should see the consequences of its actions immediately and unambiguously: fast tests, linters, type checks, a build that fails loudly. Autonomy only works when feedback is fast, specific, and hard to game. Slow or flaky feedback is worse than no feedback — it teaches agents to ignore it.
  4. Enforced architecture, local autonomy — invariants that must hold (layering, dependency direction, public API surface) get enforced centrally and mechanically (lint rules, import boundaries, CI gates) — not left to the agent's judgment or a prompt instruction it might forget. Tell agents what must be true, not how to achieve it; leave implementation freedom inside those bounds.
  5. Continuous garbage collection — dead code, stale docs, and unused scaffolding accumulate fast when agents generate a lot of it. Encode cleanup as a recurring task (a periodic "gardening" pass, a script, a lint rule) rather than relying on someone remembering to do it manually.
  6. Plans as first-class artifacts — for anything beyond a trivial change, the agent's plan should be written down and versioned (a plan file, a PR description, a design doc) before or alongside execution — not only implicit in a chat transcript that disappears. This is what makes multi-session and multi-agent work resumable.
  7. Externalized state and session continuity — long-running or multi-session agent work needs to survive the loss of chat context. Use the filesystem and git as durable memory: progress files, handoff notes, structured logs of what was tried and what failed. An agent picking up mid-task should be able to reconstruct state from the repo, not from being told.
  8. Review that scales — at low volume, human review is fine. As agent-generated throughput grows, human review becomes the bottleneck. Consider agent-to-agent review, tiered review (auto-merge trivial/reversible changes, escalate risky ones), and a throughput-first philosophy where corrections are cheap and waiting is expensive — but only once volume actually justifies it.
  9. Constraints, observability, and tool discipline — bound the model's output space (permissions, risk tiers like read-only/financial/destructive, what it can execute and where, sandboxing for anything touching secrets or prod); make its reasoning and tool calls inspectable (logs, trajectories, cost tracking); and manage context economically (offload huge tool outputs to files instead of dumping them inline, delegate context-hungry subtasks to a subagent so the main thread stays clean, keep prompt prefixes stable for caching, load skills/tools progressively rather than all upfront).
  10. Codebase harnessability — the code itself is part of the harness, not just the docs around it. Strong typing, small well-named files, clear module boundaries, and colocated tests make a codebase something an agent can navigate and reason about; a codebase that's already illegible to humans is illegible to agents too, and no amount of documentation fixes that. Investing in the code's own structure is often higher-leverage than adding docs on top of a tangled one.

Judgment calls this skill should help with

  • "Should this be in AGENTS.md, a skill, or a reference doc — and AGENTS.md or CLAUDE.md?" — AGENTS.md is the emerging cross-tool standard for the always-loaded map (read by OpenAI, Google, Cursor, and other tooling, not just Claude); where a project genuinely needs Claude-Code-specific instructions too, keep CLAUDE.md thin and pointing at AGENTS.md rather than forking the content into two files that drift apart. Either way, keep it small (under ~100 lines is a good target for most repos); skills are for procedures invoked situationally; reference docs are for depth loaded on demand. Don't put everything in the top-level file "to be safe."
  • "Does this project need agent-to-agent review or is that overkill?" — usually overkill below a certain PR volume or team size; don't propose it as a default.
  • "What's the minimum feedback loop that would have caught the last failure?" — start here before reaching for heavier infrastructure.
  • "Is this constraint better enforced by a prompt/doc, or by tooling?" — if it's a hard invariant (must never touch prod credentials, must not modify generated files), enforce it mechanically — a pre/post-tool-use hook, an import-boundary lint rule, a CI gate. If it's a style preference, a doc is enough.
  • "Is this check worth running?" — prefer computational checks (tests, types, lint: fast, deterministic, cheap) over inferential ones (an LLM review agent or judge: slower, costlier, more nuanced, and easier for an agent to game). Reach for an inferential check only where deterministic tooling genuinely can't tell — taste, architectural fit, prose quality.
  • "Are we about to build something an agent could game?" — check references/anti-patterns.md before shipping self-graded metrics, auto-merge on agent-authored tests, or any setup where the agent can edit both the work and its own evaluation.
  • "Is the harness we already added actually helping?" — don't just assume scaffolding pays for itself. Track something concrete: agent task success rate before/after, how often a human corrects the same class of mistake, time from "agent starts" to "CI green." If nothing improved, prune it rather than letting it linger as cargo cult (see pillar 5).

Always end with a concrete, prioritized recommendation scoped to what this project needs right now — not a full checklist dump.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.