AI Harness Engineering — audit & refactor

What "the harness" is: everything around the model weights that turns a token predictor into an agent that ships work — the loop, tools, context assembly, permission and containment layer, memory, verification, and multi-call orchestration. This skill is the standard to measure a project against, plus the procedure to close the gap.

This is a moving target. The current standard lives in references/standard-2026-10.md and is dated. Every component encodes an assumption about what the model can't do, so prune what a stronger model has made unnecessary (Dimension 10). When you revise the standard, copy it to standard-<new-date>.md, edit the copy, and update the changelog at the bottom of this file. Never overwrite a dated standard.

When to use

• "Review / upgrade / standardize / re-audit this project's agent harness."
• "Are we following current best practices for agents / Claude Code?"
• "Add cost tracking and cost controls to this project."
• "Set up hooks / subagents / evaluation / CLAUDE.md properly."
• Bootstrapping harness scaffolding for a brand-new project.

Not for: writing a single prompt, one-off tool calls, or non-agentic code.

The through-line

1. Build reactively, then prune. Add a component for an observed failure, record why, give it a retire-when condition.
2. Enforce with executable controls, not prose.
3. Contain at the environment layer first, then steer the model.
4. Measure the harness, not only the product.

The ten dimensions (what "good" is)

Full checklists are in the standard; this is the map.

1. Context & memory — the filesystem is memory; a short instruction file (CLAUDE.md / AGENTS.md, under ~200 lines, a table of contents), path-scoped rules and on-demand skills; plan/progress files; /clear + handoff over long corrected sessions; agent-written memory reviewed.
2. Planning — machine-checkable feature list and scope doc; interview → spec → fresh session; specs end in an end-to-end verification; plan mode only when the diff isn't one sentence.
3. Execution — summary-returning subagents with minimal tools; dynamic workflows for large fan-out; worktrees for parallel edits; parallelism for independent work, not role-play pipelines.
4. Evaluation — doer ≠ judge — a runnable check and shown evidence; the gate ladder (prompt check → /goal → Stop hook → calibrated second-opinion reviewer only where warranted); product evals with deterministic graders, measured noise, and transcripts read.
5. Deterministic guardrails — PreToolUse guards (fail closed), PostToolUse lint, Stop gate; the same guards in CI; every guard tested both ways; no prose-only rules.
6. Containment & permissions — sandbox / devcontainer / VM first; egress denied by default; credentials outside the sandbox; narrow allowlists; classifier and approval prompts as layers only; untrusted input treated as such.
7. Prompt & tool hygiene — versioned prompt registry; interfaces with test fakes; schema-validated output; minimal tool surface; untrusted text passed as data.
8. Cost — hard caps vs advisory limits labelled; priced per-call telemetry and attribution; tiering or a cheap-executor + strong-advisor pattern; circuit breaker; harness cost justified against benefit.
9. Autonomy safety & recovery — git as rollback (checkpoints miss Bash); watchdog; observability; a written unattended-run policy.
10. Harness evolution & measurement — retire-when per component; remove one at a time after model releases; measure hook blocks and skill use; evaluate components against a baseline; dated standard and re-audit cadence.

Audit workflow

Run these four phases. Keep the output as a written report (offer to save it to the target repo, e.g. docs/harness-audit-<date>.md, or the repo's re-audit log).

Phase 1 — Inventory. Discover what exists: instruction files (CLAUDE.md, AGENTS.md, .claude/rules/, module-level files, auto memory), .claude/ (skills, agents, hooks and permissions in settings, workflows), prompt registry, LLM client / gateway, pricing and usage accounting, eval harness, tracing, scope and plan files, CI checks, sandbox / container setup, and how dependencies are abstracted. Note the stack and OS (hooks, watchdogs and the sandbox are platform-specific). For a re-audit, also read the previous audit's register of assumptions and gather evidence for each (usage data, escalation rates, eval runs).

Phase 2 — Score. For each of the ten dimensions, assign Absent / Partial / Solid with one line of evidence (a path) and the most valuable missing piece. Use the checklists in references/standard-2026-10.md. Flag components that look stale (Dimension 10) as removal candidates, not only gaps.

Phase 3 — Gap report. A table: dimension → score → top gap → impact. Call out anything that risks cost blowout, unsafe autonomy or credential exposure first; those are P0 regardless of how incomplete a structural dimension looks.

Phase 4 — Refactor plan. A prioritized, sequenced list of changes (add, modify or remove), following references/refactor-playbook.md: P0 containment, safety & cost → P1 verification & evaluation → P2 structure, hygiene & pruning. Each item: what, where, and the acceptance check that proves it. Do not start editing until the plan is agreed.

Output contract

Every run of this skill produces, in order:

1. A one-paragraph maturity summary ("Solid on 7/10; P0 gap is no hard budget on headless runs; two components are removal candidates").
2. The scored dimension table.
3. The prioritized plan with acceptance checks. Then ask whether to execute P0 now or hand back the plan.

Reference index

• references/standard-2026-10.md — the current dated standard: through-line, per-dimension checklists, the verified Claude Code primitive cheat-sheet, anti-patterns, sources.
• references/standard-2026-07.md — the previous revision, kept for diffing.
• references/refactor-playbook.md — how to bring a project in line: priorities, sequencing, platform notes, acceptance checks, and a worked example on an LLM/RAG Python repo.

Changelog

• 2026-10-17 — rev 2026-10. Re-reviewed against Anthropic's 2026 engineering posts (harness design, Managed Agents, auto mode, containment, evals, infra noise), current Claude Code docs, OpenAI's harness-engineering write-up and the 2026 marmelab survey. Added the through-line (build reactively and prune; executable controls; environment-first containment; measure the harness). New Dimension 6 Containment & permissions; Dimension 9 renamed Harness evolution & measurement and renumbered 10. Evaluation reframed as a gate ladder with a conditional, calibrated reviewer and product-eval practice. Dropped the mandatory compaction policy and default initializer/sprint contracts; CLAUDE.md under ~200 lines, AGENTS.md, path-scoped rules. Cost: hard vs advisory caps, --max-budget-usd, advisor tool. Rollback: git, since checkpoints miss Bash. Refreshed primitive cheat-sheet (33 hook events, dynamic workflows, auto mode, sandbox, plugin evals).
• 2026-07-13 — rev 1. Initial standard distilled from Anthropic long-running agents guidance + Claude Code primitives (verified against official docs). Nine dimensions, audit workflow, refactor playbook.