harness-mechanic
Skill17 starsChanged 52 days agoMITView on GitHub
Harness optimization: analyze execution traces, propose rule/skill improvements. Use when user says optimize harness, harness review, improve rules, meta-harness, or /harness-mechanic. Not for trace capture (use harness-trace).
In this repository37 other files it gives its agents.
- adrSkill
- android-kotlinSkill
- apple-swiftSkill
- autoresearch-promptSkill
- blog-writerSkill
- cloud-infrastructureSkill
- commitSkill
- cover-imageSkill
- freezeSkill
- gemini-reviewSkill
- golangSkill
- harness-traceSkill
- humanizerSkill
- ios-debuggerSkill
- knowledge-syncSkill
- learning-docsSkill
- learning-loopSkill
- obsidianSkill
- orchestratorSkill
- plan-forgeSkill
- process-clippingsSkill
- project-analyzerSkill
- project-checksSkill
- pr-reviewSkill
- pythonSkill
- railsSkill
- react-nextjsSkill
- refine-requirementsSkill
- releasing-softwareSkill
- rubySkill
- scoreSkill
- second-opinionSkill
- skill-forgeSkill
- source-controlSkill
- terraformSkill
- test-design-reviewerSkill
- verify-frontendSkill
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 6 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 31
Read ALL available traces: before proposing. Anecdotal sampling produces vibes-based tweaks.
- Distinctive: line 33
Evidence or silence: Every proposal carries trace line numbers / session IDs.
- Very specific: line 40
Read latest token baseline: quality_reports/token_baselines/
- Very specific: line 41
Read recent traces: quality_reports/traces/ (last 10 sessions minimum)
- Very specific: line 42
Read prior contracts: quality_reports/harness_changes/*.md (avoid re-proposing the same change)
- Distinctive: line 43
Run the 5-stage loop above
- Very specific: line 45
If approved: apply changes, copy draft contract into quality_reports/harness_changes/, then run /skill-forge review on modified skills
- Distinctive: line 68
After a milestone week with many orchestrator sessions.
- Distinctive: line 69
When the review loop is stuck (score < 80 after 2+ rounds across multiple tasks, not just one).
- Distinctive: line 70
After noticing recurring failure patterns across sessions.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
ABOUTME: Harness optimization through execution trace analysis
ABOUTME: Evidence-based proposals for improving rules, skills, and agent definitions
Harness Mechanic
An Evolution Agent (paper §3.5.2 of arxiv 2605.18747 "Code as Agent Harness") for the claude-forge runtime. Reads execution traces, token baselines, and prior change contracts; proposes evidence-backed edits to hooks/, rules/, skills/, and agents/.
Prior art (load-bearing references)
- Meta-Harness (arxiv 2603.28052): outer-loop search over harness code via an agentic proposer that reads source, scores, and traces of all prior candidates from a filesystem. Reported +7.7 pts on text classification with 4× fewer context tokens, +4.7 on IMO problems, beats hand-engineered TerminalBench-2 baselines. The proposer-with-filesystem-access pattern is our blueprint:
quality_reports/traces/,quality_reports/token_baselines/, andquality_reports/harness_changes/ARE that filesystem. - AutoHarness (arxiv 2603.03329): iterative refinement of code harnesses from environment feedback. Less directly applicable (their domain is games, ours is text/orchestrator), but the principle (smaller-model + synthesized harness beats larger model) is the upper bound we're chasing.
- Code as Agent Harness (arxiv 2605.18747) §3.5: defines the 5-stage Evolution Agent loop this skill implements.
The 5-stage loop (paper §3.5.2)
| # | Stage | What this skill does |
| 1 | Observe | Read trace JSONL (quality_reports/traces/, schema v2; see harness-trace). Read token baseline TSV. Read prior change contracts (quality_reports/harness_changes/) for context on what's already been tried. |
| 2 | Diagnose | Attribute cost/latency/invalid-actions/test-failures/permission-denials to specific harness components. Cluster across sessions: a failure mode in 1 trace is anecdote; in 5+, it's a pattern. |
| 3 | Propose | Concrete edit: rewrite tool description, change context-packing rule, add a linter, modify retry limit, insert HITL gate, adjust routing pattern. Every proposal must carry a draft change contract (see quality_reports/harness_changes/TEMPLATE.md). |
| 4 | Evaluate | Replay against held-out traces if available; otherwise propose the smallest measurement that would falsify (matches the contract's Falsification field). A mutation no measurement has checked is still a hypothesis, so it does not reach Promote. |
| 5 | Promote | Only after user approval AND a populated change contract. Apply via standard skill edit flow. The contract's Result section gets filled in 10–20 sessions later. |
Quality bar
- Read ALL available traces before proposing. Anecdotal sampling produces vibes-based tweaks.
- One change = one failure mode (mirrors the change-contract rule). Bundling makes falsification ambiguous.
- Evidence or silence. Every proposal carries trace line numbers / session IDs. No speculation.
- Telemetry-only failures (cost, latency, denials) are first-class targets, not just task failures. Paper §3.5.1: "token-cost traces reveal when retrieval or reflection stages consume budget without improving verification outcomes."
Workflow
On-demand: /harness-mechanic
- Read latest token baseline:
quality_reports/token_baselines/ - Read recent traces:
quality_reports/traces/(last 10 sessions minimum) - Read prior contracts:
quality_reports/harness_changes/*.md(avoid re-proposing the same change) - Run the 5-stage loop above
- Present proposals + draft change contracts for approval
- If approved: apply changes, copy draft contract into
quality_reports/harness_changes/, then run/skill-forge reviewon modified skills
Prerequisites
| Check | How |
| Traces exist | ls quality_reports/traces/*.jsonl |
| Schema is v2 | Latest traces should have "v":2 (see harness-trace SKILL.md) |
| Baseline exists | ls quality_reports/token_baselines/*.tsv |
| No traces? | Run harness-trace extract on recent sessions first |
| No baseline? | Run harness-trace baseline --base-dir . first |
Governance (paper §3.5.3)
This skill is itself subject to the PEV (plan-execute-verify) loop. Treat every proposal as a code change to a safety-critical runtime:
- Changes to permission boundaries, network access, credential handling, or hook behavior → require HITL approval before activation. No exceptions.
- Changes to prompt/instruction text → can proceed with normal review, but still need a change contract.
- Changes that expand the permission surface (e.g. relax a deny rule, broaden an allowlist) → 🔴 always, contract mandatory.
When to run
- Weekly review (e.g. Friday end-of-day).
- After a milestone week with many orchestrator sessions.
- When the review loop is stuck (score < 80 after 2+ rounds across multiple tasks, not just one).
- After noticing recurring failure patterns across sessions.
Common issues
| Issue | Action |
| No traces available | Extract from session JSONL: harness-trace extract |
| Fewer than 5 sessions of traces | Defer: sample too small, retry next week. |
| All scores high, no patterns | Skip; harness is working well. Note this in a short log entry. |
| Baseline missing | Run harness-trace baseline |
| Proposal seems risky | 🔴 by default; reject and ask for alternatives. |
| Composite tasks fail but steps look OK | Use cascade analysis: check atomic skill metrics (LOCALIZE precision, REPRODUCE success, review_validity) to find the root skill deficiency. |
| Same skill weak across sessions | Target the corresponding orchestrator step or agent prompt, not broad rule changes. |
| Token cost regression with no quality gain | Paper §3.5.1: this is a first-class signal. Likely retrieval/reflection bloat; propose a context-packing rule. |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.