agentleFS
fritzprixfritzprix/libr-agent/.agents/skills/harness-eval-loop/SKILL.md

harness-eval-loop

Skill12 starsChanged 6 days agoMITView on GitHub

Build the current LibrAgent revision, run the Harbor benchmark that matches the recent work, decide whether the experiment is accepted (adopt / revert / iterate / inconclusive) against a comparable baseline, and file newly found harness problems and performance-regression causes as GitHub issues. With an interval and dataset it repeats this on a schedule. Use when the user types "/harness-eval-loop", "/harness-eval-loop 2h layman", "harness_eval_loop", "벤치 돌려서 experiment 판정", "하네스 평가 루프", or asks to run bench, judge an experiment, and open issues for what the run found.

The file

Show its rules and text

We split this skill into the rules it gives an agent

13 rules. About 0 are common, 8 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈8

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Read, in this order, and stop at the first that decides:

    line 34
  2. Very specific:

    If two datasets are equally plausible, ask once.

    line 45
  3. Very specific:

    Do not run pnpm refactor:validate or raw cargo test (project rule).

    line 74
  4. Distinctive:

    Run in the background and wait for the completion notification.

    line 82
  5. Very specific:

    Use the same flags as the baseline.

    line 88
  6. Distinctive:

    Never change timeouts, resources, tasks or verifiers to get a result.

    line 89
  7. Distinctive:

    Acceptance criteria: Check each criterion written in step 1 against trial-level evidence (tool sequences, observations), not reward alone.

    line 104
  8. Distinctive:

    Regression check: Every task that passed in the baseline must still pass.

    line 106
  9. Distinctive:

    Never fix product behavior to match a bench quirk, and never put task wording into a skill trigger or example as the "fix".

    line 133
  10. Very specific:

    If a match exists, add a comment with the new evidence (job path, trial id, tool sequence) instead of opening a new issue.

    line 141
  11. Distinctive:

    Never paste secrets, wake URLs/tokens, or the user's email.

    line 150
  12. Very specific:

    Do not push, open PRs, or edit product/verifier/skill files from this loop.

    line 155
  13. Distinctive:

    Stop and report (do not file issues or decide) when: the build fails.

    line 182

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Harness Eval Loop

Automates the tail of harbor-harness-improvement-loop: build → bench → judge → file issues. Analysis rules (harness-only scope, evidence model, comparability, never edit verifiers or official limits) are inherited from that skill; read its SKILL.md and harbor-harness-improvement-loop/references/evidence-model.md first. This skill adds execution, a decision, and issue filing.

Invocation

CommandBehavior
/harness-eval-loopOne run now. Dataset and experiment are inferred from recent context (below).
/harness-eval-loop <interval> <dataset>Run one tick now, then repeat every <interval> (30m, 2h) on <dataset>.
/harness-eval-loop stopStop the recurring loop.
--dry-runDo everything except gh issue create/comment; print the issue drafts instead.

<dataset> is a short name or a path: layman, diverse, swe, media, browser, fileop, compaction, or any benchmarks/harbor/<dir>. Map names to the pnpm bench:* scripts in package.json (e.g. layman → pnpm bench:layman, diverse → pnpm bench:diverse). If a name has no script, stop and ask; never invent a dataset id.

Recurring mode: arm it with the loop skill / ScheduleWakeup, passing the same /harness-eval-loop <interval> <dataset> prompt back. Before arming, check state (below) for an existing loop and do not arm a second one. Pick interval no shorter than one full bench run; a tick that finds the previous run still going records "run in progress" and does nothing else.

Inferring the run from context (no arguments)

Read, in this order, and stop at the first that decides:

  1. An open experiment ledger: newest .libragent/work/harbor-harness-cycle/*/experiment.md or analysis.md whose decision is iterate / inconclusive → reuse its dataset, tasks, attempts.
  2. The current branch diff (git diff main...HEAD --stat, plus git status): changes under benchmarks/harbor/<dir> or a builtin tool/skill/prompt that a dataset's tasks exercise → pick the dataset that covers them (layman-intent for skills/delegation/scheduling, fileop for readFile/editing, compaction for context/compaction, etc.).
  3. The user's last request in this conversation.

State the inferred dataset, tasks, attempts and baseline job in one short block before building. If two datasets are equally plausible, ask once; otherwise proceed. Prefer the smallest slice that exercises the change (--include / --n-tasks), then widen only if the slice is positive.

Tick procedure

1. Freeze the experiment contract

Record in .libragent/work/harbor-harness-cycle/<cycle-id>/experiment.md (cycle-id = YYYY-MM-DD-cNN): git revision + dirty flag, dataset/tasks, attempts, concurrency, assistant, execution + workspace mode, timeout/verifier settings, and the hypothesis and acceptance criteria written before the run (e.g. "t17 reaches createAgent; no task that passed regresses").

Choose the baseline: the newest complete job under jobs/ whose contract matches (same dataset/tasks/attempts/concurrency/modes/verifier). If none matches, run one on the base revision first (or mark the result inconclusive and say why). Incomplete or cancelled jobs are never baselines.

2. Build

The bench launches the release binary (src-tauri/target/release/libragent) via --isolated, so a stale binary silently benchmarks old code.

git rev-parse --short HEAD
pnpm tauri build --no-bundle      # frontend + release binary; heavy, run once per revision
  • Skip the build only if the binary is newer than every file in git diff --name-only <last-built-rev> (store lastBuiltRev in state). Doc-only or benchmarks/-only changes never need a rebuild.
  • Do not run pnpm refactor:validate or raw cargo test (project rule). For code changes run only the focused tests/lint for the changed files before building; for Rust use pnpm rust:test --test <name>.
  • Build failure → stop, report the error, file nothing about the benchmark.
  • Confirm Docker and harbor are available (docker info, harbor --version); missing → stop and say so.

3. Run the bench

Run in the background and wait for the completion notification; do not poll.

pnpm bench:<dataset> [--include '<glob>'] [--n-tasks N] [--n-attempts K]   # run_in_background
  • Use the same flags as the baseline. Concurrency stays 1 in --isolated mode.
  • Never change timeouts, resources, tasks or verifiers to get a result.
  • Ctrl+C / cancellation leaves a partial job: analyze it for diagnosis only, never as a verdict.
  • Print the job path as soon as it exists so the user can follow along.

4. Evaluate and decide

python .agents/skills/harbor-harness-improvement-loop/scripts/analyze_harbor_results.py \
  jobs/<baseline> jobs/<candidate> \
  --output .libragent/work/harbor-harness-cycle/<cycle-id>/summary.json

Then, per the improvement-loop rules:

  1. Comparability gate. Any mismatch in the contract table → decision is inconclusive.
  2. Acceptance criteria. Check each criterion written in step 1 against trial-level evidence (tool sequences, observations), not reward alone.
  3. Regression check. Every task that passed in the baseline must still pass; a flipped task is read in the trajectory before it is called noise.
  4. Cost metrics. Compare turns, tool calls, input/output/cache tokens, repeated and failed calls.
  5. Variance. With 1 attempt, differences of ≤1 task are not evidence; say inconclusive or rerun with more attempts rather than claiming a win.

Decision, exactly one: adopt (criteria met, no regression, cost not worse) · revert (regression traced to the change) · iterate (partial/new contract defect found) · inconclusive (not comparable / within variance / telemetry missing). Write analysis.md using harbor-harness-improvement-loop/references/report-template.md; every percentage carries its count.

Analysis scope applies: no model or serving-engine commentary. Findings are tool sequences, observations, and the schema/handler/response/prompt/adapter/telemetry/verifier source.

5. Triage findings before filing

For every failed or degraded trial and every cost outlier, classify (this split is mandatory):

ClassMeaningGoes to
product defecta tool/prompt/handler/response contract is broken or misleadingissue, label per layer
bench / verifier defectthe check can't be met by correct product behavior (e.g. wake flow vs idle end)issue, label bench/harbor
performance regressioncandidate worse than baseline on turns/calls/tokens/retries with a traced causeissue, with the delta
pattern onlyodd tool order but no contract brokenanalysis.md only, no issue
policy questionneeds an owner decision (e.g. act immediately on an explicit request)issue labelled question; change nothing

Never fix product behavior to match a bench quirk, and never put task wording into a skill trigger or example as the "fix". Single-trial findings are labelled hypotheses in the issue.

6. File issues

Authorized by this skill, but bounded:

  1. Dedupe first. gh issue list --state all --search "<tool or symptom> in:title,body" --limit 10. If a match exists, add a comment with the new evidence (job path, trial id, tool sequence) instead of opening a new issue. Never reopen a closed wontfix issue.
  2. At most 5 new issues per tick. Rank by breadth × confidence; the rest go to analysis.md under "backlog".
  3. Body (Korean unless the repo thread is English): title bench(harbor): <symptom> or fix(<tool>): <symptom>; sections Evidence (job, trial ids, git revision, tool sequence with args/observation sizes), Class (from step 5), Owning layer + inspected paths, Hypothesis / confidence, Proposed smallest intervention (not applied), Acceptance criteria, Counter-example check. Link the cycle's analysis.md path and the parent issue if any. Never paste secrets, wake URLs/tokens, or the user's email.
  4. Labels: reuse existing ones only (bug, enhancement, prompt, agent, question, ...); check gh label list and never create labels.
  5. --dry-run, or any doubt about whether a finding is outward-facing/sensitive → print the draft and ask.

Do not push, open PRs, or edit product/verifier/skill files from this loop. Applying a fix is a separate, user-approved step.

7. Record and report

Update .libragent/work/harness-eval-loop/state.json (git-ignored):

{
  "loopArmed": true,
  "interval": "2h",
  "dataset": "layman",
  "lastBuiltRev": "abc1234",
  "baselineJob": "jobs/…",
  "lastJob": "jobs/…",
  "openCycle": "2026-10-26-c01",
  "filedIssues": [2341],
  "lastDecision": "iterate"
}

End every tick with a short report: revision, job path, pass count vs baseline (numerator/denominator), the decision, issues opened/commented (with numbers), and what the next tick will do. In recurring mode, a tick with no change, no new findings and no decision change says so in one line.

Stop conditions

Stop and report (do not file issues or decide) when: the build fails; Docker/harbor is unavailable; the bench was cancelled or errored before any trial completed; the dataset name is unknown; the baseline is not comparable and no rerun was approved; or the only evidence is hidden reasoning text. In recurring mode also stop re-arming after 3 consecutive ticks that fail for the same environmental reason.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.