harness-runtime-optimizer
Skill0 starsChanged 3 months agoMITView on GitHub
Observe, experiment on, and optimize an agent harness runtime loop as a policy layer: routing, context, retry, stopping, tools, branching, budgeting, and execution-world policies. Designed for DSH but usable as a harness-neutral methodology.
In this repository4 other files it gives its agents.
- dsh-enhancement-analysisSkill
- dsh-plugin-potentialSkill
- publish-screenSkill
- repo-reconSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 6 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 19
Treat the harness runtime as a policy π operating over observable state s_t: action_t = π(s_t), where actions include model choice, context …
- Distinctive: line 56
Do not treat these as automatic wins.
- Distinctive: line 66
Never optimize cost/latency by silently sacrificing quality.
- Distinctive: line 67
Never claim causality from an uncontrolled before/after anecdote.
- Distinctive: line 69
No autonomous production rollout.
- Distinctive: line 70
No self-modifying policy without hard caps, logging, rollback, and user-approved objectives.
- Distinctive: line 71
If no reliable evaluator exists, improving the evaluator/instrumentation is Priority 0.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Harness Runtime Optimizer
Turn an agent harness from a static orchestration scaffold into an experimentable runtime policy layer. This skill does NOT ask only "can this project migrate to DSH?" It asks:
Given the same task distribution, where is the runtime wasting intelligence, latency, tokens, tool calls, or reliability — and which policy change measurably improves it?
The output is an evidence-backed optimization plan and, when execution is authorized, a controlled experiment loop. Never declare an optimization successful without before/after metrics.
Mental model
Treat the harness runtime as a policy π operating over observable state s_t: action_t = π(s_t), where actions include model choice, context construction, tool exposure, retry/fallback, branch/fork, stopping, budget allocation, execution environment, and merge/arbiter behavior.
Optimize a multi-objective utility such as U = task_quality - λ1*cost - λ2*latency - λ3*failure_risk - λ4*operational_complexity. Weights come from the user/project objective. If no weights are given, report the Pareto frontier instead of inventing a single winner.
Inputs
Use any available combination; do not block on missing inputs — start with static analysis, mark unobservable quantities, and propose the smallest instrumentation to measure them: project source; recon report (repo-recon); DSH enhancement report (dsh-enhancement-analysis); runtime traces (session logs, token/cost/latency, tool-call traces, evaluator scores); a representative task corpus; the DSH source (see dsh-enhancement-analysis → "Locating the DSH source").
Loop
Phase detail, checklists, and diagnosis signatures live in references/runtime-optimizer-phases.md; the intervention index lives in references/runtime-policy-catalog.md.
- Phase 0 — Optimization contract: task distribution, primary objectives, guardrails, decision rule (default provisional rule: no material quality regression; adopt only on a clear, segment-stable operational win).
- Phase 1 — Observability map:
request → context build → model request → response → tool selection → execution → observation → next turn → stopping → output; inventory per stage what is observable and which policy controls it. Deliverable: coverage table (metric → source → coverage → missing instrumentation → priority). - Phase 2 — Diagnose waste: policy failures, not code smells; every diagnosis carries trace or
file:lineevidence plus impact surface (quality/cost/latency/reliability/complexity). - Phase 3 — Interventions: across policy families (routing, context, retry/fallback, stopping, tools, branching, merge/arbiter, budget, execution world, session/memory); each states
hypothesis → trigger → policy change → expected benefit → likely failure mode → metrics → rollback condition. - Phase 4 — Experiment: paired comparison on a frozen task set; one policy dimension changed unless explicitly bundled; record the minimum scorecard below; segment by task type — averages hide regressions.
- Phase 5 — Verdict: ADOPT / REJECT / SEGMENT. Prefer SEGMENT over averaging away heterogeneous results.
- Phase 6 — Iterate: build
task/runtime features → policy choice → observed rewarddeterministically first; only add learned selection with bounded exploration, hard guardrails, and rollback.
Minimum scorecard
| Metric | Baseline P0 | Candidate P1 | Delta | Guardrail / target | Verdict |
| Task success / quality | |||||
| Total latency | |||||
| Input + output tokens | |||||
| Estimated/actual cost | |||||
| Tool calls | |||||
| Model calls | |||||
| Retry/fallback rate | |||||
| Turns to completion |
DSH-specific leverage
Runtime policy can live outside business logic: agent/request interception → model routing, budgets, fallbacks; agent/request-error + dsh-llm-retry → retry execution; llm/stream waterfall → caching/logging on raw model calls; event waterfall → instrumentation and stopping hooks; session-as-log → replayable experiments; ctx.subagents.start() (fork/spawn providers) → paired branch/model experiments; service-key injection → swappable cache, tools, filesystems, subprocess worlds.
Do not treat these as automatic wins. Measure whether the primitive actually improves the target workload. Re-check the DSH source if the exact API surface may have drifted.
Outputs
Default report path: <project>/dsh-scan/runtime-opt-<project>.md — inside the analyzed project (create dsh-scan/ if missing; a user-specified path overrides) — using templates/runtime-optimization-report.md. A complete report contains: optimization contract; runtime/observability map; baseline scorecard; diagnosed waste with evidence; ranked policy hypotheses; experiment design; before/after results when runs are authorized; adopt/reject/segment decisions; resulting policy table; next experiment queue.
Rules
- Evidence first: runtime claims need trace evidence; static architecture claims need
file:lineevidence. - One variable at a time by default; explicitly label bundled experiments.
- Never optimize cost/latency by silently sacrificing quality.
- Never claim causality from an uncontrolled before/after anecdote.
- Report heterogeneity by task segment when it changes the decision.
- No autonomous production rollout. Changes to production policy require explicit user approval.
- No self-modifying policy without hard caps, logging, rollback, and user-approved objectives.
- If no reliable evaluator exists, improving the evaluator/instrumentation is Priority 0.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

