Perf optimization (mecatl offline harness)

The companion to the regression-tracking design in docs/perf-tracking.md. This skill is the offline benchmark/scenario workflow: measure → profile → optimize → prove → guard. For diagnosing a running harness via the perf MCP server, use the perf-mcp-interpretation skill instead — different tool, different signals.

The harness (where the numbers come from)

• task bench — hot-path microbenchmarks (engine/prompt, engine/governance, engine/agent). BENCHCOUNT default 10. Offline (mockllm + memfs). Not part of task test. The engine is its own Go module, so task bench/task fuzz run these as cd engine && go test … ./prompt/ ./governance/ ./agent/. An ad-hoc re-run on an engine package must do the same: cd engine && go test -bench=… ./agent/ (an explicit ./engine/agent/ path also resolves via the committed go.work, but the ./... wildcard does not cross the module boundary).
• task perf:scenarios — five whole-loop scenarios. Four live in perf/scenarios/ (single-session-long, team-fanout, background-subagents, compaction-cycle); the fifth (tui-scrollback) lives in cmd/mecatui/ui/, and the task runs both packages — so go test ./perf/scenarios/ alone gives only four. BENCHCOUNT default 6. MECATL_PERF_JSON=.scratch/x.json writes the KPI JSON.
• Both are deterministic and offline — never reach for a live model/network.
• PGO/bench captures over engine/ packages depend on the active go.work (the committed workspace wires ./engine in alongside the root) — a GOWORK=off invocation resolves the engine module standalone and won't see the host repo.

What to gate on (allocs-first)

allocs/op, bytes/op, goroutine-delta, and cache-hit-rate are deterministic and portable — these are the real signal. ns/op and rss_* are machine-specific — treat as advisory shape, never the headline. A perf change is done only when allocs/op moves in benchstat; a wall-clock-only "win" on a shared machine is noise.

Workflow

1. Measure — pick the benchmark that actually exercises the path

Run the relevant benchmark and confirm it reflects the workload you care about. A benchmark whose per-iteration setup dwarfs the code under test, or that is all-miss by construction, will hide a real win. If no benchmark covers the path, add one first (match the existing bench_test.go style: for b.Loop(), results to a package-level sink, offline).

2. Profile — pinpoint the site, do NOT guess

Capture an allocation profile on the benchmark and follow it to a file:line:

go test -run='^$' -bench=BenchmarkX -benchmem -memprofile=.scratch/x.mprof -count=3 ./pkg/
go tool pprof -alloc_space   -top -nodecount=25 .scratch/x.mprof   # bytes
go tool pprof -alloc_objects -top -nodecount=25 .scratch/x.mprof   # object count
go tool pprof -list=FuncName .scratch/x.mprof                      # line-level

Save profiles under .scratch/ (repo rule — never /tmp). Follow the profile to the real site. The hypothesis is often wrong (see the playbook: the TUI hotspot was the string join, not SetContent as assumed). Let -list show you the exact lines.

3. Design — the smallest change at the proven hotspot

Optimize only what the profile proves is hot. Weigh the win against the real cost: a microsecond saved once per turn is invisible next to an LLM round-trip. The wrong abstraction is worse than the allocation — if the clean seam doesn't exist or the fix adds stateful invalidation surface for a marginal gain, it is a legitimate NO-GO. Say so and skip it rather than forcing it.

4. Prove — benchstat before/after, count >= 10

go test -run='^$' -bench=BenchmarkX -benchmem -count=10 ./pkg/ > .scratch/before.txt
# ... apply the change ...
go test -run='^$' -bench=BenchmarkX -benchmem -count=10 ./pkg/ > .scratch/after.txt
benchstat .scratch/before.txt .scratch/after.txt   # go install golang.org/x/perf/cmd/benchstat@latest

allocs/op / B/op must drop with a statistically significant delta. Re-run task perf:scenarios and confirm the scenario KPI moved in the expected direction. Update the baseline snapshot in docs/perf-tracking.md.

5. Guard — keep behaviour identical, prove the guard isn't vacuous

• Pure-perf changes must be byte-identical. For the TUI, task test:golden must stay green with zero golden diffs — you change HOW, never WHAT. Caveat: golden refresh runs -update first, so a plain go test ./... (no -update) against the committed goldens is what actually catches a regression; don't rely on the refresh step to catch a stale serve.
• Any cache/oracle you add must be mutation-tested. Temporarily break it (force the stale/wrong path), confirm a test goes red, then restore (back up with cp, restore with cp — never git checkout, it wipes uncommitted work). A guard that stays green when the behaviour is broken is worse than none.
• A perf change touching an EXPORTED engine/ symbol trips the api-compat CI gate (task api:check) — a failure mode a perf optimizer wouldn't expect. Keep pure-perf changes byte-identical to the engine's public surface and it never fires; if the surface legitimately changed, run task api:update and add an engine/CHANGELOG.md entry classified per engine/COMPATIBILITY.md.

Common pitfalls

• All-miss benchmark hides the win. A streaming bench that mutates state every iteration never hits the cache you added — it's the worst-case floor, flat by design. Add a steady-state bench for the cache-hit path to show the real win.
• Setup swamps the signal. Build fixtures outside b.Loop(); confirm with a quick profile that setup isn't the dominant allocator.
• Chasing ns/op on a shared machine. Gate on allocs; ns is advisory.
• Memoizing across the byte-stable prompt prefix. Any prompt-inventory cache must produce byte-identical output or it tanks the provider cache-hit rate — the exact thing perf-tracking exists to protect. Usually not worth it (see playbook).

Complementary: PGO (free compiler-level wins)

Beyond hand-optimizing a hot path, Profile-Guided Optimization lets the compiler optimize from a CPU profile (typically 2–14% CPU — but here sub-1% of wall-clock, since cost is network-dominated: a free set-and-forget win, not a latency feature). The mechanism is already wired:

• task pgo:collect builds a PROVISIONAL offline profile under .scratch/pgo/.
• cmd/mecated/default.pgo is the reserved slot — go build's -pgo=auto applies it automatically the moment a profile is dropped there (nothing in the build chain passes -pgo=off).
• Do NOT commit an offline-collected profile — it trains the compiler on the mockllm path that ships in no production binary (it can't pessimize, but it wastes the one slot). Commit only a real /debug/pprof/profile capture from a running mecated under load.

Full rationale, the per-binary decision, and the production refresh + staleness process live in perf-tracking.md "Profile-guided optimization" — read it before touching PGO.

See Also

• references/playbook.md — pprof flag cookbook + two worked examples (a real win and a real NO-GO) showing the discipline end to end.
• docs/perf-tracking.md — the KPI design, gating posture, and baselines.
• perf-mcp-interpretation skill — the live counterpart (running-harness diagnosis via the perf MCP server).