Development Environment ROI Audit

Measure whether each coding-agent component saves more loop time than it taxes. Count two taxes: context the agent pays every session, and attention the user pays when a component interrupts them, waits on them, or acts on their files without asking. Use local evidence instead of generic best practices.

Operating contract

1. Ask for or agree on an absolute OUT_DIR and analysis window before collecting data. Default to 90 days; if less data exists, report the actual window and session count.
2. Keep source and configuration read-only during measurement. Write only under OUT_DIR. Git write operations are outside this workflow.
3. For each measurement step, state which numbers it should produce, and show the raw output a conclusion rests on before interpreting it.
4. Attach at least one measured number to every conclusion. Put unsupported opinions under UNVERIFIED; never blend them into findings.
5. Discover paths and wrappers from the machine. Do not assume home-directory layouts, project names, account aliases, or config inheritance.
6. Require item-by-item approval before changing configuration, disabling components, moving files, or deleting anything. A completed audit is not blanket mutation authority.

Read references/artifact-contract.md before starting. Read references/metrics-and-decisions.md before calculating ROI or proposing a decision.

Security boundary

• Never print a complete config node when it may contain headers, env, credentials, cookies, or tokens. Print keys, value types, and lengths instead.
• Pass credentials internally from their existing config only when a read-only probe genuinely requires them. Never put secret values in commands, output files, recovery copies, or reports.
• Redact stdout and stderr before tee. If a secret is accidentally displayed, stop repeating it, identify the affected filesystem copies without showing their contents, report their count, and request explicit approval to sanitize or remove them. Until approval, mark the audit blocked on credential residue and recommend server-side rotation.
• Recovery artifacts must contain only the target component and must strip authentication fields. Do not back up an entire config to recover one entry.
• Treat tools that create orders, send messages, write repositories, manage credentials, or read personal/account data as write-capable or sensitive even if their server name sounds harmless.

Phase 0: initialize and probe capabilities

Run the bundled initializer:

python3 <skill-dir>/scripts/init_audit.py --out-dir "$OUT_DIR" --window-days 90

It creates only the audit directory, records the chosen window, and copies helper scripts to $OUT_DIR/scripts/ for reproducibility.

Discover relevant hosts and utilities with version or command-resolution checks. Probe the requested agent CLIs plus jq, python3, and rg. Only inspect --help after a real command or flag fails.

Write 00-capabilities.json. Record wrapper/function provenance without printing function bodies that may contain credentials.

Phase 1: inventory the tax side

Inventory each host independently and retain its actual status vocabulary.

MCP and plugins

• Prefer structured CLI/init interfaces over manually reconstructing config.
• Record server status, tool count, schema bytes per tool, write-capable tools, sensitive-read tools, initialization time, and errors.
• Compare exact tool names and semantic capability overlap. Keep exact overlap and inferred semantic overlap separate.
• Mark servers configured on only one host. Distinguish active config, historical metadata cache, plugin-provided config, and compatibility imports.
• Never equate an absent server in one config-layer list with absence from runtime init; validate the runtime layer when the decision depends on it.

Skills

• Measure always-resident name + description separately from on-demand body bytes.
• Evaluate whether each description has a precise trigger, overlaps another skill, under-triggers, or matches almost everything.
• Usage count zero means “zero observed in this window,” not “never used.” Preserve the observation scope.

Instructions, commands, hooks, and memory

• Discover CLAUDE.md, AGENTS.md, Cursor rules, host-global instructions, commands, agents, hooks, and memory files.
• Classify instruction sections as effective, redundant, stale, or contradicted by code. Give an evidence path for every contradiction.
• Benchmark hooks with repeated samples. Report p50/p95 and event count; a no-op hook still pays process startup tax.
• Keep generated architecture prose, copied README material, directory trees, and project summaries only when measured evidence shows they reduce loop time.

Attention tax: notifications, pending decisions, scheduled automations, toggles

These components cost the user's attention rather than the agent's context. Inventory them with the same evidence rules.

• Notifications: discover every hook, script, or bot that messages the user (desktop, chat, phone). Record the sender, trigger event, channel, and where it logs. When a sender keeps no log, record that and mark its M9 numbers UNVERIFIED; propose a log under proposed/, do not add one.
• Pending decisions: discover queues where agents park questions for the user (deferred-decision lists, run-board asks, TODO ledgers). Record queue size, age of each item, and how each item was closed.
• Scheduled automations: list LaunchAgents and LaunchDaemons the user owns (launchctl print gui/<uid>/<label>), crontab -l, systemd user timers, and agent-host schedulers. For each, record the schedule, the command, the last exit status and log path, and its mode: acts (changes files, installs, sends, merges) or reports (writes a log or asks). Record whether it can overwrite user state, such as reinstalling a component that has local edits, and whether it offers a dry-run or report-only switch.
• Settings toggles: count feature flags, env switches, and boolean settings in the discovered configs. From config history (git log, dated backups) record whether each toggle changed in the window. Print names and value types only.

Write 01-inventory.json as data only, without recommendations.

Phase 2: mine the benefit side

Discover all candidate session roots, including renamed projects, moved directories, worktrees, and account-specific homes. List candidates by mtime before assigning ownership.

For every JSONL file, inspect its first line before parsing that file:

python3 "$OUT_DIR/scripts/probe_jsonl_schema.py" <discovered-roots...> \
  > "$OUT_DIR/raw/jsonl-schema-inventory.json"

Do not reuse an expected schema after one representative file. A parser must validate each file's probed keys and skip or separately handle mismatches. Record every schema deviation.

Calculate M1-M11 exactly as defined in the metrics reference. Pay special attention to exposed tools with zero calls, wrapper calls whose actual MCP name is nested in serialized arguments, and failures hidden behind successful wrapper invocations.

Write 02-metrics.json and 02-metrics.md. Put parsing scripts in $OUT_DIR/scripts/.

Phase 3: run controlled experiments

Run only experiments supported by the installed CLI version:

• Baseline tax: same minimal prompt, bare/ignore-user-config floor, one-component variants, then full config.
• Skill trigger accuracy: replay representative redacted first prompts; calculate precision and recall.
• Feedback-loop latency: time build, full test, focused test, lint, and typecheck when the user has not excluded business repositories.
• MCP handshake: initialize and tools/list per server with isolated temp/cache paths under OUT_DIR.
• Rewrite candidates: for components flagged Rewrite in the metrics reference, run the current text and its procedure and domain variants on the same prompts. Store variants under $OUT_DIR/rewrites/<component>/, never in the live config.

Keep prompts, model/account, cwd, and timing boundaries identical. Do not compare asynchronous or configuration-drifted runs as causal A/B. Label blocked experiments and missing API keys UNVERIFIED.

Write 03-experiments.md with sanitized raw output.

Phase 4: independent interpretation

When the user requests blind review and at least two independent CLIs are available:

1. Write the dispatch prompt before launching reviewers.
2. Give each reviewer only 01, 02, and 03 data, never the orchestrator's conclusions or the other reviewer's output.
3. Set bounded turns/time/budget and keep reviewers read-only.
4. Preserve disagreements. State the exact additional measurement needed to decide each one.

Skip this phase when independence cannot be established; record why.

Phase 5: decide component by component

Build one row per MCP server, skill, command, plugin, hook, or memory component:

| component | resident tokens | window calls | usage rate | error rate | ROI | decision |

Build a second table for attention components (notification senders, decision queues, scheduled automations, toggles):

| component | interruptions/week | action rate | failures in window | mode (acts/reports) | overwrites user state | decision |

Apply the hard thresholds from the metrics reference without softening them. For a Rewrite row, record which variant changes were adopted and the observed behavior difference behind each; applying the text to live config is an approved mutation like any other. If the user wants interactive cleanup, present one component at a time with its numbers and wait for a clear retain/remove/disable decision.

After an approved mutation:

1. Resolve the exact target with a read-only check.
2. Make the smallest structured edit; do not rewrite unrelated config.
3. Verify syntax, component count before/after, current runtime listing, and credential-field count without printing values.
4. Update the report immediately. Keep historical metrics labeled historical.

Phase 6: safe artifact cleanup

Large runtime copies and dependency caches are audit waste once their evidence is embedded in reports. Before moving or deleting a candidate, report:

• real path and size
• file count and symlink count
• whether the target itself is a symlink
• Git metadata count
• report-reference count
• whether the artifact is reproducible

Use an exact validated target. Prefer moving one target to Trash over a broad permanent deletion. Never empty the user's whole Trash. If deletion is blocked by policy, report the block rather than bypassing it.

Phase 7: final report and integrity

Produce FINAL.md with:

1. Component decisions and numeric evidence, including the attention table.
2. Baseline tax before/after, window percentage, and formula.
3. Instruction contradictions with evidence paths.
4. Independent-review disagreements and required follow-up measurements.
5. Proposed instruction files under proposed/, never overwriting originals without approval.
6. Three to five regression guards.
7. A separate UNVERIFIED list.

Run:

python3 "$OUT_DIR/scripts/finalize_audit.py" --out-dir "$OUT_DIR"

The finalizer refuses to create a manifest when non-runtime artifacts contain likely Authorization/Bearer values. It excludes reproducible runtime/**, writes MANIFEST.sha256, and reports the final artifact count and size.

Completion requires: zero pending user decisions in scope, all approved changes verified, credential residue count zero, core artifacts present, and manifest verification passing.