Agent Practice Audit

Answer one question with evidence: how far is this person's coding-agent usage from what the vendors officially recommend? Two modes share one practice list, references/practices.json.

Pick the mode from what the user wants to know. "How am I doing / score me / check-up / what should I change" is audit. "Update the practice list / are the sources still valid / add what vendors published recently" is refresh. A request that only checks links (no new practices) is check_practices.py --fetch alone: it writes nothing; report the link status and the gate line, and change files only if the user then asks. When unsure, ask in one line; when the run is scheduled or headless, nobody can answer, so do the mode named in the invocation and never stop to ask.

Reply in the user's language; set lang in judgments.json to match.

Related: dev-environment-roi-audit answers "is this tool, MCP or skill worth keeping?" with ablations. This skill answers "how far is my usage from official practice?". When a finding is really about cost or startup tax of a component, point to that skill instead of judging it here.

Contract

1. Read-only towards the user's setup. Never edit AGENTS.md, CLAUDE.md, settings, hooks, Codex/Grok config, or session logs. Suggestions go into the report; the user decides.
2. Audit mode writes only under $AGENT_PRACTICE_AUDIT_DIR (default ~/.local/share/agent-practice-audit). Refresh mode additionally edits this skill's own references/ and only in a git checkout of the skill on a branch (check with git -C "$SKILL_DIR" rev-parse --abbrev-ref HEAD; the scripts refuse other targets), never an installed copy (refresh output then ships as a pull request). Never write audit output into a repository; every script that writes resolves the real path and refuses targets inside a git work tree.
3. No secrets in any output. Use scripts/config_scan.py for config; it reports sizes, counts, deny-rule coverage categories (never the rules themselves), env variable names and an allowlist of Codex keys only. If you open a config file yourself, do not print env, headers, tokens or MCP credentials.
4. Every ok/part/no needs a number or a file fact as evidence. If the logs cannot show it, use na and say why.
5. The report is local. Do not publish or attach it. The work files hold no prompt text; still keep sessions.jsonl (paths, times, hashes) on this machine.

Audit mode

SKILL_DIR is this skill's directory. RUN defaults to $AGENT_PRACTICE_AUDIT_DIR/reports/$(date +%F) (base ~/.local/share/agent-practice-audit when the variable is unset).

1. Collect. python3 "$SKILL_DIR/scripts/run_audit.py" --days 60 (use the window the user names, e.g. --days 14 for "two weeks") (add --repo DIR for repositories the user cares about, --run DIR to override RUN). One call runs extract, aggregate, config_scan and prejudge into $RUN/work/ in a few seconds. Extract auto-detects Claude Code (~/.claude*/projects/**/*.jsonl), Codex (~/.codex*/sessions/**/*.jsonl), Grok (~/.grok*/sessions/**/chat_history.jsonl plus sidecars) and WorkBuddy (~/.workbuddy/projects/*/*.jsonl); globs cover multi-account directories. It prints per-host file counts split into human, headless and subagent sessions, and not found for missing hosts. A session is in the window when its last event is. dropped-outside-window counts files touched recently whose events are older; spans-window-start counts sessions that began before the window and are kept whole. Quote those lines to the user. Roots can be overridden; see references/audit-files.md. Exit code 3 means no human sessions in the window: stop, say which roots were searched (printed) and offer a longer --days or a custom root. A report built on nothing would grade the person on silence.
2. Judge. Read only $RUN/work/judge-input.json (about 13 KB; schema in references/audit-files.md). For each practice, accept its draft or override it and say why in evidence, using the numbers in metrics and counts. Open summary.json / config-scan.json only for practices whose draft is null (they keep their detect text and a note) or when an override needs more detail. Judge mainly on human sessions; mention headless and subagent counts where they matter. Hosts flagged low_sample get a caveat, not a verdict. Write "$RUN/judgments.json": status, a short summary for the table, evidence with numbers, a concrete suggestion for part/no, and a caveat where the signal is noisy. Every ok/part/no needs evidence, every part/no a suggestion, every na an evidence or caveat; build_report.py refuses the run otherwise. Write the judgments for a reader who scans first and acts second:
   • Voice. Lead with the verdict and the number. One fact per clause (about 25 Chinese characters or 15 English words). The user, a host or a file is the subject; active voice. Keep English terms (subagent, headless, hook, sandbox) and, in zh, add a short Chinese gloss on first mention. No buzzwords, no filler summaries, no emoji.
   • headline: one sentence (up to ~35 Chinese characters / 20 English words): the biggest strength, then the biggest gap.
   • lead: two sentences. First, session counts by host and kind (human, headless, subagent). Second, the one or two fixes that climb the grade ladder fastest.
   • summary: a table-cell fragment (up to ~22 Chinese characters / 10 English words) with the decisive metric first, e.g. 66/339 个 Claude 会话超 200 次工具调用.
   • evidence: one to three short sentences with exact counts, ratios and medians from summary.json / config-scan.json; config keys and ~/... paths in code format.
   • suggestion: one or two imperative sentences that name the file, flag or command to change; empty for ok.
   • Good: {"summary": "Codex 24/24 人工会话全权限运行", "evidence": "Codex 24/24 人工会话为 approval=never 且 sandbox=danger-full-access。", "suggestion": "在 ~/.codex/config.toml 设 approval_policy = \"on-request\" 和 sandbox_mode = \"workspace-write\"。"}
   • Bad: {"summary": "权限配置存在较大风险隐患", "evidence": "经全面分析，很多会话没有开启沙箱。", "suggestion": "建议强化安全意识，完善权限管控。"} (no numbers, vague subject, nothing to do next).
3. Build. python3 "$SKILL_DIR/scripts/build_report.py" --judgments "$RUN/judgments.json" --summary "$RUN/work/summary.json" --out-dir "$RUN" (always pass the same RUN as step 1, so a custom --run or a run past midnight keeps the report beside its inputs) It scores, finds the previous run automatically (--previous none to skip), and prints score, grade and the upgrade ladder. Open the HTML locally. It also writes metrics.json (see references/audit-files.md). When a checkup command is on PATH, run checkup import "$RUN" --json and quote its one result line; exit code 10 means the user has not enabled the import yet, so pass on the enable command it prints instead of running it. An import failure never fails the audit: report "audit done, import failed" with the reason.
4. Tell the user the grade and score with coverage, the change since last time, the ladder ("fix these N items -> grade X", each step tagged config or habit change and heuristic where it applies; on equal gains config fixes and vendor-stated thresholds come first), the top three suggestions, and the report path. Mark anything you could not verify. In a scheduled run, put the same lines in the notification text instead (no session excerpts in it).

Sources in the report: a reachable source links to the live page with a text fragment. A source whose digest says linkStatus: unreachable still shows its link, marked as unavailable (原文暂时打不开), with the stored digest summary beside it. Practices with basis: digest are marked 基于摘要 (from digest).

Known false positives to keep in mind while judging: correction categories are keyword matches; a check counts when it was run, not whether it passed; a screenshot is a screenshot tool, a browser or computer-use capture, or reading an image the session produced; plan mode in Codex/Grok is not logged, so only substitutes count there. Rules ending in (heuristic) use thresholds the vendor text does not state; say so when a verdict rests on one.

Refresh mode

Two stages: one agent proposes, at least two other models verify. Nothing enters the active list on one model's word.

1. Re-check links. python3 "$SKILL_DIR/scripts/check_practices.py" --fetch --update-digests re-fetches every source, writes linkStatus (ok, redirected + finalUrl, unreachable) and lastVerifiedAt into references/sources/<slug>.md, and reports quotes that no longer match. Unreachable or dead links are flagged in the changelog, never deleted; their practices fall back to the digest (see the gate in step 7). When an unreachable page works again, re-verify its practices against the live page and set basis back to live. --update-digests writes only on a non-default branch of a git clone of the skill (start with git switch -c practices-refresh/<date>; --allow-default-branch overrides) and refuses symlink targets and installed copies outside git.
2. Propose (stage a). Read references/sources.md. Search only the allowlisted domains, starting from the seed URLs. For each candidate, fetch the page and copy a verbatim quote of at most 25 words plus a shorter fragment from inside the quote (it must match rendered page text so the #:~:text= link scrolls to it). Write the candidate in practice shape: id (kebab-case, stable), dimension (ctx, safe, doc, plan, verify, auto), title, title_zh, sources[] (source slug, page, url, quote, fragment), detect, rule (text with every number, ending in (heuristic) when a number is not in the vendor text, and params), and fix_kind (config for a one-time file or setting change, habit for a working-practice change). A practice without a mapping in scripts/prejudge.py still works: its draft is null and the judge reads detect. A second vendor saying the same thing becomes another source on the existing practice.
3. Digest. Each source page needs references/sources/<slug>.md: frontmatter slug, title, publisher, url, fetchedAt, lastVerifiedAt, linkStatus, finalUrl; a ## Summary of 5-12 bullets in your own words (no copied paragraphs); a ## Cited quotes list holding every quote practices cite from that page. Verifiers use the digest when the page cannot be opened.
4. Verify (stage b). Save the verbatim page text with python3 "$SKILL_DIR/scripts/check_practices.py" --save-pages PAGES --candidates cands.json (never a model summary; unreachable pages are skipped and verifiers use the digest). Then run python3 "$SKILL_DIR/scripts/consensus.py" run --candidates cands.json references/practices.json --pages PAGES --out "$AGENT_PRACTICE_AUDIT_DIR/refresh/$(date +%F)" --exclude <proposer> [--max-calls N]. It sends one prompt per source page and verifier, with the page once and every candidate citing it, and expects a JSON array of verdicts; a malformed batch answer counts as malformed for every candidate in it. --save-pages writes PAGES/manifest.json; a page counts as live only when its manifest runId, a fetchedAt within 24 hours, its URL and its hash all match; the rest fall back to the digest. Verifier calls add the isolation flags each CLI's --help lists, so user settings, hooks, MCP servers and skills are not loaded; a verifier with no such flags (Grok today) runs with a warning and isolated: false in decisions.json, and --require-isolation skips it. It detects the verifier CLIs (consensus.py detect shows the exact commands, with flags taken from each installed --help), sends the verify prompt on stdin in parallel (--jobs, default 4; --timeout per call, default 600 s; empty temp cwd outside any repository), and writes OUT/<id>/v-<name>.txt plus OUT/decisions.json. Never loop over verifiers by hand.
5. Decide. run applies the rule per candidate and prints one line each (accept|reject|pending-consensus id agree/total): accept iff agree >= 2 AND agree > total / 2 AND no veto. total is every verifier that ran (malformed or unparseable output, a timeout or a non-zero exit counts and never agrees); agree counts valid accept verdicts, which need real JSON booleans and all three checks true (plus rule_faithful: true when the candidate has a rule); any result with quote_found strictly false vetoes, even if the rest of it is malformed. consensus.py decide FILES --names ... re-applies the rule to saved outputs; it reads the v-<name>.status.json beside each file, so a timed-out or failed call stays malformed. Accepted candidates go into practices with consensus from the output, verifiedAt today, basis live or digest, and no needs-consensus flag. Rejected candidates go to the changelog under 未采纳 with each verifier's reason. When fewer than 2 verifiers ran, the candidate goes into the top-level pending list as pending-consensus; audit mode ignores pending.
6. Seed re-verification. Practices with "needs-consensus": true (the 20 seeds carry a one-reviewer manual check from 2026-10-31) are verified in the same run as new candidates (step 4 passes references/practices.json, which supplies every needs-consensus seed and the pending list). A rejected seed moves to the changelog with reasons; ask the user before removing an active practice. In a scheduled or headless run, keep the rejected seed active, list it under 待用户决定 in the changelog, and let the pull request carry the question.
7. Gate and record. Run python3 "$SKILL_DIR/scripts/check_practices.py" --fetch; fix what it reports and rerun, at most three times. If it still fails, stop and report the failing sources instead of weakening a quote. Gate semantics: every source must be either live-verified (page reachable, fragment and quote found) or digest-verified (page dead, unreachable or script-rendered, and its digest lists the quote under Cited quotes). It fails only for a quote or fragment missing on a reachable page, or an unreadable page whose digest is missing or lacks the quote. A practice that now rests on its digest is printed with a note: set "basis": "digest" on it (or "mixed" when only some of its sources rest on a digest) so the report shows 基于摘要. Only needs-consensus seeds may carry a consensus that fails the accept rule; the gate prints awaiting consensus: K. The summary line reads live: N/M ok, digest-verified: K, failed: 0. Then add a dated section to references/changelog.md (added, changed, 未采纳, pending, flagged links, and the gate summary line), and show the user the diff. Renaming an id breaks trend comparison; do it only with a changelog note.

Scoring

ok = 10, part = 5, no = 0, na not counted. Total and per dimension = earned / measured max x 100. Coverage = measured / total practices. Below 60% overall coverage there is no grade and no ladder, only the score marked 测量覆盖不足 / coverage too low; a dimension shows a score only with at least 2 measured practices and at least 50% coverage, otherwise —. The change since last time counts only practices measured in both runs; coverage change and added or removed ids are listed separately. A practice whose rule or detect changed since the last run is marked 规则已改 / rule changed and left out of the change. Grade labels only describe how many measured practices are met; they never mean the work can be handed off unchecked. Grades: S >= 90, A >= 80, B >= 70, C >= 60, D < 60. Dimensions: ctx (context management), safe (permissions and safety), doc (instruction files), plan (think before doing), verify (verification), auto (parallelism and automation). Equal weights are a convention for comparison, not an official standard; say so when presenting a grade.

Scheduling

The skill installs nothing. references/scheduling.md describes the intended setup: refresh runs remotely once a week (a hosted scheduled agent, or the sample GitHub Actions workflow) and delivers a pull request, never a push to the default branch; the owner merges it and installed copies pick it up through normal skill updates. Audit runs locally once a month through launchd (macOS) or cron / a systemd timer (Linux) with a headless agent call and a desktop notification when the report is ready. Install a schedule only when the user asks, and show them the exact file first.