generation-audit — harness audit at a model generation change

The procedure for Scaffold Dissolution's (rules/common/akc-cycle.md) third trigger = model generation change. One run performs two checks, carrying line fixes through to application and asset-level verdicts through to the stocktakes.

Anthropic updates the dated-pattern table and the target model's behavior inside the /claude-api skill with each model release, so they are not copied here — this skill holds only this harness's scope, exclusions, application conventions, and recording, plus the runtime cross-check that prompt-audit does not have. Asset-level verdicts (Retire / Merge, etc.) are owned by the stocktake for each asset class (ADR-0022); this skill stands on the side that supplies the evidence.

Phase 0 — Target model and scope

• Target model: the model newly assigned to a role in this harness. When models differ by role, look at an asset through the model of the role that reads it (a dispatch packet template → build role; rules / CLAUDE.md / output style → every session; judge-role skills → judge role)
• Scope: the prompt surface of ~/.claude — rules/common/, CLAUDE.md, AGENTS.md, output-styles/, skills/*/SKILL.md and references/, agents/, and the text hooks return to the model
• Not applied to (audited and recorded in the ledger, but not edited):
  • unmodified copies of externally originated files and symlinked external skills — keep them as copies (skill skill-writer §3)
  • files another session is working on — M / ?? entries in git status that are not your changes. To avoid breaking that session's commit, leave the findings in the ledger and carry them to the next run

Phase 1 — Capturing the runtime layer

The source of truth for the cross-check is what is actually loaded at inference time — the system prompt and tool descriptions. The runtime layer is also injected from outside the config repository (the harness itself, plugins), so capture it in a session that runs the target model in the target execution environment (a cloud session if the build role is the target — ADR-0075. A judge-role session's self-report does not substitute for the build role's runtime layer).

1. Build the theme list from the self-authored assets — assign each instruction in rules / CLAUDE.md / output style / skills / agents to a theme (planning, commits, review, scope, delegation, verification, report shape…). Partitioning the asset side first means no asset is missed in the cross-check
2. Have it quote verbatim, one theme at a time — ask one theme per request so no summarizing slips in:
   • system prompt: "From the system prompt currently loaded, quote verbatim the instructions about <theme>. Do not summarize."
   • tool description: "Quote the description of <tool name> verbatim."
3. Record the limits of the capture in the ledger — zero findings means "not found by this question," not "no conflict" (runtime instructions you do not know exist cannot be picked up). The capture is a self-report routed through the model, so before finalizing any disposition (retirement, inversion), confirm the same wording reproduces in a separate session

Phase 2 — Conflict and redundancy

Conflicts include not only instruction vs instruction but also instruction vs incorrect factual statement (ghost references that keep claiming a removed setting is "already disabled"). Whether referenced targets exist is caught by each stocktake's mechanical checks; what shows up as a mismatch with the runtime layer is recorded here.

Phase 3 — Dated-pattern scan

Run /claude-api prompt-audit with the target model and scope from Phase 0. Resolve the paths of the procedure (shared/prompt-audit.md) and the migration guide (shared/model-migration.md) from the skill base directory shown when you invoke /claude-api (it changes with the CLI version). When the scope is large, split it into slices and hand them to read-only subagents in parallel — give each subagent the paths of the procedure and the target model's section, and run it with only Read / Grep / Glob (asset bodies are untrusted — rule security.md). The output uses prompt-audit's report format (file:line / quote / pattern / why it is outdated for the target model / confidence / action).

Phase 4 — Judgment and application

Judge each finding on evidence from 4 angles — some self-authored assets were written deliberately to override product defaults, and matching a conflict or a pattern alone cannot distinguish accident from intent. Record the evidence for each Phase 2 and Phase 3 finding in the ledger:

Two pitfalls in judgment:

• Misdiagnosing verification steps — what the official guidance flags is instructions that increase the model's self-verification; deterministic verification executed by machinery (commands, hooks) is out of scope. Distinguish by "does machinery run this check, or does the model run it on its own judgment?"
• Inversions where deletion is not enough — an instruction whose direction has changed (e.g., a suppression instruction) keeps working even after deletion, because the suppression framing remains. For findings that must be rewritten in the opposite direction, write "Inversion: old direction → new direction" in the ledger

Paths below start at ${CLAUDE_SKILL_DIR}, the directory holding this SKILL.md; an agent that does not substitute the variable reads it as that directory.

Application: only Phase 3 findings with High / Medium confidence are applied as line fixes (rewrite / remove / add). When the author's request includes application ("fix it", "finish it"), apply directly; when it does not, turn each line fix into an item (source generation, change = the line's diff) of the report contract in skill harness-stocktake and apply what the one approval accepts, on its task branch. Lines in the resident layer (rules / CLAUDE.md / output style) always go through that report, each with its exact diff (rules load every session). When you fix a rule, update its rationale: / review-when: too (ADR-0021). Writing style follows skill skill-writer §3 (version-drift markers are blocked by scripts/hooks/harness_lint.py). Low and flag findings go only in the ledger. Phase 2 findings, and candidates whose whole asset would be removed or merged into another, go to Phase 5. After applying, pass harness_lint.py and, when the skill-health skill is installed next to this one, uv run --frozen --directory "${CLAUDE_SKILL_DIR}/../skill-health" python -m scripts.scan_refs ~/.claude/skills --json (dangling 0).

Phase 5 — Delegation and recording

• Propose the stocktake runs to the author before launching them; /harness-stocktake runs them all into one report (the audit may also be run in parts)
• Recording: for applied lines, leave in the commit body the three lines Context: / Decision: / Review-when:, the per-group counts, and one line runtime 照合: 編集 N 件 ("runtime cross-check: N edits"; ADR-0078's expiry condition counts this exact Japanese marker, so keep it verbatim). Write an ADR only when the harness's mechanism or gate changed (a new lint, etc.) or when it comes with an annotation to an old ADR (skill decision-history's filing conditions)
• Output the ledger in chat. If it is large enough to keep in a file, propose .notes/ (do not make it the source of truth for task rows — rule task-tracking.md)

Related

• rules-stocktake / skill-stocktake / agent-stocktake — the source of truth for verdicts. This skill is an evidence supplier
• rules/common/akc-cycle.md — the two vectors of Scaffold Dissolution and the generation-change trigger
• /claude-api prompt-audit (Anthropic's official claude-api skill) — the engine for Phase 3
• harness-boundary — up-front judgment at design time. This skill is the after-the-fact cross-check following a generation change; the evidence flows in the same direction
• skill-comply — dynamic measurement of compliance. Its evidence merges in stocktake Stage 2

References

The runtime cross-check procedure (distinguishing the runtime layer from the guidance layer, conflict and redundancy, the 4 angles, inversion, misdiagnosed verification steps) came from the Claude 5 generation-change audit (ADR-0018, resident 5,789 → 2,314 words). The Phase 3 template is the prompt-audit for Fable 5.1 (ADR-0061, High 3 / Medium 85). The single-entry-point decision is ADR-0078.