refine-prompts

This is the refinement step of the improvement loop, step 5 of the loop's seven steps. It runs after a generation run has been reviewed and compute-pass-rate has produced a pass-rate report. A frontier model reads where the edits concentrated, proposes targeted changes to the prompts, skills, and agents that produced those edits, and tags each change as a generalized improvement or a domain customization. The output is a refinement changeset: the plan for the next round.

It is the only step where the expensive model authors changes, and it runs only against the small sample. The volume generation in step 7 uses the cost-effective model. See ../../docs/the-improvement-loop.md for the full loop and ../../docs/models-and-cost.md for model selection.

Where this sits

This skill is not one of the seven linear pipeline stages (intake through export). It is step 5, "Refine and classify," of the improvement loop, which wraps pipeline stages 2 through 6 on a sample. The loop is: seed, generate, review, measure, refine and classify (this skill), confirm on the held-out sample, scale. You run this skill after measure and before confirm.

... → [3] Review → [4] Measure (compute-pass-rate) → [5] Refine + classify (this skill) → [6] Confirm on held-out → [7] Scale

The discipline that makes the gain real: refine on the tuned sample, confirm on the held-out sample. This skill operates entirely on the tuned sample's results. It never reads or tunes against the held-out sample. Confirmation is a separate step: re-run generation on the held-out sample with the changeset applied, then run compute-pass-rate with held_out: true. This skill ends by handing off to that step; it does not validate its own proposals.

Inputs

• The pass-rate report for the round, produced by ../compute-pass-rate/SKILL.md: the per-record pass rate against threshold (the gate), the per-field pass rates (the diagnosis), the failure clusters by field, prompt, agent, modality, and confidence band, and the calibration check. The pass-rate report has no JSON schema of its own; its content is defined in the Artifacts section of ../../docs/the-improvement-loop.md.
• The review feedback that the report was computed from, conforming to ../../schemas/review-feedback.schema.json. Read the per-field verdicts directly when you need the specific edits behind a cluster: each verdict carries the field, the stable value_id, the status (accepted, edited, rejected), the stated_confidence, and the original and final values. The final minus original deltas are the raw evidence for what a prompt change must fix.
• The run manifest for the round, conforming to ../../schemas/run-manifest.schema.json. Its outcome object carries per_field_pass_rates, failure_clusters, and calibration_note; its prompt_versions and skill_versions tell you exactly which version of each asset produced the run, so a change targets the right file at the right version.
• The application profile, conforming to ../../schemas/application-profile.schema.json, for the institution's required_fields (these define the per-record gate) and review_threshold.

Verdicts are keyed to value_id, the stable per-value id assigned at generation. A cluster of edits on subjects.03 across several records points at one recurring subject-generation defect, and that attribution holds even though the array order changed between rounds. Read clusters through value_id; never reason about an array position such as the third subject.

What it does

1. Read the gate first. Take the per-record pass rate and the threshold from the report. The per-record rate is what must move; the per-field rates are how you move it.
2. Rank the failure clusters by per-record lift, not by per-field volume. The per-record gate is multiplicative: ten required fields each correct 95% of the time yield a record that needs no edit only about 60% of the time (0.95^10), and clearing a 90% per-record bar across ten fields takes roughly 99% per field. The cluster that moves the gate most is the highest-frequency failure in a required field, because each such field gates the whole record. A high-volume failure in a non-required field costs less against the gate. Prioritize the changes that lift the per-record number, and say so in each expected_effect.
3. Treat a calibration miss as a fixable defect. If the report shows a field whose high-confidence values are still being edited, that field's generation prompt is miscalibrated and counts as a refinement target rather than triage noise. Good calibration is a deliverable of the loop, because it is what makes confidence-gated review safe.
4. Trace each cluster to the asset that produced it using the run manifest's prompt_versions, skill_versions, and the provenance on the edited values. The target is a specific file: a field block in the generation prompt, a skill, or an agent.
5. Author one change per cluster. Each change names its exact target, its rationale, and its expected pass-rate effect, and is classified (below). Prompt and skill edits follow the version discipline already used in ../../prompts/system/metadata-generation.v1.md: save a new version with a changelog entry; do not silently overwrite. The prompt edit in the usage example below saves v2 with a changelog note for exactly this reason.
6. Classify each change as generalized or domain (below), splitting mixed changes.
7. Emit the refinement changeset (shape below).

Refinement targets

Changes land on three kinds of asset:

• Prompts. The generation prompt at ../../prompts/system/metadata-generation.v1.md, its field prompts, and its modality prompts. Most field-level failures fix here: a sharper date-normalization instruction, clearer subject-versus-genre guidance, tighter handling of OCR noise in a description.
• Skills. The pipeline-stage skills, when the defect is in how a stage runs rather than in what the model is told. A systematic authority-matching miss points at ../resolve-authorities/SKILL.md; a triage that surfaces the wrong fields points at ../review-triage/SKILL.md.
• Agents. orchestrator, field-generator, metadata-critic, consistency-auditor, and review-synthesizer. A failure the critic should have caught is a metadata-critic change; a collection-coherence failure across items is a consistency-auditor change; a verdict-aggregation defect is a review-synthesizer change.

Classifying each change

Apply the generalizable-versus-domain test to every change:

Would an institution describing entirely different works benefit from this change, without it imposing your collection's assumptions?

Yes makes it a generalized improvement: it returns to this shared repo as a contribution. A clearer date-normalization instruction, better OCR-noise handling, or a sharper subject-versus-genre distinction all pass the test. No makes it a domain customization: it encodes one collection's scope, house style, local vocabulary, or donor context, and it stays in the adjacent overrides/. A rule that "board means surfboard," a donor's naming convention, or a local subject scheme all stay local.

Mixed changes are split. Many real changes carry both a general mechanism and a local binding. Extract the generalizable mechanism and route it upstream; keep the local binding in overrides/. Encode the split in the changeset rather than forcing one tag. See the test and the PR process in ../../CONTRIBUTING.md.

Output: the refinement changeset

A round's changeset is the proposal the next round runs against. It has no JSON schema of its own; its content is defined in the Artifacts section of ../../docs/the-improvement-loop.md. Each entry carries:

• target: the exact asset and, for prompts and skills, the version transition. For example prompts/system/metadata-generation.v1.md → v2 (changelog entry), or agents/metadata-critic, or overrides/prompts/subjects.local.md.
• cluster: the failure cluster this change addresses, identified the way the report identifies it (field, prompt, agent, modality, or confidence band), with the value_id-keyed evidence behind it.
• rationale: why this change fixes that cluster, grounded in the specific edits the cataloguer made (the original-to-final deltas).
• expected_effect: the expected movement, stated against the per-record gate and prioritized by per-record lift, with the per-field rate it targets. For example: "raises date per-field accept from 0.82 toward 0.97; date is required, so this is the largest available per-record lift this round."
• classification: generalized, domain, or mixed. For mixed, name both the upstream mechanism and the local binding and where each lands.

Order the changeset by expected per-record lift, highest first, so the operator and the next run spend effort where the gate moves.

After emitting the changeset, hand off: apply it, re-run generation on the held-out sample, and run compute-pass-rate with held_out: true to confirm the gain holds on data the changes were not tuned against. If the held-out per-record rate clears the threshold with margin, the loop proceeds to scale; otherwise refresh the tuned sample and run another round.

Usage example

The refinement gesture in its smallest form: a house-style mismatch surfaced in review, fixed by editing the prompt and saving a new version.

Here are five of our existing finished records (paste or point to ./examples/house-style/).
Our cataloguers write shorter, more neutral descriptions than the current output,
one or two sentences, present tense, no interpretation of mood or intent. Update
DESCRIPTION_STYLE in prompts/system-prompt-v1.md to match these examples, save it as
v2 with a note in the changelog, re-run item tel_0001, and show me v1 vs v2 side by side.

In repo terms, prompts/system-prompt-v1.md is ../../prompts/system/metadata-generation.v1.md and the institution config the prompt named (config/wvu.yaml) is the adjacent workspace's application-profile.json; DESCRIPTION_STYLE is the {{DESCRIPTION_STYLE}} placeholder filled from that profile's description_style.

This is a single changeset entry:

• target: prompts/system/metadata-generation.v1.md → v2 (changelog entry), DESCRIPTION_STYLE block.
• cluster: description field, edited in a majority of reviewed records; edits consistently shorten and neutralize.
• rationale: the original-to-final deltas show the cataloguer cutting interpretive phrasing and length; tightening the style instruction to match the five exemplar records removes the recurring edit.
• expected_effect: raises description per-field accept toward the exemplars' register; if description is required for this institution, this is a direct per-record lift.
• classification: domain. Run the test: another institution describing entirely different works gains nothing from this institution's house style, and adopting it would impose one collection's voice. The change stays in overrides/, not upstream.

Contrast a change that passes the test the other way: if the same review showed date values failing because the prompt emitted free-text dates instead of EDTF, tightening the date-normalization instruction is generalized, because every institution benefits from cleaner date normalization without inheriting this collection's assumptions, and it contributes upstream.

A mixed example: the model keeps mis-tagging a recurring object as a generic format. The general mechanism (a sharper depicts-versus-is instruction in the prompt) is generalized and goes upstream; the specific binding ("in this collection, a board is a surfboard, resolved to the appropriate AAT term by the deterministic authority stage") is domain and stays in overrides/. The changeset entry is tagged mixed and records both destinations.