compute-pass-rate

This is the Measure step of the improvement loop (step 4 of seven in ../../docs/the-improvement-loop.md). It runs after the human review of a sample and turns those verdicts into the numbers that decide what happens next: whether the sample cleared the institution's per-record gate, where the failures cluster, and whether the model's stated confidence can be trusted to triage the next review.

It is not one of the seven pipeline stages. The loop wraps pipeline stages 2 through 6 on a sample and adds this skill plus refine-prompts on top. This skill consumes the output of stage 6 (review-triage) and feeds stage refine-prompts. The frontier model orchestrating the loop reads this skill's report to author the next round of prompt, skill, and agent changes.

The report answers one operator question first: did this sample clear the threshold, and if not, which field do I fix to move it.

When to run it

• After a cataloguer finishes reviewing a sample and you have a review-feedback file for it.
• On both the tuning runs and the held-out confirmation run. The computation is identical; the interpretation differs (see "Held-out runs" below).
• Once per run. Re-running after a fresh review of the same sample is fine and overwrites the outcome.

Inputs

Computing field clusters and the calibration check needs more than the verdicts alone. Read all three, joined on item_id and value_id:

1. Review feedback for the sample (../../schemas/review-feedback.schema.json). Supplies the per-field verdicts (status is accepted, edited, or rejected), the stated_confidence the generator marked, the reviewer-supplied record_passed, and the held_out flag. This carries field and stated_confidence, so field clusters and confidence-band clusters come straight from it.
2. The metadata records for the run (../../schemas/metadata-record.schema.json). Clustering by prompt, agent, and modality is not in the review-feedback schema. Those come from each record's provenance (prompt_version, agent, generated_by) and from the record itself (type and target_schema give modality). Join each verdict to its record on item_id, then to its value on value_id. Provenance is record-granular, so prompt and agent clusters are at record granularity unless a field's prompt is individually versioned; report them at the granularity the data supports and do not over-attribute.
3. The run manifest (../../schemas/run-manifest.schema.json). Identifies the run, its round, and the models per role. You update its outcome at the end.

The threshold and the set of required fields come from the institution's application profile (../../schemas/application-profile.schema.json): review_threshold.per_record_pass_rate and required_fields. Use the run config's threshold if one is set there for the run; otherwise the application profile is the source of truth. Only required fields count toward the per-record gate.

What it computes

1. Per-record pass rate (the gate)

The share of reviewed records that need no human modification. A record passes only when every required field is accepted with no edit. For a required array field (subjects, genre_form, depicted_entities), the field passes only when every generated value in it is accepted and the cataloguer added no missing value; one edited, rejected, or missing value fails the field, and a failed required field fails the record.

per_record_pass_rate = records that passed / records reviewed.

Treat the reviewer-supplied record_passed as authoritative, and independently recompute the gate from the field verdicts as a cross-check. Where the two disagree, flag the record in the report rather than silently picking one. The review-feedback status enum (accepted, edited, rejected) cannot represent "the cataloguer added a value the model missed," so the recomputed gate can read a record as passing when the reviewer marked it failed; that disagreement is usually the unrepresented missing-value case, which is exactly why record_passed wins. Note this limitation in the report.

Compare the rate to the threshold. cleared is true when the per-record rate meets or exceeds the threshold.

2. Per-field pass rates (the diagnosis)

For each field, the share of records in which that field passed, using the array rule above for the array fields. This is record-level: a field counts once per record. This is what populates per_field_pass_rates and what points at the prompt to fix, since per-record is gated multiplicatively across fields (ten fields each right 95% of the time clears a record only about 60% of the time; see ../../docs/the-improvement-loop.md).

Keep this distinct from the per-value accept rate (the share of individual fieldValues accepted), which is value-level and is what the calibration check uses. Report both where useful and never conflate them.

3. Failure clusters

Where the edits and rejections concentrate. Cluster the failing verdicts by:

• Field: which fields drove the misses. From review-feedback directly.
• Confidence band: how misses split across high, medium, low. From review-feedback directly.
• Prompt: which prompt_version produced the failing values. From the joined record provenance.
• Agent: which agent produced them. From the joined record provenance.
• Modality: which content type or target_schema they fall under. From the joined record.

A cluster points refinement at a cause: a field whose misses concentrate under one prompt_version is a prompt to fix; misses concentrated in one modality point at that modality's prompt set.

4. Confidence-calibration check

Calibration is tracked, not trusted. Compute it two ways:

• Overall: the miss rate (edited or rejected) per confidence band. Calibration is good when misses concentrate in the low and medium bands and high-confidence values are mostly accepted. This is the same check the benchmark uses: are the misses concentrated in the fields the model marked low or medium.
• Per field: any field whose high-confidence values are still being edited. That field's prompt is the problem, and refinement targets it. A field can have a fine overall accept rate and still be miscalibrated on its high-confidence values, which is the case that breaks confidence-gated review, so surface it explicitly.

Miscalibration is a fixable defect, not noise. Good calibration is a deliverable of the loop, because it is what makes confidence-gated review safe at scale.

Outputs

Two artifacts.

1. The pass-rate report

A human-readable markdown report, written alongside the run in the adjacent workspace (for example runs/<run_id>/pass-rate-report.md). There is no schema for it; it is for the operator and the orchestrating frontier model to read. Lead with the gate result, then the diagnosis. Suggested sections:

• Verdict: per-record pass rate, the threshold, cleared or not, and by what margin. State whether this is a tuning run or the held-out run.
• Per-field pass rates: a table, sorted worst-first, so the field to fix is at the top.
• Failure clusters: the clusters from section 3, each with a one-line read on the likely cause.
• Calibration: the overall band breakdown plus any field flagged for high-confidence edits.
• Disagreements and limitations: records where recomputed gate and record_passed disagree, and the missing-value limitation note.

2. The run manifest outcome update

Populate the outcome object on the run manifest (../../schemas/run-manifest.schema.json) with exactly:

• per_record_pass_rate: the gate rate, 0 to 1.
• per_field_pass_rates: an object mapping each field name to its per-field rate.
• threshold: the threshold this run was measured against.
• cleared: true when per_record_pass_rate met or exceeded threshold.
• calibration_note: a short prose summary of the calibration check, including any field with high-confidence edits.
• failure_clusters: an array of summary strings (the rich clusters in the report collapse to short strings here, for example "subjects: 9 of 14 misses, concentrated under system/metadata-generation@1.0.0").

Set finished_at if it is not already set. Leave the rest of the manifest unchanged.

Held-out runs

The same computation runs on the held-out confirmation sample (held_out is true in the review-feedback and on the run's dataset). The difference is what clearing means: clearing the threshold with margin on data the prompts were not tuned against is the signal to go to scale (loop step 6 in ../../docs/the-improvement-loop.md). A tuning run that clears is encouraging; a held-out run that clears with margin is the go-to-scale decision. Say which kind of run the report covers so no one reads a tuning number as a scale signal.

Usage example

Operator voice, in the style of the sample prompts in the initiative's 02-pilot-program/sample-user-prompts.md (none of those eight is a pass-rate prompt; this follows their pattern):

The cataloguer just finished reviewing the round-2 telegrams sample. Read the review
feedback in runs/round-2/review-feedback.json, join it to the records in output/telegrams/
and the run manifest at runs/round-2/manifest.json, and compute pass rate against our
review_threshold from the application profile. Give me the per-record rate versus the
threshold first, then per-field rates worst-first, the failure clusters by field and
prompt, and whether confidence is calibrated. Write the report to runs/round-2/
pass-rate-report.md and update the manifest outcome. Flag any field where high-confidence
values are still getting edited.

The handoff: the report goes to refine-prompts (loop step 5), where the frontier model reads the clusters and the calibration flags and proposes the next round of changes, each classified generalizable or domain.

What this skill does not do

• It does not propose fixes. That is refine-prompts. This skill measures; it does not author changes.
• It does not re-review records or change any verdict. The cataloguer's verdicts are the input and are authoritative.
• It does not decide to scale. It produces the number and the calibration read; the institution's policy and the held-out margin decide.
• It does not present the threshold as a Notch8 accuracy guarantee. The threshold is the institution's review-policy gate and the loop's optimization target.