critique-mcq — critique multiple-choice items

This skill is an ordered checklist: each step opens when the one before it has produced its result. Every finding rests on a measurement that already exists. It produces a critique and proposed rewrites; it does not change anything until the author approves. The folder shape and what the file numbers mean are in the assessment-layout skill; the defect taxonomy is in the item-defects skill, and the measurements and the rules that keep a critique from over-reporting are in MCQ-CRITIQUE-REFERENCE.md — read both before step 5.

Whenever a step puts more than one decision to the user, it asks them in the round shape the grill-point skill gives in its step 3, except that an approval of items is asked in the approval table the assessment-layout skill gives.

A run of this skill can outlast one session's context, so every step the plan marks 🛑 writes a handoff named for this skill, HANDOFF-<skill>.md, into the run's folder, once that folder exists, before it stops: the step to resume at and where inside it, the questions waiting on the user, what the run has learned that none of its files records, and the files to read first. A decision already written into a file is pointed to rather than copied, because two copies of one answer drift apart, and every answer a grill settled is already in this skill's grill log, GRILL-LOG-<skill>.md, where the run has one; a decision settled outside a grill and not yet written anywhere goes into the handoff, or it is lost with the session. The file is rewritten whole at each stop, and the stop ends with one line, in the user's language, saying that the session can be closed here and giving the line, naming the skill and the folder, that resumes the run in a new one. The last step deletes it.

Scope: selected-response items with a single best answer. Constructed-response, practical, and matching items are listed as not reviewed rather than judged against this taxonomy.

The binding pass is the one run on the assembled paper — the items as they will be printed, after any selection and editing. Running on an item pool beforehand is useful for weeding, but a pool that passed says nothing about the paper drawn from it: selecting breaks the blueprint, and editing voids every earlier check.

Findings the user brings — their own, or another reviewer's — join a full pass: steps 1–8 run as they do for any paper, and the findings are triaged at step 9 beside the passes' own. Before step 9, each is located with the user to the span it is about, since step 9 drops a finding that cannot quote one. A finding that asks for a rule over every item, rather than a change to one, is a decision for the vault: put it to the user through the grill-point skill before it is applied across the paper. A pass over a paper whose exam set already holds a critique writes its report as 12-critique.md, or the next free 1x number after it, so the earlier report and the independent pass's number both stand; beside the paper, it is <paper filename>-critique-2.md and onward. Its step 12 leaves an independent pass already recorded as one unticked item, rather than adding a second.

Where the vault keeps a script for a count or a build this run needs, run it rather than writing another; a script written here is saved where the vault keeps its scripts, not in a session's scratch space, so the next run finds it. An item's text is changed where it lives, in the items file, rather than through a script that regenerates the file: such a script carries the paper's text a second time and is lost with the session.

Steps

1. Discover the project's conventions — read the host project's agent instructions file (AGENTS.md, or CLAUDE.md) and inspect the exam folder for: the assessment layout, the language the vault writes its documents in, the institution's cognitive-level scale and its critique paperwork, and the item rules the project or the institution fixes, such as the required number of options. A project's own assessment rules override MCQ-CRITIQUE-REFERENCE.md and the item-defects skill; an item is not flawed for obeying a rule the project imposes; and the report is written in the vault's language, defaulting to the language of the paper. The folder this run writes into is the exam set step 10 names; find it here, so a handoff left in it is seen. Where the folder this run writes into already holds this skill's handoff, the run is resuming: read it before the run's other files, read first the files it lists, and take the step it names, checked against the files that exist, since a session can end between stops after its last handoff was written. Another skill's handoff in that folder belongs to that skill's run and is left as it is. Where that folder holds an unfinished run of this skill — one whose last step has not yet written into the folder — but no handoff, infer the step from the files that exist; a finished run's files are read as the steps below direct, not resumed. Where the step was inferred, or the handoff and the files disagree, say so in the line of what could not be found. Either way the plan marks the finished steps ✓. Then show the user the plan for this run — its steps by number with a 🛑 on each one that stops for the user, then one line of what this step found and one of what it could not find, labelled in the user's language — and wait for the user's go-ahead, or take the dispatch as the go-ahead when this run was sent to work without the user, as a sub-agent; name each later step by its number as it starts.
2. Locate the paper and its anchors — the assembled paper in the form it will be given to examinees, the answer key as it stands now, the blueprint, the learning outcomes, the item pool the paper was drawn from, and the course sources the content can be checked against. Name every one of these that is missing before going further: without a blueprint and an outcome list, step 8 cannot run and the report must say so rather than infer the outcomes an item was meant to serve; without course sources, content correctness goes unverified and the report says that too.
3. Read the rendered paper once and establish provenance — read the printed or exported form, not only the text pulled out of a word-processor file: equations, figures, and marks carried by formatting live outside the text stream, and an extraction that drops them makes an intact item look broken. An item whose stem or options come out incomplete is a finding about the reading tool until the rendered page says otherwise — never a finding about the item. Match each item back to the pool and mark it used unchanged, edited after selection, or written outside the set. Items in the last two groups carry no valid earlier check, whatever was run on the pool. Where the pool is a contributed pool (08-contributed.md), no item carries a valid earlier check however it matches, since the pool was never held to the vault's standard, and an item the vault wrote to fill a gap is written outside the set. Nothing is recorded as a finding in this pass.
4. Measure every item — build the measurement table described in the reference: option lengths, terms recurring across options, words shared between stem and key only, absolute and vague terms, negation, numeric ranges, exhaustiveness, whether the lead-in is closed, and what sits in the position the key points at. Count rather than estimate, in characters where the language does not separate words; a script or a tokenizer is a speed-up here, not a requirement. One item, one row, and where a check answers yes or no the cell holds the matched span rather than a verdict. Build the table in this pass even where the exam set already carries a measurement file: a table inherited from the run that wrote the items carries that run's errors forward, and a pass that certifies a wrong value is worse than one that never ran. Keep the table — it is the evidence the findings are quoted from, and it is published with the report, so a pass that did not run is visible instead of silently absent.
5. Pass 1 — cues that reward test-wiseness — work through Group A of MCQ-DEFECTS.md in the item-defects skill, reading each check off the measurement table rather than off an impression of the item.
6. Pass 2 — irrelevant difficulty — work through Group B of MCQ-DEFECTS.md: what makes the item hard for reasons unrelated to what it measures.
7. Pass 3 — congruence, level, and correctness — work through Group C of COMMON-DEFECTS.md against the anchors from step 2, starting with whether the keyed answer survives a check against the course's own sources. Record the level on the institution's scale.
8. Pass 4 — the paper as a whole — work through Group D of COMMON-DEFECTS.md, beginning with the key-to-position check on every item, then reconciling the blueprint against the items actually selected. These defects appear only after selection and reordering, so they are checked last and never skipped because the items passed individually.
9. Triage the findings — assign Block, Revise, or Note; drop any finding that cannot quote the span it is about; merge findings a single rewrite resolves. A paper that comes back with more findings per item than it has options is over-reported, and is re-triaged rather than delivered.
10. Write the critique — into the exam set's review-stage file (default 10-critique.md), in the shape the reference's Reporting section specifies. The leading number is the lifecycle stage the vault assigns to review, rather than the next unused number in the folder — a review file numbered to follow the last authoring artifact claims to be the next step of a chain it is not part of. Find that exam set by course, year, and exam in the vault rather than only in the folder the paper sits in: the paper and its exam set routinely live in different trees, and falling back to the paper's own folder scatters the review stage away from the artifacts it belongs with. Only where no exam set exists anywhere does the report go beside the paper as <paper filename>-critique.md.
11. Stop for approval, then apply what is approved — change nothing before the author decides, item by item or in bulk, in the approval table: what each row proposes is the finding with its severity and the rewrite, and what it was measured on is the item's row of the measurement table. Re-measure every rewritten item against step 4, because repairs introduce their own flaws, and re-run the key-to-position check after any option is reordered. Never change a keyed answer silently: a suspected miskey is reported for the author to confirm.
12. Follow through and verify — every accepted change propagates to the answer key, the blueprint, the rubric, and the paper, whose options are set out again as the assessment-layout skill gives, since a changed option can change the lines it needs. Run the project's check script where one exists, then confirm: the measurement table is in the report, every pass reports a count including the empty ones, every finding carries quoted evidence and a severity, every rewritten item was re-measured, every key matches the option it points at, and the blueprint reconciles against the items the paper actually contains. Then close the review by writing into the exam set an unticked item for the independent pass — the review stage's second file, named in the assessment-layout skill — because a critique run by whoever measured the items, in the session that measured them, is the weakest form of the check, and the pass that finds a wrong measurement is the one that did not inherit it. Last, delete this skill's handoff and its grill log, GRILL-LOG-<skill>.md, from the run's folder, once anything still open in them is written into the run's own files: a handoff left behind makes the next run on this folder resume one that is finished, and a grill log left behind is a second copy of answers that drifts from the artifact, against which the next run would check its drafts.