make-exam — build a written exam set

Scope: examinations answered on paper or on screen — selected-response, constructed-response, and matching items. A practical or performance examination is a different chain and belongs to the make-practical skill: its blueprint is an alignment matrix of tasks rather than a grid of topics by level, it has no answer key of distractor rationales, and its scoring instrument and its markers are the design rather than a late step.

This skill is an ordered checklist: each step opens when the one before it has produced its result. Each artifact is derived from the one before it: items that precede a blueprint end up weighted by what was easy to write. The folder shape and what the file numbers mean are in the assessment-layout skill; what makes an outcome assessable is in the learning-outcomes skill; what a defective item looks like is in the item-defects skill; the assessment principles behind each step are in ASSESSMENT-REFERENCE.md — read it before step 5.

Whenever a step puts more than one decision to the user, it asks them in the round shape the grill-point skill gives in its step 3, except that an approval of items is asked in the approval table the assessment-layout skill gives.

A run of this skill can outlast one session's context, so every step the plan marks 🛑 writes a handoff named for this skill, HANDOFF-<skill>.md, into the run's folder, once that folder exists, before it stops: the step to resume at and where inside it, the questions waiting on the user, what the run has learned that none of its files records, and the files to read first. A decision already written into a file is pointed to rather than copied, because two copies of one answer drift apart, and every answer a grill settled is already in this skill's grill log, GRILL-LOG-<skill>.md, where the run has one; a decision settled outside a grill and not yet written anywhere goes into the handoff, or it is lost with the session. The file is rewritten whole at each stop, and the stop ends with one line, in the user's language, saying that the session can be closed here and giving the line, naming the skill and the folder, that resumes the run in a new one. The last step deletes it.

Where the vault keeps a script for a count or a build this run needs, run it rather than writing another; a script written here is saved where the vault keeps its scripts, not in a session's scratch space, so the next run finds it. An item's text is changed where it lives, in the items file, rather than through a script that regenerates the file: such a script carries the paper's text a second time and is lost with the session.

Steps

1. Discover the project's conventions — read the host project's agent instructions file (AGENTS.md, or CLAUDE.md) and inspect an existing exam set: folder layout, file naming, available templates, and whether the project carries its own assessment reference. A project's own assessment rules override ASSESSMENT-REFERENCE.md and the item-defects skill; its templates override the document shapes below. Where the folder this run writes into already holds this skill's handoff, the run is resuming: read it before the run's other files, read first the files it lists, and take the step it names, checked against the files that exist, since a session can end between stops after its last handoff was written. Another skill's handoff in that folder belongs to that skill's run and is left as it is. Where that folder holds an unfinished run of this skill — one whose last step has not yet written into the folder — but no handoff, infer the step from the files that exist; a finished run's files are read as the steps below direct, not resumed. Where the step was inferred, or the handoff and the files disagree, say so in the line of what could not be found. Either way the plan marks the finished steps ✓. Then show the user the plan for this run — its steps by number with a 🛑 on each one that stops for the user, then one line of what this step found and one of what it could not find, labelled in the user's language — and wait for the user's go-ahead, or take the dispatch as the go-ahead when this run was sent to work without the user, as a sub-agent; name each later step by its number as it starts.
2. Open the exam set folder — one folder per exam, in the place and under the name the assessment-layout skill gives, unless the vault's own rule says otherwise.
3. Read the course definition, and check the outcomes before building on them — the syllabus or its digest supplies the learning outcomes, the grade weighting, and the teaching hours per topic; the teaching plan or timetable, where the project holds one, supplies the hours each outcome actually receives; the programme-level outcomes the course is mapped to supply what the course owes upward. Take these from the documents rather than estimating, and name any that are missing rather than inferring them. Then check each outcome against the learning-outcomes skill: the four ways a verb fails, and whether teaching hours stand behind it. Check the whole set, not one outcome at a time: the mapping in both directions is a property of the set and fails silently when each outcome is read alone. Where an outcome fails any of those checks, stop and put it to the user. A compound outcome produces two exams stapled together and reads that way; an unobservable one has no defensible item; an outcome that reaches no programme outcome is unassessed however the marks total. Then ask whose outcome it is: where the user owns it, the failure is fixed before the blueprint is drawn, because a paper built on a broken outcome cannot be repaired later. Where the outcome belongs to an approved syllabus or to another person, the user cannot fix it and the run is not blocked by it — report what it would have to become, get the user's acknowledgement to proceed on it as it stands, and record the limitation in the blueprint as an open item rather than a settled one.
4. Research only what is missing — if the exam needs subject knowledge neither the course documents nor the assessment reference supply, gather it and record it in 00-research.md with sources.
5. Settle and write the blueprint (01-blueprint.md) — This step is a grill point: run the grill-point skill, and build nothing on a decision the grill has not settled. Its questions are the cells — counts, marks, formats, and the level each reaches — and the style budget the items will be written to, proposed from the papers this subject or the nearest one has actually set. Its rules hold across the vault and are recorded there: whether options carry explanations, whether stimuli are material or narrative, the share of items in stimulus sets. Its lengths differ by subject and are recorded with it: the median stem and option length in characters excluding spaces. Where no paper has been set, ask the user for one they consider well written, or carry the proposal as an open item. Write it in the shape the assessment-layout skill gives. Every cell maps to a learning outcome and carries three things: its item count, its marks, and the item format the cell's level can actually be reached by — the format is a column of the table or an entry in every cell, never one declaration covering the whole paper, because the check it exists for is made cell by cell — a format left to be decided at writing time is decided by what is quickest to write, and a Create-level cell filled with selected-response items leaves its outcome unassessed while the totals still reconcile. Marks per topic track that topic's teaching hours, and totals reconcile with the course's grade weighting.
6. Sketch the items, and let the user choose (07-sketches.md) — one row per intended item: the cell, the stem in draft, the correct answer as an idea rather than as option wording, why the point is worth asking, and the session that taught it, from the teaching plan where the project holds one. A stimulus set is sketched as one group: the stimulus itself, then one row per angle it is asked from. The correct answer stays an idea because option wording comes after the distractors, in the order the reference gives. Sketch more rows than the cells need, then stop: the user marks each row keep, drop, or change, and the file is the approval table for it. Dropped rows are deleted and changed rows rewritten in place, so the file ends as the list of items to write.
7. Write one cell as a calibration batch, and stop — three to five items from one cell into 02-items.md, each written stem first, then the distractors, then the correct option, and each measured as soon as it is written: its row goes at once into the measurement table, the next free authoring file (default 06-measurements.md) — the style budget, the longest option, and how anything flagged was repaired — so repairs made while writing stay on the record. Keep the running share of items whose key is the longest option against the band the item-defects skill gives, so a paper drifting out of it is caught while it is being written; on fewer than twenty items the share is reported but not acted on, since one item moves it too far to mean anything. Show the user the items beside their rows, in the approval table, and record in the measurement table that the batch was shown and what the user changed. What they say about wording, difficulty, and distractors applies to every item written after, so it is settled on five items rather than discovered across sixty.
8. Write the remaining items (02-items.md) — each item tagged with its learning outcome, its cognitive level, and the cell it fills, in the format that cell specifies, and each measured and recorded as it is written, the same way as the batch, before the next is started. Stimulus sets take the block form the assessment-layout skill gives. The file holds the current text of each item and nothing else, as that skill sets out for items not yet sat. Every item belongs to a cell, and the reconciliation happens when the paper is assembled, not here.
9. Measure the whole set, and complete the table — before the answer key exists, complete the measurement table the item-defects skill describes, started at step 7, over the final text of every item. Count rather than estimate — the length of every option in characters excluding spaces, never in words, since a word count measures nothing in a language that does not put spaces between words and a space is a phrase break rather than a character in one that does: how many options carry a parenthesis, a hedge, a negation or an absolute term, which words the stem shares with the key alone, which option is the longest and whether it is the key, how each flagged item was repaired, and what sits in the position the key points at — and, over the whole set, the share of items whose key is the longest option. Where a check answers yes or no rather than producing a number, the cell holds the matched span — the string, which option carries it, in how many options it appears — never a tick or the words "none found". One item, one row. Then read the counts against COMMON-DEFECTS.md and against the file for each item's own format, and rewrite what they find — re-measuring anything rewritten, because repairs introduce their own flaws. Report a count for every group including the groups that found nothing: a step that produces no table did not run, and a group with no count was not looked at.
10. Write the answer key (04-answer-key.md) — every item, with the rationale for each distractor on selected-response items.
11. Write the rubric (05-rubric.md) — for every constructed-response item, with observable criteria whose weights sum to the item's marks, every score traceable to a described level or a stated rule; and the examinee's copy shows how each criterion's marks are reached.
12. Assemble the paper (03-paper.md where the project separates the examinee-facing paper from the item bank) — propose which items the exam takes, and show the user the proposal beside the measurements of exactly those items, in the approval table — each row's cell, longest option, and key position, with the paper's longest-option share and its key positions under it; the user decides the selection, and where its longest-option share falls outside the band, the fix the item-defects skill gives is applied to the selected items before the paper is final. Then reconcile the blueprint against exactly those: marks per topic, the level distribution, and the format of each cell are recomputed for the selected set. A blueprint balanced over a larger pool says nothing about the subset drawn from it. Kept items the paper does not take move to the bank, with their keys and distractor rationales. A printed paper sets out its options as the assessment-layout skill gives. Where the paper leaves as a Word, PowerPoint, or Excel file carrying Thai text, it is assembled once that file passes the rendered check in the thai-office-files skill and its options pass the one the assessment-layout skill gives, or that check is recorded as not run the way the skill describes.
13. Verify before finishing, and record what has not happened yet — run the project's check script where one exists, then confirm: every learning outcome passed the step 3 checks, or was raised with the user and recorded as an open item; the blueprint carries the line its grill wrote, and every cell below its outcome's level names its decision record; the assembled paper's longest-option share is inside the band; every item written comes from a row of the sketches file, and the measurement table records the calibration batch as shown before the rest were written; the assembled paper's marks total to the course's grade weighting; every item traces to a learning outcome and to a cell whose format it matches; the measurement table is in the exam set and every group reports a count including the empty ones; every item appears in the answer key; and every constructed-response item has rubric criteria. Then close the set by writing into it that this skill produces an exam set and not a cleared one: the binding review runs on the assembled paper, at the stage the assessment-layout skill numbers 1x, and it is a separate pass by design, because a check run by whoever wrote the items, in the session that wrote them, is the weakest form of the check. Where the agent can start a sub-agent with a fresh context, send it the review skill the project uses — critique-mcq for selected-response papers — on the assembled paper now; otherwise, and until that pass has reported, record it as an unticked item in the exam set's own closing check, naming that skill. A review passed on only as a sentence in the session is a step that leaves no evidence, which is the failure every other check in this skill is written to avoid. Last, delete this skill's handoff and its grill log, GRILL-LOG-<skill>.md, from the run's folder, once anything still open in them is written into the run's own files: a handoff left behind makes the next run on this folder resume one that is finished, and a grill log left behind is a second copy of answers that drifts from the artifact, against which the next run would check its drafts.