analyze-results — analyse a paper after it has been sat

This skill is an ordered checklist: each step opens when the one before it has produced its result. It covers the stage the assessment-layout skill numbers 2x — the one that begins when the marking is done. Building the paper is make-exam; reviewing it before it is used is critique-mcq.

Whenever a step puts more than one decision to the user, it asks them in the round shape the grill-point skill gives in its step 3.

Scope: papers scored right or wrong per item. A practical examination scored on a rating scale needs marker agreement rather than item statistics, and is out of scope here.

A statistic is a reason to go back and read the item, never a verdict on its own. Every unusual number gets the same question first — is there a flaw in how this item is written — before the answer becomes "the students did not know this". The flaw taxonomy is in the item-defects skill; the formulas, thresholds, and the diagnostic table are in ITEM-STATISTICS-REFERENCE.md, which is read before step 5.

Steps

1. Discover the project's conventions — read the host project's agent instructions file (AGENTS.md, or CLAUDE.md) and inspect an existing exam set for: the assessment layout, where the item bank lives, the language the vault writes its documents in, this institution's own thresholds and reporting forms, and its rule on personal data. A project's own rules override ITEM-STATISTICS-REFERENCE.md, and the report is written in the vault's language, defaulting to the language of the paper. Then show the user the plan for this run — its steps by number with a 🛑 on each one that stops for the user, then one line of what this step found and one of what it could not find, labelled in the user's language — and wait for the user's go-ahead, or take the dispatch as the go-ahead when this run was sent to work without the user, as a sub-agent; name each later step by its number as it starts.
2. Pass the personal-data gate before opening anything — the scoring data names people, and the vault does not take names, identifiers, or individual scores. Read what is needed to compute, write only what is about items. Name to the user, before going further, every file being opened that carries per-person data and what will be derived from it; keep identifiers out of every intermediate file as well as the report, since a working file in the exam set is in the vault; and where the institution's export cannot be read cleanly, that is a finding about the file's encoding rather than about the data — say so instead of computing on characters that came out wrong.
3. Locate the paper and its anchors — the paper as it was actually sat, the answer key as marked, the blueprint, and the item bank. Name every one that is missing before going further: without the key as marked, nothing here can be computed; without the blueprint, step 10 cannot say whether a cut item leaves its cell empty. Establish whether the paper was scored against a standard everyone was expected to reach or against the group, because that decision reverses how step 5 reads.
4. Build the response matrix — one row per examinee, one column per item, right or wrong, plus each examinee's total and each item's chosen option. Count how many sat it. Everything below is read off this table, and it is the only place per-person data appears; it is not written into the exam set.
5. Compute difficulty for every item — the proportion answering correctly, against the reference's bands. One item, one row, and every row carries the number rather than a verdict.
6. Compute discrimination for every item — whether the examinees who scored well overall answered this item better than those who did not. Report it beside difficulty, never alone: the reference's table of what discrimination is arithmetically capable of at each difficulty is what stops a sound item being cut for a low value it could never have exceeded.
7. Analyse the options on flagged items only — for every item the two steps above flagged, how the choices were distributed and how each option relates to the total score. Read the result off the reference's diagnostic table. A distractor chosen by almost nobody made the item easier than it looks; a distractor the strongest examinees chose is a content problem until proven otherwise.
8. Compute reliability and the standard error for the paper — with the reference's caution attached, since a high coefficient is not evidence the paper measures one thing, and it rises with length on its own. Where the marks decide a pass, convert the error into a band around the cut and say who falls inside it.
9. Interpret against the number who sat it — never skip this step — with a class-sized cohort the thresholds above swing far enough to reverse a decision, and the reference gives the interval per cohort size. State that interval beside every judgement, so a value near a boundary is reported as near a boundary rather than as the category it happens to have landed in.
10. Settle each item's decision, and say what it costs — This step is a grill point: run the grill-point skill, and build nothing on a decision the grill has not settled. Dropping an item from scoring changes every examinee's mark, so each item is a question with a recommendation drawn from the statistics above — keep, revise for next time, or drop it from scoring. The decisions, and the grill's line, go into the item-analysis file the assessment-layout skill names for the stage after administration. Anything dropped is checked against the blueprint: an item removed after the fact empties a cell that was balanced when the paper was set, and the marks move somewhere. A key that turns out to be wrong is reported for the author to confirm, never changed silently, and the same rule is applied to every examinee rather than case by case once names are visible.
11. File the statistics with the item version, not the item — into the item bank the assessment-layout skill names. The values belong to the exact wording and option order that was sat; carried across an edit they describe something nobody answered.
12. Verify before finishing — run the project's check script where one exists, then confirm: the item decisions carry the line their grill wrote; no name, identifier, or individual score appears in anything written to the vault; every item has difficulty and discrimination, and every flagged item has its option breakdown; the cohort size and its interval are stated; every dropped item has been reconciled against the blueprint; every statistic is filed against an item version; and each number carries the count it was computed from, so a figure nobody could reproduce is visible rather than merely believed. Last, delete this skill's grill log, GRILL-LOG-<skill>.md, from the run's folder: a grill log left behind is a second copy of answers that drifts from the artifact, against which the next run would check its drafts.