assessment-review
Skill0 starsChanged 59 days agoMITView on GitHub
Reviewing assessment strategies, items and scoring for validity, fairness, correct keys, distractor quality and alignment. Used by CourseForge assessment reviewers at design, storyboard and QA stages.
In this repository19 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- accessibility-reviewSkill
- alignment-reviewSkill
- assessment-designSkill
- course-gap-analysisSkill
- editorial-humanizationSkill
- fact-checkSkill
- html-intake-reviewSkill
- instructional-designSkill
- instructional-graphicsSkill
- research-authoringSkill
- research-planningSkill
- research-reviewSkill
- scenario-designSkill
- storyboard-authoringSkill
- ui-reviewSkill
- ux-reviewSkill
- visual-directionSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 37
Check expected and actual scores for all-correct, all-wrong and mixed runs, and the passing threshold.
- Distinctive: line 54
Check item counts per objective, planned item types against levels, formative practice before graded assessment, the difficulty spread, the …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Assessment review
Review order
- Correctness. Is the key right according to the cited evidence? Is any distractor also defensible? This matters more than anything else.
- Alignment. Does the item measure its mapped objective at the intended level, and was the content taught earlier?
- Construction. Stem clarity, distractor quality, feedback.
- Bank-level balance. Coverage per objective, difficulty mix, redundancy, formative-graded overlap.
- Mechanics. Accessible operation, scoring behaviour (at QA).
Item defects to look for
- Wrong key; two defensible answers; a key that depends on a qualifier the stem omits.
- Stem ambiguity: undefined terms, missing scenario facts, "best" without criteria.
- Cueing: grammatical mismatch between stem and distractors; the longest or most hedged option is always right; repeated words from the stem only in the key; absolute words ("always", "never") only in distractors.
- Implausible distractors that no learner would choose, which turn the item into a two-option guess.
- "All of the above", "none of the above", "both A and C".
- Negative stems without emphasis; double negatives.
- Trivia: testing a number or name when the objective is a decision.
- Level mismatch: recall items for decision objectives.
- Untaught content: the item depends on a fact not in earlier screens.
- Formative items copied into the graded bank.
- Order dependence in graded items.
- Multiple response where every option is correct, or where the correct set is debatable.
- Sequencing where the order is conventional rather than required.
- Feedback that only says correct or incorrect; option feedback missing; rationale without citation; feedback contradicting the key.
Scoring review (QA)
- Compare the QA runner's correct-path and incorrect-path results per item with the model's keys.
- Check expected and actual scores for all-correct, all-wrong and mixed runs, and the passing threshold.
- For imported courses, inspect grading code for fuzzy text matching, keys stored separately from the data, and special cases for particular item IDs.
- Partial credit must be deliberate and explained to learners.
Severity guide
- Blocker: scoring broken so results are meaningless.
- Critical: wrong key; two defensible answers; graded item on untaught content; feedback teaching something false; an item that cannot be answered by keyboard.
- Major: level mismatch; weak distractors across an item; missing option feedback on graded items; fewer than two graded items for an objective.
- Minor: cueing that is subtle; uneven option length; feedback that could teach more.
Evidence in findings
Quote the stem, the options and the key; cite the claim or teaching block that proves the problem; for untaught content, show that no earlier screen covers it (name the closest one). Recommend the smallest change: replace one distractor with a named misconception, retarget the stem, move the key.
Strategy review (design stage)
Check item counts per objective, planned item types against levels, formative practice before graded assessment, the difficulty spread, the passing-score framing, and that no planned mechanic is inaccessible.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

