Blind evaluation protocol

Roles.

• OPERATOR (usually the agent): prepares material, runs arms, records judgments as scribe.
• JUDGE: a human, or a clean model instance with no access to preparation material. Whoever prepared or operated does not judge.
• PRINCIPAL: the human the result is for. Present at unblinding; never unblind alone.

Who judged is study data: human/model, creator/external, and the bias they declared before seeing anything.

Study layout

Create one directory per study and keep everything in it:

study-<name>/
  preregistration.md          frozen after Phase 1
  index.md                    every collection, with status
  collections/<id>/           the material judges see (clean, no badges)
  mapping/<id>.private.json   label -> arm; nobody opens until Phase 4
  judgments/                  append-only records written by the recorder
  analysis.md                 written in Phase 4 only

Phase 1 — Pre-registration (before any material exists)

Write preregistration.md, then it freezes: corrections are new files that reference the old one, never edits.

• The question, in conditional form that accepts any outcome ("under which conditions does X improve, worsen or not change Y") — never "prove X is better".
• Arms and exactly what each receives; parity of model, budget and conditions.
• Observed metadata (effective model id, versions) — a requested alias is never proof.
• Judgment dimensions + a closed flag list you write for this study. The list from the founding study — invented facts, unsupported grandiosity, irrelevant personalization, leakage, ignored constraints — is an example, not a standard.
• Proportional reading grid: what EACH possible outcome licenses you to conclude — including defeat and instability ("variance across replicas is the main finding" is a legitimate result).
• ALL post-unblinding analyses, fixed now. A stopping rule. After this the protocol is frozen: new analysis ideas become proposals for the NEXT study, never amendments to this one.

Phase 2 — Collection

• Run arms in real isolation — and audit invisible channels: CLIs may inject account identity (email, name) into the system prompt, so a "no context" arm is rarely zero-context. Record what each arm actually received, not what it was supposed to receive.
• Label variants A/B/C via a recorded shuffle; write it to mapping/<id>.private.json, which nobody opens until all judgments exist — including the operator.
• Material in collections/<id>/ must be CLEAN: no validation badges, no metadata, no cost figures, no hints. Mechanical validation stays separate, and its failures get audited after judgment (regexes have false positives: // in a comment and an SVG namespace are not external URLs).
• Do not leak condition details into the chat transcript; if it happens, declare the incident and regenerate clean material.
• Collection is not validation. Add every collection to index.md as collected_not_adjudicated — an invisible orphan collection is the seed of selection bias. Artifacts outside version control still exist: a clean working tree does not prove an experiment never ran — check the study directory before claiming "never executed".
• A replica born from an operational mistake enters as declared exploratory, never relabeled as part of the original design.

Phase 3 — Blind judgment

One judgment at a time; between judgments the judge does not re-read technical material.

Use the bundled recorder — it enforces a closed schema, refuses to overwrite, and prints a SHA-256:

node <plugin>/skills/blind-eval/scripts/record-judgment.mjs \
  --study study-<name> --file judgment.json

judgment.json carries: collectionId · judge + judgeBias · viewingOrder · finalChoice (a label, tie or none) · verbatim justification · flags per label · mappingGuess + guessConfidencePct (guess accuracy is its own outcome: it measures whether the treatment has a perceptible signature) · contamination (mandatory; "none" is also a declaration) · optional supersedes.

When transcribing dictation, record it as spoken and mark your own structuring and any transcription ambiguity as scribe notes in dimensionNotes. A missing field is asked of the judge, never filled in for them. Judgment without an instrument degrades into narrative.

Phase 4 — Unblinding and analysis

• Only after ALL judgments are recorded and hashed. Open the mapping files with the principal present.
• Run ONLY the pre-declared analyses. New findings are marked post-hoc and become hypotheses for the next study.
• Always report: consistency across replicas (variance is a result, not noise to hide); guess-versus-preference correlation (if preference tracked the perceived treatment rather than the real one, say so plainly — confident fabrication passes for personalization); recognized-but-not-preferred, if it occurs; the audit of mechanical failures.
• Record uncomfortable outcomes in the same register as comfortable ones, and state what each does NOT license you to conclude.

Known pitfalls (each one cost a real run)

• Bulk git add -A sweeps a collaborator's uncommitted work into your commit — stage by path.
• The judgment page carried per-label validation badges that were cross-referenceable with the chat transcript; it had to be regenerated clean.
• Five mechanical failures were regex false positives; an apparent "arm X fails more" correlation was coding style, not treatment.
• The baseline arm received the account email through the CLI — "no context" was actually "account context".
• Two complete collections sat invisible for three days because they lived outside version control and no one indexed them — and the operator asserted "never executed", twice.
• A test suite passing 18/18 proved nothing about three live data leaks, because every fixture used the one input style the parser handled. Write fixtures for the styles you did not plan.