Eval Viability Check

Purpose

Find out whether an evaluation can measure anything before spending money and rounds of work on it. Most failed evals are not model failures. The answer was never in the input, the input gave the answer away, or the reported gain was below what a constant guess would score.

When to use

• Before building or reviving any eval, benchmark, labeled test set, or model-as-judge task.
• Again whenever the task, the input, or the label definition changes.
• When results look surprisingly good or stubbornly bad and nobody has checked the task itself.

When not to use

• Comparing models on an established public benchmark whose validity is not in question.
• Reporting results of a task that already passed these gates and has not changed since.
• As a reason to relax labeling rules to reach a sample size. See "Label scarcity" below.

The four gates, in order

Run them cheapest and most fatal first. A failed gate stops the work; do not mine more labels or try a better model to get past it. A gate you cannot run yet is untested: say what would settle it.

Gate 1: Determinacy. Is the label a function of the input?

Take 10 cases, sampled across classes. Show a domain expert only what the model will see, with every outcome field, timestamp, and later artifact removed. Ask for the label. If the expert cannot beat the majority-class rate of that sample, the task is malformed. No model recovers information the input does not contain. Record who checked, when, and the score out of 10.

Typical failure: predicting a document's eventual status from its first draft, when later edits decide the status.

Gate 2: Leakage. Does the input contain the label?

1. Search the input for the label word, its synonyms, and the field it was mined from.
2. Run a rule that cannot really do the task, such as matching the label word. If it scores well above the majority-class rate, the input leaks.

Typical failure: a perfect score because the status line was left inside the text being classified.

Gate 3: Baseline. What does a constant guess score?

Compute majority-class accuracy on the test split and print it beside every accuracy you report. Any score at or below it is a negative result: "up from 7% to 30%" is not progress when always guessing the common class scores 84%. This gate never fails on its own; it sets the number every later result is compared against.

Gate 4: Power. Can the test set detect a difference?

Set a minimum count per class, not just in total. Before running, state what the test set can detect. One standard error of an accuracy near p on n cases is about the square root of p(1 - p)/n. At 100 cases and 70% accuracy that is about 4.6 points, so a single score is uncertain by about 9 points either way. A difference between scores on separate samples is noisier, about 1.4 times that, so roughly 13 points here. For two models on the same test set, use a paired test such as McNemar's rather than eyeballing the gap.

This gate fails when the smallest detectable difference is larger than the improvement over the baseline that the decision needs.

Verdict

• Build only when gates 1, 2, and 4 pass. An untested gate blocks Build.
• Redesign when a failed gate has a known fix, such as stripping a leaked field or collecting more cases of a thin class.
• Stop when no obtainable input carries the label, or no obtainable test set can detect the difference the decision needs.

Label scarcity: synthesize inputs, never labels

Never invent ground truth. You may construct inputs and have the expert label them.

1. Name the missing cell, not the missing class: which combination of properties does real history fail to supply?
2. Choose variation dimensions that target the failure you expect.
3. Generate the combinations first, then write each case in a separate pass. Doing both at once produces repetitive, easy cases.
4. Have the expert approve the combinations before any case is written.
5. Drop cases the expert cannot judge as realistic, then have the expert label the rest.
6. Mark every such row as synthetic input with an expert label, keep it separate from mined rows, and report it separately.

Using expert time well

• Show batches of about 20 cases, input only, with no model predictions.
• Build failure categories from the expert's own notes; do not hand them a list.
• Wait for at least five annotations before generalizing a failure pattern.
• When proposing a pattern across unreviewed cases, over-flag. A dismissal costs a word; a missed pattern costs a round.
• Mix clustered samples with random ones, because clustering only finds structure it already assumes.
• Re-show a few early cases later. Changed answers reveal a shifting rubric.
• Stop when two batches in a row surface nothing new, and report what was never sampled.
• Collect answers in a numbered list; do not build a review interface.

Never treat label volume as evidence the task is sound.

Output shape

# Eval Viability Check: [Task]

## Task
- Decision the eval informs:
- Input the model sees:
- Label and where it comes from:
- Classes and counts:

## Gates
| Gate | Result | Evidence | Action |
|---|---|---|---|
| 1. Determinacy | pass / fail / untested | [expert score out of 10, or what would settle it] | |
| 2. Leakage | pass / fail / untested | [search and trivial-rule results] | |
| 3. Baseline | [majority-class accuracy] | [test split size and class balance] | |
| 4. Power | pass / fail / untested: [smallest detectable difference] | [counts per class, and the improvement the decision needs] | |

## Verdict
[Build, redesign, or stop, and the first gate that failed or is untested]

## Next steps
1. [Smallest action that unblocks the first failed or untested gate]