Experiment Comparison Audit

Make each comparison understandable in terms of what actually changes. Preserve useful experiments while matching their interpretation to their design. A difference between variants is not automatically a flaw; it determines which question the comparison answers.

Recover the operation

Locate the relevant variant definitions, source/configuration records, and evaluation protocol. Resolve names against these records rather than trusting labels such as “fixed,” “uniform,” “no feedback,” or “without recovery.” If a historical run matters, distinguish its recorded configuration from today's implementation.

Define a comparator in plain operational language before giving its acronym or appendix pointer. For example: “The baseline assigns one of two fixed weights using a confidence threshold.” Put pointers on coefficients and detailed procedures; repeating a rule's name does not define it.

Two fixed baselines can use different rules. A preset parameterization of the proposed formula differs from a separate fixed transformation. Conversely, an identical formula does not establish identical inputs, eligible records, coefficients, or selection procedures.

Compare only consequential axes

Use the smallest matrix needed to resolve the scientific claim:

Match distinctions to the claim. Do not turn a narrow definition check into a compulsory exhaustive audit. If an axis is unknown and consequential, mark it unknown rather than assuming equality.

Equal starting pools do not establish equal eligible training data. Equal nominal updates per fit do not establish equal total search cost or effective weighted exposure. Candidate counts establish how many fits were performed, not an exact ratio of wall-clock time or energy unless those quantities were measured. Retain genuine controls, such as a shared architecture, while correcting an overbroad “identical budget” statement.

Read ablations literally

Specify the intervention and what remains. Removing a source of examples differs from removing a label on retained examples. Disabling a feature differs from changing coefficients while retaining it. A subset labeled retrospectively is not an independently run intervention.

State whether an ablation reuses full-system settings or retunes the reduced system. A removal comparison at one chosen setting answers a conditional question; it need not be discarded, but it does not automatically estimate the best reduced system or isolate selection's contribution. Combined changes do not identify each change's causal share.

Prose when drafting or revising

When drafting or revising prose, including audit explanations, read prose editing rules. Preserve scientific meaning, necessary reasoning and valid technical terms. Keep sound passages, do not add unreported properties, and limit edits to the requested scope. The local reference covers concept accuracy, clear sentences, repetition and punctuation, including the required exceptions.

Deliver the interpretation

Return the actual comparison, consequential mismatches with the draft, and the conclusion the design supports. If wording is requested, give the smallest clear replacement and preserve supported controls and results. Keep source-removal, label-removal, and selection comparisons separately named.

State which relevant records were inspected and what remains unresolved. Do not fabricate missing baseline outcomes or describe every unmatched detail as unfairness. Stop once the variants and claim-relevant differences are clear enough to interpret the reported comparison; propose additional experiments only when the user requests design advice or a specific claim cannot otherwise be assessed.

Read the worked example for a compact comparator and budget audit.