Consensus of independent audits

Goal: a private, self-contained rendered Quarto report (consensus.html) that answers, for one paper: which issues raised by the audits are confirmed, how severe each is, and whether any changes a major conclusion. Settle every disagreement with evidence, never by counting audits or by trusting the more confident report. Then judge the relative accuracy and usefulness of the included audits from that evidence.

Work through the stages in order. Start from templates/consensus_template.qmd.

Intake

1. Your working directory is a consensus folder runs/<paper_id>/consensus/<consensus_id>/ created by consensus.sh. Read inputs.json: it lists the completed runs to reconcile (runs[].run_id) and any runs skipped because their audit was unfinished. Each source run is at ../../<run_id>/. Skipped runs are not inputs; do not read them. Never read other folders under consensus/: earlier consensus runs hold the model mapping and earlier verdicts, and would break blinding and anchor your adjudication.
2. Refer to audits by run ID only. Never read registry.jsonl, never guess which model wrote an audit, and never write a model name into any file. Model names in the source runs have been replaced with run IDs or [redacted-model]. The operator attaches the run-ID → model mapping after you finish.
3. Treat the source runs as read-only. You may read anything in them: paper/, data/, manifest.json, README.md, and each run's audit/ (report, code, caches, results). To re-run a source audit's code, copy it into your folder first and run the copy.
4. Write everything under the consensus folder: consensus.qmd, R/ for adjudication code, results/ for its outputs, additional-evidence/ for retrieved material.

Stage 1 — Extract

For each run, read audit/audit.qmd (and audit.html where inline values matter) and list:

• its claims with their verdicts;
• its findings (ID, classification and rationale when present, status line, severity, affected claim, key numbers);
• what it checked and found clean: rows of its robustness table marked as not changing the verdict, its Clean checks and Verification performed appendices, and robustness output it reports.

The third list decides whether silence means "checked, no problem" or "never looked".

Stage 2 — Cross-walk

Match claims across runs by meaning, not by ID. The runs number claims and findings independently, and one run's claim may split or merge another's. Produce one consensus claim list. Then match every finding from every run to a consensus item. Two findings match when they identify the same problem in the same analysis, even when they reach it by different routes (for example estimator arithmetic versus simulation). Every finding ID from every run appears in the cross-walk table; nothing is dropped silently.

Stage 3 — Classify

Each consensus finding gets exactly one status:

• Converged: two or more audits raised it independently and none tested the same point and found it clean.
• Disputed: at least one audit raised it, and any audit explicitly tested the same point and found no problem. This holds however many audits raised it: one explicit clean result makes it disputed.
• Single-audit, untested: one audit raised it and no other audit looked.

Audits that never looked are recorded in the cross-walk but do not change the status. Never classify by majority, and never by which report sounds more certain.

Stage 4 — Adjudicate

• Disputed and untested findings: run the check yourself against the paper's data and code, in R/ with output in results/. Find out why the audits differ (different baseline, different replication unit, a code error in one run, a different reading of the paper). Give an outcome: confirmed, partly confirmed, or refuted. If an existing output in a source run already settles the question, cite that exact output instead of recomputing it, and say why it suffices.
• Converged findings: apply the multiplicity rule below even for converged findings. For other converged findings, accept them, but spot-check any whose audits disagree on magnitude or severity, and report which magnitude holds.
• Claim verdicts that differ between audits: decide the consensus verdict from the adjudicated findings and state why each audit's verdict was or was not right.
• Cache heavy computations (e.g. results/*.rds) and read them in the report. The operator re-renders the report after you finish, and every number must come out the same.

Scope: adjudicate what the audits raised and the verdicts they disagree on. Do not hunt for new issues. A new problem you notice while adjudicating goes in Noticed during adjudication, unadjudicated, one line each, but only when it could change a verdict. Leave out anything smaller. Additional evidence follows the reproducibility-audit rules: save it under additional-evidence/ with a provenance row, with no large downloads, no paid jobs, and no contacting authors.

Justify the multiple-comparison family

Before alleging missing multiple-comparison correction, establish family membership from the scientific question, the authors' inferential claims, and the analysis plan, preregistration, methods, and code where available. List the included tests and explain why they offer parallel opportunities to support the same inference. Sharing the same figure, table, dataset, or paper does not establish a family; distinct scientific questions can legitimately be tested separately.

Focus correction checks primarily on large collections of essentially identical tests performed in parallel, such as correlations across a set of variables or tests across many brain voxels. This is a priority, not a minimum family size: smaller families also need justification from their shared inferential purpose. Check any existing correction, hierarchical testing, or omnibus procedure before alleging an omission.

Do not apply Holm automatically. Justify both the family and the error criterion (e.g. family-wise error versus false discovery rate), and assess whether the authors' method addresses their claim. Report all tests in the justified family, including null results and effects in the opposite direction.

If family membership is uncertain, describe that uncertainty. An alternative grouping may be a sensitivity analysis, but do not downgrade a claim solely because it fails correction under a debatable grouping. A justified correction that changes significance must be interpreted alongside effect size, uncertainty, and the actual claim; it does not establish that the effect is absent.

Published critiques and power

Two topics are an exception to the no-new-issues scope above, because the audits are required to cover them and an omission there is a gap in the consensus, not a new issue.

Published critiques. Build the union of the audits' critique ledgers, merging the same point from different audits. If no audit searched, or the searches differ in what they found, run the critique search yourself following the reproducibility-audit Published critiques rules (commentaries, letters, replies, failed replications and reanalyses, PubPeer, methodological papers using the paper as an example), save what you find under additional-evidence/, and record the search. Assess every critique point that no audit assessed, and adjudicate points the audits assessed differently, like any other disputed finding. A supported point that affects a claim enters the cross-walk as a consensus finding.

Power and precision. For every main claim, confirm that the effective sample size, the interval for the deciding estimate, and the smallest effect the design could detect with 80% power at the paper's alpha are recorded. Where the audits omitted it or disagree, compute it yourself (a sensitivity power analysis, never observed power) in R/. When the precision is inadequate for what a main claim asserts (its magnitude, generalization, comparison between effects, or the absence of an effect), discuss it in that claim's Evidence and If weakened and let it set the verdict, even when the source audits treated it as minor or as a disclosed limitation. Insufficient power that limits several central claims is a paper-wide problem. A small sample alone is not a Defect; classify the inference it supports.

Finding classification

Classify every retained finding as Defect, Contested choice, or Undetermined, separate from evidence status and severity. Give a brief classification rationale beside the finding, including minor findings.

• Defect: something wrong under any defensible standard applicable to the stated claim and design. Identify the violated arithmetic, logical, factual, or methodological requirement and the evidence establishing the violation. Consider the strongest defensible rationale for the authors' approach before using this label; preference for another method is insufficient.
• Contested choice: researchers can reasonably disagree because more than one approach is defensible. State the authors' strongest defensible rationale, the alternative, and the assumptions or scientific aims that distinguish them. Explain any demonstrated consequence without calling the alternative a correction. A sensitivity analysis can qualify robustness, but sensitivity alone does not establish a defect.
• Undetermined: available evidence cannot settle this distinction. Specify what would resolve it; missing evidence is not proof of a defect. Use this for an unverified implementation concern, not Contested choice unless there are identifiable defensible alternatives.

A wrong denominator is a Defect; different results from two defensible covariate models are a Contested choice; unavailable code that prevents checking clustering is Undetermined. A confirmed sensitivity result can remain a Contested choice, and a Defect can be minor. A preregistration deviation alone does not prove the chosen method invalid; assess disclosure and claims of adherence separately. Split mixed findings so a factual error does not turn a defensible choice into a defect. Summaries and recommendations must preserve these distinctions: explain whether a claim changes because of a defect, dependence on contested assumptions, or unresolved evidence. Do not count all retained findings as errors.

Stage 5 — Severity and conclusion change

Converged does not mean Defect; Disputed does not mean Contested choice. Agreement between auditors is distinct from whether the authors' choice is defensible. Independently reassess the classification and its rationale even for converged findings. For older audits without these labels, derive them from the underlying evidence; do not infer Defect from severity or an error label. Record source and consensus classifications and reasons for changes in adjudication notes. Refuted allegations remain in the cross-walk as refuted; mark their classification not applicable rather than presenting them as retained defects.

In the main body, label each paper-wide or claim-specific finding and explain its classification rationale where its consequence is discussed. Include classification and rationale in minor/cosmetic issue tables and the finding cross-walk.

For every confirmed or partly confirmed finding, state its severity on the audits' scale (major / moderate / minor: the effect on the named claim). Then state whether it changes a major conclusion. It does so only when it moves that claim's consensus verdict to weak or unsupported. "Hypothesis-generating" for a claim the paper presents as established counts. A downgrade such as strong → moderate is reported, but it does not count as changing the conclusion. Apply this to every claim in the report, explaining any weakening even when it does not meet this threshold.

Stage 6 — Compare the audits

After adjudication, assess every included run, using run IDs only. Keep two judgments separate:

• Accuracy: correctness of the audit's numerical results, methodological criticisms, clean checks, severity assessments, and claim verdicts. Identify material errors, refuted allegations, overstatements, and appropriately expressed uncertainty.
• Usefulness: how much the audit helps a reader understand what holds and what changes in the paper. Credit consequential confirmed findings, informative clean checks, reproducible evidence, clear explanations, and actionable qualifications. Explain the cost of misleading allegations or distracting detail.

Base comparisons on adjudicated evidence, citing cross-walk items, claim verdicts, and exact outputs where relevant. Weight scientific consequence rather than raw finding counts, length, confidence of tone, or agreement with the majority. Compare scope and access to materials before treating an omission as a weakness; silence alone is not an error. Do not treat unresolved findings as refuted or an untested clean check as established.

Give a relative judgment for accuracy and another for usefulness. Name the leading run or tied group and explain material differences among the remaining runs. Use a tie whenever the evidence does not clearly distinguish runs; distinguish demonstrated similar performance from insufficient evidence to separate them. Accuracy and usefulness may have different leaders. Do not force an overall winner or numerical score.

State the limits: this comparison concerns these audits of this paper, not a general model ranking. Consensus adjudication is itself fallible, and accepted converged findings are not necessarily independently verified. Explain when these limits prevent a distinction.

Stage 7 — Report

Write for a reader who has not read the audits. Use plain language and short, connected paragraphs. Be concise, but give enough context to understand each claim, its evidence, and what the audit changes. There is no fixed word cap; length should follow the number and complexity of the main claims. Start from the template and use this order:

1. Paper summary: briefly explain the research question, study design, sample or data, main analyses, and the paper's central conclusions.
2. Paper-wide problems: if confirmed or partly confirmed major problems affect the paper as a whole, briefly explain each problem, the evidence establishing it, and its impact on the paper's conclusions. Distinguish demonstrated impact from uncertainty that prevents verification. If there are no such problems, omit this section. A serious problem confined to one claim belongs with that claim.
3. Main claims: give each main claim a descriptive subsection heading and these four elements, in order:
   • Claim: explain what the authors assert, including the population or scope.
   • Evidence: describe the evidence the paper offers: relevant design, measurements, comparisons, and results. Include deciding estimates and uncertainty when available, the effective sample size, and whether the design could detect the claimed effect, with a figure, table, or analysis reference. Do this for claims that survive too.
   • Holds after audit: explicitly say whether it holds, holds with qualifications, is weakened, is unsupported, or cannot be determined with the available evidence. Give the consensus strength verdict and explain its meaning for this claim. An untested claim is not a verified claim.
   • If weakened: explain the audit evidence that weakens it and the scientific impact: what narrower conclusion remains defensible, what no longer follows, and what is still unresolved. Distinguish reanalysis results from plausible concerns. If it is unchanged, say so briefly; do not invent a weakness.

Describe a paper-wide problem once in full, then explain its specific impact on each affected claim. Readers must not need to decode an issue ID or consult the appendix to understand a verdict. Brief repetition that makes a claim understandable is appropriate.

Appendix:

• Relative audit accuracy and usefulness: one row per included run, with separate accuracy and usefulness assessments and deciding evidence; then separate comparative judgments, including ties and their reasons, and the limits of the comparison.
• Published critiques: the search record and one row per critique point, with its assessment, the finding or check that tested it, and its disposition.
• Minor issues: changes to a reported number, test, or result label that do not change a claim verdict; give the location and effect.
• Noticed during adjudication: unadjudicated concerns that could change a verdict, clearly labelled as unconfirmed; otherwise "None."
• Claim verdicts by audit: per-run claim IDs and verdicts, with reasons for differences from the consensus.
• Finding cross-walk: every finding ID of every run, consensus item, status, outcome, classification and rationale, severity, affected claims, and whether it changes a major conclusion.
• Adjudication notes: evidence settling disputed or refuted findings, revised magnitudes or severities, and why audits differed. Include the family rationale for multiplicity findings, even when converged.
• Cosmetic issues: typos, labels, rounding or missing details that change no result or conclusion, with locations.
• Adjudication analyses, provenance, and session info.

Run IDs and finding IDs stay in the appendix. Consensus IDs may be used as optional links to detailed evidence, but never as substitutes for explanations in the main body.

Stage 8 — Render and verify

Render consensus.qmd to consensus.html with embedded resources. Check every number in the prose against the rendered output. Read the main body as a standalone explanation: does it summarize the paper, explain any paper-wide problems and their impact, and give each main claim its evidence, audit verdict, and the evidence and impact of any weakening? Remove unexplained shorthand and unnecessary detail without losing those elements. Confirm the cross-walk lists every finding ID of every run in inputs.json, and that multiplicity criticisms justify the family and account for the authors' correction. Confirm every published critique point has a disposition and every main claim has a power and precision row. Confirm every included run has an evidence-based accuracy and usefulness assessment, and that both relative judgments state any ties and limitations. Give the operator the file path. Do not publish it.