Reviewer Mindset

The job

Find the objection that would sink this paper, while there is still time to answer it. Not to be discouraging. A reviewer will find it anyway, and finding it yourself is the only version where you get to respond.

Self-review by an agent fails in a predictable way: it produces generous, generic feedback ("the paper is well written, consider adding more experiments"). That is worse than no review, because it manufactures confidence. Review against evidence, not impression.

Read in this order

1. Abstract and conclusion only. Do they claim the same thing? Can you state the finding in one sentence with a number in it? If not, the paper does not yet know what it is arguing.

2. Every number. For each, ask: which file produced this? Is the precision honest? Is a comparison stated that was never computed? See numbers-traceable.

3. Every figure and table. Is it referenced in the text? Does the caption say what to conclude? Is it legible at print size? Does it plot what the text says it plots?

4. The bibliography. Sample five entries and look them up. If any does not exist, stop reviewing and fix the bibliography, because there are probably more. See citation-discipline.

5. Related work against the claim of novelty. Is there a paper that already does this? Search for it specifically, do not rely on the draft's own account.

6. The method, for reimplementability. Could you build this from the description alone? List what is missing.

Questions that find real problems

• What result would have falsified the hypothesis? Was that outcome possible under this experimental design?
• Is the baseline given the same tuning effort as the method?
• Are the gains inside the variance? If variance is unreported, that is itself the finding.
• Does the conclusion generalise beyond the conditions tested?
• Is any experiment described in the plan missing from the results?
• Would the headline claim survive if the weakest experiment were removed?

Write the review usefully

Each point gets a location, a problem, and a fix:

Section 4.2, Table 2: the 12% improvement is reported without variance.
  Three seeds are mentioned in 4.1 but only means appear.
  Fix: report standard deviation, or state that the difference is
  within noise.

Not: "the evaluation could be more rigorous."

Severity, honestly assigned

• Blocking: invented citations, unsourced numbers, a claim the experiments do not support, a missing disclosure
• Major: unfair baseline, missing variance, an uncited close competitor
• Minor: prose, formatting, figure legibility

Do not inflate minor issues to look thorough, and do not soften a blocking issue to make the draft look closer to done than it is.