simulate-reviewers
Skill59 starsChanged 5 months agoApache-2.0View on GitHub
Venue-calibrated peer-review simulation. Use when a researcher says "simulate reviewers", "mock review", "review my paper like a NeurIPS reviewer", "what would Reviewer 2 say", "red-team my paper", "find weaknesses before I submit", "peer review my paper", "strengths and weaknesses / pros and cons of my paper", "what to focus on", or wants a rubric score / borderline-reject risk estimate for a conference or journal (NeurIPS, ICML, ICLR, CVPR, KDD, SIGMOD, CHI, VLDB, LNCS, TKDE, TODS...). Builds a reviewer panel calibrated to the venue family and track (harsh NeurIPS main-track skeptics vs lenient demo-track judges), scores novelty/soundness/reproducibility/clarity on the venue's scale, hunts weaknesses grounded in quoted paper text, aggregates scores into a decision-risk band with borderline-reject flags, and outputs a prioritized fix list. Advisory only; improves the paper, never predicts the outcome, re-verifies facts against the live CFP, fabricates no citations or reviewers, and submits nothing.
In this repository43 other files it gives its agents.
- docs/llms-full.txtllms.txt
- docs/llms.txtllms.txt
- add-venue-profileSkill
- anonymize-paperSkill
- assess-paperSkill
- benchmark-paperSkill
- check-originalitySkill
- draft-related-workSkill
- draft-surveySkill
- fetch-paperSkill
- find-papersSkill
- fit-page-limitSkill
- literature-reviewSkill
- make-posterSkill
- make-slidesSkill
- match-styleSkill
- orchestrate-paperSkill
- paper-profileSkill
- parse-cfpSkill
- plan-submissionSkill
- polish-proseSkill
- polish-tables-figuresSkill
- preflight-checkSkill
- prepare-artifactsSkill
- prepare-camera-readySkill
- refactor-research-codeSkill
- refactor-structureSkill
- reflect-and-improveSkill
- reflect-paperSkill
- rehearse-qaSkill
- render-workspace-htmlSkill
- select-venueSkill
- study-exemplarsSkill
- tailor-to-venueSkill
- test-research-codeSkill
- triage-reviewsSkill
- verify-citationsSkill
- verify-claimsSkill
- verify-resultsSkill
- work-with-overleafSkill
- write-abstractSkill
- write-rebuttalSkill
- write-talk-scriptSkill
The file
We split this skill into the rules it gives an agent
12 rules. About 0 are common, 6 are distinctive, and 6 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 8
Run a paper through a simulated, venue-calibrated review panel *before* submission.
- Very specific: line 21
After preflight-check passes (format is clean) but before submitting — this skill judges *content*, preflight judges *compliance*.
- Distinctive: line 35
Build the calibrated review packet: Run:
- Distinctive: line 52
If anything differs, update the profile YAML, note the discrepancy in the report, and prefer the live facts.
- Very specific: line 79
Every weakness cites a section/figure/line or quotes ≤1 sentence.
- Very specific: line 80
Never invent prior work. If a persona suspects missing related work, find real candidates with find-papers and verify them with …
- Distinctive: line 135
Every output carries the disclaimer: this is a simulation to improve the paper, not a prediction of the real outcome.
- Distinctive: line 181
Never present the simulation as a prediction ("your paper will get a 5 at NeurIPS").
- Very specific: line 183
Never fabricate citations, reviewer identities, or quotes — personas are archetypes, never named real researchers.
- Distinctive: line 190
Never submit to, or post on, any review system on the user's behalf.
- Very specific: line 199
At start: read .paper-memory/profile.yml (vertical, risk appetite, venue tier) to calibrate the panel and how hard the personas press …
- Very specific: line 209
Create .paper-memory/ on demand if absent and offer to add it to the project .gitignore.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Simulate Reviewers
Run a paper through a simulated, venue-calibrated review panel before submission. A NeurIPS main-track reviewer and a SIGSPATIAL demo-track judge reject for different reasons at different thresholds — this skill reproduces that difference: persona-driven weakness hunting, rubric scoring on the venue's own scale, and a deterministic decision-risk readout that tells the authors what to fix while there is still time.
When to use
- "What would reviewers say about this paper?" / "simulate a review"
- "Review this like a harsh NeurIPS reviewer" / "what will Reviewer 2 hate?"
- "Is this good enough for KDD, or should I aim for the short track?"
- "Find the weaknesses before the reviewers do" / "red-team my submission"
- After
preflight-checkpasses (format is clean) but before submitting — this skill judges content, preflight judges compliance.
Inputs
- The paper: a
.texsource tree, a PDF, or a draft in any readable form. Process it transiently — never copy paper text into the repo. - A venue profile:
venues/conferences/<venue>-<year>.yml(schema invenues/schema.yml). No profile? Create one withparse-cfpfirst, or run against the nearest family default and say so. - The target track (page limits and reviewer expectations differ — ask).
Process
Build the calibrated review packet. Run:
python3 scripts/review_form.py venues/conferences/<venue>-<year>.yml \ --track "<track>"This is deterministic and offline. It merges the family profile and emits the panel (personas + harshness), the venue score scale with its borderline threshold, the rubric, the per-reviewer form skeleton, and a
scores.jsontemplate. Add--jsonfor machine-readable output. Exit codes: 0 ok, 2 missing/unparsable profile or unknown track.- Re-verify against the live CFP — mandatory. Profiles and the script's
scale anchors are historical norms, never ground truth. Fetch the
profile's
cfp_url(and reviewer-guidelines page if linked) and confirm: review scale and form, blind level, rebuttal format, track expectations. If anything differs, update the profile YAML, note the discrepancy in the report, and prefer the live facts. Label every venue fact you state with a confidence tag and a clickable source:verified-live/corroborated/inferred-from-family/needs-verification. A scale number quoted to the user with no source is a bug, not a convenience. - Read the whole paper and build a claim inventory. List every claim of novelty ("first", "state-of-the-art", "outperforms"), every empirical claim, and where its supporting evidence lives. This inventory is what the personas attack. Method in references/weakness-hunting.md.
- Write each review independently, in persona. One pass per reviewer
from the packet, in order, without referring to the other reviews while
writing (real reviews are independent; convergent complaints found
independently are the strongest signal). Persona behavior, harshness
calibration, and track modifiers are specified in
references/reviewer-personas.md.
Grounding rules — non-negotiable:
- Each review opens with genuine Strengths, then Weaknesses — like a real review form. State 2–4 specific strengths (what the paper does well: novelty, a strong experiment, clarity, a useful artifact), each grounded in a section/figure the same way weaknesses are. A review that is all cons is not a real review and misleads the author about what to protect while fixing. Do not invent strengths to pad — if the paper is weak, say so, but find what genuinely works.
- Every weakness cites a section/figure/line or quotes ≤1 sentence.
- Never invent prior work. If a persona suspects missing related work,
find real candidates with
find-papersand verify them withverify-citations— or phrase the concern conditionally ("if prior work on X exists, R4 will find it") with no fake reference. - Misreadings are allowed only for the skimmer persona, and must be misreadings the actual text permits.
- Score with the rubric, then the venue scale. Score the four core dimensions (novelty, soundness, reproducibility, clarity) 1–5 per reviewer using the anchors in references/rubrics.md, then map to the venue's overall scale + confidence from the packet. Harshness calibration: at harshness 5, an unaddressed soundness weakness caps the overall at borderline-reject; at harshness 2 (demo track), it becomes a question, not a cap.
Aggregate deterministically. Fill the
scores.jsontemplate from step 1 and run:python3 scripts/aggregate_scores.py scores.jsonIt computes the confidence-weighted mean, disagreement/champion/detractor flags, drag dimensions, and the decision-risk band (likely-reject / borderline-reject / borderline-accept / likely-accept) — including the "borderline without a champion resolves downward" rule.
--exampleprints a valid input;--jsonfor machine output; exit 2 on invalid input.- Write the meta-review and the fix list. As the AC/1AC/AE persona:
synthesize the reviews, name the biggest shared concern, state whether a
champion exists. Then convert every weakness into a prioritized fix list,
each item tagged:
fix-now— addressable before submission (add ablation, soften claim, add reproducibility statement);rebuttal-defensible— survivable in this venue's rebuttal format (checkreview.rebuttal_format/ limits in the profile);structural— cannot be fixed this cycle; consider a different venue/track (hand off toselect-venue). Borderline-reject predictors to check explicitly are listed in references/weakness-hunting.md.
Output
A simulated review packet, presented in chat (written to a file only if the user asks):
- N independent reviews in the venue's form format, each opening with a Strengths section, then Weaknesses, then questions and subscores.
- The meta-review, naming the biggest strength to preserve and the biggest shared risk to fix, and stating whether a champion exists.
- The score table plus the
aggregate_scores.pydecision-risk readout. - The prioritized fix list, every item tagged
fix-now/rebuttal-defensible/structural.
Every output carries the disclaimer: this is a simulation to improve the paper, not a prediction of the real outcome.
Worked mini-example
A 9-page submission to NeurIPS main track. After python3
scripts/review_form.py venues/conferences/neurips-2026.yml --track Main and a
live-CFP check (scale and rebuttal format confirmed, tagged verified-live),
the four personas are written independently. Convergence emerges: R2 (the
empirical skeptic) and R4 (the adjacent-field expert) both, without seeing
each other, land on the same gap.
R2, Weaknesses. "Table 2 reports a single run (§5.1). With ±std over 5 seeds, does the +1.3% gap over the baseline survive? No tuning-budget parity is stated for the baseline." — soundness 2, confidence 5.
R4, Weaknesses. "The §1 claim 'first to combine X with Y' needs the 2023 work on X-under-Y. If that line exists, the novelty claim narrows to an engineering delta." (Conditional — routed to
find-papers; no fake citation stated.) — novelty 2, confidence 4.
Filling and aggregating scores.json (R1 5/4, R2 4/5, R3 6/2, R4 5/4) yields:
conf-weighted mean: 4.8 delta vs threshold: -0.133 (normalized)
DECISION RISK: BORDERLINE-REJECT
drag dimensions: soundness 2.5, reproducibility 2.75
flags: strong detractor present (R2) — objection must be rebuttal-proofThe fix list leads with the convergent finding: fix-now — add 5-seed
±std and tuning-budget parity to Table 2 (cheapest path off the soundness
floor); fix-now — verify the X-under-Y prior work and soften the "first"
claim accordingly. The meta-review notes the only above-threshold score came
from the low-confidence skimmer (no real champion), so at borderline this
resolves downward unless the soundness objection is closed before submission.
Adapt to your discipline
Panels and scales are keyed on the venue family: field. For other fields,
fork and add a calibration entry (personas + scale) for your community in
scripts/review_form.py and a venue YAML — e.g. an APA-journal panel with
action-editor + 2 reviewers and accept/minor/major/reject.
Guardrails
- Never present the simulation as a prediction ("your paper will get a 5 at NeurIPS"). Say: "the simulated panel scored it X; real panels vary widely."
- Never fabricate citations, reviewer identities, or quotes — personas are
archetypes, never named real researchers; missing-related-work claims go
through
find-papers+verify-citationsor stay conditional. - Re-verify review-process facts against the live
cfp_url(step 2 is not optional); flag any profile staleness in the report. - Process the paper transiently; quote at most one sentence per finding; never store paper text in this repo.
- Never submit to, or post on, any review system on the user's behalf.
- Format/compliance problems found along the way are out of scope — route
them to
preflight-check.
Memory
This skill uses the shared .paper-memory/ convention in the user's paper
directory, following paper-memory-convention.md.
- At start: read
.paper-memory/profile.yml(vertical, risk appetite, venue tier) to calibrate the panel and how hard the personas press novelty claims, and readlessons.mdto recall which weaknesses were already raised and whichrecurringones this author repeats (e.g. "weak ablations" or "overclaimed contributions") so the meta-review leads with them. - At end: append the durable weaknesses in the shared format
- [YYYY-MM-DD] (simulate-reviewers | <scope>) weakness -> recommendation(viareflect-and-improve'sreflect_log.py append, which dedupes and dates). A structural habit seen across drafts isrecurring; a draft-specific gap isthis-paper. Do not log per-reviewer score noise. - Create
.paper-memory/on demand if absent and offer to add it to the project.gitignore. It is local-only; never upload it or copy it into this repo.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

