agentleFS
xzljqxzljq/claude-skills/skills/research-lit-survey/SKILL.md

research-lit-survey

Skill0 starsChanged 3 months agoMITView on GitHub

Literature survey with source-quality tiering and full-text verification. Use when a conclusion will rest on the literature: mapping a field's state or mainstream research methods, selecting a baseline, deciding what to build on or refute, or checking whether an idea is novel. For a casual pointer-level related-work question, run the lightweight scan mode described inside rather than the full survey.

The file

Show its rules and text

We split this skill into the rules it gives an agent

21 rules. About 0 are common, 16 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈16

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Check that citing works actually *use* X rather than merely mention it in related work.

    line 102
  2. Distinctive:

    When only one source exists, phrase it as what it is — "one recent preprint proposes X" — rather than promoting it to consensus.

    line 107
  3. Distinctive:

    When a work has both a preprint and a published version, record both: the published landing page settles venue and tier, the arXiv ID gives …

    line 119
  4. Distinctive:

    Use when the answer will carry weight: "what is the field doing / what should the baseline be / where are the gaps".

    line 128
  5. Distinctive:

    Read the anchor's own References and Related Work sections.

    line 139
  6. Distinctive:

    If paging aborts early or the API stays unavailable, say so explicitly ("upward citation coverage incomplete — newest work may be missing").

    line 183
  7. Distinctive:

    Only count a link the paper presents as its own release ("our code is available at …"), and check the owner is plausibly the authors.

    line 218
  8. Distinctive:

    If these cannot be established: label the number "context unknown — not usable for comparison" and do not use it to rank methods.

    line 323
  9. Very specific:

    Write to a stable path in the project (e.g.

    line 333
  10. Very specific:

    Upgrade rows in place: when a work is read more deeply — bump read depth and rewrite the mechanism line from the primary source.

    line 374
  11. Distinctive:

    Correct rows rather than appending contradictions: When full-text reading overturns an earlier abstract-level entry, fix the row and note …

    line 376
  12. Very specific:

    Write it to references/SURVEY-<topic>.md, separate from LEDGER.md.

    line 393
  13. Distinctive:

    If a report was written early anyway and later evidence changes it, do not silently patch it.

    line 414
  14. Distinctive:

    Every number carries its Step 5 caveat, or the label

    line 449
  15. Very specific:

    After cloning, fill the ledger row's repo path with references/<name>/, and put the ledger key at the top of NOTES.md, so each side points …

    line 491
  16. Distinctive:

    Stop when repeated terminology search and bidirectional citation-graph expansion produce no new work that materially changes the field map, …

    line 502
  17. Distinctive:

    Search both ways: keywords *and* citation-graph crawling around the closest known works.

    line 520
  18. Distinctive:

    Search all tiers, including Tier C.

    line 522
  19. Distinctive:

    Mode ② protocol: write the Codex task as literally *"find the paper that already did this"*.

    line 583
  20. Distinctive:

    Execution: call Codex MCP tools from the MAIN conversation, never from a Claude subagent.

    line 594
  21. Very specific:

    Never send Claude's analysis first.

    line 597

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Research literature survey

Build a verified, actionable picture of a field: what is already taken, what is open, what is solid enough to build on, and what to compare against. The output is an evidence ledger — a living document that later work extends and corrects, not a snapshot that freezes when the survey ends.

Scope: match depth to consequence

Most of this file describes the full survey, which is expensive. Run it only when something downstream rests on the answer. Two modes:

ScanFull survey
When"what work is there on X?", orienting, a pointer request — nothing depends on the answer yetThe answer will drive a decision: framing a direction, choosing a baseline, asserting field state, claiming novelty, deciding what to refute or build on
EffortKeyword search + abstracts; read full text only for anything actually quotedAnchors → citation graph → tiering → full-text → number audit → ledger
Outputanswer in-conversation, nothing writtensurvey report + evidence ledger, both written to the project (Steps 6–7)
Downloadsnoneopt-in, see Step 8

Scan mode is claim-limited. It may report what exists and what a work appears to be about, with read-depth stated ("from abstracts, not verified"). It may not produce: field-state or mainstream-method claims, novelty claims, baseline recommendations, or a framing that later work builds on. Producing any of those requires escalating to the full survey.

Escalation is mandatory and immediate the moment a scan result starts being used for a decision — including later in the same conversation. Say that the earlier answer was scan-level and redo the affected part properly rather than promoting unverified rows.

Mode ② (novelty check) has no scan version: a novelty claim is load-bearing by definition.

Method

Six principles; everything below is their operational form.

  1. Structure beats keywords. A field's citation graph and a paper's own Related Work section are better maps than search ranking. Keyword search discovers candidates; it does not establish conclusions, because it misses work phrased in other terminology and gives no signal about centrality.
  2. Evidence comes in weight classes. What a source may be used for depends on where it was published and by whom — tier decides role, not correctness.
  3. Read the primary source for anything a conclusion rests on. Paraphrases drop the qualifiers that carry the meaning, and an ambiguous paraphrase invites a guess.
  4. A number is not a result until you know what it measures. Metric definition, evaluation-set size, baseline, seeds, budget-matching.
  5. Persist provenance, re-read for content. Record what was looked at, how deeply, and with what caveats — that metadata is expensive to re-derive and does not go stale. A compressed summary of what a paper says is a different thing: it is lossy, it was written before anyone knew which qualifier would matter, and quoting it downstream propagates any first-read error. Keep the full text so going back to it is free.
  6. Two independent surveyors, for coverage. Claude and Codex search in parallel. Different training data, search habits, and terminology instincts mean each surfaces work the other misses, so the union is the survey — one searcher alone is systematically partial. Disagreement between the two pictures is a useful secondary signal, but breadth is the reason to do it.

Hard rules

  • Never support a conclusion with a WebFetch summary or a search snippet. Those are for triage only.
  • Any work that becomes load-bearing must be read in full text, at that moment. Load-bearing = it defines the field's state/mainstream research methods, is the thing being refuted, is a baseline, or is an inspiration source for an idea. Do not carry forward an abstract-level understanding once a work is promoted to load-bearing.
  • Distinguish claims about a single work from claims about the field.
    • About one work — "X proposes this", "X reports these numbers", "we follow X and extend it", "X is our baseline", or "X contains a promising underexplored insight": a single paper is sufficient, provided it is read in full and its claims and any quoted numbers are audited. A full-text-verified Tier-A/B work may serve as an inspiration source even when it has little follow-up; sparse follow-up is a positive signal that the space is not yet crowded. Only verify that later work has not already occupied the intended extension. A concrete idea usually draws its primary inspiration from a small number of closely related works. These works are load-bearing and must be verified in full text. The small inspiration set describes the idea's direct intellectual lineage, not the coverage of the literature survey.
    • About the field — "X is mainstream", "the field commonly uses X", "X is a dominant research direction": the evidence needed is adoption, not replication. Good papers do not re-propose each other's methods, so counting independent works that propose the same thing is the wrong test and will almost never be satisfiable. What establishes mainstream status is uptake by independent groups: works that use X as a component, adopt it as a baseline to beat, or build directly on it; coverage in a survey; or a major-lab technical report reporting X as part of its own recipe. Check that citing works actually use X rather than merely mention it in related work. A single work proposing X never establishes that X is mainstream, no matter how strong the paper. A method posted last month with no uptake is a proposal, not a practice — describe it as one.
    • When only one source exists, phrase it as what it is — "one recent preprint proposes X" — rather than promoting it to consensus.
  • Tier is an admission gate, not a credibility endorsement. A Tier-A paper from a famous lab with open code can still have weak experiments. The number audit below applies to every tier equally.
  • Every identifier that enters the ledger must resolve to a real authoritative landing page, with title and authors confirmed on that page — regardless of whether Claude or Codex found it. Which page is authoritative depends on the identifier: arXiv abs for an arXiv ID; the publisher page for a DOI; otherwise OpenReview, ACL Anthology, PMLR, CVF Open Access, or the venue's own proceedings page. Many DOIs have no arXiv page at all, so "resolve it on arXiv" is not a general check. When a work has both a preprint and a published version, record both: the published landing page settles venue and tier, the arXiv ID gives full-text access. If an identifier resolves nowhere authoritative, treat the work as unverified and do not cite it.

Mode ①: landscape survey (full)

Use when the answer will carry weight: "what is the field doing / what should the baseline be / where are the gaps".

Step 1 — Anchors

Pick 2–3 papers certain to be central to the topic. Keyword search or prior knowledge is fine for finding anchors.

Step 2 — Citation-graph expansion

Downward (foundational work) — arXiv alone is sufficient, no API: Read the anchor's own References and Related Work sections. The Related Work section is the field's own taxonomy of itself, which is more reliable than any taxonomy inferred from search results.

Upward (newest work) — the one thing arXiv cannot provide: arXiv does not track incoming citations. Use Semantic Scholar.

Two properties of this endpoint matter and are easy to get wrong:

  • It is paginated. A single limit=N call silently truncates; a well-cited anchor will lose most of its citing papers. Page with offset until the next field disappears.
  • sort is ignored. Returned order is not newest-first, so "the newest work" only exists after fetching everything and sorting locally by year.
# page through ALL citing papers, then sort newest-first
python3 - <<'PY'
import json, time, urllib.request
ID = "arXiv:2507.17746"          # anchor
URL = ("https://api.semanticscholar.org/graph/v1/paper/{}/citations"
       "?fields=title,year,venue,citationCount,externalIds&limit=1000&offset={}")
rows, off = [], 0
while True:
    for attempt in range(5):                     # 429 without an API key
        try:
            d = json.load(urllib.request.urlopen(URL.format(ID, off), timeout=60))
            break
        except Exception:
            time.sleep(5 * (attempt + 1))
    else:
        raise SystemExit(f"INCOMPLETE: stopped at offset {off}")
    rows += [x["citingPaper"] for x in d.get("data", [])]
    if "next" not in d:
        break
    off = d["next"]; time.sleep(3)
rows.sort(key=lambda r: (r.get("year") or 0), reverse=True)
print(len(rows), "citing papers")
for r in rows[:40]:
    print(r.get("year"), "|", (r.get("venue") or "")[:18], "|", r.get("citationCount"), "|", r.get("title"))
PY

Rate-limited without an API key (HTTP 429) — the retry/sleep above handles it. If paging aborts early or the API stays unavailable, say so explicitly ("upward citation coverage incomplete — newest work may be missing"); never silently fall back to keyword search and present the result as complete.

Centrality: papers cited by several anchors independently are the field's canon. Search ranking does not give this signal.

Step 3 — Source tiering

Assign a tier to every work before it is allowed to influence anything.

Three credibility signals: provenance (well-known lab, well-known corresponding author, or institution on the list below), citations from independent work, identifiable open-source code.

TierCriteriaMay be used for
APublished at a top venue; or a major-lab technical report; or a preprint with provenance AND at least one of citations / open codeField-state evidence (uptake, see hard rules), refutation target, baseline, inspiration source
BProvenance alone — e.g. a brand-new preprint from a listed institution with no code and no citations yet; or citations/code without provenanceRelated-work differentiation, inspiration source
CNone of the three signalsAwareness only — may not enter any claim about the field, and may not serve as an inspiration source

Novelty-check exception: Tier C still enters prior-art screening. If an earlier public work implements the same mechanism, narrow the novelty claim accordingly. Its existence does not establish empirical credibility, field adoption, or suitability as an inspiration source.

Baselines and anything the work will be built on have an additional requirement: open, runnable code.

The code signal must be verified at the repository, never inferred from the paper. A printed URL is a claim, not a release. Two distinct failure modes, both common:

  • The link is not the authors'. A bare grep for github.com over a paper's HTML mostly returns reference-list links — repos the paper merely cites. Only count a link the paper presents as its own release ("our code is available at …"), and check the owner is plausibly the authors.
  • The repo is a placeholder. README-only, size == 0, or 404. Open it and confirm actual source files exist:

    gh api repos/<owner>/<repo> --jq '"size=\(.size)kb pushed=\(.pushed_at[0:10])"'
    gh api repos/<owner>/<repo>/contents --jq '[.[] | "\(.type) \(.name)"] | join(", ")'

    Code often lives in subdirectories, so a root listing without .py files is not by itself an empty repo — look inside before judging.

An unverified or placeholder link does not satisfy the code signal, and a work whose only path to Tier A ran through that link is Tier B. It can never satisfy the baseline requirement, which is about running the thing.

Tier-A institution seed list (a seed, not an exhaustive enumeration):

  • Universities, mainland China: Tsinghua, Peking University, Zhejiang University, Shanghai Jiao Tong University, Fudan, Nanjing University, USTC, Renmin University, Beihang, Harbin Institute of Technology, Xi'an Jiaotong University, HUST, Sun Yat-sen University, Tongji, Southeast University, Beijing Institute of Technology, Nankai; CAS institutes (CASIA, ICT, SIAT)
  • Universities, HK & Singapore: CUHK, HKUST, HKU, HK PolyU, City University of Hong Kong, NUS, NTU, A*STAR
  • Universities, US: Stanford, MIT, CMU, Berkeley, Princeton, Harvard, UW, UIUC, Georgia Tech, UT Austin, UMich, UCLA, UCSD, NYU, Cornell, Caltech, JHU, UMD
  • Universities, Europe & Canada: Oxford, Cambridge, ETH Zurich, EPFL, Max Planck Institutes, TU Munich, INRIA, UCL, Edinburgh, Tubingen, Toronto/Vector, Mila
  • Universities, Japan & Korea: University of Tokyo, KAIST, Seoul National University
  • Companies & labs, China: DeepSeek, Alibaba, ByteDance, Tencent, Baidu, Huawei, Ant Group, Moonshot AI, Zhipu AI, MiniMax, StepFun, SenseTime, Megvii, Xiaohongshu, Meituan, Kuaishou, 01.AI, Shanghai AI Laboratory, BAAI
  • Companies & labs, international: OpenAI, Google DeepMind, Google Research, Anthropic, Meta AI/FAIR, Microsoft Research, NVIDIA, Apple ML, AWS AI, AI2, xAI, Mistral, Cohere, Scale AI, Naver, Sony AI, IBM Research

Operating rules matter more than the list (the list will go stale):

  • The list is a seed; comparable reputation may be extrapolated.
  • Judgment criteria: multiple first-author/corresponding-author papers at top venues in the same area · institution on the list · lab with a recognized landmark work.
  • When uncertain, downgrade to Tier-B and report it to the user. Never loosen the bar unilaterally.

Step 4 — Full-text reading

Targets: everything load-bearing, plus every inspiration source (Tier A/B only) — an idea built on a misread paper is worse than no idea.

Do not route by submission year. arXiv has backfilled LaTeXML HTML to old submissions — 2017-era papers render fine — so "pre-2024 means PDF" is wrong and skips the cheap path. But the backfill is uneven: a paper whose LaTeX source does not convert still returns HTTP 200 with a near-empty <title>Untitled Document</title> stub. Availability must be decided by inspecting the extracted text, not by the status code or the year.

Do not pin v1 unless a specific version is what you mean to read. Papers get substantially revised, and the ledger's numbers should match the version actually cited. Fetch the unversioned URL (it serves the latest version) and record which version came back; append v<N> only to reach an older one deliberately.

# fetch latest version — NOT <id>v1
ID=2507.17746
curl -sL "https://arxiv.org/html/$ID" -o "full_$ID.html"

# strip to text (keeping LaTeX-ish math), then decide if the conversion is usable
python3 - "$ID" <<'PY'
import re, html, sys, pathlib
i = sys.argv[1]
s = pathlib.Path(f'full_{i}.html').read_text(encoding='utf-8', errors='ignore')
m = re.search(r'(?is)<title>(.*?)</title>', s)
title = html.unescape(m.group(1)).strip() if m else ''
s = re.sub(r'(?is)<(script|style|head)[^>]*>.*?</\1>', ' ', s)
s = re.sub(r'(?is)<math[^>]*alttext="(.*?)"[^>]*>.*?</math>', r' $\1$ ', s)
s = re.sub(r'(?is)<(p|div|li|tr|h1|h2|h3|h4|section|figcaption|td|th)[^>]*>', '\n', s)
s = re.sub(r'(?s)<[^>]+>', ' ', s); s = html.unescape(s)
s = re.sub(r'[ \t\xa0]+', ' ', s); s = re.sub(r'\n\s*\n+', '\n', s).strip()
bad = (len(s) < 20000 or not title or title.lower().startswith('untitled')
       or not re.search(r'(?im)^\s*\d*\.?\s*(experiment|evaluation|result)', s))
print(f'{i}: chars={len(s)} title={title[:60]!r} -> {"REJECT — use PDF" if bad else "OK"}')
pathlib.Path(f'txt_{i}.txt').write_text(s, encoding='utf-8')
PY

On OK, Read the text file — Method, Experiments, and the actual result tables. On REJECT, fall back: try https://ar5iv.labs.arxiv.org/html/<id>, and if that also fails, curl the PDF and Read it directly. Non-arXiv works (publisher DOI, ACL Anthology, OpenReview) go straight to the PDF.

The three thresholds are heuristics, not a spec — a genuinely short workshop paper can trip the length check. Treat REJECT as "look at the text before trusting it", never as "this paper has no full text".

Keep the extracted text. In a full survey, save it next to the ledger (e.g. references/fulltext/<id>.txt) and record the path in the ledger row. This is what makes "always re-read rather than trust the summary" actually cheap — no re-download, no re-search, just re-open. In scan mode, extraction is transient and goes to the scratchpad/tmp directory.

Step 5 — Number audit (all tiers, no exceptions)

For every headline number quoted, record:

  • what the metric actually is (pass@1? avg@k? pass@1024? a custom "verified" variant?)
  • n of the evaluation set
  • what the baseline is
  • number of seeds and the variance
  • whether the comparison is budget-matched (tokens / rollouts / judge calls / FLOPs — not merely optimizer steps)

If these cannot be established: label the number "context unknown — not usable for comparison" and do not use it to rank methods.

Relative improvements ("+20% average") must be decomposed: check whether the gain is concentrated in one small-n benchmark.

Step 6 — Evidence ledger

Full survey only. Scan mode answers in-conversation and writes nothing.

Write to a stable path in the project (e.g. references/LEDGER.md) so later sessions extend it instead of starting over.

What the ledger is for — and what it is not.

It is an index and an audit trail: which works exist, where to find them, how deeply each was actually read, what tier, and what caveats attach to their numbers. That metadata is expensive to re-derive and does not decay.

It is not a substitute for the paper, and nothing downstream may quote it as if it were:

  • Never characterize what a paper does, claims, or shows from a ledger row alone (principle 5). Re-open the stored full text. A one-line mechanism field is a routing label for finding the right paper again, not a quotable summary — and a row that looks authoritative is exactly what discourages the re-read. Re-reading is the default, not a fallback.
  • Numbers get re-checked against the paper whenever they become load-bearing for a comparison, caveat column notwithstanding.
keyid + landing pagetitleauthorsaffiliationvenuecitationstiercode (verified?)read depth (which sections)mechanism (routing label)where stated (§ / Table)headline number + caveatfull text pathrepo path

read depth ∈ {full-text, abstract, snippet} — and for full-text, name which sections were actually read, since "full-text" in practice usually means method plus experiments. Any claim resting on an abstract/snippet row must be labeled as such in the deliverable.

code (verified?) records the outcome of the repository check in Step 3, not the URL printed in the paper: verified / placeholder / 404 / none. repo path is filled in only once index-references has actually cloned the work (Step 8); it stays empty otherwise.

The where stated column exists to make re-reading cheap and targeted: it points at the section or table to re-open, so the default behaviour costs seconds rather than a re-download.

Maintenance — the ledger is never final:

  • Check for an existing ledger before surveying. Extend it; do not start a parallel one.
  • Upgrade rows in place when a work is read more deeply — bump read depth and rewrite the mechanism line from the primary source.
  • Correct rows rather than appending contradictions. When full-text reading overturns an earlier abstract-level entry, fix the row and note what changed; a wrong row left in place will be cited again.
  • Add newly published work on later passes, especially upward-citation results, which change fastest.
  • Re-tier when status changes — a preprint that gets accepted, released code, or accumulated citations moves tier.
  • Downstream documents that quoted a corrected row must be revisited, or explicitly flagged as based on the superseded version.

Step 7 — The deliverable

The ledger is not the answer. It is the index behind the answer. The answer is a separate document that states what the field is doing, what to build on, and what is open — the thing the user actually asked for and the thing later work will cite.

Write it to references/SURVEY-<topic>.md, separate from LEDGER.md. The two age differently: a report is a dated snapshot of a judgement and is allowed to go stale; the ledger is the living record that outlives it. Date the report and name the ledger revision it was built from.

Do not begin writing the report until every surveyor has reported in. This is a gate, not a preference. The report is the one artifact that is expensive to revise, because its gap list and its field-state claims are derived from coverage — a gap is only a gap relative to what was searched. Drafting it while the other surveyor is still running produces a confident document built on half the evidence, and the missing half is precisely the part most likely to overturn it: the entire reason for two independent surveyors (principle 6) is that each finds work the other misses, so the unreturned half is where the paper that occupies your gap is most likely to be.

While a surveyor is still running, do ledger work instead — adding rows, tiering, auditing numbers, reading full texts. None of that depends on total coverage, and all of it is needed either way. If a surveyor is genuinely unavailable, name that in Coverage limitations and write against the coverage you actually have.

If a report was written early anyway and later evidence changes it, do not silently patch it. Bump it to a new revision, mark the superseded one as not to be cited, and add a short table of what changed and why. Anyone who read the earlier version needs to know which of its claims died — and a gap list that quietly loses two entries between reads is worse than one that never claimed them.

Structure:

  1. Question and scope — what was asked, what was deliberately excluded.
  2. Field map — the clusters/subfields and how they relate. Take the taxonomy from the anchors' own Related Work sections (Step 2), not from search-result groupings. Name the canon per cluster: works multiple anchors cite independently.
  3. Mainstream-method claims — each one carries its adoption evidence inline (who independently uses it, as a component / a baseline to beat / a survey's coverage / a lab report's recipe), per the hard rules. A claim whose support is only the proposing paper is stated as a proposal, not a practice.
  4. Baseline recommendation — tier, landing page, code link, and the Step 5 audit of its headline numbers. Say why it is runnable, not just why it is relevant.
  5. Gaps — what is open, and what makes each gap look real rather than merely unsearched.
  6. Coverage limitations — every known hole, named: upward-citation paging that aborted, an unavailable API, a single-surveyor run when Codex was not reachable, survivors left unread. This section is mandatory and may not be empty by default; "no known limitation" is itself a claim.

Writing rules — these are what make the report trustworthy:

  • Every characterization of a paper is written while re-reading its stored full text — composing the report out of ledger rows is the exact failure the ledger design exists to prevent. Use where stated to make it cost seconds.
  • Every number carries its Step 5 caveat, or the label "context unknown — not usable for comparison".
  • Any statement resting on an abstract / snippet row is marked inline as such. Unmarked statements are read as full-text-verified.
  • Separate what the literature says from what you conclude from it. Mechanism guesses and predictions are labeled as unverified.

Mode ② deliverable is different in shape: the screening counts (hits, excluded with reasons, read in full), the closest prior work with the specific mechanism overlap quoted from its text, a verdict — novel as stated / must be narrowed to X / already occupied — and every unread survivor named.

Step 8 — Hand off to index-references (opt-in)

Not a default step. index-references clones repositories and downloads PDFs into the project — a real side effect that a survey does not by itself justify.

Trigger it only when a specific work has been selected to be run or built on — a chosen baseline, or the implementation the project will modify — and say so before doing it. Reading a paper, even a Tier-A load-bearing one, does not require pulling its repo.

Division of labour with references/<name>/NOTES.md. Both stores describe the same work, so the boundary has to be explicit or the same paper gets two descriptions that drift apart:

LEDGER.mdreferences/<name>/NOTES.md
Answerswhat this work is, and how much it may be trustedhow to run this code
Ownsidentifiers, venue, tier, read depth, number audit + caveats, full-text pathentry scripts, configs, dependencies, code map, repro command, reproduction target
Must not containbuild/run instructions duplicated from the repotier, novelty/field-state judgements, or any characterization meant to be quoted as what the paper says

Rules:

  • The ledger is the single source of truth at the paper level. A question of the form "what does this work claim / is it a valid baseline / how solid are its numbers" is answered from the ledger row plus a re-read of the full text — never from NOTES.md.
  • NOTES.md is the single source of truth at the implementation level. A question of the form "what does this script do / which config / how do I reproduce it" is answered there, not from the ledger.
  • After cloning, fill the ledger row's repo path with references/<name>/, and put the ledger key at the top of NOTES.md, so each side points at the other.
  • Reproduction targets (the paper's reported absolute numbers) legitimately appear in NOTES.md — that is what a reproduction is checked against. They carry their source table and stay inside that role: quoting them into a survey, a comparison, or a paper goes through the ledger and the full text, where the Step 5 audit lives.

Stop condition

Stop when repeated terminology search and bidirectional citation-graph expansion produce no new work that materially changes the field map, baseline choice, or set of Tier-A/B inspiration sources. There is no fixed paper-count target; stop at decision saturation rather than an arbitrary count. Report any known coverage limitation explicitly.


Mode ②: novelty check

Given a specific idea, the goal is to actively find the prior work that kills it. "I did not find it" is not evidence of novelty.

  1. Decompose the idea into separately searchable assertions; search each.
  2. Enumerate terminology variants. The same construct has different names in different subfields (e.g. all-wrong / all-correct rollout groups appear as "dead / saturated criteria" at the criterion level). Missing a variant is the dominant failure mode here.
  3. Search both ways: keywords and citation-graph crawling around the closest known works.
  4. Search all tiers, including Tier C; source tier does not filter potential prior art.
  5. Two-stage screening. A broad topic yields hundreds of hits; reading all of them is not a rule, it is a rule that gets ignored. Screen cheaply, then verify expensively — but only in one direction.

    Stage 1 — cheap exclusion (title / abstract / snippet). This pass may only rule candidates out, and only on a ground that can be stated in words: different problem, different modality, no contact with the mechanism at all. Record the exclusion reason.

    Stage 2 — full-text verification of every survivor. Whether the mechanism is genuinely the same, not merely similarly described, can only be settled in the paper.

    The asymmetry is the safety property, and it is not negotiable:

    • Cheap evidence may exclude. It may never clear a candidate — "the abstract does not sound like our mechanism" is not a finding of novelty.
    • "Probably not the same" is a survivor, not an exclusion. Uncertainty defaults to reading.
    • Report the counts: how many hits, how many excluded, by which reasons, how many read in full. Silent truncation reads as coverage that did not happen.
  6. Prioritise survivors by mechanism proximity and read the closest first, so that if effort runs out the most dangerous candidates were the ones verified. Any unread remainder must be named explicitly, not dropped.
  7. An unbounded survivor set is a signal about the claim, not just the search. If too many candidates plausibly implement the same mechanism, the novelty claim is under-specified — sharpen it until it states precisely what is claimed new, then re-screen. Narrowing the claim is the correct response; skipping reads is not.

Codex collaboration

Codex has web access and its own tools.

  • ❌ Never write "do not run shell commands", "do not inspect anything", or "answer from your own knowledge" into a Codex prompt for a task that needs retrieval. Doing so degrades it into a memory-only responder and produces unverifiable output. Note that Codex will then report that it did not search — that reflects the prompt's constraints, not its actual access.
  • ✅ Explicitly instruct Codex to search, and require resolvable arXiv IDs / URLs in its answer.
  • Anything Codex surfaces still goes through the ledger rules: resolve the identifier to its authoritative landing page and confirm title/authors there before use.

Mode ① protocol:

  1. Both sides survey independently and in parallel, each building its own picture. Merge both results into the ledger — the union is the point, since either searcher alone covers only part of the field. Where the two pictures disagree, investigate rather than pick a side. The Step 7 report waits for both (see the gate in Step 7); the ledger does not.
  2. Then send Codex the verified ledger plus Claude's landscape claims for adversarial review, targeting: which claim overreaches what the cited text says, which subfield is missing, which "mainstream method" claim rests on a single source.

Mode ② protocol: write the Codex task as literally "find the paper that already did this". Adversarial framing, not a neutral request.


Shared red lines

<!-- CANONICAL BLOCK — verbatim copy. Source of truth: codex-collab/SKILL.md. Duplicated here and in research-ideation so loading one skill suffices. Any edit must be applied to all three files identically. -->
  • Execution: call Codex MCP tools from the MAIN conversation, never from a Claude subagent. Long calls get backgrounded — do useful work meanwhile.
  • Independence: for an independent opinion, always start a FRESH thread. Never send Claude's analysis first. Use codex-reply only after the independent first response has arrived.
  • Tool access: Codex has web access and its own tools. Never write "do not run shell commands" or "answer from your own knowledge" into a prompt for a task that needs retrieval — Codex will then report that it did not search, which reflects the prompt's constraints, not its actual access. Require resolvable citations and verify them. The MCP call's own parameters impose the same constraint silently, and this is the easier mistake to make: sandbox: read-only blocks network at the shell layer, workspace-write leaves network_access off by default, and approval-policy: never removes any path to escalate. A retrieval task launched that way is crippled no matter how carefully the prompt is worded, and it usually degrades partially — Codex still answers, from a narrower channel — rather than failing loudly. For any task that must retrieve, pass sandbox: danger-full-access. Treat a Codex report of blocked access as a launch-configuration bug to fix and relaunch; that relaunch is a corrected call, not the one retry allowed under Timeout. Its "I could not find X" is never evidence of absence.
  • Deliverable transport: the MCP result returns only Codex's final turn, so a long answer delivered as chat in an earlier turn is silently lost. For anything substantial, instruct Codex to write its deliverable to a named file and reply with just the path. If a result comes back looking like housekeeping, a stray fragment, or an off-topic note, the real work probably exists on disk — go look before concluding it failed or relaunching it.
  • Clarification routing: Codex handles its own underspecification by stating assumptions (L1); Claude relays only facts already stated by the user or already measured (L2); ask the user only when the missing information changes the research question, scientific assumptions, or acceptable tradeoffs (L3). Never invent facts or user preferences to satisfy Codex.
  • Timeout: slow ≠ failed. On genuine failure retry at most ONCE in a fresh thread; if that fails too, proceed with Claude's own analysis and state explicitly that independent validation was unavailable.
  • Disagreement: Claude/Codex disagreements that affect experimental conclusions, result credibility, or direction are key decisions — surface both positions with trade-offs via AskUserQuestion. Do not self-arbitrate.
<!-- END CANONICAL BLOCK -->

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.