agentleFS
sergeeeysergeeey/Claude-cod-top-2026/skills/extensions/novelty-assessment/SKILL.md

novelty-assessment

Skill9 starsChanged 6 months agoMITView on GitHub

Assess research idea novelty through systematic literature search. DUAL independent screening in at least two vocabularies (own phrasing + adjacent-field terms), scored by relative recall against a pre-specified benchmark set. Three outcomes — novel / not_novel / recall_failure — because "nothing found" is usually a wrong-vocabulary search, not an empty niche. Use before committing to a research direction.

3 other projects use a version of this, 3 of them edited.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

6 rules. 2 are common, 3 are distinctive, and 1 is very specific to this project.

  • 2

    Common

    Most of the 3 edited versions we read keep these.

  • 3

    Distinctive

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If you have a novelty_check.py-style script installed:

    line 20
  2. Distinctive:

    Screen: every work not already *seen* (seen = the union of the S1 and S2 result lists) by title and abstract — a title-only screen does not …

    line 126
  3. Common:

    Consider both methodology novelty AND application novelty

    line 266
  4. Common:

    Check for concurrent/recent arXiv submissions

    line 267
  5. Distinctive:

    If the idea is a *method* rather than a *finding*, also search practitioner literature (library science, SE practice, clinical guidelines) …

    line 268
  6. Distinctive:

    Read this if the dual-screening step feels like overhead.

    line 276

“Common” is measured: kept by at least 6 in 10 of the 3 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Novelty Assessment

Rigorously assess whether a research idea is novel through systematic literature search.

Input

  • $0 — Research idea description, title, or JSON file

Scripts

Automated novelty check (external script optional — not shipped by this repo)

If you have a novelty_check.py-style script installed:

python novelty_check.py \
  --idea "Your research idea description" \
  --max-rounds 10 --output novelty_report.json

Otherwise, run the multi-round search-evaluate loop described in the Workflow below manually, using WebSearch/WebFetch for each round.

Literature search (external script optional — not shipped by this repo)

If you have a search_semantic_scholar.py-style script installed:

python search_semantic_scholar.py \
  --query "relevant search query" --max-results 10

Otherwise, use WebFetch https://api.semanticscholar.org/graph/v1/paper/search?query=....

References

  • Assessment prompts and criteria: ~/.claude/skills/novelty-assessment/references/assessment-prompts.md

Workflow

Step 1: Understand the Idea

  • Identify the core contribution
  • List the key technical components
  • Determine the research area and subfield

Step 1.5: Vocabulary Translation (MANDATORY — do this before any search)

Searching in your own words is the dominant failure mode of this skill. The adjacent field names the same thing differently, so a clean result proves the term is free, not the question.

Name at least three adjacent fields and write how each would phrase this idea:

Own phrasing:      "<the idea in your words>"
Field A (<name>):  "<their term>"
Field B (<name>):  "<their term>"
Field C (<name>):  "<their term>"

Standard candidate fields: information retrieval · meta-research / scientometrics · software engineering (field studies) · human factors / automation · clinical decision support · safety science · the idea's own home discipline.

Worked failures this rule comes from — all real, all one session: "retraction counts understate overturning" missed Ioannidis's "Persistence of contradicted claims" · "falsification protocol with a skeptic role" missed "Abyss Falsifier" · "cross-individual benchmark" missed "modality gap".

Then pre-specify the benchmark set now, before Step 2 launches: 3–5 known-adjacent papers that a sensitive search MUST return. Steps 2.3 and 2.5 use it; it is fixed here so it cannot be chosen after seeing what the screeners found.

Step 2: DUAL SCREENING (two independent screeners — not one, not sequential)

Why two. Measured in systematic-review methodology: single-reviewer screening misses 13% of relevant studies; dual screening misses 3%. A single agent running N rounds is single-reviewer screening no matter how large N is.

Launch two screeners in parallel, and keep them blind to each other:

screenergetsmust NOT get
S1the idea in the author's own phrasingS2's queries or results
S2the idea restated in the adjacent-field vocabulary from Step 1.5, with the author's phrasing withheldS1's queries or results

Independence is the whole mechanism — if S2 sees S1's queries it anchors on them and the 13%→3% gain is lost. Reconcile only after both have returned.

Each screener runs its own multi-round loop (query → search → review top-10 → assess overlap). Disagreement between S1 and S2 is signal, not noise: it means one vocabulary reaches literature the other does not, and both sets count.

Step 2.3: Citation chase — reach what neither vocabulary names

Keyword screening (S1, S2) finds papers that use your words. Papers that share your ancestry but not your words are reachable only through the citation graph. Run the chase after both screeners have returned (so it cannot anchor either of them) and before scoring recall.

  1. Seeds (3). The 3 papers closest to the idea (topic and method) that either screener returned — overlapping or not — at most 2 per screener, ties broken by the more recent paper. If neither screener returned any paper, use the benchmark set from Step 1.5.
  2. Resolve each seed to an OpenAlex ID: https://api.openalex.org/works/doi:<DOI> (or .../works?search=<exact title>, then check the title matches). A seed that does not resolve is logged resolved: false (unknown, not zero citations) and replaced by the next-closest paper. If the pool runs out before every seed slot is resolved, the chase is incomplete (item 6).
  3. Channels per seed — OpenAlex works filters with &per-page=25; read up to 4 pages (100 results) per pass, and record total and read for each channel:

    channelfilterreturns
    forward, most citedcites:<W-id> + &sort=cited_by_count:descworks that cite the seed (incoming)
    forward, newest (first page only)cites:<W-id> + &sort=publication_date:descthe latest citing works — new competitors have few citations and drop out of the most-cited pass
    backwardcited_by:<W-id>works the seed cites — its references (outgoing)
    relatedrelated_to:<W-id>OpenAlex's algorithmic neighbours

    Direction: cites is incoming, cited_by is outgoing. Checked against the OpenAlex docs and a live control on 2026-09-26 (Hijmans 2012, W2168997286): cites returned 646 vs the record's cited_by_count 649; cited_by returned 50 of the record's 54 referenced_works (4 references do not resolve, so coverage is partial); related_to returned 10 = related_works.

  4. Screen every work not already seen (seen = the union of the S1 and S2 result lists) by title and abstract — a title-only screen does not count. If OpenAlex has no abstract for a work, open its landing page or treat the work as borderline. "Overlapping" means the paper already claims or demonstrates the idea's core contribution, or a trivial variant of it (the Step 3 "Not Novel" test); a borderline paper counts as overlapping and is named in the justification. Record it with found_by: "citation_chase".
  5. Score the misses. A close paper (overlapping, or one that belongs in the benchmark set) that the chase surfaced and neither S1 nor S2 returned is a recall miss for both: list it in missed_by_both_screeners and add it to the benchmark set as a post-hoc addition (benchmark_added_post_hoc). It stays in the denominator for the rest of the assessment, and "returned" means returned by S1 or S2 — never by the chase — so relative_recall stays below 1.0 until a re-run of Step 2 returns it.
  6. Completion and coverage. The chase is complete only if every seed slot resolved and every channel of every seed returned a non-error response. Completion is not sufficiency: report read/total per channel, and if any forward channel has more than 100 works (so it was truncated) the decision's confidence cannot be high. An incomplete chase never supports novel: fix and retry once; if it is still incomplete, report recall_failure (Step 3) with citation_chase.skipped_reason saying what could not complete.
  7. Re-runs. Any re-run of Step 2 (Step 2.5's fix-and-re-run) repeats this chase: the seeds and the seen set change.

Limits: OpenAlex coverage of very recent papers, preprints and older non-English work is partial, so an empty chase from a resolved seed is weak evidence, not proof. In the one case tested, the chase did not cross fields. Measured on this skill's own worked example (2026-09-26, without a chase in the original run): a one-hop chase from a regulatory-genomics seed (Fulco 2019, W2991283206; forward 1 207 works, the 100 most-cited read; backward 89/89; related 10/10; the newest-first page was not part of that run) did not reach the ecology paper (Hijmans 2012) that only S2 found. A chase from an S2-vocabulary seed, which starts inside the other field, is untested. Cost: about 11 requests per seed (1 resolve, up to 4 forward most-cited, 1 forward newest, up to 4 backward, 1 related), about 33 for 3 seeds, plus screening the new titles and abstracts (up to ~700 works at full pages).

Step 2.5: Relative Recall against a benchmark set (quantifies "enough searching")

Score the benchmark set pre-specified at the end of Step 1.5 — 3–5 papers that MUST be found if the search is sensitive (papers added post hoc by Step 2.3 count as not returned). Then report:

relative_recall = (benchmark papers returned by S1 or S2) / (pre-specified + added post hoc by Step 2.3)

Recall < 1.0 → the search is not sensitive enough to support a "novel" verdict. Fix the queries and re-run (at most two re-runs; if recall is still below 1.0, report recall_failure). This converts "feels unexplored" into a number.

Step 3: Make Decision

  • Not Novel: any screener or the citation chase (Step 2.3) found significantly overlapping work
  • Novel: BOTH screeners returned nothing overlapping, AND relative recall = 1.0, AND the citation chase is complete (Step 2.3 item 6) and found nothing overlapping
  • RECALL-FAILURE (a third outcome, not a form of "novel"): nothing close was found at all. An idea worth pursuing always has adjacent literature. Zero close hits means the vocabulary is wrong, not that the niche is empty. Return to Step 1.5 — do not report this as novelty. The same outcome is reported when the search cannot be shown sensitive for another reason: recall still below 1.0 after two re-runs, or an incomplete citation chase. Say which in justification, then fix the vocabulary (Step 1.5) or the infrastructure problem (the chase) before any verdict. Precedence: not_novel wins whenever any overlap was found; recall_failure applies only when none was. A recall_failure carries confidence: low.

Step 4: Position the Idea

If novel, identify:

  • Most similar existing papers (for Related Work)
  • How the idea differs from each
  • The specific gap this idea fills

Harsh Critic Persona

Be a harsh critic for novelty. Ensure there is a sufficient contribution
for a new conference or workshop paper. A trivial extension of existing
work is NOT novel. The idea must offer a meaningfully different approach,
formulation, or insight.

Output Format

The block is a schema (| separates alternatives, ... is a placeholder), not a parseable instance.

{
  "decision": "novel" | "not_novel" | "recall_failure",
  "confidence": "high" | "medium" | "low",
  "vocabularies_searched": ["own", "<field A>", "<field B>", "<field C>"],
  "screeners": {
    "S1_own_phrasing":      {"rounds": 4, "overlapping_found": 0},
    "S2_adjacent_vocab":    {"rounds": 4, "overlapping_found": 2},
    "independent": true,
    "agreed": false
  },
  "relative_recall": {
    "benchmark_prespecified": ["...", "...", "..."],
    "benchmark_added_post_hoc": [],
    "benchmark_returned": 3,
    "score": 1.0
  },
  "citation_chase": {
    "seed_source": "screeners" | "benchmark",
    "seeds": [{"title": "...", "openalex_id": "W...", "resolved": true,
               "channels": {"forward_most_cited": {"total": 1207, "read": 100},
                            "forward_newest": {"total": 1207, "read": 25},
                            "backward": {"total": 89, "read": 89},
                            "related": {"total": 10, "read": 10}}}],
    "new_candidates": 41,
    "new_overlapping": 1,
    "missed_by_both_screeners": ["<close paper>"],
    "skipped_reason": null
  },
  "justification": "...",
  "most_similar_papers": [
    {"title": "...", "year": 2024, "overlap": "...", "found_by": "S2"}
  ],
  "differentiation": "Our idea differs because..."
}

decision: "novel" is invalid unless screeners.independent = true, both screeners returned zero overlap, relative_recall.score = 1.0, and the citation chase is complete (Step 2.3 item 6: every seed slot resolved, every channel of every seed returned without error, skipped_reason = null) and found nothing overlapping. skipped_reason may only say why the chase could not complete; a non-null value always blocks novel, and a truncated forward channel caps confidence below high.

Rules

  • Two independent screeners, always. One agent running many rounds is single-reviewer screening (13% miss rate) regardless of round count.
  • ≥3 vocabularies named in Step 1.5 before the first query is issued.
  • Round count is not evidence of sensitivity — relative recall is. The old "minimum 3 rounds" rule is retired: 3 rounds in one vocabulary failed three times in one session (2026-09-01) while feeling thorough.
  • Zero close hits ⇒ recall_failure, never novel. Real ideas have neighbours.
  • Keyword screening finds papers that share your words; the citation graph finds papers that share your ancestry. Complete the chase (Step 2.3) before any novel verdict; an unresolved seed is logged, never skipped silently, and an incomplete chase never supports novel.
  • A paper idea is NOT novel if it's a trivial extension
  • Consider both methodology novelty AND application novelty
  • Check for concurrent/recent arXiv submissions
  • If the idea is a method rather than a finding, also search practitioner literature (library science, SE practice, clinical guidelines) — techniques are often long-standardised somewhere before they are "discovered" in a research context. The "two vocabularies" rule itself turned out to be standard MEDLINE practice (free-text + MeSH) after being written up as an insight.

Worked example — the first real run (2026-09-01)

Read this if the dual-screening step feels like overhead. It is the run that justifies the cost, and the screeners disagreed.

Claim assessed: in enhancer–gene prediction, a distance-only baseline's AUC is a monotone function of the candidate-window half-width (measured 0.6725 → 0.8534 across 25→500 kb on one fixed dataset); therefore ΔAUC across papers is incomparable.

screenervocabularyroundspapers foundverdict
S1own — regulatory genomics810novel ❌
S2ecology · virtual screening · recommenders · link prediction1020+not_novel ✅

S1 was not lazy. Eight rounds, ten papers, honest [UNKNOWN] markers on what it could not verify. It was thorough inside one vocabulary — which is exactly the failure this procedure exists to catch. Thoroughness is not sensitivity.

What only S2 could reach (verified independently before acceptance):

Hijmans, Ecology 93:679–688 (2012) · 10.1890/11-0826.1 · PMID 22624221 — "spatial sorting bias". The null model uses only geographic distance; the fix (pairwise distance sampling, matching negatives' distance distribution to positives') drives its AUC to ≈0.5. Across 226 species the distance-only null beat MaxEnt for 45% and Bioclim for 67%. Shipped in CRAN as dismo::ssb().

A library function for the phenomenon is the strongest possible "already known" signal. The same effect is separately named representativeness effect (with uAUC as its published remedy), artificial enrichment (DUD-E/MUV/LIT-PCBA), and sampler-dependent evaluation (arXiv:1912.02263, arXiv:2607.27861).

Procedure metrics that made the verdict trustworthy:

  • relative_recall = 3/3 = 1.0 — every pre-specified must-find paper was returned, so not_novel rests on a sensitive search rather than on an empty result.
  • A verdict prediction was recorded before the run ("occupied structurally, possibly free specifically") and matched the outcome — making the run falsifiable in both directions instead of retrospectively unsurprising.
  • This run predates Step 2.3: it used two screeners and a recall benchmark only. See the Step 2.3 limits for what a one-hop chase from an S1-style seed would (not) have reached.

Cost accounting, measured here rather than quoted: S1 alone would have returned novel after eight rounds, sending the work into head-on collision with ecology literature from 2008–2012 — discovered at review, not before. The second screener cost one parallel call. That is the 13%→3% gain, observed rather than cited.

What survived — note the shape, it is the usual one: not the discovery, but the transfer. The correction has been standard in species-distribution modelling and virtual screening for ~20 years and is not applied in regulatory genomics. Positioning as "we discovered this" collides with prior work; positioning as "here is the sweep, and here is a known remedy this field has not adopted" survives.

Empirical basis for the dual-screening design

claimsource
single-reviewer screening misses ~13% of relevant studies; dual screening ~3%systematic-review screening methodology
92.7% of search strategies in 137 systematic reviews contained errors; 78.1% of those affected recall; commonest error = missing terms in both natural and controlled languageSalvador-Oliván, Marco-Cuenca, Arquero-Avilés, JMLA 2019 — 10.5195/jmla.2019.567, PMID 31019390
relative recall against a pre-specified benchmark set as the standard way to evaluate search-string sensitivityPMC12621535
falsely-excluded studies are common enough to have their own recovery literaturePMC9644550

Rationale for Step 2.3 (an analogy, not measured evidence for this skill): backward citation searching is a MECIR standard for Cochrane systematic reviews and forward searching a recommended adjunct (Briscoe et al. 2020, PMC7079050; Cochrane Handbook ch. 4). That concerns reference lists in systematic reviews; the yield of the same idea for prior-art search is untested here.

Related Skills

  • See also: related-work-writing
  • /hd-mavp-router — calls this as the final step of its negative_space run_mode
  • /negative-space-miner — Stage 8 delegates its mandatory Novelty Check here; without delegation the status caps at closely related

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.