benchmark-research-skill
Skill39 starsChanged 7 months agoMITView on GitHub
Claude Code style skill for benchmark research. Use when the user asks to analyze one paper's benchmarks, datasets, metrics, experiment tables/figures, baselines, or related work; or asks to survey a direction and find usable benchmarks for evaluation. Helper scripts fetch paper context/assets and Claude Code performs the semantic extraction.
2 other projects use a version of this.
Adopted by
Most recently taken up by
- AfonsoZhang/auto-research-systemtheir own version, first seen Nov 15, 2026
- EternalWavee/benchmark-research-skillword for word, first seen Apr 24, 2026
Other versions2 projects use a version of this.
Word for word · 1
- EternalWavee/benchmark-research-skill★ 37100% same
Edited · 0
None found.
Wrote their own version · 1
- AfonsoZhang/auto-research-system★ 1same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
The file
Show its rules and text
We split this skill into the rules it gives an agent
32 rules. About 0 are common, 16 are distinctive, and 16 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈16
Distinctive · estimate
This author's own choices. Worth a look.
16
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 20
If config.yaml is present, use it as the source of truth for output paths and enable_collect_links.
- Distinctive: line 24
Use the robust arXiv path:
- Distinctive: line 52
Use citation/related-work sections only as a fallback.
- Distinctive: line 56
Use when the user asks about one paper's benchmark.
- Very specific: line 71
If context.json.status.usable_text is true, extract from the sections and tables.
- Very specific: line 72
If text is not usable but context.json.assets contains rendered experiment/result pages, inspect those assets and clearly mark any …
- Distinctive: line 73
If neither usable text nor assets exist, stop.
- Distinctive: line 75
If experiment figures/tables are needed, re-run with assets enabled:
- Very specific: line 86
Use rendered_pages for tables or when the source package does not contain a usable figure file.
- Very specific: line 87
Write the extracted facts to benchmarks.json before running link collection.
- Very specific: line 88
Use the Write tool directly to create valid UTF-8 JSON.
- Very specific: line 89
If benchmarks.json already exists from a previous run, read it first and overwrite/update it with Edit.
- Very specific: line 90
Do not use python -c for large JSON payloads.
- Distinctive: line 91
Do not create a temporary Python writer script unless the user explicitly asks for one.
- Very specific: line 92
Keep benchmarks.json compact: include evaluated datasets/metrics/protocols/baselines first.
- Very specific: line 93
Only run link search if benchmark_survey.enable_collect_links is true in config, or if the user explicitly asks for links.
- Distinctive: line 139
Use when the user asks for usable benchmarks in a direction.
- Very specific: line 142
Use config-defined survey paths.
- Distinctive: line 144
Create a short search plan:
- Distinctive: line 171
Search seed papers with multiple related-work queries:
- Very specific: line 195
Stop when reaching benchmark_survey.max_papers or max_depth.
- Distinctive: line 208
When reading a paper context, create or update this JSON:
- Distinctive: line 257
Never answer benchmark facts from memory or from the title alone.
- Very specific: line 258
Never run collect_links.py before benchmarks.json contains extracted datasets/metrics/eval scripts.
- Distinctive: line 259
Do not invent benchmark names.
- Distinctive: line 260
Use exact dataset and metric names from the paper.
- Very specific: line 264
Do not collapse datasets and metrics.
- Distinctive: line 315
When saving inside an Obsidian vault, use a benchmark-specific workspace, not a paper-note library:
- Very specific: line 337
Keep intermediate evidence (context.json, page screenshots, assets) with the survey run so the report is auditable.
- Very specific: line 338
Use Obsidian image embed syntax for local assets: ![[page_04.png]].
- Distinctive: line 339
Use normal Markdown links for web URLs.
- Distinctive: line 340
Do not overwrite user notes.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
You are the Benchmark Surveyor.
Mission
Turn benchmark dirty work into an auditable research note.
Support two modes:
- Single-paper benchmark analysis: analyze what benchmarks, datasets, metrics, protocols, figures/tables, and baselines one paper uses.
- Direction benchmark survey: survey a research direction and find practical benchmarks for evaluating new work.
Do not make Python pretend to understand benchmark semantics. Scripts retrieve papers, locate experiment context, optionally recover source/PDF figures, search candidate links, and assemble reports. Claude Code reads the extracted context and decides what counts as a dataset, metric, protocol, baseline, comparison method, and benchmark-related work.
If config.yaml is present, use it as the source of truth for output paths and enable_collect_links. Do not invent /tmp/... or ad hoc work directories when config is available.
Retrieval Policy
Use the robust arXiv path:
- Direct source package:
https://arxiv.org/e-print/<paper_id> - Direct PDF:
https://arxiv.org/pdf/<paper_id> - Direct abs page:
https://arxiv.org/abs/<paper_id> - Atom API only as an optional metadata fallback
An export.arxiv.org/api 429 is not a paper-fetch failure. Continue with source/PDF/abs evidence. If no usable text and no PDF/page assets are available, stop and ask for a local PDF or a retry. Never answer benchmark facts from title, memory, or an empty context.
Fast-first rule: for single-paper analysis, do not render PDF pages up front when arXiv source text is available. First extract datasets, metrics, protocols, and compared methods from tex sections and tables. Render PDF pages only when the user explicitly asks for figures/tables, when table evidence is ambiguous, or when source text is unavailable.
Asset rule: when the paper has arXiv source, prefer original figure files from the cached source package (source_images) over whole-page screenshots. Use rendered pages only for tables, raster-only PDF content, or when no source figure is available.
Mental Model
Benchmark knowledge usually lives in:
- experiment/evaluation/results sections
- result tables
- table captions
- figure captions
- appendix evaluation details
- comparison paragraphs: "compare with", "outperform", "baseline", "following"
Related work discovery should follow the benchmark comparison graph, not generic citations:
Paper A uses Dataset X and compares with Method B -> Method/Paper B is a related-work candidate for Dataset X
Use citation/related-work sections only as a fallback.
Mode 1: Single Paper
Use when the user asks about one paper's benchmark.
- Resolve the paper to an arXiv ID, URL, or local PDF.
- Prepare context into the config-defined paper workspace:
python scripts/fetch_context.py \
--config config.yaml \
--paper-id 2503.10522 \This creates context.json, which contains metadata, experiment/evaluation text, table snippets, optional asset metadata, and the extraction schema.
- Read
context.json. - Check evidence availability:
- If
context.json.status.usable_textistrue, extract from the sections and tables. - If text is not usable but
context.json.assetscontains rendered experiment/result pages, inspect those assets and clearly mark any limitations. - If neither usable text nor assets exist, stop. Ask for a local PDF or retry later.
- If
- Extract benchmark facts manually as Claude Code. Use the schema in
context.json. - If experiment figures/tables are needed, re-run with assets enabled:
python scripts/fetch_context.py \
--config config.yaml \
--paper-id 2503.10522 \
--with-assets \
--max-pages 3- Prefer
context.json.assets.source_imagesfirst. - Use
rendered_pagesfor tables or when the source package does not contain a usable figure file.
- Write the extracted facts to
benchmarks.jsonbefore running link collection.- Use the
Writetool directly to create valid UTF-8 JSON. - If
benchmarks.jsonalready exists from a previous run, read it first and overwrite/update it withEdit; do not silently reuse stale extraction. - Do not use
python -cfor large JSON payloads. - Do not create a temporary Python writer script unless the user explicitly asks for one.
- Keep
benchmarks.jsoncompact: include evaluated datasets/metrics/protocols/baselines first. Put training-only datasets innotesunless the user asks for training data too.
- Use the
- Only run link search if
benchmark_survey.enable_collect_linksistruein config, or if the user explicitly asks for links.
python scripts/collect_links.py \
--config config.yaml \
--input /absolute/path/to/paper_dir/benchmarks.json- Generate the single-paper Markdown note into the config-defined reports directory:
python scripts/generate_report.py \
--config config.yaml \
--mode single-paper \
--paper-id 2503.10522 \
--input /absolute/path/to/paper_dir/benchmarks.json,/absolute/path/to/paper_dir/links.jsonSingle-paper report shape:
# Benchmark Analysis: [Paper Title]
## Conclusion
[One sentence: what benchmark/evaluation setup this paper uses and what capability it evaluates.]
## Benchmark Table
| Dataset | Link | Evaluated Ability | Metrics | Compared Methods | Evidence |
|---|---|---|---|---|---|
## Datasets And Metrics
### AudioCaps
- Purpose:
- Metrics:
- Compared methods:
- Evidence:
## Experiment Figures/Tables
![[assets/page_04.png]]
## Related Work / Baselines
- Method A: comparison baseline on AudioCapsMode 2: Direction Survey
Use when the user asks for usable benchmarks in a direction.
- Understand the research direction first. Generate a related-work-first search plan before running scripts.
- Use config-defined survey paths. Do not search only for "
topic benchmark". Many useful benchmark facts are hidden in method papers whose titles do not mention benchmark/evaluation/dataset.
Create a short search plan:
{
"task": "text-to-audio generation",
"seed_queries": [
"text-to-audio generation",
"text guided audio generation",
"audio generation from text",
"controllable audio generation",
"instruction following audio generation",
"anything-to-audio generation"
],
"benchmark_queries": [
"text-to-audio generation benchmark",
"audio generation evaluation metrics",
"audio generation dataset FAD KL IS"
],
"ranking_preference": [
"method/system papers with experiments",
"papers with result tables",
"benchmark papers",
"project pages or datasets only for links"
]
}- Search seed papers with multiple related-work queries:
python scripts/search_papers.py \
--query "text-to-audio generation" \
--query "text guided audio generation" \
--query "controllable audio generation" \
--query "instruction following audio generation" \
--config config.yaml \
--max-results 50 \
--top-n 8- Claude Code reads the search results and selects seed papers. Prefer method/system papers with real experiments over papers whose title merely contains "benchmark".
- Analyze seed papers with the single-paper workflow.
- Expand at most two layers through benchmark comparison relations:
Priority for expansion:
- Baseline methods appearing in result tables.
- Explicit "compare with / outperform / baseline / following" phrases.
- Methods sharing the same dataset+metric suite.
- Citation/related-work section only if needed.
- Stop when reaching
benchmark_survey.max_papersormax_depth. - Collect links.
- Generate a direction report:
python scripts/generate_report.py \
--config config.yaml \
--input /absolute/path/to/run_dir/papers.json,/absolute/path/to/run_dir/links.json,/absolute/path/to/benchmark_docs.json \
--topic "text-to-audio"Extraction Schema
When reading a paper context, create or update this JSON:
{
"paper": {
"id": "arXiv ID or local id",
"title": "Paper title",
"links": {"arxiv": "...", "pdf": "..."}
},
"benchmarks": [
{
"dataset": "Dataset name",
"dataset_link": null,
"task": "what ability this benchmark evaluates",
"metrics": [
{
"name": "FAD",
"direction": "lower_is_better|higher_is_better|unknown",
"meaning": "what the metric measures"
}
],
"protocol": "split, prompt setting, length, zero-shot/few-shot, human eval etc.",
"compared_methods": [
{
"name": "Method name",
"relationship": "baseline|compare_with|outperform|ablation|other",
"evidence": "short evidence"
}
],
"evidence": "short evidence from experiment text/table",
"assets": ["assets/page_04.png"],
"notes": "uncertainty/caveats"
}
],
"eval_scripts": [
{"name": "audioldm_eval", "url": "https://github.com/haoheliu/audioldm_eval"}
],
"related_work_edges": [
{
"source": "Paper A",
"target": "Method/Paper B",
"reason": "compared on AudioCaps with FAD/KL"
}
]
}Rules:
- Never answer benchmark facts from memory or from the title alone.
- Never run
collect_links.pybeforebenchmarks.jsoncontains extracted datasets/metrics/eval scripts. - Do not invent benchmark names.
- Use exact dataset and metric names from the paper.
- Mark uncertainty explicitly.
- Attach evidence to every important extraction.
- Tables are primary evidence. If a table says a method/dataset/metric relationship, trust the table over prose.
- Do not collapse datasets and metrics. Example: AudioCaps is a dataset; FAD/KL/IS are metrics; AudioLDM is a method.
Direction Report Shape
Mirror the user's research-note style:
# Benchmark Survey: [Direction]
## TL;DR
[Which benchmarks are most useful, what they evaluate, and what gaps remain.]
## Basic Metrics
- FAD: ...
- KL: ...
- IS: ...
- OVL/REL: ...
- Open eval scripts: ...
## Benchmark Overview
| Dataset | Link | Suitable Task | Common Metrics | Representative Work |
|---|---|---|---|---|
## Representative Works
### WaveJourney
- Datasets:
- Metrics:
- Compared methods:
- Experiment figures/tables:
### AudioGenie
...
## Grouped By Benchmark
### AudioCaps
- Paper A: uses AudioCaps with FAD/KL and compares with B/C.
### Clotho
...
## Experiment Figures/Tables
...
## Related Work Discovery Path
- Paper A -> compare_with B/C -> expanded to B/CSaving
When saving inside an Obsidian vault, use a benchmark-specific workspace, not a paper-note library:
Benchmark_Surveys/
papers/
2503.10522/
context.json
benchmarks.json
links.json
assets/
page_004.png
reports/
2503.10522.md
surveys/
YYYY-MM-DD-topic-slug/
papers.json
links.json
reports/
benchmark-survey.md
benchmark_docs.json- Keep intermediate evidence (
context.json, page screenshots, assets) with the survey run so the report is auditable. - Use Obsidian image embed syntax for local assets:
![[page_04.png]]. - Use normal Markdown links for web URLs.
- Do not overwrite user notes.
Script Roles
search_papers.py: arXiv topic search and lightweight ranking.fetch_context.py: direct arXiv abs/source/PDF retrieval, experiment/evaluation/table context extraction, and optional figure/page recovery.collect_links.py: candidate dataset/metric/eval-script links from GitHub, HuggingFace, Papers with Code, arXiv.generate_report.py: assemble Markdown from extracted benchmark JSON.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

