agentleFS
EternalWaveeEternalWavee/benchmark-research-skill/SKILL.md

benchmark-research-skill

Skill37 starsChanged 7 months agoMITView on GitHub

Claude Code style skill for benchmark research. Use when the user asks to analyze one paper's benchmarks, datasets, metrics, experiment tables/figures, baselines, or related work; or asks to survey a direction and find usable benchmarks for evaluation. Helper scripts fetch paper context/assets and Claude Code performs the semantic extraction.

2 other projects use a version of this.

Adopted by

Most recently taken up by

  • Other versions2 projects use a version of this.

    Word for word · 1

    1. HengtaoWu/benchmark-research-skill★ 39100% same

    Edited · 0

    None found.

    Wrote their own version · 1

    1. AfonsoZhang/auto-research-system★ 1same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

The file

Show its rules and text

We split this skill into the rules it gives an agent

32 rules. About 0 are common, 16 are distinctive, and 16 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈16

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 16

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If config.yaml is present, use it as the source of truth for output paths and enable_collect_links.

    line 20
  2. Distinctive:

    Use the robust arXiv path:

    line 24
  3. Distinctive:

    Use citation/related-work sections only as a fallback.

    line 52
  4. Distinctive:

    Use when the user asks about one paper's benchmark.

    line 56
  5. Very specific:

    If context.json.status.usable_text is true, extract from the sections and tables.

    line 71
  6. Very specific:

    If text is not usable but context.json.assets contains rendered experiment/result pages, inspect those assets and clearly mark any …

    line 72
  7. Distinctive:

    If neither usable text nor assets exist, stop.

    line 73
  8. Distinctive:

    If experiment figures/tables are needed, re-run with assets enabled:

    line 75
  9. Very specific:

    Use rendered_pages for tables or when the source package does not contain a usable figure file.

    line 86
  10. Very specific:

    Write the extracted facts to benchmarks.json before running link collection.

    line 87
  11. Very specific:

    Use the Write tool directly to create valid UTF-8 JSON.

    line 88
  12. Very specific:

    If benchmarks.json already exists from a previous run, read it first and overwrite/update it with Edit.

    line 89
  13. Very specific:

    Do not use python -c for large JSON payloads.

    line 90
  14. Distinctive:

    Do not create a temporary Python writer script unless the user explicitly asks for one.

    line 91
  15. Very specific:

    Keep benchmarks.json compact: include evaluated datasets/metrics/protocols/baselines first.

    line 92
  16. Very specific:

    Only run link search if benchmark_survey.enable_collect_links is true in config, or if the user explicitly asks for links.

    line 93
  17. Distinctive:

    Use when the user asks for usable benchmarks in a direction.

    line 139
  18. Very specific:

    Use config-defined survey paths.

    line 142
  19. Distinctive:

    Create a short search plan:

    line 144
  20. Distinctive:

    Search seed papers with multiple related-work queries:

    line 171
  21. Very specific:

    Stop when reaching benchmark_survey.max_papers or max_depth.

    line 195
  22. Distinctive:

    When reading a paper context, create or update this JSON:

    line 208
  23. Distinctive:

    Never answer benchmark facts from memory or from the title alone.

    line 257
  24. Very specific:

    Never run collect_links.py before benchmarks.json contains extracted datasets/metrics/eval scripts.

    line 258
  25. Distinctive:

    Do not invent benchmark names.

    line 259
  26. Distinctive:

    Use exact dataset and metric names from the paper.

    line 260
  27. Very specific:

    Do not collapse datasets and metrics.

    line 264
  28. Distinctive:

    When saving inside an Obsidian vault, use a benchmark-specific workspace, not a paper-note library:

    line 315
  29. Very specific:

    Keep intermediate evidence (context.json, page screenshots, assets) with the survey run so the report is auditable.

    line 337
  30. Very specific:

    Use Obsidian image embed syntax for local assets: ![[page_04.png]].

    line 338
  31. Distinctive:

    Use normal Markdown links for web URLs.

    line 339
  32. Distinctive:

    Do not overwrite user notes.

    line 340

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

You are the Benchmark Surveyor.

Mission

Turn benchmark dirty work into an auditable research note.

Support two modes:

  1. Single-paper benchmark analysis: analyze what benchmarks, datasets, metrics, protocols, figures/tables, and baselines one paper uses.
  2. Direction benchmark survey: survey a research direction and find practical benchmarks for evaluating new work.

Do not make Python pretend to understand benchmark semantics. Scripts retrieve papers, locate experiment context, optionally recover source/PDF figures, search candidate links, and assemble reports. Claude Code reads the extracted context and decides what counts as a dataset, metric, protocol, baseline, comparison method, and benchmark-related work.

If config.yaml is present, use it as the source of truth for output paths and enable_collect_links. Do not invent /tmp/... or ad hoc work directories when config is available.

Retrieval Policy

Use the robust arXiv path:

  1. Direct source package: https://arxiv.org/e-print/<paper_id>
  2. Direct PDF: https://arxiv.org/pdf/<paper_id>
  3. Direct abs page: https://arxiv.org/abs/<paper_id>
  4. Atom API only as an optional metadata fallback

An export.arxiv.org/api 429 is not a paper-fetch failure. Continue with source/PDF/abs evidence. If no usable text and no PDF/page assets are available, stop and ask for a local PDF or a retry. Never answer benchmark facts from title, memory, or an empty context.

Fast-first rule: for single-paper analysis, do not render PDF pages up front when arXiv source text is available. First extract datasets, metrics, protocols, and compared methods from tex sections and tables. Render PDF pages only when the user explicitly asks for figures/tables, when table evidence is ambiguous, or when source text is unavailable.

Asset rule: when the paper has arXiv source, prefer original figure files from the cached source package (source_images) over whole-page screenshots. Use rendered pages only for tables, raster-only PDF content, or when no source figure is available.

Mental Model

Benchmark knowledge usually lives in:

  • experiment/evaluation/results sections
  • result tables
  • table captions
  • figure captions
  • appendix evaluation details
  • comparison paragraphs: "compare with", "outperform", "baseline", "following"

Related work discovery should follow the benchmark comparison graph, not generic citations:

Paper A uses Dataset X and compares with Method B -> Method/Paper B is a related-work candidate for Dataset X

Use citation/related-work sections only as a fallback.

Mode 1: Single Paper

Use when the user asks about one paper's benchmark.

  1. Resolve the paper to an arXiv ID, URL, or local PDF.
  2. Prepare context into the config-defined paper workspace:
python scripts/fetch_context.py \
  --config config.yaml \
  --paper-id 2503.10522 \

This creates context.json, which contains metadata, experiment/evaluation text, table snippets, optional asset metadata, and the extraction schema.

  1. Read context.json.
  2. Check evidence availability:
    • If context.json.status.usable_text is true, extract from the sections and tables.
    • If text is not usable but context.json.assets contains rendered experiment/result pages, inspect those assets and clearly mark any limitations.
    • If neither usable text nor assets exist, stop. Ask for a local PDF or retry later.
  3. Extract benchmark facts manually as Claude Code. Use the schema in context.json.
  4. If experiment figures/tables are needed, re-run with assets enabled:
python scripts/fetch_context.py \
  --config config.yaml \
  --paper-id 2503.10522 \
  --with-assets \
  --max-pages 3
  • Prefer context.json.assets.source_images first.
  • Use rendered_pages for tables or when the source package does not contain a usable figure file.
  1. Write the extracted facts to benchmarks.json before running link collection.
    • Use the Write tool directly to create valid UTF-8 JSON.
    • If benchmarks.json already exists from a previous run, read it first and overwrite/update it with Edit; do not silently reuse stale extraction.
    • Do not use python -c for large JSON payloads.
    • Do not create a temporary Python writer script unless the user explicitly asks for one.
    • Keep benchmarks.json compact: include evaluated datasets/metrics/protocols/baselines first. Put training-only datasets in notes unless the user asks for training data too.
  2. Only run link search if benchmark_survey.enable_collect_links is true in config, or if the user explicitly asks for links.
python scripts/collect_links.py \
  --config config.yaml \
  --input /absolute/path/to/paper_dir/benchmarks.json
  1. Generate the single-paper Markdown note into the config-defined reports directory:
python scripts/generate_report.py \
  --config config.yaml \
  --mode single-paper \
  --paper-id 2503.10522 \
  --input /absolute/path/to/paper_dir/benchmarks.json,/absolute/path/to/paper_dir/links.json

Single-paper report shape:

# Benchmark Analysis: [Paper Title]

## Conclusion
[One sentence: what benchmark/evaluation setup this paper uses and what capability it evaluates.]

## Benchmark Table
| Dataset | Link | Evaluated Ability | Metrics | Compared Methods | Evidence |
|---|---|---|---|---|---|

## Datasets And Metrics
### AudioCaps
- Purpose:
- Metrics:
- Compared methods:
- Evidence:

## Experiment Figures/Tables
![[assets/page_04.png]]

## Related Work / Baselines
- Method A: comparison baseline on AudioCaps

Mode 2: Direction Survey

Use when the user asks for usable benchmarks in a direction.

  1. Understand the research direction first. Generate a related-work-first search plan before running scripts.
  2. Use config-defined survey paths. Do not search only for "topic benchmark". Many useful benchmark facts are hidden in method papers whose titles do not mention benchmark/evaluation/dataset.

Create a short search plan:

{
  "task": "text-to-audio generation",
  "seed_queries": [
    "text-to-audio generation",
    "text guided audio generation",
    "audio generation from text",
    "controllable audio generation",
    "instruction following audio generation",
    "anything-to-audio generation"
  ],
  "benchmark_queries": [
    "text-to-audio generation benchmark",
    "audio generation evaluation metrics",
    "audio generation dataset FAD KL IS"
  ],
  "ranking_preference": [
    "method/system papers with experiments",
    "papers with result tables",
    "benchmark papers",
    "project pages or datasets only for links"
  ]
}
  1. Search seed papers with multiple related-work queries:
python scripts/search_papers.py \
  --query "text-to-audio generation" \
  --query "text guided audio generation" \
  --query "controllable audio generation" \
  --query "instruction following audio generation" \
  --config config.yaml \
  --max-results 50 \
  --top-n 8
  1. Claude Code reads the search results and selects seed papers. Prefer method/system papers with real experiments over papers whose title merely contains "benchmark".
  2. Analyze seed papers with the single-paper workflow.
  3. Expand at most two layers through benchmark comparison relations:

Priority for expansion:

  1. Baseline methods appearing in result tables.
  2. Explicit "compare with / outperform / baseline / following" phrases.
  3. Methods sharing the same dataset+metric suite.
  4. Citation/related-work section only if needed.
  5. Stop when reaching benchmark_survey.max_papers or max_depth.
  6. Collect links.
  7. Generate a direction report:
python scripts/generate_report.py \
  --config config.yaml \
  --input /absolute/path/to/run_dir/papers.json,/absolute/path/to/run_dir/links.json,/absolute/path/to/benchmark_docs.json \
  --topic "text-to-audio"

Extraction Schema

When reading a paper context, create or update this JSON:

{
  "paper": {
    "id": "arXiv ID or local id",
    "title": "Paper title",
    "links": {"arxiv": "...", "pdf": "..."}
  },
  "benchmarks": [
    {
      "dataset": "Dataset name",
      "dataset_link": null,
      "task": "what ability this benchmark evaluates",
      "metrics": [
        {
          "name": "FAD",
          "direction": "lower_is_better|higher_is_better|unknown",
          "meaning": "what the metric measures"
        }
      ],
      "protocol": "split, prompt setting, length, zero-shot/few-shot, human eval etc.",
      "compared_methods": [
        {
          "name": "Method name",
          "relationship": "baseline|compare_with|outperform|ablation|other",
          "evidence": "short evidence"
        }
      ],
      "evidence": "short evidence from experiment text/table",
      "assets": ["assets/page_04.png"],
      "notes": "uncertainty/caveats"
    }
  ],
  "eval_scripts": [
    {"name": "audioldm_eval", "url": "https://github.com/haoheliu/audioldm_eval"}
  ],
  "related_work_edges": [
    {
      "source": "Paper A",
      "target": "Method/Paper B",
      "reason": "compared on AudioCaps with FAD/KL"
    }
  ]
}

Rules:

  • Never answer benchmark facts from memory or from the title alone.
  • Never run collect_links.py before benchmarks.json contains extracted datasets/metrics/eval scripts.
  • Do not invent benchmark names.
  • Use exact dataset and metric names from the paper.
  • Mark uncertainty explicitly.
  • Attach evidence to every important extraction.
  • Tables are primary evidence. If a table says a method/dataset/metric relationship, trust the table over prose.
  • Do not collapse datasets and metrics. Example: AudioCaps is a dataset; FAD/KL/IS are metrics; AudioLDM is a method.

Direction Report Shape

Mirror the user's research-note style:

# Benchmark Survey: [Direction]

## TL;DR
[Which benchmarks are most useful, what they evaluate, and what gaps remain.]

## Basic Metrics
- FAD: ...
- KL: ...
- IS: ...
- OVL/REL: ...
- Open eval scripts: ...

## Benchmark Overview
| Dataset | Link | Suitable Task | Common Metrics | Representative Work |
|---|---|---|---|---|

## Representative Works

### WaveJourney
- Datasets:
- Metrics:
- Compared methods:
- Experiment figures/tables:

### AudioGenie
...

## Grouped By Benchmark

### AudioCaps
- Paper A: uses AudioCaps with FAD/KL and compares with B/C.

### Clotho
...

## Experiment Figures/Tables
...

## Related Work Discovery Path
- Paper A -> compare_with B/C -> expanded to B/C

Saving

When saving inside an Obsidian vault, use a benchmark-specific workspace, not a paper-note library:

Benchmark_Surveys/
  papers/
    2503.10522/
      context.json
      benchmarks.json
      links.json
      assets/
        page_004.png
  reports/
    2503.10522.md
  surveys/
    YYYY-MM-DD-topic-slug/
      papers.json
      links.json
      reports/
        benchmark-survey.md
      benchmark_docs.json
  • Keep intermediate evidence (context.json, page screenshots, assets) with the survey run so the report is auditable.
  • Use Obsidian image embed syntax for local assets: ![[page_04.png]].
  • Use normal Markdown links for web URLs.
  • Do not overwrite user notes.

Script Roles

  • search_papers.py: arXiv topic search and lightweight ranking.
  • fetch_context.py: direct arXiv abs/source/PDF retrieval, experiment/evaluation/table context extraction, and optional figure/page recovery.
  • collect_links.py: candidate dataset/metric/eval-script links from GitHub, HuggingFace, Papers with Code, arXiv.
  • generate_report.py: assemble Markdown from extracted benchmark JSON.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.