agentleFS
arthur-plautzarthur-plautz/research-toolkit/.claude/skills/fetch-paper/SKILL.md

fetch-paper

Skill0 starsChanged 4 months agoMITView on GitHub

Downloads an academic paper PDF from open-access sources and saves it to docs/papers/ with the correct slug filename. Accepts an arXiv ID, DOI, or free-form title. Optionally bootstraps a reference doc skeleton.

2 other projects use a version of this.

Adopted by

Most recently taken up by

The file

Show its rules and text

We split this skill into the rules it gives an agent

20 rules. About 0 are common, 13 are distinctive, and 7 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈13

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 7

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    All network access in this skill is restricted to the following domains.

    line 8
  2. Distinctive:

    Before querying Unpaywall, always ask the user explicitly:

    line 25
  3. Distinctive:

    Only proceed if the user explicitly consents and provides or confirms their email for this request.

    line 29
  4. Distinctive:

    Do not hardcode, infer, persist, or log any email address in this skill or its outputs.

    line 31
  5. Distinctive:

    If the input was a free-form title and the resolved paper looks wrong (wrong year, different authors), confirm with the user before …

    line 69
  6. Distinctive:

    If the file already exists, report it and stop:

    line 95
  7. Distinctive:

    Verify the file is a valid PDF (not an HTML error page):

    line 113
  8. Very specific:

    If file reports HTML or ASCII text, delete the file and proceed to the next tier.

    line 118
  9. Very specific:

    Verify with file as above.

    line 128
  10. Distinctive:

    Ask the user for consent per the Personal information policy above.

    line 132
  11. Very specific:

    Verify with file as above.

    line 144
  12. Distinctive:

    If all automated tiers fail, ask:

    line 154
  13. Distinctive:

    After a successful download or file copy:

    line 167
  14. Very specific:

    Check if docs/references/{slug}.md exists:

    line 169
  15. Distinctive:

    If it exists: note the path and skip.

    line 174
  16. Distinctive:

    If it does not exist: ask the user:

    line 176
  17. Very specific:

    If yes: copy the template from .claude/skills/references/templates/reference.md, fill in the following fields from the resolved metadata, …

    line 179
  18. Very specific:

    Do not fill in Abstract, Summary, Introduction, Method, Results, Discussion, or Conclusion — leave all as {Not available — paper not read}.

    line 186
  19. Distinctive:

    Never retry in a loop. If a source fails once, move to the next tier.

    line 211
  20. Very specific:

    Ambiguous title slug: use the most recognisable abbreviation (e.g.

    line 219

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Safe URLs

All network access in this skill is restricted to the following domains. Never fetch from any domain not on this list without explicit user confirmation.

DomainPurpose
export.arxiv.orgMetadata resolution via Atom XML
arxiv.orgPDF download
api.semanticscholar.orgPaper search, OA PDF URL, author/year metadata
api.openalex.orgOA URL fallback via DOI
api.unpaywall.orgOA PDF URL fallback via DOI (requires consent — see below)
ar5iv.orgHTML paper render (last automated resort, not a PDF)

Personal information policy

Unpaywall requires an email address as a courtesy identifier in the query string (?email=...). This is the only API in the fallback chain that uses personal information.

Before querying Unpaywall, always ask the user explicitly:

"Unpaywall requires an email address as a usage identifier. May I include your email in this request? (It will be sent as a plain URL parameter to api.unpaywall.org — not stored by this skill.)"

Only proceed if the user explicitly consents and provides or confirms their email for this request. If the user declines, skip Unpaywall and move to the next fallback tier.

Do not hardcode, infer, persist, or log any email address in this skill or its outputs.


Trigger

This skill is invoked explicitly: /fetch-paper {input} where {input} is one of:

  • arXiv ID — e.g. 1703.03864 or arXiv:1703.03864
  • DOI — e.g. 10.1162/106365602320169811
  • Free-form title — e.g. "Evolving Neural Networks through Augmenting Topologies"
  • Direct PDF URL — e.g. https://arxiv.org/pdf/1703.03864

It may also fire automatically when the references skill creates a new reference doc but no corresponding PDF exists in docs/papers/.


Phase 0 — Resolution (Explore agent)

Spawn an Explore subagent (not general-purpose) with this exact brief, substituting {input}. The agent must only access domains listed in the Safe URLs table above.

Given this input — {input} (arXiv ID / DOI / title / URL) — resolve it to a canonical paper record. Query in this order:

  1. If the input contains an arXiv ID: fetch https://export.arxiv.org/abs/{arxiv_id} for metadata. The PDF URL is https://arxiv.org/pdf/{arxiv_id}.
  2. Query Semantic Scholar: https://api.semanticscholar.org/graph/v1/paper/search?query={title_or_id}&fields=title,externalIds,year,authors,openAccessPdf — extract externalIds.ArXiv, externalIds.DOI, year, authors[].name, openAccessPdf.url.
  3. If a DOI is known, query OpenAlex: https://api.openalex.org/works/doi:{doi} — extract open_access.oa_url, title, publication_year, authorships[].author.display_name.
  4. Do not query Unpaywall — it requires personal information consent that must be handled in the main conversation, not delegated to an agent.

Return ONLY a compact record with these fields (omit any that are unavailable):

  • arxiv_id: string or null
  • doi: string or null
  • authors: list of last names only (lowercase)
  • year: 4-digit integer
  • title: full title string
  • oa_pdf_url: direct PDF URL or null

The agent returns one compact record. Do not let it return raw JSON blobs or API response bodies.

If the input was a free-form title and the resolved paper looks wrong (wrong year, different authors), confirm with the user before proceeding.


Phase 1 — Slug construction and duplicate check

From the resolved record, derive the slug using the project's filename convention:

{authors}-{year}-{short-title}
  • authors: all co-authors' last names, hyphen-separated, lowercase (e.g. stanley-miikkulainen, salimans-ho-chen-sidor-sutskever)
  • year: 4-digit publication year
  • short-title: 3–5 word kebab-case slug of the paper title — use editorial judgment, not mechanical truncation

Examples from this project:

  • salimans-2017-es-scalable-alternative-rl
  • stanley-miikkulainen-2002-neat
  • stanley-damprosio-gauci-2009-hyperneat

Check for existing file:

ls docs/papers/{slug}.pdf 2>/dev/null

If the file already exists, report it and stop:

Already saved → docs/papers/{slug}.pdf

Phase 2 — PDF download (native, fallback chain)

Attempt sources in priority order, stopping at the first success. All downloads use native Bash with curl. Only access domains in the Safe URLs table.

Tier 1 — arXiv direct (if arxiv_id is known)

curl -L --max-time 60 -o docs/papers/{slug}.pdf \
  "https://arxiv.org/pdf/{arxiv_id}"

Verify the file is a valid PDF (not an HTML error page):

file docs/papers/{slug}.pdf

If file reports HTML or ASCII text, delete the file and proceed to the next tier.

Apply a 3-second delay before downloading a second arXiv PDF in the same session to respect rate limits.

Tier 2 — OA URL from agent (if oa_pdf_url was returned)

curl -L --max-time 60 -o docs/papers/{slug}.pdf "{oa_pdf_url}"

Verify with file as above. If the OA URL resolves to a domain not in the Safe URLs table, ask the user for confirmation before proceeding.

Tier 3 — Unpaywall (DOI required, consent required)

Ask the user for consent per the Personal information policy above. If consent is given and an email is provided:

curl -s "https://api.unpaywall.org/v2/{doi}?email={user_email}"

Extract best_oa_location.url_for_pdf from the JSON response, then download:

curl -L --max-time 60 -o docs/papers/{slug}.pdf "{oa_pdf_url}"

Verify with file as above.

Tier 4 — ar5iv HTML (arXiv papers only, last automated resort)

https://ar5iv.org/abs/{arxiv_id} renders the paper as HTML — it is not a PDF. Only use this tier if the user only needs to read the paper and does not need a file in docs/papers/. Note the limitation explicitly:

"Could not download a PDF. Retrieved HTML render at ar5iv.org — the paper is readable but no file has been saved to docs/papers/."

Tier 5 — User fallback

If all automated tiers fail, ask:

"No open-access PDF was found for this paper. If you have a local copy, provide the path (e.g. /home/user/Downloads/paper.pdf) and I will copy it to docs/papers/{slug}.pdf."

On user reply with a path:

cp "{user_path}" docs/papers/{slug}.pdf

Phase 3 — Reference doc bootstrap (optional)

After a successful download or file copy:

  1. Check if docs/references/{slug}.md exists:
    ls docs/references/{slug}.md 2>/dev/null
  2. If it exists: note the path and skip.
  3. If it does not exist: ask the user:

    "Create a reference doc skeleton at docs/references/{slug}.md?"

    If yes: copy the template from .claude/skills/references/templates/reference.md, fill in the following fields from the resolved metadata, and set all content sections to {Not available — paper not read}:

    • # {Title} — full paper title
    • **Authors:** — full names (not just last names)
    • **Year:**
    • **Venue:** — journal/conference if known, otherwise arXiv:{arxiv_id} or {Not available}
    • **DOI / URL:** — DOI URL or arXiv abstract URL

    Do not fill in Abstract, Summary, Introduction, Method, Results, Discussion, or Conclusion — leave all as {Not available — paper not read}. Those sections are filled separately after reading the paper.


Phase 4 — Report

Print a compact summary:

Saved  → docs/papers/{slug}.pdf  ({file size})
Ref    → docs/references/{slug}.md  [created | already exists | skipped]
Source → {tier that succeeded: arXiv | OA URL ({source}) | Unpaywall | user-provided}

Rate limits and constraints

SourceLimitHandling
arXiv PDF~1 request/3s recommendedAdd sleep 3 between consecutive downloads
Semantic Scholar1 req/sec unauthenticated (shared pool)Single call only; no retry loop
OpenAlexFair use, no stated limit for single lookupsSingle call only
UnpaywallFree API, single lookupRequires explicit user consent before each use

Never retry in a loop. If a source fails once, move to the next tier.


Slug edge cases

  • Single author: just the last name — syswerda-1991-steady-state-ga
  • Many authors: include all last names — salimans-ho-chen-sidor-sutskever-2017-es-scalable-alternative-rl. If this becomes unwieldy (5+ authors), use first-author + etal — salimans-etal-2017-es-scalable-alternative-rl — but prefer full listing when practical.
  • Ambiguous title slug: use the most recognisable abbreviation (e.g. neat for NeuroEvolution of Augmenting Topologies) rather than mechanically truncating the title.

Interaction with the references skill

The references skill governs when reference docs are created and how they must be written. This skill governs how PDFs are obtained. They are complementary — neither calls the other automatically. A natural follow-up after /fetch-paper is to read the downloaded PDF and fill in the docs/references/{slug}.md sections.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.