docs-mirror
Skill0 starsChanged 48 days agoMITView on GitHub
Mirror a documentation site or an RSS/Atom feed to local markdown in one command. Use when you need two or more pages from one site, when the user says scrape, mirror, or grab the docs, or when the target is a blog or feed. Not for a single page.
In this repository17 other files it gives its agents.
- AGENTS.mdAGENTS.md
- skills/humanizer/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- cherrypickerSkill
- copyable-markdownSkill
- deltamaxxSkill
- ffmpegSkill
- humanizerSkill
- prompt-packSkill
- socials-mirrorSkill
- spamcheckSkill
- tool-call-repairSkill
- writing-beatsSkill
- writing-for-agentsSkill
- writing-fragmentsSkill
- writing-shapeSkill
- yt-ccSkill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 28
Prefer a feed to a crawl for any blog: one request for the whole archive.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
docs-mirror
Two or more pages from one site, or any blog or feed, means run this once and read the files it writes.
python3 scripts/docs-mirror.py <url>It prints the output directory on the last line. Read README.md there first (every page with title and size), then open only the files you need. Re-running is nearly free: each response is hashed and unchanged pages are skipped, so run it again rather than assuming a mirror is current.
| Flag | Use |
--out DIR | where to write. Default $DOCS_MIRROR_BASE/<host>-docs (or <host>-feed in feed mode), falling back to $XDG_DATA_HOME/docs-mirror, then ~/.local/share/docs-mirror. Set DOCS_MIRROR_BASE to a synced folder (Dropbox, Syncthing) so every machine and agent reads one copy |
--only TEXT | only URLs (docs mode) or titles (feed mode) containing TEXT |
--jobs N | parallel requests, default 8 |
--cap N | max pages/entries, default 400 |
--feed | treat the URL as a feed, or go find the site's feed |
--delay S | seconds between requests, shared across all jobs. Default 0.3 |
--ignore-robots | skip robots.txt. Only for a site you own |
Blogs and feeds
Prefer a feed to a crawl for any blog: one request for the whole archive. A feed URL is auto-detected when it ends .xml, .rss, .atom, /feed, /rss, /atom, or the body starts as XML with <rss or <feed>.
docs-mirror https://blog.example.com/index.xml # feed, auto-detected
docs-mirror https://blog.example.com --feed # find the feed for me--feed against a bare domain tries, in order: the page's <link rel="alternate" type="application/rss+xml"> autodiscovery tag, then /index.xml, /feed, /feed.xml, /rss.xml, /atom.xml.
Most feeds carry the whole article (RSS <content:encoded> or <description>, Atom <content>), so a 20-post blog is normally one HTTP request, not 21. The script uses the feed body when it is substantial and fetches the page only when the feed shipped a summary; the README reports which happened per post. RSS 2.0 and Atom both handled. Feed mode writes YYYY-MM-DD-title-slug.md with frontmatter title, date, author, url, feed (metadata a crawl cannot recover).
Measured on a 20-post engineering blog, 2026-09-07: 38,728 words, one request.
Scrape kindly
Both defaults hold unless the user says otherwise:
- robots.txt is fetched and honored. A disallowed start URL exits 4 and stops. In docs mode, excluded URLs are dropped from the crawl and the count is printed. A short mirror is the answer, not a reason for
--ignore-robots. - Requests are spaced by
--delay, shared across workers. The clock is global, so--jobs 12cannot defeat it. Raise it for a small site.
Exit codes
| Code | Meaning | Do this |
| 0 | Pages written | Read them |
| 1 | Every page failed | Check the URL and network, and report the failure rather than empty docs |
| 2 | Nothing discovered | Not a docs root or an empty feed. Try the parent path |
| 3 | Pages returned no prose | The site renders client-side and the content is not in the HTML. Use a headless browser or the site's API |
| 4 | robots.txt says no | Stop, unless the user owns the site |
The script never writes an empty file and calls it success, so an empty file is a bug to report, not an empty page.
When a mirror comes back short
Exit 2 or 3, or a page count far below what the site's nav shows, is usually the start URL, not the site. Check these before reaching for a headless browser:
- A vanity host that redirects. Some doc hosts serve a meta-refresh stub under a custom domain and keep the pages on a generated hostname. The stub has no links, so discovery finds one page.
curl -sSL -o /dev/null -w '%{url_effective}\n' <url>gives the real host; re-run against that. - The sitemap is not where you looked. The script probes the start path and each parent before the origin root, so pass the path the docs actually live under (
/latest,/docs/v2) rather than the bare domain. - The nav is client-rendered. Then the crawl sees a handful of links and the sitemap is the only complete list. No sitemap and no links means the site really does need a browser.
Empty pages on the source site are a real result. Report the count rather than treating them as failures.
Why this exists
Fetching pages one at a time costs one model call per page, and the model re-emits every page. Server-rendered sites (Next.js, Mintlify, Docusaurus, VitePress, MkDocs) put the prose in the first HTTP response, so mirroring is cheap HTTP plus local parsing and no page passes through a model unless you read it. Measured on a 53-page product docs site, 2026-09-02: 0 failures, one command.
Portability
One Python 3 file, standard library only: no pip, node, browser, or API key. Same on macOS and Linux. Tests cover the HTML-to-markdown conversion (tables, fenced code, entity handling, index rows) and run offline in under a second:
python3 scripts/test_docs_mirror.pyDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.