seo-site-crawler
Skill0 starsChanged 7 months agoMITView on GitHub
Lightweight site URL discovery + per-page metadata pre internal linking. Tries sitemap.xml first, fallback BFS crawl s trafilatura. Output site_map.json (URL + title + h1 + meta description + canonical) cached per-project (TTL 7d). Auto-fired z $seo-content-brief; manuálne triggers - "rob site map pre <domain>", "skroluj urls na <domain>", "site crawler pre teapot.sk". Pre full technical SEO audit (broken links, alts, structured data) use $seo-tech-audit. Pre Ahrefs domain overview use $seo-audit.
In this repository5 other files it gives its agents.
- remote-mcp-clientSkill
- seo-content-briefSkill
- seo-content-copySkill
- seo-internal-linkingSkill
- seo-tech-auditSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 14
Pre-flight: ensure is_safe_url(domain) passes (SSRF guard)
- Distinctive: line 62
Politeness: default 1.0s delay between requests
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
seo-site-crawler
Lightweight site URL discovery skill. Builds site_map.json ktoré konzumuje $seo-internal-linking (a iné SEO skills) pre semantic-aware internal linking suggestions.
Workflow
- Parse intent: extract
domainz user message (https://...alebobare.sk) - Pre-flight: ensure
is_safe_url(domain)passes (SSRF guard) - Cache check: ak
~/.cache/seo-content/{project_slug}/site-crawler/site_map.jsonexistuje a je <7d → return cached (UNLESS--no-cache) - Try sitemap first (lacné, najrýchlejšie):
uv run --quiet --with httpx --with trafilatura --with lxml \ python ~/.claude/skills/seo-site-crawler/scripts/crawl.py \ <domain> [--max-pages 500] [--max-depth 3] [--no-cache] - Skript automaticky:
- Tries
<domain>/robots.txt→ readsSitemap:directive - Tries
<domain>/sitemap.xml,/sitemap_index.xmlako fallback - Ak sitemap → parse cez
lxml→ list URLs (recursive cez sitemap_index) - Ak NO sitemap → BFS crawl z homepage (depth 3, politeness 1s)
- Per URL:
trafilatura.extract_metadata→ title + h1 + description + canonical - Atomic write
site_map.json
- Tries
- Print summary (Slovak):
[site-crawler] {domain} → {N} URL (source: sitemap|bfs), cache do {date}
Output schema
{cache_dir}/site_map.json:
{
"schema_version": "1.0.0",
"domain": "lesazahrada.cz",
"crawled_at": "2026-04-26T10:30:00Z",
"source": "sitemap" | "bfs",
"url_count": 248,
"max_pages_capped": false,
"urls": [
{
"url": "https://lesazahrada.cz/strelivo/",
"title": "Hotové tovární střelivo | Lesazahrada.cz",
"h1": "Hotové tovární střelivo",
"meta_description": "Široký výběr...",
"canonical": "https://lesazahrada.cz/strelivo/",
"depth": 1,
"fetched_at": "2026-04-26T10:30:01Z"
}
]
}Hard rules
- SSRF check per URL (re-validate redirects) — uses
seo_common.url_utils.is_safe_url - Same-domain only (BFS won't follow external links)
- Robots.txt respect: parse
Disallow:patterns +Crawl-delay: - Politeness: default 1.0s delay between requests
- Hard cap:
max_pages=500(configurable), warn ak hit - Atomic writes (temp + rename),
umask 0o077,chmod 0o600 - Per-project cache namespace:
~/.cache/seo-content/{project_slug}/site-crawler/ - TTL: 30 days (sites se nemenia drasticky-rýchlo; override
SEO_SITE_CRAWLER_CACHE_TTL_SECONDS)
When to use vs siblings
$seo-site-crawler(this) — zoznam URL + lightweight metadata pre interné použitie (linking, content gap analysis)$seo-tech-audit— full technical SEO audit (broken links, alt texts, structured data, robots issues) cez black-seo-analyzer$seo-audit— Ahrefs-based domain overview (DR, backlinks, brand monitoring, traffic) cez Ahrefs MCP$seo-content-brief— per-URL keyword brief; auto-fires this skill ak chýba site_map.json
CLI reference
# Default: crawl s sitemap-first, max 500 pages, cache 7d
crawl.py lesazahrada.cz
# Force fresh crawl (skip cache)
crawl.py lesazahrada.cz --no-cache
# Sitemap-only (faster, no BFS fallback)
crawl.py lesazahrada.cz --sitemap-only
# Custom limits
crawl.py lesazahrada.cz --max-pages 1000 --max-depth 5 --politeness 2.0Files
seo-site-crawler/
├── SKILL.md (this file)
├── CHANGELOG.md
├── references/
│ └── crawl-defaults.md
├── schemas/
│ └── site-map.schema.json
└── scripts/
├── crawl.py
├── sitemap_parser.py
└── tests/
└── test_sitemap_parser.pyDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

