agentleFS
Teapot-AgencyTeapot-Agency/teapot-seo-skills/skills/seo-site-crawler/SKILL.md

seo-site-crawler

Skill0 starsChanged 7 months agoMITView on GitHub

Lightweight site URL discovery + per-page metadata pre internal linking. Tries sitemap.xml first, fallback BFS crawl s trafilatura. Output site_map.json (URL + title + h1 + meta description + canonical) cached per-project (TTL 7d). Auto-fired z $seo-content-brief; manuálne triggers - "rob site map pre <domain>", "skroluj urls na <domain>", "site crawler pre teapot.sk". Pre full technical SEO audit (broken links, alts, structured data) use $seo-tech-audit. Pre Ahrefs domain overview use $seo-audit.

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Pre-flight: ensure is_safe_url(domain) passes (SSRF guard)

    line 14
  2. Distinctive:

    Politeness: default 1.0s delay between requests

    line 62

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

seo-site-crawler

Lightweight site URL discovery skill. Builds site_map.json ktoré konzumuje $seo-internal-linking (a iné SEO skills) pre semantic-aware internal linking suggestions.

Workflow

  1. Parse intent: extract domain z user message (https://... alebo bare.sk)
  2. Pre-flight: ensure is_safe_url(domain) passes (SSRF guard)
  3. Cache check: ak ~/.cache/seo-content/{project_slug}/site-crawler/site_map.json existuje a je <7d → return cached (UNLESS --no-cache)
  4. Try sitemap first (lacné, najrýchlejšie):
    uv run --quiet --with httpx --with trafilatura --with lxml \
      python ~/.claude/skills/seo-site-crawler/scripts/crawl.py \
      <domain> [--max-pages 500] [--max-depth 3] [--no-cache]
  5. Skript automaticky:
    • Tries <domain>/robots.txt → reads Sitemap: directive
    • Tries <domain>/sitemap.xml, /sitemap_index.xml ako fallback
    • Ak sitemap → parse cez lxml → list URLs (recursive cez sitemap_index)
    • Ak NO sitemap → BFS crawl z homepage (depth 3, politeness 1s)
    • Per URL: trafilatura.extract_metadata → title + h1 + description + canonical
    • Atomic write site_map.json
  6. Print summary (Slovak): [site-crawler] {domain} → {N} URL (source: sitemap|bfs), cache do {date}

Output schema

{cache_dir}/site_map.json:

{
  "schema_version": "1.0.0",
  "domain": "lesazahrada.cz",
  "crawled_at": "2026-04-26T10:30:00Z",
  "source": "sitemap" | "bfs",
  "url_count": 248,
  "max_pages_capped": false,
  "urls": [
    {
      "url": "https://lesazahrada.cz/strelivo/",
      "title": "Hotové tovární střelivo | Lesazahrada.cz",
      "h1": "Hotové tovární střelivo",
      "meta_description": "Široký výběr...",
      "canonical": "https://lesazahrada.cz/strelivo/",
      "depth": 1,
      "fetched_at": "2026-04-26T10:30:01Z"
    }
  ]
}

Hard rules

  • SSRF check per URL (re-validate redirects) — uses seo_common.url_utils.is_safe_url
  • Same-domain only (BFS won't follow external links)
  • Robots.txt respect: parse Disallow: patterns + Crawl-delay:
  • Politeness: default 1.0s delay between requests
  • Hard cap: max_pages=500 (configurable), warn ak hit
  • Atomic writes (temp + rename), umask 0o077, chmod 0o600
  • Per-project cache namespace: ~/.cache/seo-content/{project_slug}/site-crawler/
  • TTL: 30 days (sites se nemenia drasticky-rýchlo; override SEO_SITE_CRAWLER_CACHE_TTL_SECONDS)

When to use vs siblings

  • $seo-site-crawler (this) — zoznam URL + lightweight metadata pre interné použitie (linking, content gap analysis)
  • $seo-tech-audit — full technical SEO audit (broken links, alt texts, structured data, robots issues) cez black-seo-analyzer
  • $seo-audit — Ahrefs-based domain overview (DR, backlinks, brand monitoring, traffic) cez Ahrefs MCP
  • $seo-content-brief — per-URL keyword brief; auto-fires this skill ak chýba site_map.json

CLI reference

# Default: crawl s sitemap-first, max 500 pages, cache 7d
crawl.py lesazahrada.cz

# Force fresh crawl (skip cache)
crawl.py lesazahrada.cz --no-cache

# Sitemap-only (faster, no BFS fallback)
crawl.py lesazahrada.cz --sitemap-only

# Custom limits
crawl.py lesazahrada.cz --max-pages 1000 --max-depth 5 --politeness 2.0

Files

seo-site-crawler/
├── SKILL.md (this file)
├── CHANGELOG.md
├── references/
│   └── crawl-defaults.md
├── schemas/
│   └── site-map.schema.json
└── scripts/
    ├── crawl.py
    ├── sitemap_parser.py
    └── tests/
        └── test_sitemap_parser.py

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.