agentleFS
Sign inStart free
bestdeejay-designbestdeejay-design/agent-skills/skills/seo-crawl/SKILL.md

seo-crawl

Skill6 starsChanged 3 months agoMITView on GitHub

Crawlability and site structure for SEO: robots.txt, noindex, canonical, redirect chains, sitemap.xml; internal linking, click depth, orphan pages, URL hierarchy, breadcrumbs; competitor comparison (gaps, overlapping keywords, content gaps). Commands documented in commands/. Triggers: 'seo crawl', 'crawlability', 'robots txt', 'site structure', 'URL structure', 'competitor seo', 'индексация', 'структура сайта', 'конкуренты seo', 'краулинг'.

1 other project uses a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use this skill for the crawl/visibility layer of SEO: whether search engines can reach the site, how it is internally linked, and how it …

    line 14
  2. Very specific:

    Do not use for copy quality or schema authoring.

    line 42
  3. Distinctive:

    Every finding contains URL/file, HTTP status or source evidence, affected link path, severity, and next action.

    line 50

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

SEO Crawl — crawlability, structure & competitors

Use this skill for the crawl/visibility layer of SEO: whether search engines can reach the site, how it is internally linked, and how it compares to competitors.

Commands (documented in commands/)

CommandWhat it does
seo-crawlCrawlability: robots.txt, noindex, canonical, redirect chains, sitemap.xml
seo-structureInternal linking, click depth, orphan pages, URL hierarchy, breadcrumbs
seo-compareCompetitor comparison: gaps, overlapping keywords, content gaps (URL mode)

Modes: URL (external sites) and File (local projects); seo-compare is URL-only.

When to use

  • User asks for "seo crawl", "crawlability", "robots txt", "site structure", "URL structure", "competitor seo", "индексация", "структура сайта".
  • Indexation, linking structure or competitor gaps are the task.

Do NOT use

  • For the full technical audit / CWV / reports / fixes — that is seo-audit.
  • For schema.org / meta tags — that is seo-schema.
  • For content / keywords / headings / images — that is seo-content.

Boundaries

  • Do not use for copy quality or schema authoring; use seo-content or seo-schema.

Crawl workflow and safety

Declare crawl scope, user agent, rate limit, robots policy, maximum pages, and whether the input is a local build or a permitted URL. Separate robots exclusions, HTTP failures, canonical conflicts, redirect chains, orphan detection, and competitor observations; do not merge them into one score.

Output gate

Every finding contains URL/file, HTTP status or source evidence, affected link path, severity, and next action. Report pages not crawled and why. Never bypass robots.txt, authentication, rate limits, or access controls, and never call a competitor gap an observed fact without fetched evidence.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free