agentleFS
JakeLabateJakeLabate/Claude-SEO-Skills/skills/site-architecture-audit/SKILL.md

site-architecture-audit

Skill3 starsChanged 5 months agoMITView on GitHub

Audit a website's architecture and crawlability for SEO. Crawls the site and inspects robots.txt and XML sitemaps to find hostname/protocol duplication, URL structure problems, excessive crawl depth, weak directory taxonomy, blocked or conflicting crawl and index directives, and sitemap-to-crawl parity gaps. Use when the user asks to audit, analyze, or improve site architecture, site structure, URL structure, crawlability, indexability, robots.txt, XML sitemaps, canonicalization, or how a site is organized.

  • Installs packages

The file

We split this skill into the rules it gives an agent

10 rules. About 0 are common, 3 are distinctive, and 7 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 7

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use this skill when the user asks to:

    line 12
  2. Distinctive:

    Check crawlability and indexability: robots.txt, meta robots, canonical signals, conflicts between them

    line 16
  3. Distinctive:

    Before starting, confirm with the user:

    line 25
  4. Very specific:

    Use scripts/crawl_architecture.py to crawl from the homepage (same registrable host only), probe the four hostname/protocol variants, fetch …

    line 36
  5. Very specific:

    Use scripts/audit_architecture.py to evaluate every check against the inventory:

    line 46
  6. Very specific:

    Write a report following the structure in references/report-template.md:

    line 78
  7. Very specific:

    Remove Disallow: /blog from robots.txt.

    line 91
  8. Very specific:

    Add the 60 indexable pages listed below to sitemap.xml and reference the sitemap from robots.txt

    line 94
  9. Very specific:

    If the user wants the report as a .docx (for example, to share with stakeholders or attach to a ticket), save the Markdown report to a file …

    line 100
  10. Very specific:

    If the user wants the raw findings as a spreadsheet (for filtering, sorting, or triage in Sheets or Excel), convert the audit's …

    line 110

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Site Architecture Audit

Audit the structural layer of a website (URL design, hostname and protocol canonicalization, robots.txt, XML sitemaps, crawl and index directives, depth distribution, and directory taxonomy) and produce an actionable SEO report.

When to use this skill

Use this skill when the user asks to:

  • Audit or analyze a site's architecture, structure, or how it is organized
  • Review URL structure quality (depth, readability, parameters, trailing slashes, case)
  • Check crawlability and indexability: robots.txt, meta robots, canonical signals, conflicts between them
  • Validate XML sitemaps and how well they match what is actually crawlable
  • Confirm hostname and protocol canonicalization (http vs https, www vs non-www)
  • Understand click-depth distribution and directory grouping across the site

This skill covers the structural layer. For the link-by-link view of broken links, orphan pages, anchor text, and redirect chains, use internal-link-audit. For outbound links use external-link-audit. The two are complementary: architecture explains the shape of the site, internal-link explains the flow of equity within it.

Inputs to collect

Before starting, confirm with the user:

  1. Site URL : a live site root URL (e.g., https://example.com). A live crawl is required because robots.txt, sitemaps, and redirect behavior cannot be assessed from local files alone.
  2. Scope : full site or a specific section (e.g., /blog/).
  3. Crawl limits : max pages (default 500) and max depth (default 10) for the crawl.
  4. Business context : site type (e-commerce, blog, local business, SaaS) and the few page templates that matter most, so depth and taxonomy expectations can be set.

Workflow

Step 1: Crawl the site and gather architecture signals

Use scripts/crawl_architecture.py to crawl from the homepage (same registrable host only), probe the four hostname/protocol variants, fetch and parse robots.txt, and collect every XML sitemap URL (following sitemap index files):

python3 scripts/crawl_architecture.py https://example.com --max-pages 500 --max-depth 10 --output architecture_inventory.json

For every crawled page the inventory records: requested URL, final URL, HTTP status, redirect chain, click depth, scheme, host, path, query, path-segment count, canonical, meta robots, title, and the raw resolved internal link targets (used to detect URL-form inconsistency). It also records the host-variant probe, the parsed robots.txt rules and Sitemap: directives, and the full sitemap URL set.

Step 2: Run the audit checks

Use scripts/audit_architecture.py to evaluate every check against the inventory:

python3 scripts/audit_architecture.py architecture_inventory.json --output audit_report.json

Useful flags: --max-url-depth 4 (path-segment threshold), --max-click-depth 3 (click-depth threshold), and --tracking-params utm_source,utm_medium,gclid,fbclid to extend the tracking-parameter list.

Step 3: Evaluate the audit checks

Evaluate each check defined in references/audit-checks.md. The core checks are:

CheckSeverity
Multiple indexable hostname/protocol variants (no canonical host enforcement)High
Important pages blocked by robots.txtHigh
Conflicting crawl/index directives (disallow + noindex, noindex + canonical elsewhere, canonical to non-200/redirect/noindex)High
Sitemap lists non-indexable URLs (non-200, redirected, noindexed, or non-canonical)High
Indexable pages missing from the XML sitemap (sitemap/crawl parity gap)Medium
Excessive click depth or top-heavy depth distributionMedium
URL structure problems (uppercase, underscores, spaces/encoding, over-long, over-deep, ID-only slugs)Medium
Inconsistent URL forms in internal links (mixed trailing slash, mixed http/https, mixed www)Medium
Crawlable parameter URLs (tracking, session, sort/filter) creating duplicate spacesMedium
Missing or non-200 robots.txtMedium
No XML sitemap foundMedium
Pages missing a self-referencing canonicalLow
Sitemap not referenced in robots.txtLow
Thin or inconsistent directory taxonomyLow
Sitemap over size limits (50,000 URLs / 50 MB) without a sitemap indexLow

Step 4: Produce the report

Write a report following the structure in references/report-template.md:

  1. Summary : pages crawled, canonical host, sitemap/robots status, issue counts by severity
  2. Canonicalization and crawl controls : host-variant table, robots.txt findings, index-signal conflicts
  3. URL and depth structure : depth histogram, URL-quality issues, directory map
  4. Sitemap health : parity gaps and non-indexable entries
  5. Prioritized action list : ordered by severity and estimated impact

Step 5: Recommend fixes

For each issue, give a concrete, copy-pasteable recommendation, e.g.:

  • Redirect http://example.com and https://www.example.com to https://example.com with a single 301 so only one host variant resolves
  • Remove Disallow: /blog from robots.txt; it is blocking 42 linked, indexable articles
  • Drop noindex from /services/seo or remove it from the sitemap; the two signals contradict each other
  • Lowercase and de-underscore /Services/SEO_Audit to /services/seo-audit and 301 the old URL
  • Add the 60 indexable pages listed below to sitemap.xml and reference the sitemap from robots.txt

Optional: export the report

As a Word document (.docx)

If the user wants the report as a .docx (for example, to share with stakeholders or attach to a ticket), save the Markdown report to a file and convert it:

python3 scripts/md_to_docx.py report.md --output report.docx

scripts/md_to_docx.py uses only the Python standard library (no pip install) and renders headings, tables, lists, links, bold/italic, and code blocks. Offer this whenever a user asks for a Word doc, a .docx, or a shareable/downloadable report.

As a CSV of findings (.csv)

If the user wants the raw findings as a spreadsheet (for filtering, sorting, or triage in Sheets or Excel), convert the audit's audit_report.json directly:

python3 scripts/findings_to_csv.py audit_report.json --output findings.csv

scripts/findings_to_csv.py is also standard-library only. It writes one row per finding, with check and severity columns prepended and list fields (e.g. the pages sharing a duplicate value) joined with ; . Unlike the .docx, which reformats the written report, the CSV is a direct dump of the structured findings: offer it whenever a user wants the data itself, a spreadsheet, or to slice findings by check or severity.

Resources

  • scripts/md_to_docx.py: convert the Markdown report into a Word (.docx) document (standard library only)
  • scripts/findings_to_csv.py: flatten the audit findings JSON into a CSV, one row per finding (standard library only)
  • references/audit-checks.md : full definitions, thresholds, and rationale for every audit check
  • references/report-template.md : report output structure
  • scripts/crawl_architecture.py : crawl the site, probe host variants, parse robots.txt and sitemaps, build the inventory
  • scripts/audit_architecture.py : run audit checks against an architecture inventory JSON file

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.