site-architecture-audit
Skill3 starsChanged 5 months agoMITView on GitHub
Audit a website's architecture and crawlability for SEO. Crawls the site and inspects robots.txt and XML sitemaps to find hostname/protocol duplication, URL structure problems, excessive crawl depth, weak directory taxonomy, blocked or conflicting crawl and index directives, and sitemap-to-crawl parity gaps. Use when the user asks to audit, analyze, or improve site architecture, site structure, URL structure, crawlability, indexability, robots.txt, XML sitemaps, canonicalization, or how a site is organized.
- Installs packages
In this repository19 other files it gives its agents.
- canonical-tag-auditSkill
- content-quality-auditSkill
- core-web-vitals-auditSkill
- external-link-auditSkill
- full-seo-auditSkill
- heading-structure-auditSkill
- image-seo-auditSkill
- internal-link-auditSkill
- keyword-cannibalization-auditSkill
- llms-txt-auditSkill
- meta-data-auditSkill
- mixed-content-auditSkill
- open-graph-auditSkill
- pagination-auditSkill
- redirect-auditSkill
- robots-txt-auditSkill
- schema-markup-auditSkill
- sitemap-auditSkill
- soft-404-auditSkill
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 3 are distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 12
Use this skill when the user asks to:
- Distinctive: line 16
Check crawlability and indexability: robots.txt, meta robots, canonical signals, conflicts between them
- Distinctive: line 25
Before starting, confirm with the user:
- Very specific: line 36
Use scripts/crawl_architecture.py to crawl from the homepage (same registrable host only), probe the four hostname/protocol variants, fetch …
- Very specific: line 46
Use scripts/audit_architecture.py to evaluate every check against the inventory:
- Very specific: line 78
Write a report following the structure in references/report-template.md:
- Very specific: line 91
Remove Disallow: /blog from robots.txt.
- Very specific: line 94
Add the 60 indexable pages listed below to sitemap.xml and reference the sitemap from robots.txt
- Very specific: line 100
If the user wants the report as a .docx (for example, to share with stakeholders or attach to a ticket), save the Markdown report to a file …
- Very specific: line 110
If the user wants the raw findings as a spreadsheet (for filtering, sorting, or triage in Sheets or Excel), convert the audit's …
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Site Architecture Audit
Audit the structural layer of a website (URL design, hostname and protocol canonicalization, robots.txt, XML sitemaps, crawl and index directives, depth distribution, and directory taxonomy) and produce an actionable SEO report.
When to use this skill
Use this skill when the user asks to:
- Audit or analyze a site's architecture, structure, or how it is organized
- Review URL structure quality (depth, readability, parameters, trailing slashes, case)
- Check crawlability and indexability: robots.txt, meta robots, canonical signals, conflicts between them
- Validate XML sitemaps and how well they match what is actually crawlable
- Confirm hostname and protocol canonicalization (http vs https, www vs non-www)
- Understand click-depth distribution and directory grouping across the site
This skill covers the structural layer. For the link-by-link view of broken links, orphan pages, anchor text, and redirect chains, use internal-link-audit. For outbound links use external-link-audit. The two are complementary: architecture explains the shape of the site, internal-link explains the flow of equity within it.
Inputs to collect
Before starting, confirm with the user:
- Site URL : a live site root URL (e.g.,
https://example.com). A live crawl is required because robots.txt, sitemaps, and redirect behavior cannot be assessed from local files alone. - Scope : full site or a specific section (e.g.,
/blog/). - Crawl limits : max pages (default 500) and max depth (default 10) for the crawl.
- Business context : site type (e-commerce, blog, local business, SaaS) and the few page templates that matter most, so depth and taxonomy expectations can be set.
Workflow
Step 1: Crawl the site and gather architecture signals
Use scripts/crawl_architecture.py to crawl from the homepage (same registrable host only), probe the four hostname/protocol variants, fetch and parse robots.txt, and collect every XML sitemap URL (following sitemap index files):
python3 scripts/crawl_architecture.py https://example.com --max-pages 500 --max-depth 10 --output architecture_inventory.jsonFor every crawled page the inventory records: requested URL, final URL, HTTP status, redirect chain, click depth, scheme, host, path, query, path-segment count, canonical, meta robots, title, and the raw resolved internal link targets (used to detect URL-form inconsistency). It also records the host-variant probe, the parsed robots.txt rules and Sitemap: directives, and the full sitemap URL set.
Step 2: Run the audit checks
Use scripts/audit_architecture.py to evaluate every check against the inventory:
python3 scripts/audit_architecture.py architecture_inventory.json --output audit_report.jsonUseful flags: --max-url-depth 4 (path-segment threshold), --max-click-depth 3 (click-depth threshold), and --tracking-params utm_source,utm_medium,gclid,fbclid to extend the tracking-parameter list.
Step 3: Evaluate the audit checks
Evaluate each check defined in references/audit-checks.md. The core checks are:
| Check | Severity |
| Multiple indexable hostname/protocol variants (no canonical host enforcement) | High |
| Important pages blocked by robots.txt | High |
| Conflicting crawl/index directives (disallow + noindex, noindex + canonical elsewhere, canonical to non-200/redirect/noindex) | High |
| Sitemap lists non-indexable URLs (non-200, redirected, noindexed, or non-canonical) | High |
| Indexable pages missing from the XML sitemap (sitemap/crawl parity gap) | Medium |
| Excessive click depth or top-heavy depth distribution | Medium |
| URL structure problems (uppercase, underscores, spaces/encoding, over-long, over-deep, ID-only slugs) | Medium |
| Inconsistent URL forms in internal links (mixed trailing slash, mixed http/https, mixed www) | Medium |
| Crawlable parameter URLs (tracking, session, sort/filter) creating duplicate spaces | Medium |
| Missing or non-200 robots.txt | Medium |
| No XML sitemap found | Medium |
| Pages missing a self-referencing canonical | Low |
| Sitemap not referenced in robots.txt | Low |
| Thin or inconsistent directory taxonomy | Low |
| Sitemap over size limits (50,000 URLs / 50 MB) without a sitemap index | Low |
Step 4: Produce the report
Write a report following the structure in references/report-template.md:
- Summary : pages crawled, canonical host, sitemap/robots status, issue counts by severity
- Canonicalization and crawl controls : host-variant table, robots.txt findings, index-signal conflicts
- URL and depth structure : depth histogram, URL-quality issues, directory map
- Sitemap health : parity gaps and non-indexable entries
- Prioritized action list : ordered by severity and estimated impact
Step 5: Recommend fixes
For each issue, give a concrete, copy-pasteable recommendation, e.g.:
- Redirect
http://example.comandhttps://www.example.comtohttps://example.comwith a single 301 so only one host variant resolves - Remove
Disallow: /blogfrom robots.txt; it is blocking 42 linked, indexable articles - Drop
noindexfrom/services/seoor remove it from the sitemap; the two signals contradict each other - Lowercase and de-underscore
/Services/SEO_Auditto/services/seo-auditand 301 the old URL - Add the 60 indexable pages listed below to
sitemap.xmland reference the sitemap from robots.txt
Optional: export the report
As a Word document (.docx)
If the user wants the report as a .docx (for example, to share with stakeholders or attach to a ticket), save the Markdown report to a file and convert it:
python3 scripts/md_to_docx.py report.md --output report.docxscripts/md_to_docx.py uses only the Python standard library (no pip install) and renders headings, tables, lists, links, bold/italic, and code blocks. Offer this whenever a user asks for a Word doc, a .docx, or a shareable/downloadable report.
As a CSV of findings (.csv)
If the user wants the raw findings as a spreadsheet (for filtering, sorting, or triage in Sheets or Excel), convert the audit's audit_report.json directly:
python3 scripts/findings_to_csv.py audit_report.json --output findings.csvscripts/findings_to_csv.py is also standard-library only. It writes one row per finding, with check and severity columns prepended and list fields (e.g. the pages sharing a duplicate value) joined with ; . Unlike the .docx, which reformats the written report, the CSV is a direct dump of the structured findings: offer it whenever a user wants the data itself, a spreadsheet, or to slice findings by check or severity.
Resources
scripts/md_to_docx.py: convert the Markdown report into a Word (.docx) document (standard library only)scripts/findings_to_csv.py: flatten the audit findings JSON into a CSV, one row per finding (standard library only)references/audit-checks.md: full definitions, thresholds, and rationale for every audit checkreferences/report-template.md: report output structurescripts/crawl_architecture.py: crawl the site, probe host variants, parse robots.txt and sitemaps, build the inventoryscripts/audit_architecture.py: run audit checks against an architecture inventory JSON file
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

