soft-404-audit
Skill3 starsChanged 5 months agoMITView on GitHub
Audit a website for soft 404s (pages that return HTTP 200 but are really "not found", empty, or error pages). Crawls pages and sends a missing-page probe to find error pages served with a 200 status, a server that returns 200 for every unknown URL instead of a real 404/410, thin pages matching the site's error template, and empty/near-empty pages. Use when the user asks to audit, check, or fix soft 404s, soft 404 errors in Search Console, pages that should be 404 but return 200, fake/empty pages, or missing pages that don't return the right status code.
- Installs packages
In this repository19 other files it gives its agents.
- canonical-tag-auditSkill
- content-quality-auditSkill
- core-web-vitals-auditSkill
- external-link-auditSkill
- full-seo-auditSkill
- heading-structure-auditSkill
- image-seo-auditSkill
- internal-link-auditSkill
- keyword-cannibalization-auditSkill
- llms-txt-auditSkill
- meta-data-auditSkill
- mixed-content-auditSkill
- open-graph-auditSkill
- pagination-auditSkill
- redirect-auditSkill
- robots-txt-auditSkill
- schema-markup-auditSkill
- site-architecture-auditSkill
- sitemap-auditSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 4 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 14
Use this skill when the user asks to:
- Distinctive: line 18
Check whether the site returns a real 404 for missing URLs
- Distinctive: line 25
Search Console flagged works well.
- Very specific: line 66
Write a report following references/report-template.md: lead with the site-wide status-code behavior, then confirmed soft 404s, then …
- Distinctive: line 77
Thin/empty pages: verify they aren't JavaScript-rendered (this crawler doesn't run JS).
- Very specific: line 85
If the user wants the report as a .docx (for example, to share with stakeholders or attach to a ticket), save the Markdown report to a file …
- Very specific: line 95
If the user wants the raw findings as a spreadsheet (for filtering, sorting, or triage in Sheets or Excel), convert the audit's …
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Soft 404 Audit
Audit a website for soft 404s and produce an actionable report. A soft 404 is a
page that returns 200 OK but is really a "not found"/empty page: it wastes
crawl budget and can get real URLs dropped from the index.
When to use this skill
Use this skill when the user asks to:
- Audit or fix soft 404s (including Search Console "Soft 404" reports)
- Find pages that return 200 but should return 404/410
- Check whether the site returns a real 404 for missing URLs
- Find empty or error-template pages served as live pages
Inputs to collect
- Site URL or URL list: a live site root to crawl (auto-seeds from
/sitemap.xml), or a text file with--url-list. A URL list of the pages Search Console flagged works well. - Scope: pages to crawl (default 500,
--max-pages). - Thin threshold: word count below which a 200 page is "thin" (default 50,
--thin-words); raise it for content-heavy sites.
Workflow
Step 1: Crawl and probe
Use scripts/extract_pages.py:
python3 scripts/extract_pages.py https://example.com --max-pages 500 --output soft404_inventory.json
# already crawled once (e.g. in a full SEO audit)? skip the crawl and reuse the shared cache:
# python3 scripts/fetch_pages.py https://example.com --output page_cache.json
# python3 scripts/extract_pages.py --from-cache page_cache.json --output soft404_inventory.jsonIt records each page's status, title, first H1, visible word count, and whether error phrases appear in the title/H1, then sends one request to a deliberately non-existent URL to learn how the site handles missing pages.
Step 2: Run the audit checks
python3 scripts/audit_soft404.py soft404_inventory.json --output audit_report.jsonStep 3: Evaluate the audit checks
Evaluate each check in references/audit-checks.md. Core checks:
| Check | Severity |
| Soft 404 (error page served as 200) | High |
| Missing URLs return 200 (site-wide) | High |
| Thin page matching the error template | Medium |
| Empty / near-empty page | Low |
Step 4: Produce the report
Write a report following references/report-template.md: lead with the site-wide
status-code behavior, then confirmed soft 404s, then suspected ones, and a
prioritized action list.
Step 5: Recommend fixes
- Missing URLs return 200: configure the server/framework to return a real
404for unmatched routes (render the custom 404 page with a 404 status). This single fix clears the whole class. - Confirmed soft 404: return
404(or410if permanently gone), or restore the page if the URL should resolve. - Thin/empty pages: verify they aren't JavaScript-rendered (this crawler doesn't run JS); add content or noindex genuine placeholders.
- Base every recommendation on observed signals; never invent page content.
Optional: export the report
As a Word document (.docx)
If the user wants the report as a .docx (for example, to share with stakeholders or attach to a ticket), save the Markdown report to a file and convert it:
python3 scripts/md_to_docx.py report.md --output report.docxscripts/md_to_docx.py uses only the Python standard library (no pip install) and renders headings, tables, lists, links, bold/italic, and code blocks. Offer this whenever a user asks for a Word doc, a .docx, or a shareable/downloadable report.
As a CSV of findings (.csv)
If the user wants the raw findings as a spreadsheet (for filtering, sorting, or triage in Sheets or Excel), convert the audit's audit_report.json directly:
python3 scripts/findings_to_csv.py audit_report.json --output findings.csvscripts/findings_to_csv.py is also standard-library only. It writes one row per finding, with check and severity columns prepended and list fields (e.g. the pages sharing a duplicate value) joined with ; . Unlike the .docx, which reformats the written report, the CSV is a direct dump of the structured findings: offer it whenever a user wants the data itself, a spreadsheet, or to slice findings by check or severity.
Resources
scripts/md_to_docx.py: convert the Markdown report into a Word (.docx) document (standard library only)scripts/findings_to_csv.py: flatten the audit findings JSON into a CSV, one row per finding (standard library only)references/audit-checks.md: full definitions and rationalereferences/report-template.md: report output structurescripts/extract_pages.py: crawl pages and probe missing-page behaviorscripts/audit_soft404.py: run soft-404 audit checks
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

