agentleFS
JakeLabateJakeLabate/Claude-SEO-Skills/skills/llms-txt-audit/SKILL.md

llms-txt-audit

Skill3 starsChanged 5 months agoMITView on GitHub

Audit a website's /llms.txt for AI/LLM discoverability. Fetches and parses the llms.txt markdown file (per llmstxt.org) to find a missing file, a file served as HTML instead of markdown, a missing required H1 title, a missing blockquote summary, broken (non-200) or redirecting links, links missing descriptions, malformed list items that aren't proper markdown links, links to the wrong host, links that are noindexed or blocked by robots.txt, relative instead of absolute URLs, duplicate links, an empty file with no sections, the wrong content type, and whether a companion /llms-full.txt exists. Use when the user asks to audit, analyze, check, validate, create, or fix an llms.txt or llms-full.txt file, or to make a site discoverable and usable by AI assistants and LLMs.

  • Installs packages

1 other project uses a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

9 rules. About 0 are common, 2 are distinctive, and 7 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 7

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use this skill when the user asks to:

    line 12
  2. Distinctive:

    Before starting, confirm with the user:

    line 43
  3. Very specific:

    Write a report following the structure in references/report-template.md:

    line 100
  4. Very specific:

    Missing file: create /llms.txt with at minimum an H1 title, a one-line > summary, and one or more ## sections of curated - name: …

    line 111
  5. Very specific:

    Missing H1: add a single top-level # Project Name heading: it is the only required element.

    line 113
  6. Very specific:

    Relative URLs: use absolute https:// URLs so the file works when fetched out of context.

    line 116
  7. Very specific:

    llms-full.txt: if the user wants full-content ingestion, generate /llms-full.txt with the expanded page contents.

    line 117
  8. Very specific:

    If the user wants the report as a .docx (for example, to share with stakeholders or attach to a ticket), save the Markdown report to a file …

    line 124
  9. Very specific:

    If the user wants the raw findings as a spreadsheet (for filtering, sorting, or triage in Sheets or Excel), convert the audit's …

    line 134

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

llms.txt Audit

Audit a website's /llms.txt file and produce an actionable report for AI/LLM discoverability.

When to use this skill

Use this skill when the user asks to:

  • Audit, validate, or check an llms.txt or llms-full.txt file
  • Make a site discoverable and usable by AI assistants, chatbots, and LLMs
  • Find broken, redirecting, or blocked links inside llms.txt
  • Confirm llms.txt follows the llmstxt.org structure (H1 title, summary, link sections)
  • Decide what should go into llms.txt vs. llms-full.txt

Background: what llms.txt is

llms.txt is a proposed standard (https://llmstxt.org) for a markdown file at the site root (/llms.txt) that gives LLMs a concise, curated map of the site's most useful content. Unlike robots.txt (which controls crawling) or sitemap.xml (which lists every URL), llms.txt is hand-curated and human-readable. Its structure:

# Project Name              ← required: the only mandatory element

> A short summary.          ← recommended blockquote summary

Optional free-form markdown with key context (no headings).

## Docs                     ← H2 section of curated links
- [Quickstart](https://example.com/quickstart): How to get started.
- [API reference](https://example.com/api): Full API docs.

## Optional                 ← an "Optional" section may be skipped by LLMs with limited context
- [Changelog](https://example.com/changelog): Release history.

A companion /llms-full.txt may contain the full expanded content in one file. This skill audits structure and link health against that spec.

Inputs to collect

Before starting, confirm with the user:

  1. Site URL or llms.txt URL: a live site root (e.g., https://example.com, which fetches /llms.txt) or a specific llms.txt URL (--llms).
  2. Scope: how many referenced links to probe (default 300, capped with --max-urls).
  3. Probing: whether to fetch each link to check status/redirect/noindex (default on). Use --no-probe for a fast structure-only pass.

Workflow

Step 1: Fetch and parse llms.txt

Use scripts/collect_llms.py:

python3 scripts/collect_llms.py https://example.com --max-urls 300 --output llms_inventory.json

Point directly at an llms.txt with --llms:

python3 scripts/collect_llms.py https://example.com/llms.txt --llms --output llms_inventory.json

The script fetches /llms.txt without auto-following redirects (so a redirected or HTML-served file is visible), records status / content type / size, parses the H1 title, blockquote summary, and H2 sections with their list items, then probes every referenced link (status, redirect target, canonical, meta robots, X-Robots-Tag, robots.txt disallow). It also checks whether /llms-full.txt exists.

Step 2: Run the audit checks

Use scripts/audit_llms.py:

python3 scripts/audit_llms.py llms_inventory.json --output audit_report.json

Step 3: Evaluate the audit checks

Evaluate each check defined in references/audit-checks.md. The core checks are:

CheckSeverity
/llms.txt missing (non-200)High
Served as HTML, not markdown/plain textHigh
Missing required H1 titleHigh
Referenced link returns 4xx/5xx/errorHigh
/llms.txt itself redirectsMedium
Referenced link redirects (not a final 200)Medium
Missing blockquote summaryMedium
Link list item missing a descriptionMedium
Malformed list item (not a markdown link)Medium
Link on the wrong hostMedium
Linked page is noindex or robots-blockedMedium
Wrong content typeMedium
Relative URL instead of absoluteLow
No sections / no linksLow
Duplicate linkLow
/llms-full.txt missingLow

Step 4: Produce the report

Write a report following the structure in references/report-template.md:

  1. Summary: file status, H1, sections, links found/probed, issue counts by severity
  2. File structure: H1, summary present, section list, llms-full.txt presence
  3. Issues: one section per failing check, listing the link/item, the problem, and the fix
  4. Prioritized action list: ordered by severity and impact

Step 5: Recommend fixes

For each issue, give a concrete fix:

  • Missing file: create /llms.txt with at minimum an H1 title, a one-line > summary, and one or more ## sections of curated - [name](url): description links. Serve it as text/markdown or text/plain at the root with a 200.
  • Served as HTML / wrong content type: serve the raw markdown file, not a rendered page; set Content-Type: text/markdown (or text/plain).
  • Missing H1: add a single top-level # Project Name heading: it is the only required element.
  • Broken / redirecting / blocked links: replace each with its final, canonical, indexable 200 URL (or remove it). Links in llms.txt should point straight to the live destination.
  • Missing descriptions / malformed items: rewrite each item as - [Descriptive title](absolute-url): one-line description so an LLM can tell what each link is for.
  • Relative URLs: use absolute https:// URLs so the file works when fetched out of context.
  • llms-full.txt: if the user wants full-content ingestion, generate /llms-full.txt with the expanded page contents.
  • Base every recommendation on the observed file; quote the exact line to change and do not invent URLs.

Optional: export the report

As a Word document (.docx)

If the user wants the report as a .docx (for example, to share with stakeholders or attach to a ticket), save the Markdown report to a file and convert it:

python3 scripts/md_to_docx.py report.md --output report.docx

scripts/md_to_docx.py uses only the Python standard library (no pip install) and renders headings, tables, lists, links, bold/italic, and code blocks. Offer this whenever a user asks for a Word doc, a .docx, or a shareable/downloadable report.

As a CSV of findings (.csv)

If the user wants the raw findings as a spreadsheet (for filtering, sorting, or triage in Sheets or Excel), convert the audit's audit_report.json directly:

python3 scripts/findings_to_csv.py audit_report.json --output findings.csv

scripts/findings_to_csv.py is also standard-library only. It writes one row per finding, with check and severity columns prepended and list fields (e.g. the pages sharing a duplicate value) joined with ; . Unlike the .docx, which reformats the written report, the CSV is a direct dump of the structured findings: offer it whenever a user wants the data itself, a spreadsheet, or to slice findings by check or severity.

Resources

  • scripts/md_to_docx.py: convert the Markdown report into a Word (.docx) document (standard library only)
  • scripts/findings_to_csv.py: flatten the audit findings JSON into a CSV, one row per finding (standard library only)
  • references/audit-checks.md: full definitions and rationale for every audit check
  • references/report-template.md: report output structure
  • scripts/collect_llms.py: fetch, parse, and probe a site's llms.txt
  • scripts/audit_llms.py: run audit checks against an llms.txt inventory JSON file

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.