llms-txt-audit
Skill3 starsChanged 5 months agoMITView on GitHub
Audit a website's /llms.txt for AI/LLM discoverability. Fetches and parses the llms.txt markdown file (per llmstxt.org) to find a missing file, a file served as HTML instead of markdown, a missing required H1 title, a missing blockquote summary, broken (non-200) or redirecting links, links missing descriptions, malformed list items that aren't proper markdown links, links to the wrong host, links that are noindexed or blocked by robots.txt, relative instead of absolute URLs, duplicate links, an empty file with no sections, the wrong content type, and whether a companion /llms-full.txt exists. Use when the user asks to audit, analyze, check, validate, create, or fix an llms.txt or llms-full.txt file, or to make a site discoverable and usable by AI assistants and LLMs.
- Installs packages
1 other project uses a version of this.
Adopted by
Most recently taken up by
- romankurnovskii/get-llms-txttheir own version, first seen Oct 28, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- romankurnovskii/get-llms-txt★ 3same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository19 other files it gives its agents.
- canonical-tag-auditSkill
- content-quality-auditSkill
- core-web-vitals-auditSkill
- external-link-auditSkill
- full-seo-auditSkill
- heading-structure-auditSkill
- image-seo-auditSkill
- internal-link-auditSkill
- keyword-cannibalization-auditSkill
- meta-data-auditSkill
- mixed-content-auditSkill
- open-graph-auditSkill
- pagination-auditSkill
- redirect-auditSkill
- robots-txt-auditSkill
- schema-markup-auditSkill
- site-architecture-auditSkill
- sitemap-auditSkill
- soft-404-auditSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 2 are distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 12
Use this skill when the user asks to:
- Distinctive: line 43
Before starting, confirm with the user:
- Very specific: line 100
Write a report following the structure in references/report-template.md:
- Very specific: line 111
Missing file: create /llms.txt with at minimum an H1 title, a one-line > summary, and one or more ## sections of curated - name: …
- Very specific: line 113
Missing H1: add a single top-level # Project Name heading: it is the only required element.
- Very specific: line 116
Relative URLs: use absolute https:// URLs so the file works when fetched out of context.
- Very specific: line 117
llms-full.txt: if the user wants full-content ingestion, generate /llms-full.txt with the expanded page contents.
- Very specific: line 124
If the user wants the report as a .docx (for example, to share with stakeholders or attach to a ticket), save the Markdown report to a file …
- Very specific: line 134
If the user wants the raw findings as a spreadsheet (for filtering, sorting, or triage in Sheets or Excel), convert the audit's …
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
llms.txt Audit
Audit a website's /llms.txt file and produce an actionable report for AI/LLM discoverability.
When to use this skill
Use this skill when the user asks to:
- Audit, validate, or check an
llms.txtorllms-full.txtfile - Make a site discoverable and usable by AI assistants, chatbots, and LLMs
- Find broken, redirecting, or blocked links inside
llms.txt - Confirm
llms.txtfollows the llmstxt.org structure (H1 title, summary, link sections) - Decide what should go into
llms.txtvs.llms-full.txt
Background: what llms.txt is
llms.txt is a proposed standard (https://llmstxt.org) for a markdown file at the site root (/llms.txt) that gives LLMs a concise, curated map of the site's most useful content. Unlike robots.txt (which controls crawling) or sitemap.xml (which lists every URL), llms.txt is hand-curated and human-readable. Its structure:
# Project Name ← required: the only mandatory element
> A short summary. ← recommended blockquote summary
Optional free-form markdown with key context (no headings).
## Docs ← H2 section of curated links
- [Quickstart](https://example.com/quickstart): How to get started.
- [API reference](https://example.com/api): Full API docs.
## Optional ← an "Optional" section may be skipped by LLMs with limited context
- [Changelog](https://example.com/changelog): Release history.A companion /llms-full.txt may contain the full expanded content in one file. This skill audits structure and link health against that spec.
Inputs to collect
Before starting, confirm with the user:
- Site URL or llms.txt URL: a live site root (e.g.,
https://example.com, which fetches/llms.txt) or a specific llms.txt URL (--llms). - Scope: how many referenced links to probe (default 300, capped with
--max-urls). - Probing: whether to fetch each link to check status/redirect/noindex (default on). Use
--no-probefor a fast structure-only pass.
Workflow
Step 1: Fetch and parse llms.txt
Use scripts/collect_llms.py:
python3 scripts/collect_llms.py https://example.com --max-urls 300 --output llms_inventory.jsonPoint directly at an llms.txt with --llms:
python3 scripts/collect_llms.py https://example.com/llms.txt --llms --output llms_inventory.jsonThe script fetches /llms.txt without auto-following redirects (so a redirected or HTML-served file is visible), records status / content type / size, parses the H1 title, blockquote summary, and H2 sections with their list items, then probes every referenced link (status, redirect target, canonical, meta robots, X-Robots-Tag, robots.txt disallow). It also checks whether /llms-full.txt exists.
Step 2: Run the audit checks
Use scripts/audit_llms.py:
python3 scripts/audit_llms.py llms_inventory.json --output audit_report.jsonStep 3: Evaluate the audit checks
Evaluate each check defined in references/audit-checks.md. The core checks are:
| Check | Severity |
| /llms.txt missing (non-200) | High |
| Served as HTML, not markdown/plain text | High |
| Missing required H1 title | High |
| Referenced link returns 4xx/5xx/error | High |
| /llms.txt itself redirects | Medium |
| Referenced link redirects (not a final 200) | Medium |
| Missing blockquote summary | Medium |
| Link list item missing a description | Medium |
| Malformed list item (not a markdown link) | Medium |
| Link on the wrong host | Medium |
| Linked page is noindex or robots-blocked | Medium |
| Wrong content type | Medium |
| Relative URL instead of absolute | Low |
| No sections / no links | Low |
| Duplicate link | Low |
| /llms-full.txt missing | Low |
Step 4: Produce the report
Write a report following the structure in references/report-template.md:
- Summary: file status, H1, sections, links found/probed, issue counts by severity
- File structure: H1, summary present, section list, llms-full.txt presence
- Issues: one section per failing check, listing the link/item, the problem, and the fix
- Prioritized action list: ordered by severity and impact
Step 5: Recommend fixes
For each issue, give a concrete fix:
- Missing file: create
/llms.txtwith at minimum an H1 title, a one-line>summary, and one or more##sections of curated- [name](url): descriptionlinks. Serve it astext/markdownortext/plainat the root with a200. - Served as HTML / wrong content type: serve the raw markdown file, not a rendered page; set
Content-Type: text/markdown(ortext/plain). - Missing H1: add a single top-level
# Project Nameheading: it is the only required element. - Broken / redirecting / blocked links: replace each with its final, canonical, indexable
200URL (or remove it). Links in llms.txt should point straight to the live destination. - Missing descriptions / malformed items: rewrite each item as
- [Descriptive title](absolute-url): one-line descriptionso an LLM can tell what each link is for. - Relative URLs: use absolute
https://URLs so the file works when fetched out of context. - llms-full.txt: if the user wants full-content ingestion, generate
/llms-full.txtwith the expanded page contents. - Base every recommendation on the observed file; quote the exact line to change and do not invent URLs.
Optional: export the report
As a Word document (.docx)
If the user wants the report as a .docx (for example, to share with stakeholders or attach to a ticket), save the Markdown report to a file and convert it:
python3 scripts/md_to_docx.py report.md --output report.docxscripts/md_to_docx.py uses only the Python standard library (no pip install) and renders headings, tables, lists, links, bold/italic, and code blocks. Offer this whenever a user asks for a Word doc, a .docx, or a shareable/downloadable report.
As a CSV of findings (.csv)
If the user wants the raw findings as a spreadsheet (for filtering, sorting, or triage in Sheets or Excel), convert the audit's audit_report.json directly:
python3 scripts/findings_to_csv.py audit_report.json --output findings.csvscripts/findings_to_csv.py is also standard-library only. It writes one row per finding, with check and severity columns prepended and list fields (e.g. the pages sharing a duplicate value) joined with ; . Unlike the .docx, which reformats the written report, the CSV is a direct dump of the structured findings: offer it whenever a user wants the data itself, a spreadsheet, or to slice findings by check or severity.
Resources
scripts/md_to_docx.py: convert the Markdown report into a Word (.docx) document (standard library only)scripts/findings_to_csv.py: flatten the audit findings JSON into a CSV, one row per finding (standard library only)references/audit-checks.md: full definitions and rationale for every audit checkreferences/report-template.md: report output structurescripts/collect_llms.py: fetch, parse, and probe a site's llms.txtscripts/audit_llms.py: run audit checks against an llms.txt inventory JSON file
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

