howseen-aihowseen-ai/howseen-geo-skills/crawler-check/SKILL.md

crawler-check

Skill1 starsChanged 54 days agoMITView on GitHub

Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others) are actually allowed to read a website. Use this whenever the user asks if ChatGPT/Claude/Perplexity can crawl or read their site, wants to check robots.txt for AI bots, suspects they're blocking AI, or is debugging why they aren't cited. Point Claude at a domain and it fetches robots.txt, parses it correctly per user-agent, and reports what each AI crawler can and can't access.

In this repository9 other files it gives its agents.
- bing-checkSkill
- buyer-questionsSkill
- citation-checkSkill
- content-capsuleSkill
- entity-coverageSkill
- faq-schemaSkill
- llms-txtSkill
- retrievabilitySkill
- schema-builderSkill

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.

≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.

Common
Distinctive
Very specific

Distinctive:
If AI crawlers can't read your site, nothing else matters — you can't be cited from a page no model has seen.
line 8
Very specific:
Fetch https://domain.com/robots.txt: If it 404s, note that — no robots.txt means everything is allowed by default (which is usually fine …
line 26

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.

Files

SKILL.md

crawler-check

If AI crawlers can't read your site, nothing else matters — you can't be cited from a page no model has seen. This skill checks the real access rules.

The AI crawlers that matter

Bot	Who it feeds
`GPTBot`	ChatGPT training / browsing
`OAI-SearchBot`	ChatGPT search
`ChatGPT-User`	ChatGPT live retrieval on user request
`ClaudeBot` / `Claude-Web`	Claude
`PerplexityBot` / `Perplexity-User`	Perplexity
`Google-Extended`	Gemini / AI Overviews training
`Bingbot`	Bing → powers ChatGPT search (see `bing-check`)
`Applebot-Extended`	Apple Intelligence
`Amazonbot`, `Bytespider`, `Meta-ExternalAgent`	other AI systems

Steps

Fetch https://domain.com/robots.txt. If it 404s, note that — no robots.txt means everything is allowed by default (which is usually fine for AI visibility).
Parse it correctly — per user-agent block. This is where most tools get it wrong: a Disallow under User-agent: SomeBot applies only to that bot. A directive under User-agent: Nutch does not block GPTBot. Do not report a bot as blocked unless a rule under its own user-agent (or the wildcard User-agent: *) actually disallows the path. Read each block's directives only against the agent that block names.
Determine, per AI crawler above: allowed on the whole site, allowed on key paths, or disallowed — and quote the exact directive that decides it.
Do a reachability sanity check on the homepage and one key page: does the raw HTML contain the real content, or is it empty until JavaScript runs? Content that only appears after JS is effectively invisible to many crawlers even when robots.txt allows them (flag it, hand off to retrievability).

Output format

## Crawler access — [domain]

robots.txt: [found / not found]

| Crawler | Access | Deciding rule |
|---|---|---|
| GPTBot | ✅ allowed | (no matching disallow) |
| ClaudeBot | ⛔ blocked | Disallow: / under User-agent: ClaudeBot |
| ... | | |

⚠️ [Any accidental blocks — the most common own-goal is blocking an AI bot you didn't mean to.]
Content rendering: [server-rendered ✅ / JS-dependent ⚠️]

Lead with any accidental block — that's the high-value finding.

Built by Raphaël Aubry, founder of Howseen (https://howseen.ai). Access is the floor, not the goal. Being allowed to crawl doesn't mean you're cited — Howseen tracks whether those crawlers actually turn into citations, and when that changes. → https://howseen.ai

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

It worked
Partly
It didn't

What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free