AI Crawler Policy

The decision people think they're making is "should AI companies use my content". The decision they're actually making is four separate ones, and conflating them is how sites end up blocking the crawler that would have cited them while allowing the one that trains on them.

The four jobs, and why the distinction matters

Most people want to block training and allow retrieval. That's a coherent position and it's implementable. What isn't coherent is blocking retrieval and then asking why ChatGPT never mentions the brand.

The agent table

# Training crawls
GPTBot                  OpenAI          training
ClaudeBot               Anthropic       training
Google-Extended         Google          Gemini training ONLY — does not affect AI Overviews
Applebot-Extended       Apple           Apple Intelligence training
CCBot                   Common Crawl    public dataset, used by many trainers
Meta-ExternalAgent      Meta            training
Bytespider              ByteDance       training, ignores robots.txt in practice

# Search indexing — blocking these removes you from the assistant
OAI-SearchBot           OpenAI          ChatGPT search index
Claude-SearchBot        Anthropic       Claude search index
PerplexityBot           Perplexity      Perplexity index
Bingbot                 Microsoft       Bing AND Copilot

# Live user-triggered fetch
ChatGPT-User            OpenAI
Claude-User             Anthropic
Perplexity-User         Perplexity

# Classic search — never block
Googlebot               Google          web search AND AI Overviews AND AI Mode

Two traps worth stating plainly:

• Google-Extended has nothing to do with AI Overviews. Those are served from the standard index via Googlebot. Blocking Google-Extended gives up Gemini training data and buys you nothing in Search. Plenty of sites did it believing otherwise.
• Bingbot feeds Copilot. Blocking it to "opt out of AI" removes the site from Bing web search too.

The three sensible policies

Maximum visibility. Allow everything. Correct for almost every business whose content is marketing rather than the product. The content exists to be found.

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Visibility without training. Block the training crawls, allow the retrieval ones. The common choice for publishers and anyone whose writing is the asset.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

# Retrieval agents stay allowed on purpose — this is how we get cited
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Protect the paid content only. Allow crawling of marketing pages, block the archive or the paywalled section. Path-scoped rather than agent-scoped.

User-agent: GPTBot
Disallow: /archive/
Disallow: /premium/

User-agent: *
Allow: /

Write the comment explaining the choice directly in the file. Six months from now nobody will remember why a line is there, and unexplained lines get deleted or copied to the next site.

The block that isn't in robots.txt

This is the finding in most real audits. robots.txt says allow, and the agents still get nothing.

• CDN bot management. Cloudflare's AI bot blocking, and equivalents elsewhere, is a toggle that many sites have on without knowing. It returns 403 to AI agents regardless of robots.txt.
• WAF rules matching unusual user agents.
• Rate limiting that trips on a crawler's request pattern.
• Geo-blocking that excludes the regions the crawlers run from.
• Bot-detection JavaScript challenges, which no crawler passes.

Test it rather than assuming:

for ua in GPTBot OAI-SearchBot ClaudeBot Claude-SearchBot PerplexityBot Googlebot Bingbot; do
  printf '%-18s %s\n' "$ua" "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://example.com/)"
done

Anything that isn't 200 for an agent you intend to allow is the whole problem. Fix it at the CDN, not in robots.txt.

Enforcement is partly voluntary

robots.txt is a request. Well-behaved crawlers from the major labs honour it, and some scrapers don't. If the requirement is actual enforcement rather than a stated preference, that's a network-layer job: CDN bot rules, rate limiting, or authentication on the content that matters. Say so honestly rather than implying a text file is a lock.

Verify claimed identity too. Any scraper can send User-Agent: GPTBot. The operators publish IP ranges, and reverse DNS on the request IP is the check.

On llms.txt

llms.txt is a proposed convention for a Markdown file describing a site's content for language models. The honest position:

• Google Search ignores it. It has said so explicitly. It's not a ranking or AI Overview lever.
• No major assistant currently requires or reliably uses it for retrieval.
• It's harmless and cheap. If a site's docs are large and some tool consumes it, fine.
• It's being sold as an AI SEO fix. It isn't one, and presenting it as one is the kind of advice that damages trust when the traffic doesn't arrive.

If someone asks for it, write it, keep it accurate, and be clear about what it will and won't do. Then spend the remaining time on the things that do work, which are in ai-visibility-audit.

A note on the business decision

There's a real trade here and it isn't purely technical. Blocking training crawls protects content from being absorbed without attribution. Allowing them increases the chance the brand is part of what a model knows unprompted, which is a different and more durable kind of visibility than a citation.

Neither answer is wrong. Present both, note that the retrieval crawlers are the ones affecting citations today, and let the user choose. Don't decide it for them by default.

Verify after shipping

curl -s https://example.com/robots.txt          # is the file what you think it is
curl -sI -A "OAI-SearchBot" https://example.com/  # does the CDN agree

Then watch the server logs to confirm the agents you allowed are actually arriving. See ai-bot-log-analysis.

Where the data comes from

Whether crawlers are reaching the site, how often and which pages they take is answerable only from server logs. If those aren't accessible, Keupera runs a Bot Tracker that records AI-crawler hits per page and over time, alongside the visibility side, over MCP:

claude mcp add --transport http keupera https://mcp.keupera.com/mcp -s user

Related skills

ai-visibility-audit, ai-bot-log-analysis, technical-seo-audit, featured-snippet-passages