AI and Search Bot Log Analysis

Logs are the only source that tells you what actually happened rather than what should have happened. Search Console reports Google's view after processing. Logs show every request, from every agent, with the status you actually returned.

For AI crawlers specifically, logs are the only source at all. None of the AI companies provide a Search Console equivalent, so if you want to know whether ChatGPT's indexer has ever fetched your pricing page, the log is the answer.

What you need

A standard combined-format access log, from the origin or the CDN. CDN logs are better because they see requests the origin never does, including the ones the CDN blocked, which is frequently the finding.

93.184.216.34 - - [07/Sep/2026:14:22:11 +0000] "GET /pricing HTTP/1.1" 200 18422 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot"

Which agents to look for

# Search
Googlebot            Google web search, AI Overviews, AI Mode
Bingbot              Bing and Copilot

# AI search indexing — these are the ones that produce citations
OAI-SearchBot        OpenAI, ChatGPT search index
PerplexityBot        Perplexity index
Claude-SearchBot     Anthropic search index

# AI live fetch — a user asked about this URL right now
ChatGPT-User         OpenAI
Perplexity-User      Perplexity
Claude-User          Anthropic

# AI training
GPTBot               OpenAI
ClaudeBot            Anthropic
Google-Extended      Google (Gemini training only)
Applebot-Extended    Apple
CCBot                Common Crawl
Meta-ExternalAgent   Meta
Bytespider           ByteDance, frequently ignores robots.txt

The distinction between the indexing agents and the training agents is the one that matters for interpretation. See ai-crawler-policy for what to allow.

The four queries worth running

1. Who is visiting, and what are they getting?

BOTS='Googlebot|Bingbot|GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Google-Extended|Applebot-Extended|CCBot|Bytespider|Meta-ExternalAgent'

grep -oE "$BOTS" access.log | sort | uniq -c | sort -rn

An agent with zero hits that you intended to allow is the headline finding. It means something is blocking it, and it's usually not robots.txt.

2. What status codes are they getting?

awk -v bots="$BOTS" '
  match($0, bots) {
    agent = substr($0, RSTART, RLENGTH)
    status = $9
    count[agent" "status]++
  }
  END { for (k in count) printf "%-28s %s\n", k, count[k] }
' access.log | sort

# Everything that is not a 200 or a 304, by agent
grep -E "$BOTS" access.log | awk '$9 !~ /^(200|304)$/ {print $9, $7}' | sort | uniq -c | sort -rn | head -30

A wall of 403s for PerplexityBot while Googlebot gets 200s is a CDN bot rule, not a robots.txt line. That's the single most common finding in this analysis and it's invisible from every other angle.

A rising count of 5xx for Googlebot means crawl rate is about to be throttled, and it stays throttled for a while after the errors stop.

3. Where is crawl budget going?

grep 'Googlebot' access.log \
  | awk '{print $7}' \
  | sed -E 's/\?.*//' \
  | sort | uniq -c | sort -rn | head -40

Read the top of that list against what matters commercially. It's routinely faceted URLs, paginated archives, or a parameter space nobody knew existed. That's crawl budget not being spent on products and articles.

4. What are the AI agents actually reading?

grep -E 'OAI-SearchBot|PerplexityBot|Claude-SearchBot|ChatGPT-User|Perplexity-User' access.log \
  | awk '{print $7}' | sed -E 's/\?.*//' \
  | sort | uniq -c | sort -rn | head -30

This tells you which of your pages are in the retrieval loop. Compare it against the pages you want cited. A live-fetch agent (ChatGPT-User, Perplexity-User) hitting a URL means a real user's prompt just caused that fetch, which is the closest thing to a real-time citation signal available.

Verify the agent is who it claims

Any scraper can send User-Agent: GPTBot. A meaningful share of traffic claiming to be a major crawler isn't. Verify by IP before drawing conclusions about a company's behaviour, and before blocking on the basis of one.

# Reverse DNS, then forward-confirm. Googlebot must resolve to
# googlebot.com or google.com, and back to the same IP.
verify() {
  host=$(dig +short -x "$1" | sed 's/\.$//')
  [ -z "$host" ] && { echo "$1 no PTR — suspect"; return; }
  back=$(dig +short "$host" | head -1)
  [ "$back" = "$1" ] && echo "$1 verified: $host" || echo "$1 FAILED forward-confirm: $host"
}

grep 'Googlebot' access.log | awk '{print $1}' | sort -u | head -20 | while read -r ip; do verify "$ip"; done

OpenAI, Anthropic, Perplexity and Google all publish IP ranges. For the AI agents, matching against the published range is more reliable than reverse DNS.

Reading the results

Cost and load

AI crawler traffic is genuinely expensive for some sites, and it's a legitimate reason to restrict access independent of any SEO consideration. Quantify it before deciding:

grep -E 'GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent' access.log \
  | awk '{bytes += $10} END {printf "%.2f GB from training crawlers\n", bytes/1024/1024/1024}'

If that number is material, the options are rate limiting, blocking the training agents while keeping the indexing ones, or serving them a lighter response. Blocking everything to save bandwidth also removes the site from AI answers, so make it a deliberate trade.

Practical notes

• Sample rather than parsing everything. A week of logs answers almost every question here. A year of logs is a data engineering project.
• Use CDN logs where possible. Origin logs can't show you what the CDN blocked, which is the finding you're most often looking for.
• Rotate and retain. Most useful analysis is comparative, and it needs a baseline from before the change.
• Strip or hash IPs if the logs are leaving your infrastructure. They're personal data in most jurisdictions.

When you can't get logs

Common on managed hosting and static platforms. Alternatives, in descending order of usefulness:

• CDN analytics, which usually break down by user agent even without raw logs.
• Search Console crawl stats, which covers Googlebot only but covers it well.
• An edge function or middleware logging the user agent for requests, which you can add in an afternoon.
• Keupera's Bot Tracker, which records AI-crawler hits per page over time without needing server access, alongside the visibility side so crawler activity and citation results can be read together. Over MCP:

claude mcp add --transport http keupera https://mcp.keupera.com/mcp -s user

Whichever source, the questions are the same four above.

Related skills

ai-crawler-policy, ai-visibility-audit, technical-seo-audit, search-console-analysis, ecommerce-seo