sitetillersitetiller/ai-visibility-skills/skills/ai-logs/SKILL.md

ai-logs

Skill0 starsChanged 2 days agoMITView on GitHub

Count real AI and search crawler visits in server access logs and separate verified bots from fakes using each vendor's published IP ranges (OpenAI, Google, Bing, Perplexity). Also counts visitors referred by ChatGPT, Perplexity, Copilot, Gemini and Claude. Use when the user asks 'is ChatGPT/GPTBot actually crawling my site', 'which AI bots visit us', 'are these bots real', 'how much traffic comes from ChatGPT', or shares an access.log / nginx / Apache / Cloudflare log export.

In this repository44 other files it gives its agents.
- AGENTS.mdAGENTS.md
- agent-readySkill
- ai-citationsSkill
- ai-crawlersSkill
- ai-trackerSkill
- backlinksSkill
- cannibalizationSkill
- canonicalsSkill
- citabilitySkill
- comparison-pagesSkill
- competitorsSkill
- content-briefSkill
- content-decaySkill
- content-qualitySkill
- core-web-vitalsSkill
- ecommerce-seoSkill
- ga4-aiSkill
- ghost-publishSkill
- gsc-bingSkill
- hreflangSkill
- imagesSkill
- indexingSkill
- internal-linksSkill
- llms-txtSkill
- local-seoSkill
- measure-rollbackSkill
- onpageSkill
- platformsSkill
- pseo-quality-gateSkill
- quick-winsSkill
- redirects-migrationSkill
- schemaSkill
- seo-planSkill
- seo-reportSkill
- setupSkill
- site-architectureSkill
- sitemapsSkill
- static-site-publishSkill
- sxoSkill
- tech-auditSkill
- titles-metaSkill
- topic-clustersSkill
- weekly-checkSkill
- wordpress-publishSkill

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.

≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.

Common
Distinctive
Very specific

Distinctive:
Run the analysis (downloads the vendors' IP lists once per run):
line 18
Very specific:
Read per bot: hits, verified, spoofed, unverifiable, ips, status mix and top paths.
line 23
Distinctive:
Never present unverifiable counts as confirmed bot visits.
line 44

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.

Files

SKILL.md

AI crawlers in your logs

robots.txt says who may visit. Logs say who did. Anyone can put "GPTBot" in a user agent, so a hit only counts as OpenAI's when the IP is inside OpenAI's published ranges. This skill reports both.

Inputs

One or more access logs in common or combined log format (nginx and Apache defaults). Gzip: decompress first.
Other formats (Cloudflare Logpush JSON, CDN CSV): convert to combined format, or ask for the columns ip, time, request, status, referer, user agent and convert with a short script.

Steps

Run the analysis (downloads the vendors' IP lists once per run):
```
npx sitetiller logs access.log access.log.1 --json > ai-logs.json
```
Offline: --no-verify counts by user agent only and marks every hit unverifiable.
Read per bot: hits, verified, spoofed, unverifiable, ips, status mix and top paths.
Interpret:

Finding	Meaning	Action
`verified` hits on search bots (OAI-SearchBot, PerplexityBot, Bingbot, Googlebot)	The engines can read the site	Check that top paths are the pages you want cited
Search bot hits mostly `4xx/5xx` or `403`	Bot reaches the server but is refused	Check WAF/CDN rules and robots.txt (`ai-crawlers` skill)
`spoofed` > 0	Scrapers using a vendor's name	Safe to rate-limit or block those IPs; it does not affect the real bot
`unverifiable`	Owner publishes no IP list (Anthropic, Apple, Amazon, Meta, others)	Treat counts as an upper bound
ChatGPT-User / Perplexity-User hits on a page	A person asked an assistant about that page	Strong signal the page is being shown or discussed
`aiReferrals`	People clicked through from an assistant	These are visits; compare with GA4 (`ga4-ai` command)
No AI bot hits at all over weeks	Blocked, or the site is not discovered	Check access first, then sitemaps and Bing indexing

Output to the user

Period covered and lines parsed.
Table per bot: purpose, hits, verified, spoofed, unverifiable, top 3 paths, error rate.
AI referral visits by source and landing page.
Three concrete actions.

Guardrails

Logs contain IP addresses (personal data in many jurisdictions). Process them locally, do not paste raw logs into third-party tools, and report aggregates.
Never present unverifiable counts as confirmed bot visits.
IP lists change; the tool downloads them on each run. Cite the list creationTime if the user needs an audit trail.

References

references/sources.md

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

It worked
Partly
It didn't

What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free