technical-seo-ai-crawler-audit
Skill14 starsChanged 7 months agoMITView on GitHub
Diagnose and prioritize the technical issues blocking a site from being crawled, indexed, ranked, and cited. Modern technical SEO has two audiences: Google's crawler/indexer and AI answer engines (GPTBot, PerplexityBot, ClaudeBot, etc.). Both need HTML-visible content and a clean crawl path.
In this repository28 other files it gives its agents.
- AGENTS.mdAGENTS.md
- ad-spend-allocatorSkill
- ai-search-visibility-aeo-geo-llmoSkill
- bofu-seo-aeo-strategySkill
- campaign-analyzerSkill
- content-effort-auditSkill
- google-ads-account-auditSkill
- google-ads-campaign-builderSkill
- google-ads-keyword-engineSkill
- google-ads-optimizerSkill
- google-ads-scriptsSkill
- google-merchant-centerSkill
- link-building-digital-prSkill
- local-seo-google-business-profileSkill
- meta-ads-account-auditSkill
- meta-ads-campaign-builderSkill
- meta-ads-creative-engineSkill
- meta-ads-optimizerSkill
- meta-ads-playbookSkill
- meta-ads-tracking-setupSkill
- on-page-seo-aeo-optimizationSkill
- paid-media-reporterSkill
- paid-search-ads-playbookSkill
- prompt-pickerSkill
- seo-aeo-content-strategySkill
- seo-keyword-research-intent-mappingSkill
- seo-max-ktSkill
- seo-reporting-measurementSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 3 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 21
JavaScript rendering audit: Check what renders in raw HTML vs client-side.
- Distinctive: line 209
A site audit without GSC access is incomplete: If you don't have GSC, your first deliverable is "set up GSC." Everything else is guesswork.
- Distinctive: line 211
Log file analysis reveals what crawler-based tools cannot: If you're diagnosing crawl budget, ranking loss, or AI bot access, server logs …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Why this matters
- 72% of websites fail at least one critical technical SEO factor (third-party site-health benchmarks, 2025).
- AI crawlers read raw HTML only — they do not execute JavaScript. If critical content renders client-side, it's invisible to ChatGPT, Perplexity, and most AI assistants. SSR or SSG is mandatory for AI visibility.
- Cloudflare changed its default to block AI bots. Many sites accidentally block GPTBot/PerplexityBot/ClaudeBot without realizing it. Check this before anything else if AI visibility is a goal.
- WordPress specifically: only 43.44% pass Core Web Vitals (Custom Web Audits). Google research shows a 32% higher bounce at 3-second load time.
- Core Web Vitals thresholds (field data, 75th percentile): LCP ≤2.5s, INP ≤200ms (replaced FID in March 2024), CLS ≤0.10.
Process
- Baseline snapshot. Capture current state before touching anything: GSC coverage counts, CWV scores, indexed pages, top-ranking pages, crawl budget utilization. You cannot measure improvement without a baseline.
- Crawlability audit. robots.txt rules, crawl budget, server errors (4xx/5xx), crawl depth, orphan pages, sitemap health.
- Indexation audit. GSC coverage report, noindex issues, canonical conflicts, duplicate content, parameter handling, soft 404s.
- Core Web Vitals. Field data from CrUX/GSC first — not Lighthouse lab scores. Lab scores are a starting point for diagnosis, not the KPI.
- Site architecture review. Click depth from homepage, internal link distribution, URL structure, faceted navigation, pagination handling, breadcrumb implementation.
- JavaScript rendering audit. Check what renders in raw HTML vs client-side. View-source test on key templates.
- Structured data audit. Schema validation, coverage by page type, error rate, rich result eligibility.
- AI crawler accessibility. robots.txt rules for GPTBot/PerplexityBot/ClaudeBot/Google-Extended/Bytespider. Cloudflare settings. Server log analysis for AI bot user agents. llms.txt file presence.
- Mobile-first verification. Mobile Googlebot is primary. Confirm mobile renders equivalent content, not a stripped version.
- Security. HTTPS site-wide, no mixed content, valid cert, HSTS if appropriate.
- Prioritized fix list. Critical → High → Medium → Low, ordered by impact and effort.
Frameworks
Audit priority matrix
Always fix in this order — later work is wasted if earlier layers are broken:
- Crawl/index blocking issues (robots.txt errors, accidental noindex, server errors). If Google can't crawl or index, nothing else matters.
- AI crawler access (if AI visibility is a goal). Block discovery = no citation.
- Core Web Vitals failures on key templates (LCP, INP, CLS).
- Duplicate content / canonical conflicts that split link equity.
- Site architecture (click depth, internal linking).
- Structured data coverage and errors.
- Refinements (image optimization, security headers, minor schema).
Core Web Vitals thresholds
| Metric | Good | Needs improvement | Poor |
| LCP (Largest Contentful Paint) | ≤ 2.5s | 2.5s – 4.0s | > 4.0s |
| INP (Interaction to Next Paint) | ≤ 200ms | 200ms – 500ms | > 500ms |
| CLS (Cumulative Layout Shift) | ≤ 0.10 | 0.10 – 0.25 | > 0.25 |
Thresholds apply to the 75th percentile of field data (real-user monitoring), not lab scores. Get field data from: GSC Core Web Vitals report (free, authoritative), CrUX dashboard, PageSpeed Insights (field tab, not lab), or your RUM tool.
INP replaced FID in March 2024. Do not use FID as a KPI.
AI crawler access checklist
| Check | What to verify | How |
| robots.txt | GPTBot, PerplexityBot, ClaudeBot, Google-Extended, Bytespider, CCBot not blocked | Fetch https://domain.com/robots.txt, grep for each user-agent |
| Cloudflare AI bot blocking | "Block AI Bots" toggle is OFF in Cloudflare dashboard (default: ON as of 2024) | Cloudflare dashboard → Security → Bots → "Block AI Bots" |
| Server logs | AI bot user agents are hitting the site (ChatGPT-User, PerplexityBot, ClaudeBot) | Grep last 30 days of access logs for each UA string |
| JavaScript rendering | Critical content visible in raw HTML | Right-click → View Source on key templates; search for visible text |
| llms.txt | File at site root providing AI-readable site structure | Fetch https://domain.com/llms.txt |
| Logged-in / paywall content | Gated content is not accessible to AI crawlers (by design) | Confirm this is intentional |
robots.txt example (permissive for AI):
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /Note on robots.txt semantics: Absence of a Disallow for a user-agent means allowed. The explicit Allow: / block above is belt-and-suspenders — useful for clarity and for sites that have a global disallow earlier in the file.
Common technical issues → fix actions
| Issue | Fix |
| Duplicate content across URL variants | Canonical tags to preferred URL; 301 redirect when safe |
| Thin content (<300 words, no unique value) | Noindex, consolidate with related page, or rewrite |
| Redirect chains (3+ hops) | Collapse to single 301 hop |
| Mixed internal linking (http + https, www + non-www) | Standardize on one version; redirect the others |
| Orphan pages (no internal links pointing to them) | Add internal links from topically relevant pages |
| Infinite faceted navigation generating URLs | noindex + parameter handling; disallow crawler traps |
| Slow LCP on key templates | Preload hero image, remove render-blocking CSS/JS, optimize server response |
| Poor INP on interactive pages | Break up long tasks, defer non-critical JS, audit third-party scripts |
| CLS from dynamically-inserted content | Reserve space for ads/images with CSS aspect-ratio or fixed dimensions |
| Schema errors (GSC Enhancement reports) | Fix per Schema.org spec; validate with Rich Results Test |
| Content rendered client-side (invisible to AI) | Migrate to SSR/SSG; at minimum, server-render the critical above-fold content |
Monitoring cadence
- Weekly: GSC for new Coverage errors, manual actions, security issues; uptime.
- Monthly light check: Key page indexation, CWV scores (mobile + desktop), new issues in GSC Enhancements, AI bot hits in logs, Cloudflare bot settings unchanged.
- Quarterly deep audit: Full crawl (desktop crawler), log file analysis, schema coverage, internal link audit, redirect chains.
- Emergency audit: Post-migration, post-redesign, post-CMS change, post-CDN change, post-algorithm update with traffic drop.
Tool reference (free-first)
| Purpose | Free | Paid (if needed) |
| Crawl errors + indexation | Google Search Console (authoritative) | — |
| Full site crawl | Free desktop crawler (URL limits vary) | Commercial crawl suite |
| Field CWV data | GSC Core Web Vitals report, CrUX dashboard | — |
| Lab CWV / diagnosis | PageSpeed Insights, Chrome DevTools Lighthouse | — |
| Rendering check | Chrome DevTools, URL Inspection in GSC | Prerender.io, Rendertron |
| Schema validation | Rich Results Test, Schema.org validator | — |
| Log file analysis | awk/grep on access logs | Log analyzers, SIEM (e.g. Splunk) |
| Competitive technical comparison | — | Commercial SEO audit tools |
GSC is the only fully authoritative source — everything else is a proxy. Always ground findings in GSC when possible.
Output format
## Technical SEO Audit Report
**Site:** [URL]
**Audit Date:** [Date]
**Crawl Tool:** [Desktop crawler / cloud crawl suite / etc.]
**Pages Crawled:** [X]
**Data Sources:** GSC, CrUX, [other]
### Health Score: [X/100]
### 🔴 Critical — Blocks crawling/indexation/AI citation
1. **[Issue]** — [X pages affected]
- Evidence: [GSC report / crawl finding / log line]
- Fix: [Specific action]
- Effort: [Low/Medium/High]
- Priority: **Immediate**
### 🟡 High Priority — Impacts rankings or CWV
1. **[Issue]** — [X pages affected]
- Fix: [Specific action]
- Priority: **This sprint**
### 🟢 Medium / Low — Refinements
...
### AI Crawler Status
| Bot | Allowed in robots.txt | Cloudflare | Seen in logs (last 30d) | Notes |
|-----|----------------------|------------|-------------------------|-------|
| GPTBot | | | | |
| PerplexityBot | | | | |
| ClaudeBot | | | | |
| Google-Extended | | | | |
### JavaScript Rendering Check
| Template | Renders in raw HTML | AI-visible | Action |
|----------|---------------------|------------|--------|
| Homepage | | | |
| Article | | | |
| Product | | | |
| Category | | | |
### Core Web Vitals (Field Data, 75th percentile)
| Metric | Mobile | Desktop | Status |
|--------|--------|---------|--------|
| LCP | | | |
| INP | | | |
| CLS | | | |
### Schema Coverage
| Page type | Schema type | Implemented | Valid | Errors |
|-----------|-------------|-------------|-------|--------|
### Monitoring Plan
- **Weekly:** [metrics]
- **Monthly:** [checks]
- **Quarterly:** [full audit scope]
- **Next audit:** [date]Example — mid-sized SaaS audit
Site: example-saas.com (2,400 URLs, WordPress on Cloudflare) Baseline: 780 indexed, GSC Coverage shows 120 "Crawled — not indexed," CWV mobile all red
Key findings:
- 🔴 Cloudflare blocking all AI bots by default — zero AI bot hits in logs for 60 days. ChatGPT, Perplexity, Claude cannot reach the site at all.
- 🔴 200 thin blog archive pages (
/tag/*,/author/*) indexed, cannibalizing real content. GSC shows "Duplicate without user-selected canonical" on 180 of them. - 🟡 LCP 3.8s mobile on the primary blog template — unoptimized hero image (1.2MB PNG, should be WebP ~180KB) and render-blocking theme CSS.
- 🟡 React-based pricing page — pricing numbers render client-side. Invisible to AI crawlers that do not execute JS.
- 🟢 Schema coverage is strong on articles but missing on product/pricing pages.
Prioritized fixes:
- [Immediate] Disable "Block AI Bots" in Cloudflare dashboard. Add explicit
Allowdirectives in robots.txt for GPTBot, PerplexityBot, ClaudeBot, Google-Extended. Verify with log sampling within 7 days. - [Immediate] Noindex tag/author archive pages in WordPress settings (Yoast → Taxonomies).
- [This sprint] Compress hero images site-wide; convert to WebP; preload on blog template. Target mobile LCP <2.5s.
- [This sprint] Server-render pricing page critical content. At minimum, output pricing numbers in raw HTML with JS progressive enhancement.
- [Next sprint] Add Product + Offer schema to pricing page; add FAQPage schema to help center.
Monitoring: Weekly GSC Coverage + CWV check. Monthly log sample for AI bot hits. Quarterly deep audit.
Guidelines
- Fix crawl/index blockers before optimizing CWV. A fast page that isn't indexed is worth nothing. Sequence matters.
- Use field data (real users), not lab data, for CWV KPIs. Lighthouse lab scores are a diagnostic starting point, not the target. GSC/CrUX is authoritative.
- Check AI bot access before assuming "AI isn't finding us." Cloudflare's default-block setting catches many teams by surprise.
- AI visibility ≠ SEO visibility. An SSR'd raw-HTML page is the floor for AI citation. If critical content is in a
<div id="root">only, AI tools cannot see it. - A site audit without GSC access is incomplete. If you don't have GSC, your first deliverable is "set up GSC." Everything else is guesswork.
- Don't chase a 100/100 Lighthouse score. Chase CWV pass rates in field data. Perfect lab scores at the cost of real-user performance is the wrong trade.
- Log file analysis reveals what crawler-based tools cannot. If you're diagnosing crawl budget, ranking loss, or AI bot access, server logs are the ground truth.
- Treat llms.txt as experimental. Its ecosystem adoption is early. Implement it, but do not rely on it — standard robots.txt and HTML-renderable content remain primary.
- Never skip the mobile check. Google's mobile-first indexer is the primary index. "Looks fine on desktop" is irrelevant.
- Quarterly audits are the default cadence for most sites. Monthly for large e-commerce or news sites. Post-migration audits are non-negotiable.
Cross-skill handoffs
- → seo-onpage-optimization: Hand off specific pages with schema gaps, thin content, or rendering issues for page-level fixes.
- → aeo-ai-search-visibility: Hand off AI crawler access findings and JS-rendering gaps as inputs to citation readiness.
- → seo-reporting: Hand off baseline metrics and CWV field data as tracked KPIs.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

