technical-seo
Skill0 starsChanged yesterdayMITView on GitHub
Check whether search engines can crawl and index a site: status codes, redirects and redirect chains, canonical tags, noindex, robots.txt rules and the XML sitemap compared with what a crawl finds. Use when the user asks about indexing or crawling problems, 'why is my page not indexed', 'pages not showing in Google', 'Crawled - currently not indexed', 'Excluded by noindex tag', 'Blocked by robots.txt', canonical issues, 'Duplicate without user-selected canonical', redirect problems, a robots.txt review, or a sitemap check.
11 other projects use a version of this.
Adopted by
Most recently taken up by
- 6t9xstar/one-step-seotheir own version, first seen Nov 8, 2026
- uprinter/ai-agents-teamtheir own version, first seen Nov 8, 2026
- nisarth/claude-skills-storetheir own version, first seen Nov 8, 2026
- astDeniss/business-skillstheir own version, first seen Nov 1, 2026
- manifoldmcp/marketing-skillstheir own version, first seen Oct 17, 2026
- MartinOlivero/saas-buildertheir own version, first seen Oct 13, 2026
Other versions11 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 11
- indranilbanerjee/digital-marketing-pro★ 934same name
- agencia-conversion/agentic-seo-skills★ 51same name
- searchfit/searchfit-seo★ 9same name
- astDeniss/business-skills★ 8same name
- 6t9xstar/one-step-seo★ 5same name
- manifoldmcp/marketing-skills★ 5same name
- MartinOlivero/saas-builder★ 4same name
- nisarth/claude-skills-store★ 1same name
- novatic-cu/nova-skills★ 1same name
- uprinter/ai-agents-team★ 1same name
1 more not listed
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository6 other files it gives its agents.
- ai-search-visibilitySkill
- broken-linksSkill
- on-page-seoSkill
- schema-markupSkill
- seo-auditSkill
- site-migration-checkSkill
The file
We split this skill into the rules it gives an agent
7 rules. 0 are common, 4 are distinctive, and 3 are very specific to this project.
0
Common
Most of the 9 edited versions we read keep these.
4
Distinctive
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 16
Look for the Crawl Cove crawl tools: names ending in crawl_site, load_export, get_issues, get_page, list_broken_links, list_pages.
- Distinctive: line 52
Use 301 or 308 for permanent moves.
- Very specific: line 55
All four host variants (http/https, www/bare) must end at one origin in a single hop.
- Distinctive: line 65
Do not combine noindex with a canonical to another URL.
- Distinctive: line 71
In the sitemap, not found by the crawl: If the crawl was not truncated, these are orphan pages: no internal link path.
- Very specific: line 113
Ask for, or fetch if you can: the URL(s) in question, robots.txt, and the sitemap URL.
- Distinctive: line 115
Read robots.txt and the sitemap against the checklists above.
“Common” is measured: kept by at least 6 in 10 of the 9 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Technical SEO: indexability and crawlability
Answer one question for each important URL: can a search engine reach it, is it allowed to index it, and is this the URL it will choose? Work down the chain in order, because a failure early on makes the later checks irrelevant.
robots.txt allows fetch -> URL returns 200 (after at most one redirect) -> no noindex -> canonical points to itself -> listed in the sitemap and linked internallyStep 0: which mode are you in
Look for the Crawl Cove crawl tools: names ending in crawl_site, load_export, get_issues, get_page, list_broken_links, list_pages.
- Crawl mode: tools present. Use the procedure below.
- Page mode: tools absent (claude.ai chat does not run a plugin's local tools). Use "Page mode" at the end and say you are checking individual URLs, not the site.
State the mode in your first line.
Crawl mode procedure
- Crawl:
crawl_site{ "url": "<home page>", "maxPages": 200 }. LeaveignoreRobotsoff: you want to see what a rule-abiding crawler is kept out of. RecordpageCount,truncatedand the "Skipped per robots.txt" list from the answer. - Status codes:
list_pages{ "indexable": false, "limit": 200 }gives every page that is not a clean, indexable 200. Thenget_issues{ "type": "broken-page", "limit": 200 }for the 4xx, 5xx and fetch errors with their detail. - Redirects:
get_issues{ "type": "redirect-chain", "limit": 200 }. For each,get_pageshows the final URL and hop count. Single redirects are not listed as issues. If the user wants those cleaned up too, callget_pageon the URLs in question: a redirected one has a "Final URL" line. - Noindex:
get_issues{ "type": "noindex", "limit": 200 }. Classify each URL as intended (cart, checkout, account, internal search, thank-you, thin tag or filter pages, staging) or suspicious (anything the business wants found). Verify suspicious ones withget_page. - Canonicals:
get_issues{ "type": "missing-canonical", "limit": 200 }. Then spot-check canonical targets withget_pageon the home page, two or three template representatives (a category, a product or post, a paginated page) and any parameter URLs found vialist_pages{ "urlContains": "?" }. For each, compareCanonicalwithURL. - robots.txt: the crawl tools obey robots.txt but do not show it. Fetch
<origin>/robots.txtyourself with your web fetch tool or a shell (curl -sS <origin>/robots.txt) and read it against the checklist below. If you cannot fetch, ask the user to paste it. - Sitemap vs crawl: find the sitemap from the
Sitemap:lines in robots.txt, else try/sitemap.xmland/sitemap_index.xml. Fetch it (follow one level of sitemap index). Compare its URL set withlist_pagesoutput: see "Sitemap comparison". - Report in the format below.
What to look for, in order of damage
robots.txt
Disallow: /underUser-agent: *on a live site: nothing can be crawled. The usual cause is a staging file shipped to production.- A 5xx on robots.txt: Google treats the whole site as disallowed while it lasts. A 404 is fine and means "no rules".
- Disallowing CSS, JS or image directories that pages need to render.
- A disallowed URL can still be indexed (URL only, no snippet) if other sites link to it. robots.txt controls crawling, not indexing.
- A page that is both disallowed and noindexed: the crawler never fetches it, so it never sees the noindex. To remove a page from the index, allow crawling and serve noindex.
- Rules are per user-agent group, and the most specific group wins outright: a
User-agent: Googlebotgroup replaces the*group for Googlebot, it does not add to it. Within a group the longest matching path wins, and Allow wins a tie. Crawl-delayis ignored by Google.Noindex:lines in robots.txt are not supported.
Status codes
- 5xx and timeouts: urgent, often intermittent. Persistent 5xx gets URLs dropped and slows crawling of the whole host.
- 404 vs 410: both drop from the index; neither is a problem for a page that is really gone. The problem is internal links pointing at them (use the broken-links skill).
- Soft 404: a "not found" or empty page that returns 200. The crawl shows these as normal pages. Clues: the same title on many odd URLs, or titles like "Page not found" under
list_pages. - 403 and 429 in bulk mean the server throttled or blocked the crawler. Report as "could not verify", not as broken pages.
Redirects
- Use 301 or 308 for permanent moves. A 302 or 307 left in place for months is treated as permanent eventually, but states the wrong intent and delays consolidation.
- Chains of 2 or more hops: point the first URL straight at the final one, and update internal links to the final URL. Typical chain:
http://tohttps://towww.to trailing slash, each a separate rule. Collapse them into one rule. - A redirect to an irrelevant page (everything to the home page) is treated as a soft 404.
- All four host variants (
http/https,www/bare) must end at one origin in a single hop. The crawler stays on one origin, so test the other three yourself with a fetch orcurl -sI.
Noindex
- The crawl reads
<meta name="robots">in the HTML. It does not read theX-Robots-Tagresponse header or agooglebot-specific meta tag. If a page is reported indexable but the user says Google reports "Excluded by noindex", check the header:curl -sI <url>. - A noindex page that stays noindexed for a long time ends up with its links not followed either.
Canonicals
- Self-referencing canonical on every indexable page is the safe default.
- A canonical is a hint. Google ignores it when the pages differ or when other signals (internal links, sitemap, redirects) point the other way.
- Faults that lose pages: canonical on every page pointing to the home page; canonical to a URL that redirects, 404s or is noindexed; canonical with the wrong protocol or host; paginated pages canonicalised to page 1 (products on page 2 and later lose their only crawl path); relative canonicals resolved against the wrong base.
- Do not combine noindex with a canonical to another URL. Pick one.
Sitemap comparison
Build three lists and report counts plus up to ten examples of each:
- In the sitemap, not found by the crawl. If the crawl was not truncated, these are orphan pages: no internal link path. If it was truncated, you cannot tell; say so.
- Found by the crawl, indexable, not in the sitemap. Pages the sitemap forgot.
- In the sitemap but should not be: URLs that redirect, return 4xx or 5xx, are noindexed, or whose canonical points elsewhere. A sitemap should list only final, indexable, canonical URLs.
Also note: more than 50,000 URLs or 50 MB uncompressed in one file is invalid; lastmod helps only when it is accurate; Google ignores priority and changefreq. A sitemap is discovery help, not a ranking factor, and listing a URL does not get it indexed.
Report format
# Technical SEO check: <site>
Mode: live crawl, <N> pages (<complete | stopped at the 200-page cap>), robots.txt and sitemap <fetched | not available>.
## Verdict
<two sentences: can search engines crawl and index this site, and the biggest blocker>
## Blockers (fix now)
| URL or pattern | Problem | Evidence | Fix |
## Risks (fix soon)
| URL or pattern | Problem | Evidence | Fix |
## Checked and fine
<one line each: robots.txt, status codes, redirects, noindex, canonicals, sitemap>
## Could not verify
<what the tools cannot see, as it applies>Evidence is the observed value: the status code, the robots meta string, the canonical URL, the robots.txt line. Never a guess.
Wrong conclusions to avoid
- "Indexable" in the tool output means HTTP 200 with no meta noindex. It is not "indexed". Only Search Console's URL Inspection or Pages report says what Google has indexed.
- "Crawled - currently not indexed" in Search Console is usually a content quality or duplication judgement, not a technical block. If the chain above is clean for that URL, say that and stop looking for a technical cause.
- A URL skipped per robots.txt was blocked for this crawler's group (
crawlcove-cli, else*). Read the file before saying Googlebot is blocked. - A redirecting URL carries its destination's tags in the crawl data. Do not report a canonical or noindex "on" a redirecting URL; report it on the final URL.
- Missing titles and H1s on nearly every page point to JavaScript rendering, which the crawler does not do. Google renders JavaScript, later and less reliably; do not report those pages as unindexable.
- For a site bigger than the 200-page cap, an orphan-page or sitemap-coverage claim needs the full crawl. Say once that the Crawl Cove desktop app crawls past the cap (Free plan: 1 project, 1,000 URLs a crawl, £0) and that its JSON export loads with
load_export; do not repeat it.
Page mode (no crawl tools)
- Say you are checking specific URLs and have not crawled the site.
- Ask for, or fetch if you can: the URL(s) in question,
robots.txt, and the sitemap URL. For each page you need the response status, the redirect hops, and the page source<head>. If the user can run commands,curl -sIL <url>gives status and hops. - Walk the chain at the top of this skill for each URL and report the first point where it fails, with the observed value.
- Read robots.txt and the sitemap against the checklists above. A sitemap comparison is limited to "these N sitemap URLs I tested"; do not claim orphan pages.
- Report "Checked N URLs" and list them. Give no site-wide counts.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

