indexing
Skill0 starsChanged 3 months agoMITView on GitHub
When the user wants to fix indexing issues from Search Console, use noindex, or implement Google Indexing API. Also use when the user mentions "fix indexing," "not indexed," "Crawled - currently not indexed," "discovered - currently not indexed," "index coverage," "noindex," "noindex tag," "pages not indexed," "why not indexed," "request indexing," or "Google Indexing API." For sitemap, use xml-sitemap.
7 other projects use a version of this.
Adopted by
Most recently taken up by
- gaznilmuzammil12-prog/seo-skills-by-gaznilword for word, first seen Nov 16, 2026
- sitetiller/ai-visibility-skillstheir own version, first seen Nov 15, 2026
- epicx-labs/keytomic-claude-plugintheir own version, first seen Nov 1, 2026
- austintgriffith/ethskillstheir own version, first seen Oct 8, 2026
- BankrBot/openclaw-skillstheir own version, first seen Oct 8, 2026
- BankrBot/skillstheir own version, first seen Apr 28, 2026
Other versions7 projects use a version of this.
Word for word · 1
- gaznilmuzammil12-prog/seo-skills-by-gaznil★ 0100% same
Edited · 0
None found.
Wrote their own version · 6
- BankrBot/skills★ 1.2ksame name
- BankrBot/openclaw-skills★ 1.2ksame name
- austintgriffith/ethskills★ 303same name
- ffsshhttiikk/opencode-agents-skills★ 2same name
- epicx-labs/keytomic-claude-plugin★ 1same name
- sitetiller/ai-visibility-skills★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository85 other files it gives its agents.
- AGENTS.mdAGENTS.md
- docs/llms-full.txtllms.txt
- docs/llms.txtllms.txt
- llms-full.txtllms.txt
- llms.txtllms.txt
- ab-testingSkill
- accessibilitySkill
- ai-seoSkill
- analyticsSkill
- attributionSkill
- blog-analyzeSkill
- blog-audioSkill
- blog-auditSkill
- blog-brandSkill
- blog-briefSkill
- blog-calendarSkill
- blog-cannibalizationSkill
- blog-chartSkill
- blog-clusterSkill
- blog-discourseSkill
- blog-factcheckSkill
- blog-flowSkill
- blog-geoSkill
- blog-googleSkill
- blog-imageSkill
- blog-locale-auditSkill
- blog-localizeSkill
- blog-multilingualSkill
- blog-notebooklmSkill
- blog-outlineSkill
- blog-personaSkill
- blog-repurposeSkill
- blog-rewriteSkill
- blog-schemaSkill
- blog-seo-checkSkill
- blogSkill
- blog-strategySkill
- blog-taxonomySkill
- blog-translateSkill
- blog-writeSkill
- canonical-tagSkill
- competitorsSkill
- content-optimizationSkill
- content-strategySkill
- copywritingSkill
- core-web-vitalsSkill
- croSkill
- directory-submissionsSkill
- eeat-signalsSkill
- entity-seoSkill
- featured-snippetSkill
- google-search-consoleSkill
- heading-structureSkill
- indexnowSkill
- internal-linksSkill
- keyword-researchSkill
- local-seoSkill
- meta-descriptionSkill
- mobile-friendlySkill
- open-graphSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
8 rules. 0 are common, 6 are distinctive, and 2 are very specific to this project.
0
Common
Most of the 3 edited versions we read keep these.
6
Distinctive
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 21
Check for project context first: If .claude/project-context.md or .cursor/project-context.md exists, read it for site URL and indexing …
- Distinctive: line 69
Rationale: Not all site content should be indexed.
- Distinctive: line 70
Caution: Avoid noindex on important content pages
- Very specific: line 86
noindex,follow vs noindex,nofollow: Use noindex,follow for most cases—excludes from SERP but allows link equity.
- Distinctive: line 90
When intentionally removing a page from the web, choose the method based on whether a relevant alternative exists and whether the page …
- Distinctive: line 100
Before removing: Check the URL's search traffic, backlinks, internal links, and conversion value.
- Distinctive: line 110
Remove deleted URLs from XML sitemap.
- Distinctive: line 111
Update internal links to point directly to the final URL (avoid relying on redirects)
“Common” is measured: kept by at least 6 in 10 of the 3 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
SEO Technical: Indexing
Guides indexing troubleshooting and fix actions. For how to find and diagnose issues in GSC, see google-search-console.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Scope (Technical SEO)
- Fix actions: noindex, canonical, content quality, URL Inspection; verify robots.txt does not block (see robots-txt)
- Noindex: Page-level index control; which pages to exclude and how. Complements robots-txt (path-level crawl control) and google-search-console (Coverage diagnosis)
Initial Assessment
Check for project context first: If .claude/project-context.md or .cursor/project-context.md exists, read it for site URL and indexing goals.
Identify issue from GSC (see google-search-console for Coverage report, issue types, diagnosis workflow). Then apply fix below.
Crawled - Currently Not Indexed
| Cause | Action |
| Low quality, duplicate, off-topic | Improve content, fix duplicates, set correct canonical |
| Static assets (CSS/JS) | See below |
| Feed, share URLs with params | Usually OK to ignore; or noindex, canonical to main URL |
| Important content pages | Use URL Inspection, verify canonical/internal links/sitemap, Request indexing |
Static Assets (Next.js / Vercel)
Vercel adds unique dpl= params to static assets per deploy, creating many "Crawled - currently not indexed" URLs.
| Do | Don't |
Keep robots.txt allowing /_next/ | Do not block /_next/ (breaks CSS/JS loading). See robots-txt |
| Accept static assets in GSC as expected | Do not block /_next/static/css/ or ?dpl= |
| Use X-Robots-Tag for static assets | CSS/JS should not be indexed; no SEO impact |
Static assets in "Crawled - currently not indexed" is normal and expected.
Other Issue Types (from GSC Coverage)
| Issue | Fix |
| Excluded by «noindex» tag | Remove noindex if accidental; keep if intentional |
| Blocked by robots.txt | See robots-txt; remove Disallow for important paths |
| Redirect / 404 | Fix URL or add redirect |
| Duplicate / Canonical | Set correct canonical; usually OK |
| Soft-404 | Page returns 200 but content says "not found" or empty—Google may treat as 404. Fix: return 404 status for truly missing pages; or add real content for 200 pages |
Soft-404
A soft-404 occurs when a page returns HTTP 200 but the content indicates the page doesn't exist (e.g. "Page not found" message, empty state). Google may treat it as 404 and exclude from index.
| Fix | When |
| Return 404 | Page truly doesn't exist; use proper 404 status |
| Add content | Page is intentional (e.g. empty search results); ensure substantive content or use noindex |
| Redirect | If URL moved, use 301 to correct destination |
Noindex Usage
- How:
metadata.robots = { index: false }or<meta name="robots" content="noindex">or X-Robots-Tag - Rationale: Not all site content should be indexed; noindex is a valid choice for many pages
- Caution: Avoid noindex on important content pages
- With robots.txt: robots.txt = path-level crawl control; noindex = page-level index control. Do not block noindex pages in robots.txt—crawlers must access the page to read the directive. Use both: robots for /admin/, /api/; noindex for /login/, /thank-you/, etc. See robots-txt for when to use which.
- nofollow ≠ noindex: nofollow controls link equity only; it does not prevent indexing. To exclude from search, use noindex. See page-metadata for meta robots implementation.
Page Types That Typically Need Noindex
| Category | Page Types | Typical Meta | Reason |
| Auth & Account | Login, Signup, Password reset, Account dashboard | Login: noindex,nofollow; Signup: noindex,follow | No search value; login indexed = security risk; signup follow allows crawl of Privacy/Terms links |
| Admin & Private | Admin, Staging, Test pages, Internal tools | noindex,nofollow | Not for public; avoid discovery |
| Conversion Endpoints | Thank-you, Confirmation, Checkout success, Download gate | noindex,follow | Post-conversion; no SERP value; allow link equity |
| System & Utility | 404, Internal search results, Faceted/filter URLs | noindex,follow or noindex,nofollow | Thin/duplicate; 404 = error state |
| Legal | Privacy, Terms, Cookie Policy (optional) | Often noindex,follow | Low-value indexed; reduces clutter |
| Duplicate & Thin | Printer-friendly, Parameter URLs, Near-duplicate | noindex,follow or canonical | Duplicate content; canonical preferred when possible |
| Low-Value | Media kit, Feedback board (external), Thin press | noindex or index for brand queries | Case-by-case |
noindex,follow vs noindex,nofollow: Use noindex,follow for most cases—excludes from SERP but allows link equity. Use noindex,nofollow only for login (security), staging, or temporary test pages.
Page Removal Decision Framework
When intentionally removing a page from the web, choose the method based on whether a relevant alternative exists and whether the page should remain accessible:
| Scenario | Method | Rationale |
| Has a closely related replacement page | 301 redirect | Preserves accumulated link signals and user flow |
| Content merged into a new page | 301 redirect | Direct old URL to the new canonical location |
| Permanently deleted, no alternative | 410 Gone | Explicitly signals permanent removal to search engines |
| Deleted, uncertain if permanent | 404 Not Found | Safe default; can reinstate later if needed |
| Still accessible but should not be indexed | noindex | Page remains available to users; excluded from SERP |
Before removing: Check the URL's search traffic, backlinks, internal links, and conversion value. If the page has value, consider updating or merging rather than removing.
Common mistakes:
- 404-ing pages that have relevant alternatives (wastes accumulated signals)
- Redirecting all deleted pages to the homepage (breaks user intent)
- Creating redirect chains (A → B → C) instead of direct redirects
- Removing pages without cleaning up internal links pointing to them
- Using
robots.txtto block noindex pages (crawler must access the page to read the noindex directive)
Post-removal cleanup:
- Remove deleted URLs from XML sitemap; update and resubmit
- Update internal links to point directly to the final URL (avoid relying on redirects)
- For 301 redirects, ensure the target URL is in the sitemap
- In GSC, use URL Inspection to verify important pages; use Removals tool for temporary quick-hide (not permanent — use proper HTTP status or noindex)
Google Indexing API
| Type | Typical use |
| JobPosting | Job boards |
| BroadcastEvent | Live platforms |
Requirements: Enable Indexing API, create service account, add owner in Search Console, request quota (default 200 URLs/day).
Output Format
- Action items: Prioritized fixes
- References: Page indexing report
Related Skills
- google-search-console: Find and diagnose indexing issues in GSC
- robots-txt: Path-level crawl control; when to use robots.txt vs noindex; do not block /_next/ or noindex pages
- page-metadata: Meta robots implementation; noindex vs nofollow
- xml-sitemap: Submit and maintain sitemap
- indexnow: Faster indexing for Bing
- canonical-tag: Resolve duplicate content
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

