Search Console + Bing + IndexNow

Run this after the site is live on its production domain (DNS on Cloudflare, production branch deployed). It makes Google and Bing aware of the site and keeps Bing's index fresh automatically.

Only register the live domain. main and every *.pages.dev preview are noindexed by the kit's functions/_middleware.ts, and after go-live the project alias <project>.pages.dev 301s to the live domain (once CANONICAL_URL is set — new-website/references/CLOUDFLARE_FIRST_DEPLOY.md, "After go-live"). Do not add any pages.dev host as a property; they can't be indexed.

Human-in-the-loop: creating the Google/Microsoft accounts and adding the DNS record are actions only the owner can do (like the GitHub/Cloudflare accounts in SETUP.md). The agent drafts the exact TXT value and the sitemap URL; the owner pastes/clicks.

The verification record can go in earlier. A Domain property is verified by DNS alone, so its TXT record (and Bing's CNAME, for the manual route) can be written on setup day, before the site is live: the new-website setup script writes it, DNS only, with the owner's yes (--record "TXT @ google-site-verification=…", new-website/references/CLOUDFLARE_FIRST_DEPLOY.md, "S"). The owner still copies the value from Google and clicks Verify; the sitemap waits until the site is live.

Console UI labels are localized — the click paths below are quoted from the English UI; if the owner's account language differs, translate the labels for them (German: Property hinzufügen, DNS → Einträge → Eintrag hinzufügen, Aus der Google Search Console importieren, Caching → Konfiguration).

1. Google Search Console (Google)

1. Go to https://search.google.com/search-console and sign in.
2. Add property → "Domain" (covers http+https, www+apex, all subdomains — the right default). Google shows a TXT record to add to DNS.
3. Add the TXT record in Cloudflare (DNS is on Cloudflare): dashboard → your domain → DNS → Records → Add record → Type TXT, Name @, Content = the google-site-verification=… string Google gave you → Save. Back in GSC click Verify (propagation is usually seconds on Cloudflare).
   • Alternative: a URL-prefix property verified by the HTML <meta> tag (drop it in Base.astro's <head>) or an HTML file in public/. Domain+TXT is preferred — broader coverage, survives scheme/subdomain changes.
4. Submit the sitemap: GSC → Sitemaps → enter sitemap-index.xml → Submit. The kit's @astrojs/sitemap emits /sitemap-index.xml at the site root; confirm it loads in a browser first. Google discovers pages from it over the next days.
5. Look at "Search generative AI" in the property's settings, and opt out only if the owner wants out of Google's AI answers. A property can inherit it from a parent property, so check what yours shows. Opting out keeps your content out of AI Overviews, AI Mode and Discover's AI features, usually within a few days. Google says the control "isn't used as a ranking or inclusion signal affecting other parts of Search" and "doesn't affect AI training"; that is Google-Extended (Google).

2. Bing Webmaster Tools (Bing + the IndexNow consumers)

Easiest path — import from Google Search Console (auto-verifies and pulls the sitemap, so no second verification dance):

1. Go to https://www.bing.com/webmasters and sign in (Microsoft account).
2. Choose "Import from Google Search Console" → Continue, sign in to the Google account that owns the GSC property, grant access, select the site → Import.
3. Within minutes the site is added, auto-verified, and existing sitemaps are imported. Confirm under Sitemaps; if it didn't carry over, Submit sitemap → https://<domain>/sitemap-index.xml.

Manual alternative (no Google account): Add a site, verify by the XML file (place in public/), the <meta> tag (in Base.astro), or a CNAME DNS record, then submit the sitemap as above.

3. IndexNow — one Cloudflare toggle (Crawler Hints)

IndexNow lets the site push "this URL changed" to search engines instead of waiting to be crawled. On a Cloudflare-proxied site you get it for free with Crawler Hints — Cloudflare generates and hosts the IndexNow key and sends the notifications for you, so there is nothing to add to the repo:

1. Cloudflare dashboard → your domain → Caching → Configuration.
2. Find Crawler Hints and enable it.

That's it. Cloudflare now notifies the IndexNow network (Bing, Yandex, Naver, Seznam) whenever your content is created/updated/deleted. Requires the records to be proxied through Cloudflare (orange cloud) — true for a Cloudflare Pages site on a Cloudflare-managed domain.

• Google does NOT use IndexNow — it relies on its own crawl signals, so the GSC sitemap (step 1) is how Google stays current. IndexNow speeds up Bing & friends.
• No manual key file: don't add an indexnow.txt/key to public/ — Cloudflare's managed key would conflict. (Only self-host a key if you're NOT using Crawler Hints.)

Also in the Cloudflare dashboard — keep crawlers allowed

The AI crawler settings are under your domain → Security → Settings → filter by Bot traffic → Configure AI bot policies (the names as a dashboard showed them on 2026-10-30). For the open, indexable posture this kit wants, confirm these (the defaults can block). docs/CLOUDFLARE-CHECKLIST.md in the Croftweaver folder, section 4, has the details:

• Search, Agent and Training → "Allow (do not block)". Since 15 September 2026, Cloudflare says, Block and Block on pages with ads on Training also keep out Googlebot, Bingbot and Applebot, "so either setting impacts search as well as training" (Cloudflare). Its way to refuse AI training and stay in Google, Bing and Apple search is Disallow for Training, with Bot Preference Sync on (next bullet); the kit has not tested how crawlers are treated, and the file Sync writes also shuts out Baiduspider and PetalBot, the crawlers of Baidu's and Huawei's Petal search. That is the training choice ai-seo describes (Step 4). Bing does not read the no-training preference yet; Cloudflare says Microsoft's support is "targeted for early 2027".
• Enable Bot Preference Sync → off, unless you chose Disallow for Training. With it on, Cloudflare puts its own lines in front of the repo's own public/robots.txt; with Disallow, those lines are what publish the no-training preference. Choosing Disallow does not turn it on by itself.
• If your dashboard still shows the older Block AI bots setting, make sure it is off. A dashboard on 2026-10-30 no longer showed it.

Assistant: offer to do these Cloudflare dashboard clicks for the user if your runtime can drive a browser. The Crawler Hints toggle and the AI bot policies above are dashboard actions, not repo edits — so offering to perform them beats just listing steps. Pick the path that matches your runtime:

• Claude Code with Claude in Chrome — offer that extension: ask the user to install it (it's what gives real click/navigate access; plain screen control can only read a browser).
• Another agent (Codex, Antigravity, …) — offer your own browser-automation tool if you have one; otherwise walk the user through the clicks.

Whichever path, the shape is the same:

1. The user opens the browser and logs in to Cloudflare themselves — never handle their credentials or 2FA.
2. With their explicit go-ahead and the target zone, navigate to Caching → Configuration (Crawler Hints) and Security → Settings → Bot traffic → Configure AI bot policies, read the current values back, and only then set any that don't match the posture above.

If no browser automation is available or the user declines, fall back to walking them through the clicks. Never attempt these dashboard changes through blind screen control.

4. Confirm it's working

• GSC: the Pages / Indexing report shows URLs moving to "Indexed" over a few days; Sitemaps shows "Success" with the discovered-URL count.
• Bing: Site Explorer lists your URLs; Sitemaps shows the submitted file.
• IndexNow: Bing Webmaster Tools → IndexNow (or URL submission) shows notifications arriving once Crawler Hints has fired on a content change.

Scope notes

• This is search-engine registration, not on-page SEO — the head/sitemap/schema contract is website-seo-geo; technical-SEO/Core-Web-Vitals is seo-audit; GEO/AI answers is ai-seo. Run those before this so what gets indexed is already correct.
• GSC is also the free, no-script, no-consent-banner analytics option from the new-website decision interview (Q5): if the owner only needs "is anyone visiting?", this skill is the whole analytics setup.