Web Discovery Artifacts

Generate consistent discovery output from verified route and metadata sources.

When Portable Web Toolkit manages the project, use its generators and discovery-doctor as the specialist authority. Do not create a second route inventory, discovery generator, or deployment gate.

Determine intent

1. Establish whether the site is public, private, staged, or intentionally non-indexed.
2. Identify authoritative route, origin, locale, content, ownership, image, and update-date sources.
3. Inventory framework integrations and existing generated artifacts.
4. Choose only outputs with a clear consumer and maintenance path.
5. Prefer a supported framework integration over a custom generator when it satisfies the requirements.

Do not present llms.txt or other emerging conventions as universally adopted standards.

Candidate outputs

• robots.txt
• sitemap.xml or sitemap index
• canonical and alternate-language links
• Open Graph and platform-specific social metadata
• JSON-LD matching visible page content
• /.well-known/security.txt
• llms.txt or a project-defined AI-readable index
• content or search feeds
• humans.txt when the project has a reason to publish it

Build one source of truth

• Derive related outputs from shared route and metadata data.
• Avoid hand-maintained URL duplication.
• Include only canonical, public, resolvable routes.
• Make environment origin and indexing policy explicit.
• Keep build-time and runtime generation consistent.
• Fail generation on material invalidity only when the repository intends that as a build gate.
• Keep structured data factual, page-specific, and aligned with visible content.

Use references/validation.md for protocol-specific checks.

Environment safety

Production should publish the intended canonical origin and crawler policy. Preview, staging, and development should prevent accidental indexing where appropriate and must not leak private routes, internal APIs, secret-bearing URLs, or production-only data.

Validate

Check:

• syntax and schema
• canonical host and protocol consistency
• route coverage and exclusions
• duplicate, redirected, or broken URLs
• content type and cache behavior
• environment-specific robots and canonical behavior
• image dimensions and resolvability
• JSON-LD correspondence with page content
• security-contact ownership and expiry
• generated-output freshness

When deployment is authorized, verify live responses rather than only build artifacts. Report selected outputs, source of truth, environment behavior, checks, and unresolved gaps.