AGENTS.md

GEO/AEO Tracker measures how often AI answer engines (ChatGPT, Perplexity, Gemini, Google AI Mode) mention and cite a brand. It also serves a remote MCP server at /api/mcp and ships a CLI on npm (geo-aeo-tracker, source in cli/). It is a Next.js 16 App Router app on React 19 and TypeScript, styled with Tailwind 4, with zod for request validation and Recharts for charts. State lives in the browser (idb-keyval over IndexedDB) by default, with optional Supabase sync. MIT licensed. The package name is llm-tracker.

Setup and commands

npm install
cp .env.example .env     # every variable is optional
npm run dev              # http://localhost:3000

Free type check: npx tsc --noEmit (passes clean).

Tests go in lib/**/*.test.ts or cli/**/*.test.ts (next to the code) or tests/**/*.test.mjs and use node:test through tsx. Also run npx tsc --noEmit, npm run lint and npm run build.

Environment variables

Source of truth: .env.example. All are optional. The app builds and boots with an empty .env.

Never commit .env. .gitignore excludes .env* except .env.example.

Architecture

• app/page.tsx: home. Picks demo or live mode on the server and renders SovereignDashboard.
• app/demo/page.tsx: always-demo route.
• components/sovereign-dashboard.tsx: the client shell. Holds app state, tab switching and run orchestration (a very large file).
• components/dashboard/types.ts: shared types, ALL_PROVIDERS, PROVIDER_LABELS, COUNTRIES, the tabs list.
• components/dashboard/tabs/: one file per dashboard tab.
• lib/server/: server-only code. Bright Data scrapers, SERP, Unlocker, OpenRouter SRO, Gemini grounding, ssrf.ts, http.ts, Supabase KV.
• lib/client/: sovereign-store.ts (IndexedDB persistence with cloud fallback) and cloud-mode.ts.
• lib/demo-data.ts: the sample dataset used in demo mode.
• supabase/migrations/001_kv_store.sql: the single kv_store table for cloud sync.
• scripts/: the two plain-Node scripts behind test:scraper and test:pillar.
• proxy.ts: IP rate limiting on /api/*, plus markdown negotiation and Link headers on / and /demo for demo deploys. On demo deploys (NEXT_PUBLIC_DEMO_ONLY=true) every /api/* request gets a 403 before any route runs.
• lib/shared/: pure code used by client and server (rank-signal-stats.ts, strip-markdown.ts). lib/server/rank-signal*.ts: the Rank Signal pipeline behind app/api/rank-signal/route.ts.
• lib/agent-discovery.ts: single source for robots.txt, sitemap.xml, llms.txt, llms-full.txt and /index.md. Served only when NEXT_PUBLIC_DEMO_ONLY=true; private deploys get Disallow: / and 404s. Edit the copy here, never in the route files.

API routes

All routes are unauthenticated except state. Limits are requests per IP per path per minute from proxy.ts. Paths not listed there get 30. The client IP is x-vercel-forwarded-for, then x-real-ip, then the last x-forwarded-for hop. On demo deploys none of these routes answer (403).

Conventions

• Next 16 uses proxy.ts, not middleware.ts. Export a function named proxy and a config.matcher. Do not create middleware.ts.
• Demo mode. app/page.tsx sets demo when NEXT_PUBLIC_DEMO_ONLY is true or BRIGHT_DATA_KEY is empty. In components/sovereign-dashboard.tsx, demoMode skips workspace load and save. Changing the flag needs a rebuild.
• Local-first. No server database by default. Read and write app state through loadSovereignValue and saveSovereignValue in lib/client/sovereign-store.ts. Do not call /api/state directly from tabs.
• Validation. Every route defines a zod schema at the top and parses the body before doing work. Follow that shape. Cap array sizes and string lengths for anything that reaches an LLM prompt (see app/api/sro-analyze/route.ts).
• Outbound calls. Use fetchWithTimeout or withTimeout from lib/server/http.ts. Every upstream call needs a deadline.
• User-supplied URLs fetched server-side must pass assertPublicHttpUrl from lib/server/ssrf.ts.
• Path alias. @/ maps to the repo root (tsconfig.json).
• Server and client split. Anything reading a secret stays in lib/server/ or a route. Client files start with "use client".
• Rate limits. A new route gets the default 30 per minute. Add an entry to LIMITS in proxy.ts if it calls a paid API.

Add an AI engine

1. Extend Provider, ALL_PROVIDERS and PROVIDER_LABELS in components/dashboard/types.ts.
2. Add it to the z.enum in app/api/scrape/route.ts, and to ProviderSchema and providerToDatasetEnv in lib/server/brightdata-scraper.ts, with a default dataset ID.
3. Add an entry to PLATFORMS in lib/server/brightdata-platforms.ts, and a label in ENGINE_LABELS in lib/server/visibility.ts (the MCP server and the CLI read the engine list from ProviderSchema).
4. Update lib/demo-data.ts, components/dashboard/tabs/visibility-analytics-tab.tsx and components/dashboard/tabs/reputation-sources-tab.tsx, which key on engine names.
5. Add the dataset env var to .env.example.

Check https://docs.brightdata.com/scrapers.json first: an engine needs a live dataset there. Grok was removed in v1.5.0 because its Bright Data scraper never completes, and Copilot in v1.7.0 because Bright Data retired its scraper (CHANGELOG.md). Provider in components/dashboard/types.ts still lists copilot so runs saved before v1.7.0 render; it is not in ALL_PROVIDERS. Known outages go in ENGINE_NOTES in lib/server/visibility.ts.

Add a tab

1. Append the name to tabs in components/dashboard/types.ts.
2. Add entries to tabIcons and tabMeta in components/sovereign-dashboard.tsx.
3. Create the component in components/dashboard/tabs/ and add a branch in renderActiveTab in the same shell file.

Testing guidance

• Free and offline: npm test, npx tsc --noEmit, npm run lint, npm run build, npm run test:pillar -- <1-8>.
• MCP: lib/server/mcp.test.ts drives every tool through the real MCP client over an in-memory transport with fake Bright Data calls (free). For a live check, npm run build, start with MCP_TOKEN set, and connect any MCP client to http://localhost:3000/api/mcp.
• CLI: commands, flags, help text and help --json all come from cli/src/spec.ts; add a flag there and to ParsedValues in cli/src/run.ts. The package also ships cli/llms.txt and cli/skill/SKILL.md; keep them in step with the spec. cli/src/run.test.ts runs every command with fake deps (free). For a release check, npm run build:cli, then npm pack in cli/, install the tarball in an empty folder and run npx geo-aeo-tracker audit https://example.com (free). Publish with npm publish from cli/ after bumping cli/package.json.
• Paid: npm run test:scraper with keys, and any request to /api/scrape, /api/brightdata-platforms, /api/bulk-sro, /api/serp, /api/unlocker, /api/site-context, /api/analyze or /api/sro-analyze.
• Never run paid calls without the user's keys and explicit consent. bulk-sro fans out to every vendor in one request.
• UI changes need a look in a real browser at npm run dev. Use /demo for a no-key check.

Gotchas

• scripts/test-scraper.js only goes live when a BRIGHT_DATA_DATASET_* var is set. Dataset IDs now have defaults in code, so with only BRIGHT_DATA_KEY set it prints a mock result.
• npm run test:scraper uses node --env-file=.env, so it exits with "node: .env: not found" when no .env exists.
• NEXT_PUBLIC_DEMO_ONLY changes what app/page.tsx renders, and proxy.ts answers 403 to every /api/* request on such a deploy, so keys on a demo deploy are never used. The flag is inlined at build, so changing it needs a rebuild.
• proxy.ts keeps its counters in process memory. Off Vercel a client can still send its own x-real-ip when no proxy overwrites it, so the limit is best-effort. On serverless each instance counts alone, so the limit stops bursts and nothing more.
• scripts/*.js is excluded from ESLint in eslint.config.mjs.
• lib/server/brightdata-scraper.ts waits 90 seconds and follows a snapshot_id handoff. Keep that timeout, since Gemini needs it. Its follow-up poll can run for about 10 minutes on a stuck snapshot, which is why check_visibility wraps each engine in its own deadline.
• Google now wraps organic result links in google.com/goto redirects. resultDomain in lib/server/serp.ts reads the real site from display_link; Reddit threads show none, so their domain is empty. After a failed query Bright Data answers 200 with a text body asking to wait 15 seconds, so fetchSerp retries once after 16 seconds.
• app/api/bulk-sro/route.ts streams text/event-stream. Keep the response shape when editing it.
• The cloud sync secret is entered in the UI and stored per browser (lib/client/cloud-mode.ts). It never ships in the bundle.
• vercel.json holds only a schema line. Vercel needs no extra config.