GCP production triage: monitor, classify, fix

The vercel logs era is over; production is 20 Cloud Run services in project aerial-vehicle-466722-p5 (region us-central1). This skill is the loop an agent runs to answer "is production healthy?" and to act on what it finds. gcloud is already authenticated in this workspace.

Step 1: run the monitor

When the user asks "what's wrong with three.ws?" (or anything equally broad), run the DEEP sweep; it checks everything, not just what happened to log:

npm run triage:gcp -- --json --deep --since 6h   # agents: always --json

For a quick pulse check mid-incident, the fast form is still fine:

npm run triage:gcp -- --json --since 1h

The base monitor merges three signals and classifies every distinct problem signature:

1. https://three.ws/api/healthz: the platform's own subsystem roll-up (database, cache, Helius, x402 ring, world, sniper).
2. A WARNING+ log sweep across every Cloud Run service, fingerprinted so repeats group into one finding.
3. HTTP request logs: 5xx groups per route, 429s.

--deep adds nine concurrent read-only probes, each wrapping an existing standalone audit, normalized into the same findings stream:

A probe that cannot run becomes an investigate finding (a blind spot is not "healthy"); one skipped for missing local secrets is reported as skipped. --skip pages,custodial-keys drops probes when you must (e.g. re-running in a tight loop). The deep sweep takes a few minutes; that is the point.

Exit 0 = healthy or self-healing noise only. Exit 1 = findings[] contains actionable items. Each finding carries class, count, services, sample, and a concrete action. Deep findings carry signature: "deep-<probe>".

Surfaces even the deep sweep does not cover, worth running when the complaint points at them: npm run smoke:mcp (remote MCP endpoints), npm run smoke:x402-facilitator (facilitator verifies a real signed payment without broadcasting it, so it is safe to run any time), npm run audit:web or audit:web:login (real-browser page audit with the QA account).

Step 2: act per class, in this order

Step 3: root-causing an investigate finding

• Read the exact logs with the reader CLI (vercel logs equivalent):

  npm run logs -- -s <service> --errors --since 6h        # errors, one service
  npm run logs -- -s <service> --app --warnings           # app logs only
  npm run logs -- --all --grep "<term>" --since 1d        # search the fleet
  npm run logs:tail                                       # live tail three-ws-api

• 5xx with no matching app-level error means the handler itself returned 5xx (upstream dependency reply), not a crash. Read the handler in api/ for what makes it return that status, and check which upstream it wraps.
• Generation failures: the forge_creations table carries per-generation backend/status/error/prompt (DATABASE_URL from the Cloud Run service env).
• Crash signatures (Uncaught signal, OOM, no available instance): check the revision, memory, and instance ceilings via gcloud run services describe <service> --region us-central1.
• Rollback and LB/DNS/TLS runbook: docs/ops/gcp-production.md.

Step 4: close the loop

• Fixed something in code? Commit locally with explicit paths (never git add -A). Do not push or deploy without owner approval; leave the ship as one command in the report.
• Found a NEW signature that is expected degradation (a fallback firing correctly)? Add it to KNOWN_SIGNATURES in scripts/gcp-triage.mjs AND document it in docs/ops/production-log-triage.md, so the monitor never flags it as investigate again. That is how this system learns.
• Healthz unreachable? That is an outage: check three-ws-api revisions (gcloud run revisions list --service three-ws-api --region us-central1) and the load balancer per the production runbook before anything else.

Report format

Lead with the verdict (healthy / degraded / outage). Then: what you fixed, what you committed, what needs the owner (exact commands), and what is self-healing noise. Solana-facing findings first.