Monitoring a wizard release

Write the plan before you test the release candidate. Post it on the release PR. Then check it after the merge.

The plan

For each metric, write the insight you will watch, the baseline from the previous version, and the change that counts as a regression. Cover these five:

1. Usage. Runs started per program.
2. Run duration. p50 and p90 wall clock per run.
3. Cost. LLM cost per run.
4. Completion rate. Runs started that finish with status = success.
5. Outcomes. Success, cancelled, and error, with errors split by error code.

Split each one by version. Also split by program_id, harness, and model when the release touches them.

Filters for every query

• build = prod. Other values are dev, ci, and headless.
• Person email does not contain @posthog.com.
• program_id when the release touches one program.

Key events

All of these fired in the last 14 days. Every wizard event carries run_id, version, build, run_surface, program_id, and command.

Traps:

• wizard: agent completed fires once per task under the orchestrator. Its duration_ms is task time, not run time. Never use it to count runs.
• wizard: agent started fires before login, so it is not a sign that work began.
• $ai_generation has run_id but no version. Join it to wizard: started on run_id to get cost per version.
• The orchestrator sets model = chosen-per-task. Per-task models are on wizard: orchestrator task models.
• There is no context mill version property.
• wizard: mcp complete counts coding agents only, and reports the first install attempt. A retry from the results screen fires wizard: mcp install retried with client and success. AI assistants (Claude Desktop/Web, ChatGPT) are counted by wizard: mcp assistant opened.
• Since the MCP step started installing in every detected coding agent, mcp_outcome = skipped is close to zero (there is no picker to decline) and mcp_installed_clients lists every detected coding agent instead of the ones a user ticked. Compare these within a version, not across that release.

Insights to build or reuse

One query covers runs, outcomes, and cost per version. Change the window and the program as needed:

WITH runs AS (
  SELECT toString(properties.run_id) AS run_id,
         argMin(toString(properties.version), timestamp) AS version,
         argMin(toString(properties.program_id), timestamp) AS program
  FROM events
  WHERE timestamp >= now() - INTERVAL 7 DAY
    AND event = 'wizard: started' AND properties.build = 'prod'
  GROUP BY run_id),
fin AS (
  SELECT toString(properties.run_id) AS run_id,
         argMax(toString(properties.status), timestamp) AS status
  FROM events
  WHERE timestamp >= now() - INTERVAL 7 DAY AND event = 'setup wizard finished'
  GROUP BY run_id),
cost AS (
  SELECT toString(properties.run_id) AS run_id,
         sum(toFloat(properties.$ai_total_cost_usd)) AS usd
  FROM events
  WHERE timestamp >= now() - INTERVAL 7 DAY AND event = '$ai_generation'
    AND toString(properties.$ai_product) LIKE 'wizard:%'
  GROUP BY run_id)
SELECT r.version, count() AS runs,
       round(countIf(f.status = 'success') / count() * 100, 1) AS success_pct,
       round(countIf(f.status = 'error') / count() * 100, 1) AS error_pct,
       round(countIf(f.status = 'cancelled') / count() * 100, 1) AS cancelled_pct,
       round(avgIf(c.usd, c.usd > 0), 2) AS cost_per_run_usd
FROM runs AS r
LEFT JOIN fin AS f ON f.run_id = r.run_id
LEFT JOIN cost AS c ON c.run_id = r.run_id
WHERE r.program = 'posthog-integration'
GROUP BY r.version
HAVING runs > 50
ORDER BY r.version DESC

About 10% of runs send no setup wizard finished, so the three shares do not add up to 100.

Starting points

• Wizard v2 overview. Usage by program, abort count, task completions, and AI cost by program.
• wizard-switchboard. Completion, duration, and cost by runner arm, and orchestrator internals. Its filters fix the start date to 2026-07-22 and the program to posthog-integration. Set your own date range.

Tiles that are broken or misleading, as of 2026-10-15:

Compare versions

• Break down by version, or filter to the new version and the previous one. The previous version is your baseline.
• New versions roll out as users run npx @posthog/wizard@latest, so the old version keeps running for days. Compare the two versions over the same days. Do not compare this week with last week.
• For a context mill release, the wizard version does not change. Compare the same wizard version before and after the release time. Add an annotation at the release time so it shows on every chart.
• Watch closely for the first 4 hours: usage, error share, and new error codes. Check again at 24 hours, when completion, duration, and cost have enough runs. Close the plan after 3 days if nothing moved.
• Wait for about 300 runs of the new version on posthog-integration before you trust a completion or cost change. Smaller programs need a longer window.