/bench-check — Performance Regression Check

Run the project's benchmark suite, compare results against a saved baseline, flag regressions beyond a threshold, and optionally update the baseline. Prevents silent performance degradation between releases.

Arguments

• $ARGUMENTS may contain:
  • --save-baseline — run benchmarks and save as the new baseline (no comparison)
  • --compare-only — compare against baseline without updating it
  • --threshold N — regression threshold percentage (default: 15%)
  • No arguments — compare against baseline, then update it if no regressions

Phase 0 — Pre-flight

1. Confirm we're in the codegraph repo root
2. Check that benchmark scripts exist:
   • scripts/benchmark.js (build speed, query latency)
   • scripts/incremental-benchmark.js (incremental build tiers)
   • scripts/query-benchmark.js (query depth scaling)
   • scripts/embedding-benchmark.js (search recall) — optional, skip if embedding deps missing
3. Parse $ARGUMENTS:
   • SAVE_ONLY=true if --save-baseline
   • COMPARE_ONLY=true if --compare-only
   • THRESHOLD=N from --threshold N (default: 15)
4. Check for existing baseline at generated/bench-check/baseline.json
   • If missing and not --save-baseline: warn that this will be an initial baseline run

Phase 1 — Run Benchmarks

Run each benchmark script and collect results. Each script outputs JSON to stdout.

1a. Build & Query Benchmark

output=$(timeout 300 node scripts/benchmark.js 2>&1)
exit_code=$?

If exit_code is 124: record "timeout" for this suite and skip to the next suite. Else if exit_code is non-zero: record "error: $output" for this suite and skip to the next suite.

Extract:

• buildTime (ms) — per engine (native, WASM)
• queryTime (ms) — per query type
• nodeCount, edgeCount — graph size

1b. Incremental Benchmark

output=$(timeout 300 node scripts/incremental-benchmark.js 2>&1)
exit_code=$?

If exit_code is 124: record "timeout" for this suite and skip to the next suite. Else if exit_code is non-zero: record "error: $output" for this suite and skip to the next suite.

Extract:

• noOpRebuild (ms) — time for no-change rebuild
• singleFileRebuild (ms) — time after one file change
• importResolution (ms) — resolution throughput

1c. Query Depth Benchmark

output=$(timeout 300 node scripts/query-benchmark.js 2>&1)
exit_code=$?

If exit_code is 124: record "timeout" for this suite and skip to the next suite. Else if exit_code is non-zero: record "error: $output" for this suite and skip to the next suite.

Extract:

• fnDeps scaling by depth
• fnImpact scaling by depth
• diffImpact latency

1d. Embedding Benchmark (optional)

output=$(timeout 300 node scripts/embedding-benchmark.js 2>&1)
exit_code=$?

If exit_code is 124: record "timeout" for this suite and skip to the next suite. Else if exit_code is non-zero: record "error: $output" for this suite and skip to the next suite.

Extract:

• embeddingTime (ms)
• recall at Hit@1, Hit@3, Hit@5, Hit@10

Timeout: Each benchmark gets 5 minutes max (timeout 300). Exit code 124 indicates timeout — record "timeout" for that suite and continue.

Errors: If a benchmark script fails (non-zero exit), record "error: <message>" and continue with remaining benchmarks.

Phase 2 — Normalize Results

Build a flat metrics object from all benchmark results:

{
  "timestamp": "<ISO 8601>",
  "version": "<from package.json>",
  "gitRef": "<current HEAD short SHA>",
  "metrics": {
    "build.native.ms": 1234,
    "build.wasm.ms": 2345,
    "query.fnDeps.depth3.ms": 45,
    "query.fnImpact.depth3.ms": 67,
    "query.diffImpact.ms": 89,
    "incremental.noOp.ms": 12,
    "incremental.singleFile.ms": 34,
    "incremental.importResolution.ms": 56,
    "graph.nodes": 500,
    "graph.edges": 1200,
    "embedding.time.ms": 3000,
    "embedding.recall.hit1": 0.85,
    "embedding.recall.hit5": 0.95
  }
}

Adapt the metric keys to match whatever the benchmark scripts actually output — the above are representative. The goal is a flat key→number map for easy comparison.

Phase 3 — Compare Against Baseline

Skip this phase if SAVE_ONLY=true or no baseline exists.

For each metric in the current run:

1. Look up the same metric in the baseline
2. Guard against division-by-zero: if baseline == 0, mark the delta as "N/A — baseline was zero" and treat the metric as informational only (not a regression or improvement)
3. Otherwise compute: delta_pct = ((current - baseline) / baseline) * 100
4. Classify:
   • Regression: metric increased by more than THRESHOLD% (for time metrics) or decreased by more than THRESHOLD% (for recall/quality metrics)
   • Improvement: metric decreased by more than THRESHOLD% (time) or increased (quality)
   • Stable: within threshold

Direction awareness: For latency metrics (ms), higher = worse. For recall/quality metrics, higher = better. For count metrics (nodes, edges), changes are informational only — not regressions.

Regression table

Phase 4 — Verdict

Pre-condition check

Before evaluating verdicts, verify that at least one benchmark produced valid numeric results. If metrics is empty (all suites produced "error" or "timeout" records):

• Print: BENCH-CHECK ABORTED — no valid benchmark results (all suites failed or timed out)
• Do NOT proceed to Phase 5 — the baseline must not be overwritten with empty data
• Stop here and skip to Phase 6 to write the report with verdict ABORTED.

Based on comparison results:

No regressions found

• Print: BENCH-CHECK PASSED — no regressions beyond {THRESHOLD}% threshold
• If not COMPARE_ONLY: update baseline with current results

Regressions found

• Print: BENCH-CHECK FAILED — {N} regressions detected
• List each regression with metric name, baseline value, current value, delta %
• Do NOT update the baseline
• Suggest investigation:
  • git log --oneline <baseline-ref>..HEAD to find what changed
  • codegraph diff-impact <baseline-ref> -T to find structural changes
  • Re-run individual benchmarks to confirm (not flaky)

First run (no baseline)

• If COMPARE_ONLY is set: print a warning that no baseline exists and stop here — do not proceed to Phase 5 or Phase 6. No baseline is saved and no report is written.
• Otherwise: print BENCH-CHECK — initial baseline saved and save current results as baseline

Save-baseline with existing baseline (--save-baseline)

• Print: BENCH-CHECK — baseline overwritten (previous: <old gitRef>, new: <new gitRef>)
• Save current results as the new baseline (overwrite existing)

Phase 5 — Save Baseline

Skip this phase if COMPARE_ONLY is set. Compare-only mode never writes or commits baselines. Skip this phase if regressions were detected in Phase 4. The baseline is only updated on a clean run. Skip this phase if the ABORTED pre-condition was triggered in Phase 4. The baseline must not be overwritten with empty data.

When saving (initial run, --save-baseline, or passed comparison):

Write to generated/bench-check/baseline.json:

{
  "savedAt": "<ISO 8601>",
  "version": "<package version>",
  "gitRef": "<HEAD short SHA>",
  "threshold": $THRESHOLD,
  "metrics": { ... }
}

Also append a one-line summary to generated/bench-check/history.ndjson:

{"timestamp":"...","version":"...","gitRef":"...","metrics":{...}}

This creates a running log of benchmark results over time.

After writing both files, commit the baseline so it is a shared reference point:

git add generated/bench-check/baseline.json generated/bench-check/history.ndjson
git diff --cached --quiet -- generated/bench-check/baseline.json generated/bench-check/history.ndjson || git commit generated/bench-check/baseline.json generated/bench-check/history.ndjson -m "chore: update bench-check baseline (<gitRef>)"

git add first so that newly created files (first run) are staged; --cached then detects them correctly. Without this, git diff --quiet ignores untracked files and the baseline is never committed on the first run.

Phase 6 — Report

Skip this phase (write no report) if COMPARE_ONLY was set and no baseline existed, AND the ABORTED pre-condition was not triggered. That early-exit case was already handled in Phase 4 — writing a "BASELINE SAVED" report here would be misleading since no baseline was saved. When ABORTED, always write the ABORTED report regardless of other flags.

Write a human-readable report to generated/bench-check/BENCH_REPORT_<date>.md.

If the ABORTED pre-condition was triggered (no valid benchmark results): write a minimal report — this check MUST come before the SAVE_ONLY/first-run check, because when all benchmarks fail on a --save-baseline or first run, SAVE_ONLY would also be true but no baseline was actually saved:

# Benchmark Report — <date>

**Version:** X.Y.Z | **Git ref:** abc1234 | **Threshold:** $THRESHOLD%

## Verdict: ABORTED — no valid benchmark results

All benchmark suites failed or timed out. See Phase 1 error records for details.

## Raw Results

<!-- Error/timeout records from each suite -->

If SAVE_ONLY is set or no prior baseline existed (first run): write a shortened report — omit the "Comparison vs Baseline" and "Regressions" sections since no comparison was performed:

# Benchmark Report — <date>

**Version:** X.Y.Z | **Git ref:** abc1234 | **Threshold:** $THRESHOLD%

## Verdict: BASELINE SAVED — no comparison performed

## Raw Results

<!-- Full JSON output from each benchmark -->

Otherwise (comparison was performed): write the full report with comparison and verdict:

# Benchmark Report — <date>

**Version:** X.Y.Z | **Git ref:** abc1234 | **Threshold:** $THRESHOLD%

## Verdict: PASSED / FAILED

## Comparison vs Baseline

<!-- Full comparison table with all metrics -->

## Regressions (if any)

<!-- Detail each regression with possible causes -->

## Trend (if history.ndjson has 3+ entries)

<!-- Show trend for key metrics: build time, query time, graph size -->

## Raw Results

<!-- Full JSON output from each benchmark -->

Phase 7 — Cleanup

1. If report was written, print its path
2. If baseline was updated, print confirmation
3. Print one-line summary: PASSED (0 regressions) | FAILED (N regressions) | BASELINE SAVED | ABORTED (all suites failed)

Rules

• Never skip a benchmark — if it fails, record the failure and continue
• Timeout is 5 minutes per benchmark — use appropriate timeout flags
• Don't update baseline on regression — the user must investigate first
• Recall/quality metrics are inverted — a decrease is a regression
• Count metrics are informational — graph growing isn't a regression
• The baseline file is committed to git — it's a shared reference point; Phase 5 commits it on clean (non-regressed) runs where COMPARE_ONLY is not set
• history.ndjson is append-only — never truncate or rewrite it
• Generated files go in generated/bench-check/ — create the directory if needed