Branch Regression

Runs the same example suite on a base branch (usually main) and the current branch, then compares run results, exit codes, new WARN/ERROR log lines, and screenshots. Screenshots are not expected to match 100% (sims are non-deterministic) — they should be close barring an issue.

Requires: Linux host with display + GPU (editor screenshots), nix develop, repo root as cwd.

Workflow

- [ ] 0. Preflight: clean tree, record current branch
- [ ] 1. Base pass: checkout base, build, capture
- [ ] 2. Head pass: checkout original branch, build, capture
- [ ] 3. Compare + visually inspect flagged screenshots
- [ ] 4. Report (and confirm you are back on the original branch)

0. Preflight

git status --porcelain   # MUST be empty
HEAD_BRANCH=$(git branch --show-current)
BASE_BRANCH=main         # or the branch the user named
OUT=ai-context/branch-regression/$(date +%Y-%m-%d)-$BASE_BRANCH-vs-$HEAD_BRANCH
mkdir -p "$OUT"
if [ -f "$OUT/shell-id.txt" ]; then
  ELODIN_SHELL_ID=$(cat "$OUT/shell-id.txt")
else
  ELODIN_SHELL_ID="${ELODIN_SHELL_ID:-branch-regression-$(date +%Y%m%d-%H%M%S)}"
  printf '%s\n' "$ELODIN_SHELL_ID" > "$OUT/shell-id.txt"
fi
export ELODIN_SHELL_ID

If there is any staged or uncommitted work, STOP and ask the user to commit or stash it themselves. Never stash, commit, or discard on their behalf.

Use the same OUT and ELODIN_SHELL_ID when resuming an interrupted run. Never create a fresh shell ID for each branch or example. A caller may choose any path-safe name by exporting ELODIN_SHELL_ID before starting, for example:

export ELODIN_SHELL_ID="branch-regression-$(date +%Y%m%d-%H%M%S)"

Named shell directories persist for later passes and resumed sessions. Shells that start without an explicit ID use a numeric process/session ID and contain target/shells/$ELODIN_SHELL_ID/garbage-collectable; the Nix shell may remove those after the process dies.

1 & 2. Capture each branch

For each branch (base first, then head):

git switch "$BASE_BRANCH"            # then later: git switch "$HEAD_BRANCH"
nix develop --command bash -lc \
  'just install && bash .cursor/skills/branch-regression/capture_branch.sh "$1"' \
  _ "$OUT/base"
# head pass writes to "$OUT/head"

capture_branch.sh writes <example>.png, <example>.log, <example>.exit per example plus commit.txt. Default example set is the known-good editor gallery (ball, three-body, drone, rc-jet, apollo-lander, video-stream, sensor-camera, cube-sat, voyager, geo-frames); pass example names to override. Headless-only examples (frames, linalg, stablehlo, cube-sat-pysim) are run with elodin run for logs/exit only — add them explicitly if wanted.

Build and capture must run in the same nix develop invocation. Each Nix shell has its own virtual environment, so a separate capture shell cannot import the Python package installed by just install. Both branch passes must also inherit the same named ELODIN_SHELL_ID, which reuses the virtual environment and bin directory instead of creating a new session. Still run just install after switching branches so each pass tests binaries built from its own commit.

Rules baked into the script (do not work around them):

• One live editor/sim at a time — everything binds TCP 2240. It waits for the port to free between examples and group-kills leftovers (s10 children respawn on a plain kill).
• Stale DB cleanup for video-stream and voyager before each run.
• ELODIN_SCREENSHOT_EXIT=1 + watchdog; a missing/empty PNG is a capture failure, not proof of a regression — retry once before flagging.

Always git switch back to $HEAD_BRANCH when done or on any error.

3. Compare

nix develop --command uv run python .cursor/skills/branch-regression/compare_runs.py \
  "$OUT/base" "$OUT/head" --rmse-threshold 0.05 --html "$OUT/report.html" \
  > "$OUT/report.md"

Prints a markdown table (exit codes, new WARN/ERROR count, screenshot RMSE, verdict), writes report.html, and exits 1 if anything is flagged. The HTML report must include base/head screenshot thumbnails for every example; each thumbnail links to the full-size image. Always generate both report formats.

Then, for every flagged example, Read both PNGs and judge visually:

• Same scene composition (objects, trails, view cube, graph panels populated)?
• Status bar healthy (RAM > 0, ticks advancing)?
• Is the pixel delta explained by sim phase (a ball mid-bounce vs apex) or is content actually missing/broken?

RMSE above threshold with equivalent-looking scenes = note and pass. Missing geometry, blank viewport, dead graphs, or a new panic = regression.

4. Report

Summarize per example: base vs head exit code, new WARN/ERROR lines (quote them), RMSE, visual verdict. State the two commits compared (from commit.txt). Distinguish regressions (head worse than base) from pre-existing issues (present in both).

Interpreting log diffs

compare_runs.py only reports WARN/ERROR lines that are new in head (timestamps stripped, deduped) — noisy-but-stable warnings on both branches do not flag. Lines that differ only by a pointer/tick number may still slip through; use judgment.

Gotchas

• Rebuilding between branches is mandatory; a stale target/release/elodin or Python wheel silently tests the wrong branch. commit.txt in each capture dir is the audit trail.
• voyager needs SPICE kernels under examples/voyager/nasa_spice_data/; video-stream needs the GStreamer plugins from nix develop. If a prerequisite is missing on both branches, drop the example rather than flagging it.
• Screenshot delay: script default 20 s (ELODIN_SCREENSHOT_DELAY); heavy examples may need 25 s. Same delay on both branches, or RMSE is meaningless.
• Output lives under gitignored ai-context/; never commit captures.
• When resuming after a crash, locate the existing output directory and restore ELODIN_SHELL_ID from shell-id.txt before entering nix develop.