Regression Testing for LLM Systems

The Core Problem

LLM systems have two sources of variance that look identical from the outside:

1. Code/prompt changes — something you did caused the behavior to change
2. LLM non-determinism — same code, same prompt, different output due to sampling variance

You cannot distinguish between these without a controlled protocol. Every "fix" that isn't measured is a guess.

Step 0: Establish Determinism First

Before running any comparison, verify your system is as deterministic as possible.

Check these in order:

1. Temperature = 0 — is it set in your LLM config?
2. Seed parameter — is a fixed seed set? (OpenAI: seed=42, best-effort but significantly reduces variance)
3. No shared mutable state — verify no module-level mutation, no shared checkpointer across runs
4. No parallel runs — sequential only; parallel runs share API quota and cause cascading rate limit errors

Verify determinism before proceeding: Run the same batch file twice with identical parameters. If outputs differ significantly, you have a determinism problem that must be fixed before any comparison is meaningful.

# Run 1
python terminal_app.py --batch-file test-inputs/core.txt \
  --logs-dir logs/determinism-check/run1 --max-critic-rounds 3

# Run 2 (immediately after)
python terminal_app.py --batch-file test-inputs/core.txt \
  --logs-dir logs/determinism-check/run2 --max-critic-rounds 3

# Compare outputs manually

If runs differ: do not proceed. Fix determinism first (seed, temperature, state isolation).

Step 1: Define Your Test Suite

A test suite must have:

• Fixed cases — same queries every time, in the same order
• Known expected outputs — what "correct" looks like for each case
• Difficulty labels — easy cases (should always pass) and hard cases (aspirational)
• Minimum size — at least 5-7 cases covering different query types

Easy cases are your regression canaries. If an easy case fails after a change, something is wrong regardless of whether the hard cases improved.

Step 2: Establish a Baseline

Before making any change, run the full test suite and record:

baseline_commit: <hash>
date: <ISO date>
results: [case1: sufficient, case2: sufficient, case3: no_evidence, ...]
pass_rate: N/M
notes: <any known instability>

This is your ground truth. Every subsequent run is compared against this.

Step 3: Controlled Comparison Protocol

When comparing two versions (A vs B), these MUST be identical across both runs:

If any parameter differs, the comparison is not valid for causal claims.

Document every run with a _metadata.txt:

commit: <hash> <message>
command: <full command>
batch_file: <path>
max_critic_rounds: <N>
sleep_seconds: <N>
model: <model>
temperature: <N>
seed: <N>
started_at: <ISO timestamp>
results: <per-case outcomes>
pass_rate: N/M

Step 4: Interpreting Results

A change is an improvement if:

• Pass rate increased AND
• No previously-passing easy case regressed

A change is neutral if:

• Pass rate unchanged within ±1 case (could be noise)

A change is a regression if:

• Any previously-stable easy case now fails
• Pass rate dropped

A result is inconclusive if:

• Only 1-2 runs available (need ≥3 for statistical signal)
• Parameters differed between runs
• Determinism not verified

Step 5: Bisect Protocol (for regressions)

When a regression appears and you don't know which change caused it:

1. Identify the last known-good commit (last run where all easy cases passed)
2. Identify the first known-bad commit (first run where regression appeared)
3. List all commits between them that touched LLM behavior (prompts, node logic, routing)
4. Binary search: checkout midpoint commit, run stability batch, check result
5. Repeat until you find the first bad commit

Key insight: Not all commits affect LLM behavior. Commits that only change display logic, docs, or non-LLM utilities can be skipped in the bisect.

Document the bisect:

last_good: <commit> — <description> — <pass_rate>
first_bad: <commit> — <description> — <pass_rate>
bisect_result: <commit> — <what it changed> — confirmed by <N> runs

Step 6: Statistical Thresholds

For production systems, define minimum acceptable pass rates per case difficulty:

A single run proving nothing is the most common mistake. Budget for multiple runs before claiming improvement.

Cost estimation before running:

N cases × M LLM calls per case × P runs × cost_per_call = total cost

Calculate this before starting any validation cycle. If cost is prohibitive, reduce P (runs) or N (cases) but document the tradeoff.

Known Failure Modes