writing-tests

---
name: writing-tests
description: Use when writing or changing tests in this repo, deciding what to test or where a test belongs, adding a regression test for a bug fix, reviewing the test portion of a PR, deciding which golden outputs to bless and reviewing the blessed diff, or adding a benchmark. Trigger phrases include "add a test", "write tests for", "is this covered", "does this need a test", "review the tests here", "which layer should this test go in". Also use for the mechanics of the golden framework itself — the //! > file format, creating test_file_test! runners, or how CAIRO_FIX_TESTS works (covered in references/golden-framework.md).
---

# Writing tests for the Cairo compiler

Goal: the right tests, not more tests. A test earns its keep by failing when
behavior regresses and staying green through refactors. This repo's history
shows the dominant failure mode is missing cases (error paths, boundaries),
not over-testing — when in doubt about a rejection path, write the case.

## Modes

- **Writing tests** (developing a change): continue below.
- **Reviewing tests** (judging a PR's test portion): walk
  `references/review-checklist.md` top to bottom.

## Test-placement decision tree

Pick exactly one home layer per behavior — the lowest layer that can observe
it. A behavior covered at its home layer is not re-asserted at other layers.

1. **Pure Rust logic** (span arithmetic, id/key construction, data structures,
   formatting helpers) → Rust unit test in the sibling `foo_test.rs` file.
   History: off-by-one bugs in helpers (#10017 span end, #10003 n_steps)
   escaped because only golden layers were exercised and they never observe
   helper internals. See `references/rust.md`.
2. **Stage output or diagnostic** (parser tree, semantic/lowering diagnostic,
   lowering text, sierra-gen phase output, plugin expansion, formatter output)
   → new case in the existing golden test_data file of that stage's crate.
   A new case in an existing file beats a new file: the runner, wiring, and
   review habits already exist. See `references/rust.md`; framework mechanics
   in `references/golden-framework.md`.
3. **Language or runtime semantics** (what executing the Cairo code does:
   corelib behavior, arithmetic, storage/events/syscalls) → Cairo `#[test]`
   in `corelib/src/test/` or `crates/cairo-lang-starknet/cairo_level_tests/`.
   The VM is the only execution oracle this repo has — codegen goldens prove
   output stability, not that the generated code runs correctly (#10069:
   empty-enum `Store` derive produced invalid code for 3 years while its
   plugin golden stayed green). See `references/cairo.md`.
4. **Libfunc-level codegen, gas, or ap-change** → one minimal case in
   `tests/e2e_test_data/libfuncs/`. This is the de-facto oracle for
   `sierra-gas`/`sierra-ap-change`, which have almost no tests of their own.
5. **Whole-program regression from a reported issue** →
   `tests/bug_samples/issueNNNN.cairo`.
6. **Performance-sensitive path** → benchmark, only per
   `references/benchmarks.md` (benches run nightly, not on PRs).

Go up a layer only when the behavior is the _interaction_ of stages; reserve
e2e (`tests/e2e_test_data/`) for behavior that individual stage suites cannot
express. When a plugin/codegen change alters what generated code _does_ at
runtime, add the executed Cairo-level test in addition to the codegen golden —
that pair is two different behaviors (shape vs. semantics), not duplication.

## Deriving the cases

Derive cases from the intended contract (doc comment, diagnostic's purpose,
libfunc signature), not from the implementation you just wrote — tests derived
from the code inherit its bugs. Cover, in order of demonstrated payoff here:

1. **Every explicit error/rejection path.** The repo's #1 escaped-bug class:
   new features ship exhaustively tested happy paths and untested rejection
   paths (#10211 ICE on wrong generic-arg count, #9998 unterminated string,
   #10118 illegal dict type via `Default`). For each new diagnostic, add the
   golden case that triggers it; for each `Err`/`Option::None` branch, a case
   that reaches it.
2. **Boundaries**: empty, zero, one, max, max±1 (#10069 zero-variant enum,
   #10017 offset exactly at span end). Zero-variant/empty-collection inputs
   are cheap golden cases — add them when deriving, not after the bug. For
   per-element validations, one case where the offender is not the first
   element; for parser changes, one case where the input truncates
   mid-construct (the partial_trees suites exist for this).
3. **One representative per equivalence class** of the happy path. Add
   another case only when it can fail for a different reason.
4. **Feature interactions** with orthogonal machinery the change touches:
   coupons, gas-disabled mode, `#[cfg]`, snapshots, debug-info flags
   (#10081 coupon_call broke the profiler; #9957 a debug-info flag no test
   ever enabled). If your change adds a flag or mode, at least one test runs
   with it on.

## Exactness rules

- Assert outcomes, not internals: a test any behavior-preserving refactor
  breaks protects the implementation, not the behavior. House proof of a pure
  refactor is "regenerating all goldens yields zero diff" (#10120) — that only
  works if goldens assert behavior.
- Keep each golden case minimal and single-feature, so a future diff is
  attributable to one cause; name it for the behavior and condition
  (`wrong_generic_arg_count_through_alias`), never an issue number alone.
- Bug fix → ship the regression test that fails without the fix, at the home
  layer. 5 of 9 escaped defects sampled from history shipped fixes with no
  test, leaving the bug class unguarded; reviewers here ask "add an affected
  test if this does anything" — pre-empt them.
- Reuse the existing fixtures: `*DatabaseForTesting`, `setup_test_function`,
  shared e2e DBs. A helper may hide boilerplate, never the input and expected
  value that make this test this test.
- No unseeded randomness, wall-clock time, or test-order dependence; tests
  share salsa DBs, so never mutate shared state in a test body.
- Deletion is part of the change: when your change makes cases redundant
  (two cases that now fail together, a case subsumed by a new one), remove or
  merge them in the same PR.

## Property testing (not yet adopted)

No property-testing framework exists in this workspace, so do not write
property tests today, and do not add the dependency inside an unrelated PR —
adoption is a team decision. What you can do: when a change touches one of the
candidates below, suggest adoption _once_ (a review note or follow-up task,
not a nag on every PR). Recommended framework: **proptest** (composable
strategies suit compiler-shaped inputs better than quickcheck's one-strategy-
per-type). Evidence-backed candidates, each motivated by an escaped bug:

- Span/offset arithmetic round-trips — #10017 (inclusive-vs-exclusive end).
- Formatter idempotence: `format(format(x)) == format(x)`.
- Bounded-int/integer libfunc ranges over full domains — the #10069 class.
- Sibling-predicate agreement (e.g. `is_var_free` vs `is_fully_concrete` on
  inputs where they must coincide) — #10010, where a copy-pasted helper
  silently diverged.

Once adopted: use a property test where the contract is a relationship over
all inputs (round-trips, idempotence, two paths agreeing) — one property
replaces a family of example cases; keep a few named examples as
documentation; seed generators with domain-valid inputs; commit the failure
persistence file so found bugs stay found; then regenerate this skill so this
section becomes full guidance.

## References

- `references/review-checklist.md` — walk when reviewing a PR's tests.
- `references/rust.md` — Rust unit/golden mechanics: skeletons, commands, the
  bless-then-review workflow.
- `references/golden-framework.md` — the `cairo_lang_test_utils` file-based
  framework itself: `//! >` format, writing runners, CAIRO_FIX_TESTS.
- `references/cairo.md` — Cairo-language tests: corelib, cairo_level_tests,
  bug_samples, cairo-test runner flags.
- `references/benchmarks.md` — when a change warrants a benchmark and how the
  nightly perf gate works.

More agent context in starkware-libs/cairo

4 other files this repository gives its agents.