testing

---
name: testing
description: "MUST READ before declaring any TouchDesigner build, fix or show done, and before any soak or performance test: the verification ladder (structure, errors, frame, motion, data, performance, soak, end to end), run_soak_test and the observer-effect rules that keep measuring from perturbing the show, the iterate-by-capture loop, and what done means."
---
<!-- Generated by Embody/Envoy - Do not remove this comment - sha:276fb21dda167ef8 -->

# Testing: a show is verified by running it, not by looking at it once

A network that renders one good frame has passed one test of eight. Shows fail at hour three, not minute one: a buffer that grows 2 MB a minute, a feedback loop that drifts, an `absTime` that loses precision, a cook cascade that only starts when the cue list reaches scene 4. Every stable installation you have seen was soaked before it was trusted. Testing is not a phase after building; it is how you know the build is real.

## The ladder

| Rung | Question | Evidence |
|---|---|---|
| 1 Structure | Does the network I meant exist? | `get_network_layout` (no overlaps, forward wires, docks hugged), `get_connections`, `read_tdxn` for the authored state |
| 2 Errors | Is anything red? | `get_op_errors` with `recurse=true`: cook errors, `kind: 'script'` tracebacks AND `shaderErrors`. Fix, re-run, clean |
| 3 Frame | Does it render what I intended? | `capture_top` on the output (a `Quality: FAIL` is black or flat and never passes); `capture_op` for every other family; `sample_grid` for numbers |
| 4 Motion | Does it animate, and settle? | Two captures seconds apart must differ; feedback and particles judged after settling; a loop wrap watched twice (`/visual-aesthetics`) |
| 5 Data | Are the values right, not just present? | `get_chop_data` (per-channel stats, `compare_to`), `get_dat_content(format='stats')`, `get_pop_data` metadata. Never a blind dump |
| 6 Performance | Does it hold frame rate with headroom? | `get_project_performance` before and after every heavy step, against the `performance.md` thresholds |
| 7 Soak | Does it STAY right for as long as the show runs? | `run_soak_test` for minutes to hours: PASS means flat memory, zero drops, fps above the floor |
| 8 End to end | Does the show sequence work, cue by cue, on the show machine? | Drive the real inputs and assert the outputs at each step; relay to the show machine with Convoy |

Rungs 1 to 3 after every build step. Rungs 4 to 6 before saying done. Rungs 7 and 8 before anyone says show-ready.

## Soak testing without slowing the system down

The observer effect is the trap: the tools that watch a frame can be what drops it. A `capture_top` downloads a texture and stalls the GPU for a frame; `numpyArray()`, `TOP.sample()`, `cook(force=True)`, `get_pop_data(samples>0)` and a `describe_op_type` sweep all cost frames; an open viewer adds cook demand; a Trail CHOP on the Perform CHOP is itself a cook. A soak that reads the system heavily measures its own footprint.

`run_soak_test` is built around that constraint:

- It samples Envoy's existing Perform CHOP on the frame hook: three channel reads per frame (microseconds), so dropped frames and the worst frame time are exact, not sampled; memory and op counts once per interval; nothing cooked, captured or created. Its cost sits below what you can measure.
- Every Envoy call that could disturb the run (a capture, a cook, a script, a save, a create, a DAT write) is logged as a perturbation, and samples within 2 s of one leave the `clean` statistics the verdict uses. The capture you took mid-soak shows up under `perturbations`, not as a show defect. Read-only counters (`get_project_performance`, `get_job_status`) are free.
- It runs as a job: `run_soak_test(duration_s=1800, label='act 2 loop')` returns a `job_id` at once; `get_job_status(job_id)` shows `progress` (elapsed, last sample, summary and verdict so far) while it runs and `result` when it finishes; `run_soak_test(stop=True)` ends it early. Save first if the run will be long; a job record survives a server restart, a soak does not survive TD quitting.

The verdict applies the performance rule's thresholds to the whole run rather than to one read. FAIL: frames dropped outside perturbation windows above 0.1% of frames (any drop is at least a WARN, because a show drops none); more than 5% of clean samples under 90% of the target fps; GPU headroom under 20%; CPU or GPU memory climbing (a slope over 1 MB per minute AND more than 10 MB total). `hotspots_start` and `hotspots_end` name the COMPs to compare when something grew.

Reading a soak:

- Memory that climbs and never falls is a leak: an unbounded Python list or `storage`, a Trail or Cache buffer with no window, a Feedback loop with decay at 1.0, a movie pre-read that keeps growing. Compare `hotspots_end` with `hotspots_start`, then `get_op_performance(include_children=True)` on the suspect.
- fps that sags slowly is accumulation (particle counts, instance counts, an expression cook cascade); fps that steps down at a moment is a state change (a cue, a movie switch). Match the time in `samples` to what the show was doing.
- Drops clustered at one time with no perturbation nearby are the show's own hitch: a file load, a network burst, a Python callback doing I/O on the main thread.
- A verdict on a 5-minute soak is a smoke test. A show that runs for hours is soaked for hours, at least once, on the machine that will run it.

Measuring inside TD without Envoy: a fresh Perform CHOP outputs only `fps` until you enable `msec`, `droppedframes`, `gpumemused` and `cpumemused` on it; a Trail CHOP after it shows history and adds a cook, so remove it from the show build. `project.realTime` off cooks every frame regardless of duration; that is the movie-export mode, never the show mode (`/movie-export`).

## Iterative design is testing at a shorter cadence

Build in passes and capture after each one (`/visual-aesthetics`): write the intent in three lines, get composition reading, then value, then colour, then motion, then finish. Judge the frame against the brief, change the smallest knob set, capture again. A pass that was not captured did not happen. The `_effects` footer on every write tool says when your last change broke a shader or dropped fps; read it before the next change, not after ten more.

For anything animated, prove it moves: two captures seconds apart with different content. A network can cook clean, error-free and frozen (`td-python.md`, Cook Model).

## End to end and show rehearsal

- Drive the show's own inputs: pulse the cue parameter, send the OSC message, start the timer, play the audio file (`execute_python` against the show's API when no operator does it). Assert the outputs at each step with `capture_top` and `get_chop_data`, not by reading the network.
- Test the transitions, not only the states: cue 3 to cue 4 while cue 3's feedback is still settling is where the black frame lives.
- Test on the show machine, at show resolution, with the show's other processes running. Pin the session to a Convoy node with `convoy_select_node` and the ordinary tools (`run_soak_test`, `capture_top`, `run_tests`, `save_project`) run there; a laptop soak says nothing about the media server.
- Test the restart: a show that survives one boot and one hour is not a show that survives a power cycle at 6 pm. Save, quit, reopen, and confirm `get_op_errors` clean and the first frame right (Embody restores externalized COMPs on open; TDXN reconstruction finishes around frame 60).
- Keep the harness out of the export: temporary frame drivers, probes and OP Viewer TOPs live outside the COMP or are deleted before `save_externalization` (`/specimen-authoring`).

## Unit and regression tests for the code under the network

Python that carries logic (extensions, callbacks, parsers) gets tests like any other code. Pure modules run under pytest on the project venv (`op.Embody.VenvPython`); TD-touching code is exercised with `execute_python` assertions or an in-TD runner and verified against live state (the op exists, the value changed), never against printed output alone. Embody's own suite is the model: thousands of methods, a save-gated destructive tier, and `run_tests(background=True)` plus `get_job_status` from any client.

## Done means

- `get_op_errors` clean on the COMP you built, recursively.
- The output captured and judged; for motion, two differing captures after settling.
- `get_project_performance` within the baseline: fps, frame time and dropped frames flat, GPU headroom above 20%.
- For a show: a soak of the real run length on the real machine with a PASS verdict, and the cue sequence driven end to end.
- The harness removed, the network saved, and `diff_tdxn` showing nothing unsaved.