consult-user-mcp / test-cases
CLAUDE.md39 starsChanged 4 months agoMITView on GitHub
JSON fixtures for every dialog type, plus two harnesses over them. Two consumers of the fixtures, which is the thing worth knowing: the harnesses and the Tray App's debug menu all read from cases/. The two harnesses answer different questions, and the second one is the one that fails a build: A state is a fixture plus how to drive it: which pane is open, what keys to type, how long to settle. That is why states.tsv has more rows…
In this repository13 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- container/CLAUDE.mdCLAUDE.md
- dialog-cli/CLAUDE.mdCLAUDE.md
- dialog-cli-windows/CLAUDE.mdCLAUDE.md
- docs/CLAUDE.mdCLAUDE.md
- macos-app/CLAUDE.mdCLAUDE.md
- mcp-server/CLAUDE.mdCLAUDE.md
- sketch-cli/CLAUDE.mdCLAUDE.md
- windows-app/CLAUDE.mdCLAUDE.md
- beadsSkill
- release-appSkill
- windows-buildSkill
The file
We split this CLAUDE.md into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 61
The debug menu loads from here: Never hardcode dialog JSON in macos-app/Sources/AppDelegate.swift.
≈ “Common” is an estimate: at least 1 in 5 of the 27,641 CLAUDE.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Test Cases
What this is
JSON fixtures for every dialog type, plus two harnesses over them. Two consumers of the fixtures, which is the thing worth knowing: the harnesses and the Tray App's debug menu all read from cases/.
The two harnesses answer different questions, and the second one is the one that fails a build:
test-runner.sh | layout-audit.sh | |
| Question | does it look right? | does it fit? |
| Method | spawns real dialogs, screencapture, OCR | renders off-screen, measures the view tree and the bitmap |
| Verdict | word presence, margin strips | overflow, overlap, text-fit, escapes |
| Takes the keyboard | yes — one dialog per case, on screen | no — windows are parked off-display |
| Exit code | capture failures only | fails on any layout violation |
Mental model
cases/<type>/<name>.json a raw dialog request, exactly as the CLI receives it
test-runner.sh iterates cases/, spawns each dialog, screenshots it
capture-dialog.swift the screenshot helper
verify-checklist.md what a human should look for per dialog type
screenshots/<timestamp>/ output (gitignored)
layout-audit.sh the asserting harness — render every state, then check
skin-states/states.tsv the state manifest: fixture + settle + pane + key script
skin-states/render/ off-screen renderer (main.swift) and the probe (probe.swift)
skin-states/check-layout.ts the rules; turns probe output into pass/fail
skin-states/waivers.json known-bad states, each with a reason and an issue id
skin-states/audit/<skin>/ output: <state>.png beside <state>.layout.json (gitignored)A state is a fixture plus how to drive it: which pane is open, what keys to
type, how long to settle. That is why states.tsv has more rows than cases/
has files — one fixture shot four ways is four states.
States with no pane and no key script are rendered in parallel; the rest run one at a time. Their settle delays were measured on an idle machine, and under load the probe catches a surface mid-transition — every state that went red the first time the audit ran wide was a pane or a key script, and every one of them was actually correct. Parallelism is not allowed to change the verdict, so the split is by whether a state has anything to settle, not by a tuning knob.
A fixture is not a wrapper format — it is the literal JSON argument passed to the [[Dialog CLI]]. That is why the same file works for both the runner and the debug menu.
Directory names are the test-case names, not CLI commands. test-runner.sh maps between them around line 88:
| Directory | CLI command |
confirm/ | confirm |
choose/ | choose |
text-input/ | textInput |
questions/ | questions |
notify/ | notify |
tweak/ | tweak |
sketch/ | (Sketch CLI) |
Important invariants
- The debug menu loads from here. Never hardcode dialog JSON in
macos-app/Sources/AppDelegate.swift. A fixture added here is the way to get a dialog into the debug menu. - A new dialog type needs four things: a
cases/<type>/directory with at least one fixture, an entry in thetest-runner.shcommand mapping, amakeWindowcase inskin-states/render/main.swift, and a debug-menu item. Miss the mapping and the runner skips the whole type without reporting anything; miss themakeWindowcase and the audit reportsunknown kindand moves on. - A fixture is not a state. Adding JSON to
cases/gets it into the debug menu and the screenshot runner. It does not get it into the layout audit — that needs a row inskin-states/states.tsv. A fixture with no row is never measured. - Fixtures cover shapes, not just happy paths. The value of
long-body,unbreakable-tokens,forty-options,wide-glyphs,unbalancedis that they break layout. Keep adding the ugly ones — window sizing is the most fragile thing in the dialog CLI. - The skin is pinned, not inherited. Both harnesses default to
caret, the New Interface. Before that was true, every visual run silently exercised only the classic chassis, and the skin people had actually switched on was the one nothing tested. - Some fixtures have companion setup files (a
.cssfor tweak cases) that the runner copies into place. Keep them beside the JSON.
What the audit can and cannot see
Worth knowing before trusting a green run, and before writing a rule:
fittingSizecan never report overflow on its own.createAutoSizedWindowsizes the window fromfittingSize, so the two agree by construction. Overflow only means something once the window is clamped at the height cap. The rule is kept because that clamped case is real, not because it fires often.- SwiftUI
Textis invisible to the view walk. It is drawn into the hosting view's layer, not into a childNSView. Only theNSViewRepresentablewidgets —CaretTarget,CaretNoteEditor, theFocusable*family — can be measured. A clipped SwiftUI label is caught by its consequences (an overlap, a box that stopped fitting), not directly. - Overlap between interactive views is the highest-signal rule. It is what catches a note pane opening across the footer: nothing is clipped, the geometry stays self-consistent, and two things a person can click simply land on the same pixels.
- The pixel probe derives the background from the image, so it works on any skin and theme without being told the palette. It cannot see a collision — pixels do not record what drew them.
- The rules are calibrated for caret and only caret. Telling a row scrolled out of a list from a control the layout pushed outside the window has no general answer. Caret lays an oversized list out at full height and hangs the scroll view outside the surface, which is what the probe keys on; classic keeps it inside and clips at the layer with a translucent footer over the top. Run the audit against classic or bracket and every long-list state fails while looking perfectly correct in the screenshot — 14 and 8 respectively. That is why
SKINSdefaults to caret. Tracked in cum-6ql. - There is no
edge-inkrule, though the probe still measures it. It produced zero true positives across two skins and a wall of false ones. A rule nobody can trust is worse than no rule.
Common change patterns
Reproducing a layout bug → add a fixture that triggers it and a states.tsv row, then run the audit and confirm it fails before fixing. visual-bugs/ collects known-bad renders.
Adding a fixture → drop the JSON in the right cases/ directory for the debug menu and the screenshot runner; add a states.tsv row to get it measured.
A bug you are not fixing today → add it to waivers.json with the rules it trips, a reason, and the issue id. A waiver that stops matching fails the run, so the fix gets noticed rather than quietly widening the waiver list.
Verification
Run these in the container. bun run test:container — or a subset with
SUITES="unit layout" bun run test:container. See ../container/README.md.
bun run test:container # everything, in the VM — the verdict
bun run test:layout # assert: caret, every state
SKINS="caret classic" bun run test:layout # more than one — expect noise, see below
bun run test:visual # capture + OCR ⚠ takes the screen
bun run test:keyboard # keyboard contract ⚠ takes the keyboardtest:layout is the exception that is safe on the host: it renders off-screen,
so nothing appears and nothing takes focus. The other two spawn real dialogs.
On a machine somebody is using they seize the screen for minutes, and a stray
focus change silently corrupts the run — the visual suite spent months
photographing one stuck window because of exactly that.
From here: ./layout-audit.sh [name-filter], ../container/launch.sh run.
The audit prints a FAIL/WAIVED/STALE line per state and exits non-zero.
The screenshot runner captures, OCRs each shot against the words that should be
on it, and now fails when it cannot read them — a shot of the wrong window is a
red run rather than a green one. Read verify-checklist.md against its output.
A pass is not a clean surface. The audit waives every state with a known
open bug and prints the list each run. bun run test:layout currently comes
back green with a dozen states waived across six open issues. Read that list
before treating a pass as a release signal.
Related context
verify-checklist.md— per-dialog-type review checklist../dialog-cli/CLAUDE.md— what consumes these fixtures../macos-app/CLAUDE.md— the debug menu.claude/rules/dialog-parity.md
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

