agentleFS
Omni-CraftOmni-Craft/OmniCraft/.claude/skills/cli-setup-verify/SKILL.md

cli-setup-verify

Skill1 starsChanged 4 months agoApache-2.0View on GitHub

Verify the OmniCraft CLI's setup/onboarding flow, terminal UI/UX, and critical user journeys in a completely isolated, reproducible loop. Drives the real `omnicraft` binary through a PTY (pexpect) inside a throwaway OMNICRAFT_CONFIG_HOME / OMNICRAFT_DATA_DIR sandbox that never touches the user's real ~/.omnicraft, captures ANSI-stripped frames for UX inspection, and proves a change is verifiable via a before→fix→after baseline diff. Load when developing or reviewing a CLI setup/onboarding/REPL/picker change (omnicraft/cli.py, omnicraft/onboarding/*, omnicraft/repl/*, scripts/install_oss.sh), reproducing a cold-start/first-run UX bug, or confirming a fix actually lands. Several agents can run it concurrently on separate worktrees.

  • Pipes a download into a shell
  • Reads credentials
  • Deletes or force-pushes

1 other project uses a version of this, 1 of them edited.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

6 rules. About 0 are common, 1 is distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Every run fingerprints the real ~/.omnicraft before and after (stat-only, no content reads): the top-level config files and the set of …

    line 50
  2. Distinctive:

    If a finding has no machine check yet, add one (see "Adding a scenario") so the fix becomes provable instead of asserted.

    line 133
  3. Very specific:

    Run a slice with the project's gated runner, e.g.

    line 159
  4. Very specific:

    Always drive the CLI through the driver (which redirects HOME + the config/data knobs), never a bare omnicraft setup — that would write to …

    line 179
  5. Very specific:

    If you can't reach the surface under test (no harness, no credential, headless limit), the scenario must report skipped — do not claim a …

    line 187
  6. Very specific:

    Write scenario_<name>(args, sandbox, result) in verify_cli.py: drive the CLI (reuse

    line 194

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Verifying the OmniCraft CLI setup & UX in a closed loop

The OmniCraft CLI's first impression is: curl | sh → run omnicraft → pick a model credential → start a session. This skill lets an agent enter that flow, examine the UI/UX, and prove whether a change is verifiable — without a browser, without real credentials, and without ever touching the developer's real ~/.omnicraft.

The engine is verify_cli.py (next to this file). It drives the real omnicraft binary through a pseudo-terminal (pexpect) inside a throwaway sandbox, captures what renders, runs assertions, and prints one machine-readable SUMMARY {json} line.

The whole point is a verifiable loop, not a one-shot check:

  1. Run a scenario on the unfixed code → baseline (--label before).
  2. Make the change.
  3. Run the same scenario → --label after.
  4. Diff the two SUMMARY lines. A fix is "verifiable" only if a concrete check or note flips between the two runs. If it doesn't flip, you can't prove the fix did anything — go back to step 2.

Why this is safe (read first)

The real ~/.omnicraft here can be many GB (chat DB, runner logs, native harness state). The sandbox isolates every write three ways:

  • HOME is redirected into the sandbox by default. This is the load-bearing one. OMNICRAFT_CONFIG_HOME / OMNICRAFT_DATA_DIR (omnicraft/cli.py _CONFIG_HOME_ENV_VAR / _DATA_DIR_ENV_VAR) redirect config + data — but the CLI's diagnostics logger ignores them: it writes a per-invocation cli-*.log under state_dir(), hardcoded to Path.home()/.omnicraft/logs (omnicraft_ui_sdk/terminal/_config.py). So only redirecting HOME keeps a non-help command (config list, the setup PTY spawns, server stop) from writing into the real home. The driver does this for you.
  • --strip-path reduces PATH so node/tmux/claude/codex read as "not installed" → the true fresh-machine cold start.
  • Ambient model keys (ANTHROPIC_API_KEY, …) are stripped from the child env unless you pass --keep-env-creds.

--inherit-home opts out of HOME isolation — use it only to reach a real credentialed REPL via ambient ~/.claude / ~/.databrickscfg auth. It is less safe: a non-help command then writes cli-*.log into the real ~/.omnicraft/logs.

Every run fingerprints the real ~/.omnicraft before and after (stat-only, no content reads): the top-level config files and the set of logs/cli-*.log diagnostic files. A new config file/mtime or a new cli-*.log basename trips real_config_untouched: false. With the default isolation that never happens; under --inherit-home it correctly does — which is exactly the violation the guard is meant to catch. If that check is ever false, stop and investigate. Run check-isolation first to confirm the loop is safe on your machine.

Prerequisites

  • You're in the worktree whose code you want to test (each parallel agent on its own worktree). The driver runs omnicraft from --repo's checkout.
  • A Python with pexpect — the project's .venv/bin/python bundles it (pexpect>=4.9 in pyproject.toml). Run the driver with that interpreter.
  • An omnicraft binary: the driver auto-finds <repo>/.venv/bin/omnicraft, or pass --omnicraft <path>.
  • The setup / picker / help / cold-start scenarios need no credentials and no harness. Only repl-commands needs a working harness + credential: pass --inherit-home (ambient ~/.claude auth) and/or --keep-env-creds (env API key) with --agent. It reports skipped, never a false pass, when the prompt isn't reachable.

Quick start

REPO=/path/to/your/worktree
PY=$REPO/.venv/bin/python
DRV=$REPO/.claude/skills/cli-setup-verify/verify_cli.py

# 0. Prove the sandbox is safe on this machine (do this once).
#    HOME is isolated by default — no flag needed.
$PY $DRV --scenario check-isolation --repo "$REPO"

# 1. See exactly what a brand-new user sees on a fresh machine.
$PY $DRV --scenario cold-start --strip-path --keep-sandbox --repo "$REPO"
#    → reads the printed `artifacts` path, then `cat <that>/cold_start.txt`

# 2. Lint the top-level help (and any subcommand's).
$PY $DRV --scenario help-snapshot --repo "$REPO"
$PY $DRV --scenario help-snapshot --subcommand server --repo "$REPO"

Each run prints SUMMARY {…} and exits non-zero if any check failed (a skipped scenario exits 0). Pipe to … | grep '^SUMMARY' | python -m json.tool to read it.

Scenario catalog

ScenarioWhat it drivesKey checks / notesMaps to findings
check-isolationomnicraft config list in the sandbox (no PTY)config_list_ran, sandbox_config_home_used, real_config_untouchedsafety gate for everything
cold-startomnicraft setup via PTY on a simulated fresh machineonboarding_rendered, harness_menu_present; note guided_default_affordancecold-start dead-end; missing "recommended start here"
setup-snapshotomnicraft setup, optional --nav-down N arrow stepsmenu_rendered; saves a frame per steppicker markers/footer/alignment; narrow-terminal at 80×24
help-snapshotomnicraft [--subcommand] --help (no PTY)help_rendered, no_param_leak, no_update_dup; note top_level_command_count:param leak, duplicate update/upgrade, command sprawl
repl-commandsomnicraft run <agent> REPL, sends /help + /quithelp_lists_commands; note quit_advertisedREPL discoverability (/help, /quit)

--list-scenarios prints them too. Captured frames land in the printed artifacts dir as both <name>.txt (ANSI-stripped, for reading/asserting) and <name>.ansi.txt (raw, to see real colors with less -R).

The verifiable loop — a worked example

Finding: "server --help leaks Sphinx :param/:returns into user help."

# BEFORE the fix (on the unfixed code):
$PY $DRV --scenario help-snapshot --subcommand server --label before --repo "$REPO"
#   → "no_param_leak": {"ok": false, ...}      ← bug reproduced (the baseline)

# ... make the change (move :param docs into # comments) ...

# AFTER the fix:
$PY $DRV --scenario help-snapshot --subcommand server --label after --repo "$REPO"
#   → "no_param_leak": {"ok": true, "detail": "clean"}   ← flipped → fix is verifiable

The same shape proves the update/upgrade duplicate (no_update_dup), the cold-start dead-end (guided_default_affordance note flips absent→present), or REPL /quit discoverability (quit_advertised note flips no→yes). If the check/note doesn't flip, the fix isn't proven — that is the signal to keep working, and it's exactly the judgment the loop exists to force.

If a finding has no machine check yet, add one (see "Adding a scenario") so the fix becomes provable instead of asserted.

Examining UI/UX deliberately

  • Narrow terminal is the default. The driver uses 80×24 — the size a new user's window actually is, and where banner overflow and picker redraw-past-the-bottom bugs appear. Re-run with --cols 120 --rows 40 to compare the roomy layout; diff the two frames.
  • Read the frame, don't just trust the check. cat <artifacts>/cold_start.txt shows the literal screen — the all-✗ menu, the footer hint (Esc back at the root), the marker (❯), alignment of the status gutter. The frame is the UX evidence.
  • Compare pickers for consistency. setup-snapshot --nav-down 3 captures the harness menu as you move; eyeball marker/footer/highlight drift against the theme and resume pickers (different engines render differently).

Covering all critical user journeys

This skill owns the setup / onboarding / first-run / TUI journeys. The repo already has complementary CUJ coverage — use both:

  • Live setup/UX journeys → this skill's scenarios (cold-start, setup, pickers, help, REPL discoverability).
  • Deeper end-to-end journeys → tests/e2e/test_journey_*.py (first session to code, resume/disconnect, fork/explore, file upload, collaboration, …). Run a slice with the project's gated runner, e.g. uv run --frozen --extra dev python -m pytest tests/e2e/test_journey_first_session_to_code.py -q.
  • Reusable PTY helpers live in tests/e2e/omnicraft/_pexpect_harness.py (spawn_omnicraft_run, wait_for_ready, submit_prompt, await_turn_complete, clean_exit) and the snapshot comparator in tests/e2e/omnicraft/_snapshot.py — prefer extending those over re-inventing.

To drive a surface this skill doesn't script yet, spawn it by hand with the sandbox env and the keys the driver exports (KEY_UP/KEY_DOWN/KEY_ENTER/ KEY_ESC), then drain() and save_frame() the result.

Teardown — non-negotiable

  • The driver force-kills the PTY child and its descendants, and runs omnicraft server stop against the sandbox to reap any spawned background server. After a run, confirm nothing leaked: pgrep -af "omnicraft.*(server|runner|host._daemon)" — anything bound to your sandbox's data dir is yours to kill.
  • The sandbox temp dir is deleted unless --keep-sandbox. If you keep one for inspection, rm -rf it when done.
  • Always drive the CLI through the driver (which redirects HOME + the config/data knobs), never a bare omnicraft setup — that would write to the real ~/.omnicraft. If you pass --inherit-home, expect cli-*.log writes to the real ~/.omnicraft/logs and a real_config_untouched: false — that's the guard working, not a bug.

Honesty

If you can't reach the surface under test (no harness, no credential, headless limit), the scenario must report skipped — do not claim a CUJ passed. The strongest evidence for a fix is a reproduced baseline (before) plus the flipped after; report both SUMMARY lines, not a summary of a summary.

Adding a scenario

Write scenario_<name>(args, sandbox, result) in verify_cli.py: drive the CLI (reuse pexpect.spawn(... env=sandbox.env, dimensions=(args.rows, args.cols)), drain(), save_frame(), the KEY_* constants), record findings with result.add(name, ok, detail) (fails the run) or result.notes.append(...) (informational, for before/after flips), register it in SCENARIOS, and add a row to the catalog above. Keep one assertion per real, observable behavior so a fix is provable as a single check flip.

Code under test

  • First-run dispatch / no-arg routing: omnicraft/cli.py (run, the first-run plan, _run_configure_harnesses_interactive).
  • Onboarding: omnicraft/onboarding/* (setup.py, interactive.py, configure_models.py, provider_selection.py, detected.py).
  • TUI / REPL & pickers: omnicraft/repl/* (_repl.py, _theme_picker.py, _resume_picker.py), omnicraft/_terminal_picker_theme.py.
  • Installer: scripts/install_oss.sh.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.