Manual CLI Testing & Tasting (/manual-cli-test / agent-cli-kick-tires)

Unit tests and mocked integration tests are necessary, but they are never sufficient for a CLI tool.

Unit tests only verify assertions the author remembered to write against synthetic fixtures. A CLI command can have 100% unit test pass rates while still emitting output that is contradictory on adjacent lines, bloated with bot noise, missing critical fields needed for the next step, or actively misleading when run against real repository and server state.

/manual-cli-test ("tasting the CLI" or "kicking the tires") is a disciplined workflow where you—the LLM—step out of the code-author mindset and into the consumer mindset. You build a standalone snapshot of the CLI, exercise it across a matrix of realistic and adversarial states in a safe sandbox, read every line of stdout and stderr critically, fix what feels wrong, and re-taste until the CLI is crisp, honest, and ergonomic.

Core Philosophy: Why You Must "Taste" the CLI

1. Mocks Hide Real-World Messiness: Live Gerrit CLs, Buganizer issues, LUCI CI runs, and Git worktrees contain bot spam, HTML entities, reverse chronological sorting, detached HEAD states, and half-finished rebases that hand-crafted unit test structs rarely model.
2. Consumer Goal Alignment: Every CLI command exists to support a downstream decision (by a human developer or an autonomous workflow like /crank). Tasting asks: "If I only had this stdout, could I take the exact right next step immediately without running three extra diagnostic commands?"
3. Curse of Knowledge: Because you just wrote the code, your brain automatically fills in missing context when reading the output. Deliberate tasting—especially via a naive "fresh-eyes" subagent—exposes where the output is ambiguous or self-contradictory.

Phase 1: Safe Execution Strategy (Decoupling Binary & Repo State)

The Branch-Dependent Binary Trap

When testing a CLI subcommand that lives on your current feature branch (such as a new ./gh or ./pw subcommand), do not test branch-switching, checkout, rebase, or --prepare flows by invoking the repo wrapper (./gh) inside your active development worktree:

1. The Binary Disappears: Switching branches or checking out an older CL in the active worktree replaces the source tree—causing ./gh to rebuild an older commit where your new subcommand or fix does not exist!
2. Worktree Contamination: Simulating dirty files, merge conflicts, or branch checkouts in your primary worktree risks clobbering uncommitted work or dirtying MODULE.bazel.lock.
3. Bazel Cache Churn: Repeatedly switching branches in your primary worktree invalidates incremental build state.

The Decoupled Binary + Disposable Worktree Sandbox Pattern

Always separate the binary under test from the Git worktree being manipulated:

1. Compile & Snapshot the Binary to /tmp: Build the target once in your primary feature worktree and copy the compiled executable to /tmp:

   bazelisk build --noshow_progress --noshow_loading_progress //pw_ghish:gh-ish
   cp -f bazel-bin/pw_ghish/gh-ish_/gh-ish /tmp/gh-under-test
   chmod +x /tmp/gh-under-test

   (Whenever you edit code during the fix loop, re-run this one-liner to refresh /tmp/gh-under-test.)
2. Use Secondary Worktrees for Read-Only Cross-Worktree Sensing: If git worktree list or ./gh wt list shows another clean worktree while your primary worktree has an active CL, you can test read-only commands there via a subshell:

   (cd /path/to/other/worktree && /tmp/gh-under-test <subcmd>)

3. Create a Disposable git worktree Sandbox in /tmp for Mutations: For any scenario that switches branches, creates commits, dirties files, triggers merge/rebase conflicts, or tests mutating flags (like --prepare or checkout), create a lightweight detached worktree in /tmp with an automatic EXIT cleanup trap:

   SANDBOX="/tmp/cli-taste-sandbox-$$"
   cleanup() {
     git worktree remove --force "$SANDBOX" 2>/dev/null || rm -rf "$SANDBOX"
     git worktree prune 2>/dev/null || true
     git branch -D test-cli-wip test-cli-conflict 2>/dev/null || true
   }
   trap cleanup EXIT
   git worktree add --detach "$SANDBOX" origin/main

4. Verify Your Sandbox Setup Actually Produced the Intended State:
   • Common Trap: When simulating a git rebase conflict by editing a file on origin/main~3 and rebasing onto origin/main, if that file was not actually modified between origin/main~3 and origin/main, Git will fast-forward/rebase cleanly without a conflict—and you will accidentally test a clean branch thinking you tested a conflict!
   • Fix: Dynamically pick a file that actually changed in the upstream range:

     (
       cd "$SANDBOX"
       git checkout -q -b test-cli-conflict origin/main~1
       FILE=$(git diff --name-only origin/main~1 origin/main | head -n 1)
       echo "# SIMULATED CONFLICT" > "$FILE"
       git commit -q -a -m "pw_foo: Simulated conflict on $FILE"
       git rebase origin/main || true  # Guaranteed UU conflict on $FILE
       /tmp/gh-under-test <subcmd>
       git rebase --abort || true
     )

5. Check Primary Worktree Hygiene Afterward: After running your suite, run git status -s in your primary worktree and revert any incidental lockfile touches (e.g. git checkout -- MODULE.bazel.lock).

Phase 2: Designing a Multi-Axis Scenario Matrix

Before running commands, design a concrete scenario list of 8–12 cases spanning four orthogonal axes:

Axis 1: Local Repository & Worktree State

• Active Open CL: Branch is 1 commit ahead of origin/main with an open Gerrit CL (status=NEW).
• Clean / Merged Idle State: Worktree is at origin/main (ahead=0) or on a branch whose CL was already MERGED or ABANDONED.
• Dirty Local WIP (ahead=0, dirty=true): Modified tracked files on a brand new branch with zero commits ahead of origin/main.
• Stale Branch (Clean vs. Conflicting): Branch is $N$ commits behind origin/main (test both a cleanly mergeable branch and one where in-memory git merge-tree --write-tree HEAD origin/main detects a conflict).
• Mid-Rebase Conflict (REBASE_IN_PROGRESS): Active git rebase paused on an unmerged (UU) file conflict.

Axis 2: Target & Input Variety

• Your Own Open CL vs. Teammate's Open CL: Discover real CL numbers first (/tmp/gh-under-test pr list --limit 10) so you test against real review threads, CI checks, and ownership metadata.
• Merged / Closed Targets: Point the CLI at an already-MERGED CL and an already-CLOSED Buganizer issue.
• Cross-Linked Targets: Point the CLI at an OPEN bug that already has an open Gerrit CL linked to it (should recommend adopting the CL, not creating a duplicate CL!).
• Alternate Input Formats: Shortlinks (pwrev/12345, b/12345), full URLs, or external artifact URLs (e.g. Google Chat thread URLs).
• Non-Existent (404) & Malformed Inputs: Test pwrev/999999999, b/999999999, and not-a-valid-target.

Axis 3: Modes, Flags & Environment Overrides

• --help Inspection: Always read <subcmd> --help first. Are all flags, defaults, and examples accurate?
• Output Modes: Default human/LLM compact output vs. --json vs. --status.
• Scope Flags & Env Vars: Test flags like --fleet / --all and environment overrides (e.g. GH_ISH_ONCALL_FILE=/tmp/mock-oncall.cfg).

Axis 4: Mutating Flags, Safety Gates & Idempotency

• Blocked Safety Gates: Run mutating flags (e.g. --prepare <target>) while on a dirty branch or active open CL and verify it refuses to clobber work with a clear reason.
• Clean Execution: Run the mutating flag in the clean /tmp sandbox and verify the mutation succeeds.
• Second Consecutive Run (Idempotency): Immediately run the exact same mutating command a second time when already on the target branch—and also test switching to a branch that already existed locally with commits.

Phase 3: The "Taste, Critique, Fix, Re-Taste" Loop

Run your scenario matrix in batches, read the literal output for each scenario, and evaluate it against the 6 Tasting Questions:

1. Internal Consistency: Do any two lines in the output contradict each other?
2. Goal Support: Does the output give the consumer (e.g. /crank or a human engineer) everything needed to execute the next step without extra CLI round-trips?
3. Actionability of Directives: Do the recommended Next: steps make sense for this exact state, or are they generic boilerplate that tells the user to do something invalid (like fixing a closed bug or checking out a 404 CL)?
4. Context Separation: Does local worktree state accidentally bleed into remote target inspection (or vice versa)?
5. Signal-to-Noise & Token Diet: Is the output polluted by bot boilerplate, HTML tags, redundant JSON arrays, or reverse-chronological comment threads?
6. Latency & Redundant RPCs: Did any scenario stall for seconds making unnecessary network calls (e.g. querying CI checks for an already-merged CL) or timing out on cold SSO calls?

Field Guide: 9 Classic Bugs Only "Tasting" Catches

These real bugs were all discovered in a single ./gh sense tasting session after 100% of unit tests were already passing:

Closing the Loop

After fixing all issues discovered during tasting:

1. Add unit tests covering every edge case you uncovered so they never regress.
2. Rebuild /tmp/<tool>-under-test.
3. Re-run the failed scenarios in the sandbox and read the new output side by side to confirm every fix works in live execution.

Phase 4: Strategic Use of Subagents for CLI Tasting

When invoke_subagent is available, always use TypeName: "self" and Workspace: "inherit" (never "branch" or "share", which create ad-hoc checkouts with cold Bazel caches). Subagents unlock three high-leverage CLI testing patterns:

Pattern A: The "Fresh-Eyes / Naive Consumer" Subagent (Recommended)

Because the author agent knows the implementation internals, it often overlooks confusing flag names, cryptic status badges, or missing next-step instructions. To get an unbiased evaluation, spawn a subagent that has not read the implementation source files:

• What to give the subagent:
  1. The path to the decoupled binary (/tmp/<tool>-under-test).
  2. A dedicated disposable sandbox directory (/tmp/cli-sandbox-fresh).
  3. The high-level user goal (e.g., "You are an agent running /crank. Run /tmp/gh-under-test sense --help to learn the tool, then run it across these 5 scenarios in /tmp/cli-sandbox-fresh. Do NOT read the Go source code. Report every place where the output is confusing, contradictory, noisy, or leaves you unsure what command to run next.").
• Why it works: The subagent experiences the CLI strictly through --help, stdout, stderr, and exit codes—just like a real downstream consumer.

Pattern B: Parallel Sandbox Scenario Runners

When testing a large scenario matrix or slow network/build flows, fan out 2–3 subagents in parallel (Workspace: "inherit"), giving each subagent its own isolated /tmp/cli-sandbox-<N> worktree and a slice of the scenario matrix (e.g., Subagent 1 tests local Git/rebase conflict states; Subagent 2 tests live Gerrit CL and Buganizer target states; Subagent 3 tests --prepare safety gates and idempotency).

Pattern C: Trajectory-Aware Retrospective Subagent

After an intensive manual CLI testing and debugging session in the parent conversation, you can delegate documentation, skill updates, or regression test authoring to a subagent while preserving the full history of what went wrong:

• Pass the parent conversation's transcript path (<appDataDir>/brain/<parent-conversation-id>/.system_generated/logs/transcript.jsonl) and step range in the subagent prompt.
• Instruct the subagent to read those transcript lines via view_file so it can see the exact raw CLI outputs, hypotheses, and fixes without cluttering the parent agent's context window.