Troubleshoot the ad harness

The harness drives the engine's app-server in its own home (~/.agent-daemon/engine). Most failures are one of four things: login, sandbox, hooks, or an engine error that only shows up in the engine's own log. Check them in that order. Don't guess from the agent's chat output.

If an error says "The engine reported a problem", open /warnings in the TUI or read the details: lines in CLI output for the engine's original text. ad doctor shows its tested version.

Version probes must capture child stderr explicitly: Node's execFileSync echoes it when stdio is omitted. Doctor retains probe stderr as labelled details, and live checks must preserve engine error codes/raw fields for the façade's mapping.

Each row below names its entry in docs/troubleshooting.md (#N) where the full text is.

When to use

Any harness command errors out, hangs, or finishes without doing the work ("my shell was denied", "couldn't run the tests").

Procedure

1. Login. ad auth status (what ad uses now, and whether each account works). Not logged in → ad auth login <provider> (openai, openrouter, chatgpt, …; ad auth login --help lists them). An account that "needs its key again" → the command it prints. The harness login is separate from your own install's login.
2. Engine + hooks. ad doctor. Look at the engine lines:
   • the engine version should equal the pin in runtime/package.json; if it doesn't, cd runtime && npm install.
   • Engine hooks: k/N trusted with k < N → the next ad run re-trusts them; if it stays low, the hook commands changed (re-run ad run once, then check again). The row names the hooks the engine skips until trusted. After an upgrade the hooks file is regenerated at the next start; hooks written for older ad keep working.
   • The hook rows (every hook runs the dispatcher, runtime/bin/hook.mjs <event>): Hook routes (a step run twice), Hook script (gone, or in a temporary copy), Rules file, Memory plugin hooks, the rules check's mode, and Hook log (the last lines of ~/.agent-daemon/hooks/hook.log: what a hook couldn't do, which no host shows).
3. Sandbox (Windows). ad sandbox status. Not ready → ad sandbox setup. Unattended runs (ad loop, workers, schedules) refuse to start without it.
4. Reproduce small, in a scratch dir and never the user's repo, so test prompts don't land in its memory:

   mkdir <scratch>/probe && cd <scratch>/probe && git init -q
   ad run "Run: node --version. Report the output." --json

5. Read the engine's log when the output doesn't explain it. It's logs_2.sqlite in the harness home, table logs (columns id, level, target, feedback_log_body). sqlite3 may not be installed; use Node:

   cd ~/.agent-daemon/engine
   node --experimental-sqlite --no-warnings -e '
   const { DatabaseSync } = require("node:sqlite");
   const db = new DatabaseSync("logs_2.sqlite", { readOnly: true });
   for (const r of db.prepare("select id, level, target, feedback_log_body b from logs where level in (?, ?) order by id desc limit 10").all("ERROR", "WARN"))
    console.log(r.id, r.level, r.target, "::", r.b.slice(r.b.lastIndexOf("}:") + 2, r.b.lastIndexOf("}:") + 500));'

   Never print auth.json or tokens.
6. Match it to a known cause (below), fix, and re-run step 4. For a run that used several accounts, ad sessions show --last says which target ran each turn and why a turn failed.

Known causes

Memory

Accounts, providers, limits and fallback

API keys live in ad's local gateway, never in the engine: provider errors are often the gateway's own words ("ad: …").

ad tui

The terminal UI runs the same engine in the same home, so steps 1–5 apply. These are its own symptoms; details in docs/troubleshooting.md and docs/tui.md.

Reproduce TUI problems with ad tui in a scratch folder. Never run the user's own install or use ~/.codex.

Workers, skills and moving ad

Examples

Example 1: "ad run did nothing"

The agent says its shell was denied, and every command asks to run outside the sandbox. Step 5 shows CreateProcessAsUserW failed on a pwsh.exe under WindowsApps. ad --version says older than 2.1.1 → upgrade (re-run the one-liner), then re-run the step 4 probe from the same PowerShell.

Example 2: "tests won't run inside the agent"

node --test → spawn EPERM while node --version works. That's the unelevated sandbox. Tell the agent to run the test file directly, or verify the tests yourself after the turn.

Anti-patterns

• Debugging in the user's repo. Probe prompts get captured as memory there.
• Editing ~/.codex, or running the probe with the user's own install. It's the user's own install; the harness never runs the engine there, and codexEnv() refuses it. Debug with the harness home or a temp engine home only.
• Turning the sandbox off (--sandbox danger-full-access) to make an error go away. Find the cause.
• Running a guarded command for the user. Account changes, ad fallback enable, ad review accepts and ad memory redact refuse from the agent's own shell on purpose; hand the user the command.
• Pasting log rows wholesale. They can be long; print the tail of the message, and never auth data.