Integration-test a cockpit runtime path against the mock CLI

Unit tests prove a helper returns the right value. Static review proves the diff looks correct. Neither proves the feature actually works end to end. For anything whose value is runtime behaviour — the cockpit agent calling a tool, a session spawning with the right flags, a job posting to the inbox, the permission gate allowing/denying — the only honest evidence is driving the real Claude CLI and watching it happen. This harness does that with a scripted mock Anthropic API so it is deterministic and offline.

When to use

• A behavioural acceptance criterion: "the assistant calls a tool and it succeeds", "asking X returns Y", "a cockpit-agent tool is not denied".
• A spawn-time contract: the CLI receives --system, --mcp-config, a model, a permission mode.
• A regression you can only see at the wire level (what the CLI sent to the API, what came back, what rendered).

If you only need to assert a pure function's output, write a normal tests/*.test.ts vitest unit test instead. Use this harness when the proof requires a running CLI.

The harness (tests/integration/)

startHarness() (in tests/integration/harness.ts) boots three things in isolated tmpdirs: a mock Anthropic API on a random port that replays scripted SSE, a seeded COCKPIT_CONFIG_DIR + CLAUDE_CONFIG_DIR (password, a mock provider pointing the CLI at the mock, default model), and a cockpit server (node dist/server.js) on a random port. The Playwright fixture in tests/integration/fixtures.ts exposes a per-test harness and an already-authenticated page (it injects the cockpit_session cookie), so you navigate straight to authenticated routes.

Write a test

Model on tests/integration/hello.spec.ts. Import the fixture, skip when no CLI is present, script the mock, create a session, drive the UI, assert both the rendered output and what the CLI sent.

import { execSync } from "node:child_process";
import { textResponse, toolUseResponse } from "../mock-api/builder";
import { expect, test } from "./fixtures";

const CLAUDE_BIN = process.env.CLAUDE_BIN ?? "claude";
const CLAUDE_AVAILABLE = (() => {
  try { execSync(`${CLAUDE_BIN} --version`, { stdio: "ignore" }); return true; } catch { return false; }
})();
test.skip(!CLAUDE_AVAILABLE, `claude binary not found at ${CLAUDE_BIN} (set CLAUDE_BIN env)`);

test("the cockpit agent calls a config tool and it is not denied", async ({ page, harness }) => {
  // Script the mock: turn 1 emits a tool_use, turn 2 a final text answer.
  harness.mock.setScript([
    { events: toolUseResponse("list_jobs", {}) },
    { events: textResponse("You have 2 scheduled jobs.") },
  ]);

  const res = await page.request.post(`${harness.cockpitUrl}/api/sessions`, {
    data: { cwd: harness.configDir, cockpitAgent: true, runtime: "pty" },
  });
  const { sessionId } = await res.json();

  await page.goto(`${harness.cockpitUrl}/sessions/${sessionId}?cwd=${encodeURIComponent(harness.configDir)}`);
  const input = page.getByTestId("message-input");
  await expect(input).toBeVisible();
  await page.waitForTimeout(5000); // let the eager PTY spawn settle before sending
  await input.fill("list my jobs");
  await page.getByTestId("btn-send").click();

  await expect(page.getByText("You have 2 scheduled jobs.")).toBeVisible({ timeout: 30_000 });

  // Assert what reached the API: the system prompt and the tool result round-tripped,
  // so the tool was NOT denied. getRequests() returns { url, body } for each call.
  const calls = harness.mock.getRequests().filter((r) => r.url.split("?")[0] === "/v1/messages");
  expect(calls.length).toBeGreaterThanOrEqual(2); // tool turn + follow-up turn
  expect(calls[0].body).toContain("Cockpit Assistant"); // --system prompt arrived
});

Scripting the mock (tests/mock-api/builder.ts)

• textResponse(text, stopReason?) — a plain assistant text turn.
• toolUseResponse(name, input, opts?) — an assistant turn that calls a tool (drives the permission/tool path).
• askUserQuestionResponse(...), errorResponse(status, message), messageStop(usage) for the other cases.
• harness.mock.setScript([{ events: <builder> }, ...]) sets one entry per conversation turn; the last entry repeats if the CLI makes more calls.
• harness.mock.getRequests() returns every request the CLI made, with url and body. This is your wire-level assertion surface: the --system prompt, the user message, and tool results all show up in /v1/messages bodies. CLI calls carry a ?beta=true query, so match on r.url.split("?")[0].

Asserting spawn args without a browser

If you only need to prove the CLI was spawned with a flag (e.g. --system or --mcp-config is present for cockpitAgent sessions), a fast vitest unit test that spies on the spawn args is cheaper than a full browser run. Model on tests/session-manager-mcp-lifecycle.test.ts. Use the integration harness when the proof requires the CLI to actually run and respond.

Run it

• npm run test:integration (the pretest:integration script runs npm run build first; the harness runs node dist/server.js in production mode, so rebuild after any source edit).
• Needs the claude binary on PATH, or CLAUDE_BIN=/path/to/claude. Tests test.skip when it is absent — a skipped integration test is not a passing one. Do not treat a green run with everything skipped as verification; confirm the CLI was actually present.
• COCKPIT_IT_DEBUG=1 surfaces cockpit's logs to stdout when a test hangs.
• runtime defaults to pty; pass { runtime: "stream" } to startHarness (or session create) to exercise the stream path.

Gotchas

• 60s per-test timeout. The PTY needs a settle window after session create before the first message:send, or the send races the spawn — the waitForTimeout(5000) in the example is load-bearing.
• The mock replays a fixed script; it is not a real model. Script exactly the turns your path needs.
• Production-mode server means stale dist/ serves old code. If your change does not show up, you forgot to rebuild.

Reference

tests/integration/harness.ts, tests/integration/fixtures.ts, tests/integration/hello.spec.ts, tests/mock-api/builder.ts, tests/mock-api/server.ts.