writing-slicc-tests
Skill37 starsChanged 20 days agoApache-2.0View on GitHub
Use when writing or updating Vitest tests, setting up VirtualFS/fake-indexeddb tests, or when test:coverage fails a floor. Includes the ratchet rule (never hand-lower coverage-thresholds.json), VirtualFS dbCounter isolation pattern, RestrictedFS security test patterns, tool testing patterns, and the fake-LLM e2e framework. Also triggered by error strings like 'coverage below threshold', 'ENOENT', or 'fake-indexeddb'.
In this repository9 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- docs/CLAUDE.mdCLAUDE.md
- .github/copilot-instructions.mdCopilot instructions
- adding-slicc-featuresSkill
- cdp-smoke-testSkill
- demo-recordingSkill
- deploying-tray-workerSkill
- slicc-handoffSkill
- verifying-before-pushSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
28 rules. About 0 are common, 13 are distinctive, and 15 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈13
Distinctive · estimate
This author's own choices. Worth a look.
15
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 9
Use these patterns, conventions, and procedures when writing tests for SLICC.
- Very specific: line 30
Import fake-indexeddb: when VirtualFS is used: import 'fake-indexeddb/auto'
- Distinctive: line 39
When changing web components or DOM rendering, run the browser-mode suites — they require automated tests, not just manual verification.
- Distinctive: line 44
When testing filesystem code, import fake-indexeddb and create a VirtualFS with a unique dbName:
- Distinctive: line 84
Test path access control and ACL boundaries:
- Very specific: line 145
Path traversal: Test /../.. escapes → should throw ENOENT.
- Distinctive: line 151
Test tool execution with filesystem integration:
- Very specific: line 197
Test command execution: call tool.execute() with args
- Very specific: line 198
Check isError flag for error conditions
- Distinctive: line 199
Use file operations within tool tests (pipes, redirects)
- Distinctive: line 200
Test compound operations (e.g., zip → unzip)
- Distinctive: line 204
Test supplemental commands with mocked context:
- Distinctive: line 267
Mock context: Create minimal mock with only needed properties
- Distinctive: line 269
Test arg parsing separately: Test command-line parsing logic without booting the just-bash runtime
- Distinctive: line 270
Check exit codes and output: Verify both success and error paths
- Distinctive: line 304
Create helpers for constructing typed test messages:
- Very specific: line 382
If one suite verifies both extension and non-extension paths, call vi.resetModules() between cases so each import sees the intended runtime …
- Very specific: line 402
Run npm run test:coverage:<package> — this invokes vitest --coverage (v8 provider) then coverage-gate.mjs, which reads the package's floors …
- Very specific: line 487
Keep the count at 1 unless the flake is provably external (a real browser, a real port).
- Very specific: line 490
Never enable retries locally.
- Very specific: line 520
Keep the dynamic form only where the test needs a genuinely fresh module instance, i.e.
- Very specific: line 544
Never answer either pattern by raising testTimeout: that makes the failure rarer and harder to read without removing the wasted seconds.
- Very specific: line 550
When CI is set, the root test.reporters adds vitest's json reporter and writes per-test durations to test-timing/vitest.json (gitignored).
- Distinctive: line 643
Avoid hardcoding test data.
- Distinctive: line 666
Run a single test with verbose output:
- Very specific: line 672
Add console.log() in test code — output appears in terminal:
- Very specific: line 852
If a scenario needs a /preview/* page rendered for assertion, seedVFS continues to work for content the _test page_ itself loads — it just …
- Very specific: line 922
A failed turn never renders the reply you are waiting for: When a turn dies — provider error, fixture overflow, retries exhausted — the …
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
writing-slicc-tests
Use these patterns, conventions, and procedures when writing tests for SLICC.
Quick Reference
| Command | Purpose |
npm run test | Run Node-mode Vitest projects |
npm test -w @slicc/webcomponents | Run Chromium browser-mode component tests |
npm test -w @ai-ecoverse/spoon | Run Chromium launcher tests |
npx vitest run packages/webapp/tests/fs/virtual-fs.test.ts | Run a single test file |
npx vitest run packages/webapp/tests/fs/ | Run all tests in a directory |
npx vitest run --reporter=verbose | Show full stack traces |
npm run test:coverage:<package> | Check a TypeScript package's coverage floor |
./packages/dev-tools/tools/swift-coverage-check.sh <pkg-dir> <bundle> | Check Swift package coverage floors |
npm run test:server-integration | Node-server integration tests |
npm run test:e2e | Run Playwright and fake-LLM E2E tests |
Set Up the Framework
- Framework: Vitest with
globals: true,environment: node - Convention:
foo.test.tsinpackages/*/tests/mirroring thesrc/structure - Import fake-indexeddb when VirtualFS is used:
import 'fake-indexeddb/auto'
Note: npm run test runs the default Node-mode Vitest projects only. It does
not include:
- Browser-mode suites:
npm test -w @slicc/webcomponentsandnpm test -w @ai-ecoverse/spoon(real Chromium via@vitest/browser) - Server integration:
npm run test:server-integration - Playwright E2E:
npm run test:e2e
When changing web components or DOM rendering, run the browser-mode suites — they require automated tests, not just manual verification.
Set Up VirtualFS Tests
When testing filesystem code, import fake-indexeddb and create a VirtualFS with a unique dbName:
import 'fake-indexeddb/auto';
import { describe, it, expect, beforeEach } from 'vitest';
import { VirtualFS } from './virtual-fs.js';
describe('VirtualFS', () => {
let vfs: VirtualFS;
let dbCounter = 0;
beforeEach(async () => {
// Create fresh VirtualFS with unique DB name for test isolation
vfs = await VirtualFS.create({
dbName: `test-vfs-${dbCounter++}`,
wipe: true,
});
});
it('writes and reads text files', async () => {
await vfs.writeFile('/test.txt', 'Hello VirtualFS!');
const content = await vfs.readFile('/test.txt');
expect(content).toBe('Hello VirtualFS!');
});
it('writes and reads binary files', async () => {
const data = new Uint8Array([10, 20, 30]);
await vfs.writeFile('/binary.dat', data);
const result = (await vfs.readFile('/binary.dat', { encoding: 'binary' })) as Uint8Array;
// LightningFS may return a view into a larger buffer, compare actual bytes
expect(result.length).toBe(data.length);
expect(Array.from(result)).toEqual(Array.from(data));
});
});Key pattern: increment dbCounter in beforeEach to ensure each test gets an isolated IndexedDB instance.
Test RestrictedFS Security
Test path access control and ACL boundaries:
import 'fake-indexeddb/auto';
import { describe, it, expect, beforeAll } from 'vitest';
import { VirtualFS } from './virtual-fs.js';
import { RestrictedFS } from './restricted-fs.js';
describe('RestrictedFS', () => {
let vfs: VirtualFS;
let restricted: RestrictedFS;
beforeAll(async () => {
vfs = await VirtualFS.create({ dbName: 'test-restricted-fs', wipe: true });
// Set up scoop directories
await vfs.mkdir('/scoops/andy-scoop', { recursive: true });
await vfs.mkdir('/shared', { recursive: true });
await vfs.writeFile('/scoops/andy-scoop/file.txt', 'hello');
await vfs.writeFile('/shared/data.txt', 'shared data');
await vfs.writeFile('/root-file.txt', 'root');
// Restrict to scoop + shared paths
restricted = new RestrictedFS(vfs, ['/scoops/andy-scoop/', '/shared/']);
});
it('reads files within allowed dirs', async () => {
const content = await restricted.readFile('/scoops/andy-scoop/file.txt', { encoding: 'utf-8' });
expect(content).toBe('hello');
});
it('throws ENOENT for reads outside allowed dirs (not EACCES)', async () => {
await expect(restricted.readFile('/root-file.txt')).rejects.toThrow('ENOENT');
});
it('prevents path traversal (returns ENOENT)', async () => {
await expect(restricted.readFile('/scoops/andy-scoop/../../root-file.txt')).rejects.toThrow(
'ENOENT'
);
});
it('prevents writing outside allowed dirs', async () => {
await expect(restricted.writeFile('/root-file.txt', 'hacked')).rejects.toThrow('EACCES');
});
// Parent directory traversal needed for 'cd'
it('stat on parent dir of allowed path works (cd needs this)', async () => {
const stat = await restricted.stat('/scoops');
expect(stat.type).toBe('directory');
});
it('readDir on parent dir filters to only allowed children', async () => {
const entries = await restricted.readDir('/scoops');
const names = entries.map((e) => e.name);
expect(names).toContain('andy-scoop');
});
});Key patterns:
- ENOENT vs EACCES: Outside reads → ENOENT. Outside writes → EACCES.
- Path traversal: Test
/../..escapes → should throw ENOENT. - Parent traversal: Reading parent dirs is allowed (needed for
cd). Writing parent dirs is blocked. - readDir filtering: Parent directories show only children leading toward allowed paths.
Test Tools
Test tool execution with filesystem integration:
import 'fake-indexeddb/auto';
import { describe, it, expect, beforeEach } from 'vitest';
import { VirtualFS } from '../fs/index.js';
import { AlmostBashShellHeadless } from '../shell/almost-bash-shell-headless.js';
import { createBashTool } from './bash-tool.js';
import type { ToolDefinition } from '../core/types.js';
describe('Bash Tool', () => {
let fs: VirtualFS;
let shell: AlmostBashShellHeadless;
let bash: ToolDefinition;
let dbCounter = 0;
beforeEach(async () => {
fs = await VirtualFS.create({
dbName: `test-bash-tool-${dbCounter++}`,
wipe: true,
});
shell = new AlmostBashShellHeadless({ fs });
bash = createBashTool(shell);
});
it('executes echo', async () => {
const result = await bash.execute({ command: 'echo hello world' });
expect(result.isError).toBeFalsy();
expect(result.content).toContain('hello world');
});
it('reports errors with isError', async () => {
const result = await bash.execute({ command: 'cat /nonexistent' });
expect(result.isError).toBe(true);
});
it('supports file creation and reading', async () => {
await bash.execute({ command: 'echo "test content" > /test.txt' });
const result = await bash.execute({ command: 'cat /test.txt' });
expect(result.content).toContain('test content');
});
});Key patterns:
- Test command execution: call
tool.execute()with args - Check
isErrorflag for error conditions - Use file operations within tool tests (pipes, redirects)
- Test compound operations (e.g., zip → unzip)
Test Shell Command Argument Parsing
Test supplemental commands with mocked context:
import { describe, it, expect } from 'vitest';
import { createWhichCommand } from './which-command.js';
import type { IFileSystem } from 'just-bash';
function createMockCtx(
overrides: {
registeredCommands?: string[];
fs?: Partial<IFileSystem>;
} = {}
) {
const fs: Partial<IFileSystem> = {
resolvePath: (base: string, path: string) => (path.startsWith('/') ? path : `${base}/${path}`),
...overrides.fs,
};
return {
fs: fs as IFileSystem,
cwd: '/home',
env: new Map<string, string>(),
stdin: '',
getRegisteredCommands: () => overrides.registeredCommands ?? ['ls', 'cat', 'node', 'git'],
};
}
/** Create a minimal VirtualFS mock that yields the given file paths from walk(). */
function createMockVfs(files: string[]) {
return {
exists: async () => true,
walk: async function* () {
for (const f of files) yield f;
},
} as unknown as VirtualFS;
}
describe('which command', () => {
it('resolves built-in command to /usr/bin/<name>', async () => {
const cmd = createWhichCommand();
const result = await cmd.execute(['node'], createMockCtx());
expect(result.exitCode).toBe(0);
expect(result.stdout).toBe('/usr/bin/node\n');
});
it('returns error for no arguments', async () => {
const cmd = createWhichCommand();
const result = await cmd.execute([], createMockCtx());
expect(result.exitCode).toBe(1);
expect(result.stderr).toContain('missing argument');
});
it('finds .jsh file on VFS', async () => {
const mockVfs = createMockVfs(['/workspace/skills/test-skill/hello.jsh']);
const cmd = createWhichCommand(mockVfs);
const result = await cmd.execute(['hello'], createMockCtx());
expect(result.exitCode).toBe(0);
expect(result.stdout).toBe('/workspace/skills/test-skill/hello.jsh\n');
});
});Key patterns:
- Mock context: Create minimal mock with only needed properties
- Mock VFS: Return specific files from
walk()for file discovery tests - Test arg parsing separately: Test command-line parsing logic without booting the just-bash runtime
- Check exit codes and output: Verify both success and error paths
Apply Mocking Patterns
Using vi.fn() for function mocks
import { describe, it, expect, vi } from 'vitest';
describe('Module with callbacks', () => {
it('calls the callback with data', async () => {
const callback = vi.fn();
await myAsyncFunction(callback);
expect(callback).toHaveBeenCalledWith('expected data');
});
});Using vi.spyOn() for partial mocks
import { describe, it, expect, vi } from 'vitest';
describe('Module with internal methods', () => {
it('calls internal helper', async () => {
const spy = vi.spyOn(module, 'privateHelper');
await publicFunction();
expect(spy).toHaveBeenCalled();
});
});Message helper functions for typed content
Create helpers for constructing typed test messages:
function createMessage(role: 'user' | 'assistant' | 'toolResult', text: string): AgentMessage {
return {
role,
content: [{ type: 'text' as const, text }],
} as any;
}
function createToolResult(text: string): AgentMessage {
return {
role: 'toolResult',
content: [{ type: 'text' as const, text }],
} as any;
}
describe('createCompactContext', () => {
it('calls generateSummary when threshold exceeded', async () => {
const compact = createCompactContext({
model: mockModel,
getApiKey: () => 'test-key',
contextWindow: 200000,
});
const baseMsg = 'x'.repeat(65000);
const messages = Array.from({ length: 12 }, () => createMessage('user', baseMsg));
const result = await compact(messages);
expect(mockGenerateSummary).toHaveBeenCalledOnce();
expect(result.length).toBeLessThan(messages.length);
expect((result[0].content as any)[0].text).toContain('<context-summary>');
});
});Key pattern: Helper functions reduce boilerplate and make tests more readable.
Test These Categories
- Pure logic: Utilities, adapters, data transformations
- Path handling: Filesystem operations, ACL checks, path normalization
- Tool execution: Command execution, error handling, output parsing
- Message processing: Agent message formatting, context compaction
- Error conditions: ENOENT, EACCES, type mismatches
Manually Verify These Categories
- just-bash runtime: the shell interpreter itself (covered by tool tests)
- Chrome API: DebuggerClient, service workers — EXCEPT
state-machine and lifecycle-reconciliation logic (e.g., the
leader-tab pinning and service-worker lifecycle), which MUST be unit-tested
with mocked
chrome.*APIs. Seepackages/webapp/tests/ui/offscreen-client.test.tsfor the mock pattern (the extension's own SW suites live in ai-ecoverse/slicc-extension). - xterm.js: Terminal rendering (manually verified)
Web components and DOM rendering are covered by browser-mode Vitest suites
(@slicc/webcomponents, @ai-ecoverse/spoon). Changes to those packages require
automated browser tests, not just manual verification.
For skipped categories, ensure manual verification in both CLI and extension modes before committing.
Import-Time Runtime Gating in Extension Tests
Some modules compute runtime mode at import time (for example
const isExtension = isExtensionRealm()). For those modules, extension-path
tests must stub chrome before import, then re-import the module after
vi.resetModules().
Pattern:
(globalThis as any).chrome = { runtime: { id: 'test-extension-id', sendMessage, onMessage } };
vi.resetModules();
const mod = await import('../../src/providers/oauth-service.js');If one suite verifies both extension and non-extension paths, call
vi.resetModules() between cases so each import sees the intended runtime
globals.
Enforce Coverage
Coverage thresholds are enforced in CI and stored in coverage-thresholds.json at the
repo root. Never hand-lower these values — use the commands below to measure and let
the ratchet raise floors automatically.
Follow the Floors and Nightly Ratchet
packages/dev-tools/tools/coverage-ratchet.mjs (driven by
.github/workflows/coverage-ratchet.yml) is the single source of truth for floor
maintenance. It only ever raises floors toward measured coverage — whole-point steps
with ~0.5–1.5 pp headroom via a half-point safety margin — and opens a PR when anything
changed.
Check TypeScript Packages
Run npm run test:coverage:<package> — this invokes vitest --coverage (v8 provider)
then coverage-gate.mjs, which reads the package's floors from
coverage-thresholds.json. CI runs the same script as the package's only test step.
Anything after the package name is forwarded verbatim to vitest, so a run-wide option can be added without paying for a second pass over the same suite:
node packages/dev-tools/tools/coverage-gate.mjs webapp --reporter=json --outputFile=report.jsonCheck Swift Packages
Run packages/dev-tools/tools/swift-coverage-check.sh <package-dir> <test-bundle-name>,
which executes swift test --enable-code-coverage followed by xcrun llvm-cov report.
Tests/ and .build paths are excluded; the TOTAL row is checked against the
lines/functions/regions floors from coverage-thresholds.json (passed explicitly or
read from the file). The swift-launcher floor stays low because most of the bundle is
SwiftUI views that resist unit tests.
Both arguments are required:
./packages/dev-tools/tools/swift-coverage-check.sh \
packages/swift-server SliccServerPackageTests
./packages/dev-tools/tools/swift-coverage-check.sh \
packages/swift-traykit SliccTrayVFSPackageTests
./packages/dev-tools/tools/swift-coverage-check.sh \
packages/swift-launcher SliccstartPackageTestsKeep Tests Isolated From Shared State
Isolation is per-suite, and each suite buys it differently:
| Suite | Isolation |
| Vitest projects | One worker thread + a fresh module registry per file; VirtualFS gets a unique dbName |
dev-tools | fileParallelism: false — two gates overwrite the same on-disk baseline (documented) |
swift test packages | Per-test UserDefaults(suiteName:) suites, erased in tearDown |
ios-app (xcodebuild test) | Serial + random execution order + per-test suites + a clean app container per run |
ios-app is the one that needs care. Both bundles are pinned serial
(-parallel-testing-enabled NO): simulator clones race the XCUITest runner's install
and the losing clone fails preflight with Busy before a test runs. Serial execution
hides order dependence, so three things stand in for parallelism, and
npm run lint:ios-test-isolation fails the build when the first two disappear:
- Random execution order — declared per test target in
packages/ios-app/project.yml. The Xcode project is generated and uncommitted, so nothing else pins it. No writes to
UserDefaults.standard. A unit test runs inside the host app's process, so that is the app's own persistent domain: shared by the whole bundle and kept on disk in the simulator container. Take an ephemeral suite instead:let defaults = try makeIsolatedDefaults(flags: ["uiTestSessionsFixture": true]) let backend = try XCTUnwrap(UITestHooks.sessionsFixtureBackend(defaults: defaults))makeIsolatedDefaults(SliccFollowerTests/IsolatedTestDefaults.swift) names the suite after the test, seeds the flags, and erases the domain in a teardown block that also runs when the test fails. Production reads stay on.standard, which is where a UI test's launch arguments land.- A clean app container per run —
ios-sim-prepare.shboots the device and uninstalls both bundles beforexcodebuild test, so an interrupted run cannot leave a fixture flag on disk for the next one. Both CI legs go through it.
Retry Flaky Tests
Retries are configured per vitest project in vitest.config.ts and gated on CI so
local runs still fail fast:
| Project | Retries in CI | Why |
node-server | 1 | Spawns Chrome/Electron, binds ports, waits on child-process handshakes |
Playwright E2E (packages/webapp/tests/e2e/) | 2 | Real browser + CDP + model staging under load |
every other project (webapp, shared, cherry, …) | 0 | Deterministic in-process suites — a failure is a bug, not noise |
Rules for changing this:
- Do not add retries to a project to make a red suite green. A retry hides nondeterminism; fix the ordering, fake the timer, or isolate the resource instead.
- Keep the count at 1 unless the flake is provably external (a real browser, a real port).
Retried-but-passing tests still show up as
retriedin the run report, so the signal survives; a higher count buries it and doubles worst-case wall-clock. - Never enable retries locally.
CI_RETRIESinvitest.config.tsresolves to0withoutCI, sonpm run teston a laptop reports the first failure.
Keep Tests Inside the Timeout Budget
testTimeout is vitest's default 5000ms; the root config never raises it. A test that
deterministically spends 2-3s of that budget passes on an idle laptop and fails on a loaded
one, in a different file each run. That moving target is the signature of budget exhaustion,
not of a shared-resource race — the victim is whichever slow test met peak contention.
Two patterns burn the budget without asserting anything.
1. Heavy await import() inside a test body. Vitest charges module transform and
evaluation to whichever test triggers it, so the first test in a file pays for the whole
graph and the rest cost nothing. In packages/webapp/tests/shell/ipk/, newShell() imported
VirtualFS and AlmostBashShellHeadless lazily: the first call cost 1347ms, every later
one 0ms. Hoist them to static top-level imports — vi.mock() factories are hoisted above
imports so mocks still apply, and the cost moves to the collection phase, which no per-test
timeout covers.
// Wrong — the first test in the file absorbs the entire module graph.
async function newShell() {
const { VirtualFS } = await import('../../../src/fs/index.js');
}
// Right — collection pays for it; no test does.
import { VirtualFS } from '../../../src/fs/index.js';Keep the dynamic form only where the test needs a genuinely fresh module instance, i.e. it
calls vi.resetModules() (see tests/ui/wc/wc-follower.test.ts). Give those an explicit
per-test timeout instead.
2. Real production backoff in a test that is not testing backoff. Retry schedules are
injectable for exactly this reason: setupStandalonePrelude takes sleep, and
SessionTrayDurableObject takes webhookDeliveryWaitMs. Omit them and the test sits out the
full production budget — 3.1s (CDP_BRIDGE_CONNECT_RETRY_DELAYS_MS) and 3s
(WEBHOOK_DELIVERY_WAIT_MS) — inside a 5s timeout. Inject the fast value, and leave the
timing assertion to the dedicated unit test (connectWithBoundedRetry, which takes delays).
3. Spawning a real browser from the default suite. playwright-command.test.ts's
iframe integration block launches real headless Chrome. Under memory/CPU pressure the OS
SIGKILLs it mid-launch and the whole suite fails with Chrome exited with code null before
reporting CDP port — a machine-state failure that says nothing about the code under test. It
is gated on CI || SLICC_TEST_CHROME_INTEGRATION=1, so CI (ubuntu-latest ships Chrome and
sets SLICC_CDP_LAUNCH_TIMEOUT_MS) keeps it as a real gate while npm run test on a laptop
does not spawn browsers. Run it locally with:
SLICC_TEST_CHROME_INTEGRATION=1 npx vitest run \
packages/webapp/tests/shell/supplemental-commands/playwright-command.test.tsNever answer either pattern by raising testTimeout: that makes the failure rarer and harder
to read without removing the wasted seconds. Audit with the timing JSON below — nothing in the
default suite should sit near a second without an explicit, justified per-test timeout.
Read Test Timing
When CI is set, the root test.reporters adds vitest's json reporter and writes
per-test durations to test-timing/vitest.json (gitignored). reporters and outputFile
are root-only options in vitest 4, so this covers every project — including the
vitest run --project <name> invocations made by
packages/dev-tools/tools/coverage-gate.mjs.
CI uploads the file from the webapp and node-server jobs as
test-timing-webapp / test-timing-node-server
(if: always(), so a failing run still produces it).
Shape: testResults[] is one entry per file with startTime / endTime, and
assertionResults[] is one entry per test with a duration in milliseconds.
{
"numTotalTests": 341,
"testResults": [
{
"name": "/…/packages/shared-ts/tests/base64.test.ts",
"startTime": 1785156911234,
"endTime": 1785156911814,
"assertionResults": [
{
"fullName": "base64 codec round-trips an empty Uint8Array",
"status": "passed",
"duration": 1.2396669999999972
}
]
}
]
}To find the slowest tests in a downloaded artifact:
node -e "const r=require('./test-timing/vitest.json').testResults.flatMap(f=>f.assertionResults.map(a=>[a.duration,a.fullName]));r.sort((a,b)=>b[0]-a[0]);console.log(r.slice(0,20))"Reproduce it locally with CI=1 npm run test.
Run Tests
| Command | Purpose |
npm run test | Run Node-mode Vitest projects |
npm test -w @slicc/webcomponents | Run Chromium browser-mode component tests |
npm test -w @ai-ecoverse/spoon | Run Chromium launcher tests |
npx vitest run packages/webapp/tests/fs/virtual-fs.test.ts | Run single test file |
npx vitest run packages/webapp/tests/fs/ | Run all tests in directory |
npx vitest run --reporter=verbose | Verbose output with full stack traces |
npx vitest run --reporter=dot | Minimal output (one . per test) |
Organize Test Files
Tests live in packages/*/tests/ mirroring the src/ structure:
packages/webapp/tests/fs/
virtual-fs.test.ts
restricted-fs.test.ts
packages/webapp/tests/tools/
bash-tool.test.ts
file-tools.test.ts
packages/webapp/tests/shell/supplemental-commands/
which-command.test.ts
skill-command.test.ts
packages/webapp/tests/core/
context-compaction.test.ts
logger.test.tsTest with the Design-Time Chat Fixture
Load the app with ?ui-fixture=1 (also accepts ?ui-fixture or ?ui-fixture=true) to swap
the chat view for a synthetic session covering every message variant — user/assistant bubbles,
markdown + code blocks, all four tool-call states, the six lick channels, delegation, queued
messages, and a streaming tail.
Messages are defined in packages/webapp/src/ui/chat-fixture.ts (createChatFixture()) and
persist to a dedicated session-ui-fixture ID so real scoop storage is untouched. Clicking any
real scoop cleanly exits fixture mode. Vite HMR picks up CSS changes live against the fixture.
When adding new message UI variants: extend createChatFixture() and add a matching
assertion in packages/webapp/tests/ui/chat-fixture.test.ts so the fixture harness stays
comprehensive. The test file asserts that every variant type is present in the fixture — a
new type that appears in the fixture but has no test entry causes the test to fail fast.
Build Test Data Fixtures
Avoid hardcoding test data. Use generators or helper functions:
function generateLargeText(sizeChars: number): string {
return 'x'.repeat(sizeChars);
}
function createTestFile(name: string, content: string) {
return { name, content };
}
describe('File operations', () => {
it('handles large files', async () => {
const largeContent = generateLargeText(1_000_000);
await vfs.writeFile('/large.txt', largeContent);
const result = await vfs.readFile('/large.txt');
expect(result).toBe(largeContent);
});
});Debug Tests
Run a single test with verbose output:
npx vitest run --reporter=verbose packages/webapp/tests/fs/virtual-fs.test.tsAdd console.log() in test code — output appears in terminal:
it('does something', async () => {
const result = await operation();
console.log('result:', result); // visible in test output
expect(result).toBe(expected);
});Watch mode for rapid iteration:
npx vitest watch packages/webapp/tests/fs/virtual-fs.test.tsMake changes to test or source → Vitest re-runs automatically.
Choose Integration or Unit Tests
- Unit tests (the default): Test one module in isolation with mocked dependencies
- Integration tests (acceptable): Test filesystem + shell + tool together if they can't be tested separately
Example of acceptable integration test:
describe('Bash tool integration', () => {
it('reads from VirtualFS via shell', async () => {
// Test that bash tool talks correctly to VirtualFS
// This requires both components together
const fs = await VirtualFS.create({ dbName: 'test', wipe: true });
const shell = new AlmostBashShellHeadless({ fs });
const bash = createBashTool(shell);
await bash.execute({ command: 'echo hello > /file.txt' });
const content = await fs.readFile('/file.txt');
expect(content).toBe('hello\n');
});
});But avoid testing implementation details across many layers. Keep most tests focused and fast.
Test the Agent Loop with the Fake-LLM E2E Framework
End-to-end agent-loop tests run the real WC composer + kernel-worker agent against a deterministic OpenAI-compatible fake LLM server. The agent loop is identical to production — fixtures only change which assistant turns stream back.
See docs/architecture.md for the standalone topology and protocol context.
Three pieces compose the framework:
- Fake server (
packages/webapp/tests/e2e/fake-llm/): SSE-streaming OpenAI-compatible server with permissive CORS. Started by a secondwebServerentry inplaywright.config.tson port 5781. - Fixture (
packages/webapp/tests/e2e/fake-llm/fixtures/*.json): ordered list of scripted assistantturns(text + optionaltool_calls). Turns are matched cursor-first; per-turnwhenUserMessageMatches(substring or{ pattern, flags }regex) selects a specific turn for a specific user input. - Playwright harness (
packages/webapp/tests/e2e/fake-llm-helpers.ts):seedLocalLlmProvider,submitUserMessage,waitForTurnComplete,runUserInputFixture, andreadCdpPageState. - Two-instance topology (
packages/webapp/tests/e2e/two-instance-helpers.ts): a second SLICC runtime joined to the leader through the real tray hub, plus the cone / freezer-rail / tab-strip / terminal helpers the multi-cone scenarios share. See "Write a Two-Instance Scenario" below. - Terminal-only smokes (no fake LLM): drive
globalThis.__slicc_terminal_view.executeCommandInTerminalafterslicc-dock.selectItem('term')— same seam as the chat panel's "run in terminal". Examples:git-clone-live.test.ts,python-print.test.ts(browseripk add pyodide@<root pin>thenpython3 -c "print(1 + 1)").
Write a Scenario
See packages/webapp/tests/e2e/reference-scenario.test.ts for the
working reference. It runs a single Playwright test against a
3-phase fixture and interleaves chat-transcript + CDP-state
assertions after each phase:
- Phase 1 — user input
"open the reference page"matchesturns[0](substring matcher); the agent runs onebashplaywright-cli tab-newopening Page A (titleFAKE LLM REFERENCE TARGET).turns[1](cursor follow-up) delivers"Done. Page A is open.". The test then asserts both strings are in the thread and thatreadCdpPageStatesees exactly one target at the Page Adata:URL. - Phase 2 — user input
"open the comparison pages"matchesturns[2], which emits twobashtool calls in one turn (Page BFAKE LLM COMPARE ALPHA, Page CFAKE LLM COMPARE BETA) with smallcontentChunkSizeandtoolArgumentsChunkSizeto stress SSE delta reassembly.turns[3](cursor) closes the phase with"Done. Pages B and C are open.". The test asserts the closing text and thatreadCdpPageStatefiltered to the two Page B/C URLs reports both distinct titles. - Phase 3 — user input
"give me a summary"matchesturns[4]via the object-form regex matcher ({ "pattern": "summar(y|ize)", "flags": "i" }); the turn returns text only (no tool call). The test asserts the summary text in the thread and thatreadCdpPageStatefiltered to all threedata:URLs enumerates Pages A/B/C with their three distinct titles (sorted for a stable assertion).
Boilerplate from reference-scenario.test.ts (note: uses gotoLeader,
not page.goto('/'), and resets the fixture cursor before each attempt):
import { expect, test } from '@playwright/test';
import {
readCdpPageState,
resetFakeLlm,
seedLocalLlmProvider,
submitUserMessage,
waitForTurnComplete,
} from './fake-llm-helpers.js';
import { gotoLeader, seedSkipSwReload, waitForSW } from './helpers.js';
// Binds Chrome's CDP at the default port the helper probes AND the
// port the `node-server --serve-only --cdp-port=9222` proxy expects.
test.use({ launchOptions: { args: ['--remote-debugging-port=9222'] } });
test.describe('my scenario', () => {
// Rewind the fake-LLM turn cursor before every attempt so Playwright
// retries don't resume mid-fixture and fail with fixture_overflow.
test.beforeEach(async () => {
await resetFakeLlm();
});
test('scripted tool calls drive CDP navigations', async ({ page }) => {
await seedLocalLlmProvider(page, { modelId: 'fake-coder-reference' });
await seedSkipSwReload(page);
// MUST use gotoLeader — it appends ?bridge=ws://…/cdp&bridgeToken=…
// so the page-realm BrowserAPI dials the node-server CDP bridge.
// page.goto('/') will NOT connect the agent to CDP.
await gotoLeader(page);
await waitForSW(page);
await page.waitForSelector('slicc-input-card');
await expect(page.locator('slicc-chat-thread')).toContainText('Welcome to SLICC');
// Drive one phase at a time so per-phase assertions can interleave.
await submitUserMessage(page, 'open the reference page');
await waitForTurnComplete(page);
await expect(page.locator('slicc-chat-thread')).toContainText('Done. Page A is open.');
await expect
.poll(async () =>
(await readCdpPageState({ filter: (t) => t.type === 'page' })).map((t) => t.title)
)
.toContain('FAKE LLM REFERENCE TARGET');
// Phases 2 and 3 follow the same submit + wait + assert pattern.
});
});For full inputs, fixture content, and assertions, read the working
test directly. The fixture lives at
packages/webapp/tests/e2e/fake-llm/fixtures/reference-scenario.json
and uses onOverflow: "error" so a fixture/test mismatch fails fast
with a 400 from the fake server rather than hanging.
Assert CDP State
readCdpPageState polls http://127.0.0.1:9222/json and returns the
browser's CDP target list. Use it whenever you need to observe the
agent-driven Chrome from the outside — it is runtime-agnostic and
works against any Chrome with --remote-debugging-port set, including
a cone-driven Chrome the test process never launched.
Why data:text/html,… instead of a seeded /preview/* page: tabs
opened via CDP Target.createTarget (which is what playwright-cli
tab-new does) are not claimed by the preview-vfs service worker —
the SW only controls clients that loaded the SLICC bootstrap. A data:
URL carries its HTML inline, so the agent-driven tab gets a
deterministic title without depending on SW interception. When a test
needs to assert on Playwright-controlled DOM, navigate the existing
page via bash playwright-cli goto … instead (and assert via
page.locator(...) before the navigation breaks the test page).
If a scenario needs a /preview/* page rendered for assertion,
seedVFS continues to work for content the test page itself loads
— it just doesn't reach CDP-spawned tabs.
Write a Two-Instance Scenario
Anything that crosses the tray — follower rails, cone mirroring, a follower
driving leader state — needs two runtimes, not two tabs.
two-instance-helpers.ts builds that pair on top of the servers the harness
already runs; multiple-cones-follower.test.ts is the worked example.
await bootMultiConeLeader(page, { fixture, tray: true });
const follower = await joinAsFollower(browser, await leaderJoinUrl(page));
try {
await expect.poll(() => switcherLabels(follower.page)).toEqual(await switcherLabels(page));
} finally {
await follower.close();
}Four things make it work, and each is a trap if you rebuild it by hand:
- The tray hub is real. The harness's
wrangler devISpackages/cloudflare-worker, Durable Objects included, so the leader mints a tray againsthttp://localhost:<wrangler>and the worker serves the SPA at/join/<token>.bootMultiConeLeader({ tray: true })appends?trayWorkerUrl=, which beats both the stored value and the node-server's/api/runtime-config— without it the page points its tray at the PRODUCTION hub and the run means nothing (the origin rule, see.agents/skills/cdp-smoke-test/tier3-multi-harness.md). - The follower needs its own browser CONTEXT.
wc-tray.tselects one leader per origin per profile over the Web Locks API; a second page in the leader's context defers instead of following.joinAsFolloweropens a context, and the caller closes it in afinallyso a failing assertion cannot leak a runtime into the next test. - Read the join URL from
localStorage, not from a module global.LeaderTrayManagerruns in the page realm;slicc.leaderTrayStatusis the shim the page keeps current, andleaderJoinUrlpolls it forstate === 'leader'. - This follower has no local CDP surface. It can render, select and drive leader state; it can never host a teleported tab or advertise targets. Do not write a federated-CDP assertion against it.
Two more rules the cone scenarios paid for:
- Assert the scripted REPLY, not
[data-processing]. A fake-LLM turn can open and close between two polls of the attribute, sowaitForTurnComplete({ mustObserveTurnRise: true })fails turns that ran perfectly. Thechat()helper waits for the reply text first, then lets the turn settle. - The workbench terminal is a LAZY mount.
__slicc_terminal_viewonly appears after the term surface is activated —openTerminal()(and thereforeexecInTerminal()) firesdock.selectItem('term')first. A bare wait on the global just burns its timeout.
Give a two-instance test its own test.setTimeout (TWO_INSTANCE_TEST_TIMEOUT_MS,
10 min; single-runtime cone specs use CONE_TEST_TIMEOUT_MS, 5). The config's
30s default covers one turn, not a cone lifecycle plus a tray handshake — a
healthy CI run of the follower scenario takes ~1.5 min against ~10s locally, so
size the ceiling for a LOADED runner, not merely a slow one.
Bound every step, or a hang tells you nothing. Playwright's default
actionTimeout / navigationTimeout is 0 — unbounded, cut off only by the
test timeout. A goto or .click() left at the default turns any stall into a
bare Test timeout of Nms exceeded whose stack points at whatever ran last
(usually your finally). This dropped #2328 out of the merge queue three
attempts in a row before the cause was even visible. Pass an explicit timeout
to every navigation and action.
A failed turn never renders the reply you are waiting for. When a turn dies
— provider error, fixture overflow, retries exhausted — the thread renders a
<slicc-error-card> and the scripted text never arrives, so toContainText
waits out the whole budget. Use expectReply(), which races the expected text
against that card and throws naming the agent error.
Capture both runtimes' console output with watchBrowserDiagnostics() and
re-throw through diagnostics.annotate(err). Without it a CI failure's cause
lives only in the trace artifact, which for the e2e job is ~190 MB.
Tune retries per spec when an attempt is expensive: the follower spec sets
test.describe.configure({ retries: 1 }) because three attempts at a 10-minute
ceiling would consume half the e2e job's budget and starve the specs after it.
Run the E2E Framework
npm run test:e2eplaywright.config.ts boots three webServer entries, mirroring the
production "Standalone" topology now that node-server no longer serves
the UI:
- wrangler dev on 8787 — serves the built
dist/ui(the leader / UI origin and the PlaywrightbaseURL) with SPA fallback, exactly as the production worker does. Requiresnpm run build -w @slicc/webappfirst. Runs under the supervisor inwrangler-server.ts— see "Survive a wrangler crash" below. Uses--local --env stagingbecause staging hasroutes: []: the worker must preserve bothlocalhost:<port>and preview<token>.localhost:<port>hosts. Do not replace this with--route localhost; that rewrites preview hosts into the app origin and returns the SPA instead. Local mode runs DO/R2 in workerd, not the deployed staging hub. - node-server
--serve-only --cdp-port=9222on 5710 — the thin/cdpbridge +/apisurface only.SLICC_BRIDGE_TOKENarms the/cdpupgrade gate + cross-origin/apitoken check, andBRIDGE_DEV_ALLOWED_ORIGINSallowlists the wrangler origin so its cross-origin requests pass. - fake LLM on 5781 — the agent talks to it via the seeded
local-llmprovider. Override the fixture with theFAKE_LLM_FIXTUREenv var.
Because the UI is served cross-origin from the bridge, scenarios that drive
the agent (CDP) or the fetch proxy boot via the gotoLeader helper, which
appends ?bridge=ws://localhost:5710/cdp&bridgeToken=… so the page-realm
BrowserAPI dials the bridge and proxied-fetch routes /api/* at the
node-server origin (the same launch params node-server appends in
production standalone mode).
The project pins workers: 1 so only one CDP-binding scenario runs
at a time. The node-server --serve-only --cdp-port=9222 proxy can
only point at one Chrome at a time, and every scenario that drives
the agent's playwright-cli (reference-scenario.test.ts,
preview-serve.test.ts) launches Playwright Chrome with
--remote-debugging-port=9222 — running them in parallel would
collide on the port and on the proxy's outbound target. The fake-LLM
webServer entry also sets reuseExistingServer: false so each run
starts with a fresh turn cursor and fixture.
In CI the dedicated e2e job (in .github/workflows/ci.yml) runs a medium
suite on pull requests (SLICC_E2E_PR=1) and the full suite on
merge_group. The PR gate ignores the merge-queue-only basenames in
packages/webapp/tests/e2e/mq-only-specs.ts (multiple-cones*,
compaction-robustness, roving-tray-webhook, sprinkle-details, speech) so it
fills spare capacity under webapp / node-matrix without owning the
required ci signal; new medium specs join PRs by default. It triggers on changes to any runtime the harness drives or bundles —
webapp, vfs-root, assets, shared-ts, spoon, webcomponents,
cloud-core, node-server, cloudflare-worker — plus dependency manifests,
patches, relevant TS/Vitest config, and the CI workflow. The workflow records
that scope in a dedicated e2e paths-filter output; keep it narrower than
root-config so a coverage-floor-only change does not start a browser run.
The multi-cone leg (multiple-cones*.test.ts, #2313) rides the merge-queue
job with no extra gating: it exercises the multiple-cones flag end to end — cone
create / switch / drop, the rail's session actions and their freezer outcomes,
lick addressing across cones, and the leader + follower pair above.
Playwright retries twice in CI (retries in the config); locally it
fails fast. The real-Kokoro speech round-trip rides the full merge-queue
run as a conditional leg: when the speech path filter matches, the run
sets RUN_REAL_SPEECH_E2E=1 (un-skipping speech-roundtrip.test.ts) and
frees runner disk first so Chromium's free-disk-derived storage quota can
hold the staged weights.
Survive a wrangler Crash
wrangler dev (workerd) dies mid-suite (#2372). Left alone, the first crash
turns every later spec into a ~200 ms ERR_CONNECTION_REFUSED failure — a wall
of red naming eighteen innocent tests. The likely cause is
workers-sdk#15202:
workerd exits with kj/async-io-unix.c++: … Broken pipe when the consumer of
its stdout (Playwright's webServer capture) stops draining the pipe.
Three pieces contain it, and one rule applies when you write a spec:
- Import
testfrom./fixtures.js, never from@playwright/test. That is the whole rule.fixtures.tsre-exportsexpectunchanged and adds one auto fixture that probesHEAD <baseURL>/statusbefore and after every test. wrangler-server.tssupervises wrangler instead of Playwright spawning it directly: it owns (and always drains) wrangler's stdout, mirrors it to.wrangler/e2e-logs/wrangler-output.log, re-spawns workerd when it exits, and servesPOST /restartonSLICC_E2E_WRANGLER_SUPERVISOR_PORT(default: leader port + 1).leader-health.tsdecides what happens on a dead origin: restart through the supervisor and fail only the current spec with an error namedWRANGLER_CRASHED. If the restart fails, the remaining specs fail instantly with the same named error instead of burning their timeouts — the job then names its cause. Its logic is dependency-injected and unit-tested inpackages/webapp/tests/e2e-harness/leader-health.test.ts.
So a crash costs one spec (which CI then retries), not the run. Evidence lands
in .wrangler/e2e-logs/: wrangler's own wrangler-<ts>.log, the mirrored
output, and crash-report.md (one section per unexpected workerd exit, with
the output tail). The CI job prints the crash report and uploads the directory
as the wrangler-logs-e2e artifact.
Grep a red e2e job for WRANGLER_CRASHED before believing the failing spec
names are meaningful.
Verify Risks with the Reference Scenario
- localStorage → kernel-worker shim sync: the test only passes if
the seeded
slicc_accounts+selected-modelmake it from page storage into the worker'slocalStorageshim — otherwise the agent never resolves thelocal-llmmodel and never calls the fake server. waitForTurnCompletemasking failures: the per-phase chat-transcript + CDP-state assertions are positive — each one only succeeds when the prior scripted turn actually ran end to end, so a silent "turn never started" surfaces as a timeout, not a false green.- CDP port alignment: the helper, the
node-server --serve-onlyproxy, and the test'slaunchOptions.argsall agree on 9222, and the per-phasereadCdpPageStatefilters key on each target's distinct title/URL so multi-tab assertions stay unambiguous. - SSE delta reassembly: phase 2's turn sets small
contentChunkSizeandtoolArgumentsChunkSizeso the agent has to reassemble both streamed content and multi-tool-call argument fragments before the bash commands run. - Matcher coverage: the three matched turns exercise the
substring form (phases 1 + 2) and the object-form regex
(
{ pattern, flags }, phase 3) ofwhenUserMessageMatches.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

