writing-tests
Skill59 starsChanged 12 days agoMITView on GitHub
Use when writing, changing, or running Go tests in clawker: test helpers, fakes, moq mocks, command test tiers, the e2e harness, golden files, and the storage oracle.
43 other projects use a version of this.
Adopted by
Most recently taken up by
- quarkiverse/quarkus-langchain4jtheir own version, first seen Nov 16, 2026
- chrisezedi/claude-skillstheir own version, first seen Nov 16, 2026
- RMATI-11861/Claude-Skillstheir own version, first seen Nov 16, 2026
- ekirton/Pouletheir own version, first seen Nov 15, 2026
- ZaxbyHub/opencode-swarmtheir own version, first seen Nov 15, 2026
- iliaal/ai-skillstheir own version, first seen Nov 14, 2026
Used with
Projects with it also use reviewing-code, commits and 7 more.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
RiccardoGrinSkillreviewing-code
Runs an adversarial code review loop that spawns independent reviewer and fixer subagents, iterating until only nitpicks remain. Scores findings by confidence, fixes real issues, and re-reviews with fresh eyes — all internal, no GitHub comments. Use when asked to review code, self-review, adversarial review, or polish code before pushing
Also in 4 projects that use this
JetBrainsSkillcommits
Use this skill whenever the user asks to commit changes, write or fix a commit message, amend or rename a commit, or do a workflow that includes committing in the IntelliJ repository. This is a thin repo-specific overlay: use IntelliJ commit format, write full commit messages by default, and keep requested suffixes such as IJ-MR trailers in a final separate paragraph.
Also in 3 projects that use this
OpenAISkillrunning-tests
Guide for running tests in the openai-dotnet repository. Use this when asked to run, debug, or validate tests, or when writing new tests. Explains test modes (Playback, Record, Live), how to identify recorded vs non-recorded tests, environment variable configuration, and what to do when recordings are missing or stale.
Also in 3 projects that use this
BlockSkilldebugging
Debugging guidelines and best practices
Also in 6 projects that use this
sahinSkillexecute
Also in 3 projects that use this
stericishereSkillgraphify
any input (code, docs, papers, images) → knowledge graph → clustered communities → HTML + JSON + audit report
Also in 3 projects that use this
Other versions43 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 43
- remotion-dev/remotion★ 73ksame name
- PostHog/posthog★ 41ksame name
- JetBrains/intellij-community★ 21ksame name
- quarkusio/quarkus★ 16ksame name
- mcp-use/mcp-use★ 11ksame name
- DetachHead/rebased★ 6.5ksame name
- chattocorp/chatto★ 3ksame name
- starkware-libs/cairo★ 1.9ksame name
- bholmesdev/hubble.md★ 1.5ksame name
- amplifthq/openmeld★ 1.4ksame name
33 more not listed
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository17 other files it gives its agents.
- AGENTS.mdAGENTS.md
- clawkerd/AGENTS.mdAGENTS.md
- controlplane/AGENTS.mdAGENTS.md
- docs/AGENTS.mdAGENTS.md
- test/AGENTS.mdAGENTS.md
- .github/copilot-instructions.mdCopilot instructions
- agent-filesSkill
- audit-memorySkill
- cli-outputSkill
- dev-checksSkill
- firewall-uatSkill
- initiativeSkill
- monitoring-checksSkill
- store-editorSkill
- test-hunterSkill
- wizard-commandsSkill
- code-reviewSkill
The file
We split this skill into the rules it gives an agent
12 rules. 0 are common, 3 are distinctive, and 9 are very specific to this project.
0
Common
Most of the 43 edited versions we read keep these.
3
Distinctive
This author's own choices. Worth a look.
9
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 35
When every node provides fakes/mocks/stubs, any tier can independently test by mocking the entire chain below it.
- Very specific: line 58
Every command constructor accepts runF to intercept execution:
- Distinctive: line 122
Every future test at any tier benefits
- Very specific: line 202
Use projectmocks.NewMockProjectManager() when you only need interface-level behavior in unit tests.
- Very specific: line 215
Use projectmocks.NewMockProject(name, repoPath) when your test needs a project with read accessors populated.
- Very specific: line 225
Use projectmocks.NewTestProjectManager(t, gitFactory) when tests must exercise real Register/Remove/List round-trips with file-backed …
- Distinctive: line 388
All command tests (gomock fully removed from codebase)
- Very specific: line 484
Use t.TempDir() for WorkDir to avoid os.Getwd() issues in tests
- Distinctive: line 850
All container commands should test BOTH flag-based and positional patterns:
- Very specific: line 962
Must use dev.clawker label prefix (not com.whailtest) — docker-layer methods like ListContainers call ClawkerFilter() which filters by …
- Very specific: line 965
No import cycles: internal/docker/mocks -> internal/docker + pkg/whail + pkg/whail/whailtest is clean
- Very specific: line 970
Default NewFakeClient volume/network inspect defaults sufficient for EnsureConfigVolumes and EnsureNetwork flows
“Common” is measured: kept by at least 6 in 10 of the 43 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
writing-tests
Extended test patterns and examples. For the mandatory rules, see the Testing rules section of the root
AGENTS.md.
Testing Philosophy: DAG-Driven Test Infrastructure
The Core Principle
Clawker's packages follow a strict DAG (Directed Acyclic Graph):
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Foundation │ ──▶ │ Middle │ ──▶ │ Composite │ ──▶ │ Commands │
│ Packages │ │ Packages │ │ Packages │ │ │
├─────────────┤ ├─────────────┤ ├─────────────┤ ├─────────────┤
│ git, logger │ │ bundler │ │ docker, │ │ cmd/* │
│ iostreams │ │ │ │ workspace │ │ │
│ config │ │ │ │ │ │ │
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
│ │ │
▼ ▼ ▼
gittest/ docker/mocks/ Factory DI
config/mocks/ whailtest/ + runF seamEach node in the DAG must provide test infrastructure for its dependents.
When every node provides fakes/mocks/stubs, any tier can independently test by mocking the entire chain below it.
Test Seams in Clawker
1. Factory Pattern DI
Factory fields are closures that return dependencies. Tests inject fakes:
f := &cmdutil.Factory{
IOStreams: tio,
Client: func(ctx context.Context) (*docker.Client, error) {
return fake.Client, nil // Fake from docker/mocks
},
Config: func() (config.Config, error) {
return configmocks.NewBlankConfig(), nil
},
// ... other fields
}2. runF Test Seam
Every command constructor accepts runF to intercept execution:
// Tier 1: Flag parsing only (intercept before run)
var captured *RunOptions
cmd := NewCmdRun(f, func(ctx context.Context, opts *RunOptions) error {
captured = opts
return nil // Don't actually run
})
// Tier 2: Full execution with injected deps
cmd := NewCmdRun(f, nil) // nil = real run function with Factory's fakes3. Package Test Utilities
Each package with complex dependencies provides test infrastructure:
| Package | Test Location | What It Provides |
internal/testenv | testenv/ | New(t, opts...) → isolated XDG dirs + optional Config/ProjectManager |
internal/docker | mocks/ | FakeClient, SetupContainerList, fixtures |
internal/config | mocks/ | NewBlankConfig(), NewFromString(projectYAML, settingsYAML), NewIsolatedTestConfig(t), ConfigMock |
internal/git | gittest/ | InMemoryGitManager |
internal/project | mocks/ | NewMockProjectManager(), NewMockProject(name, repoPath), NewTestProjectManager(t, gitFactory), RegistryMock (moq) |
internal/state | mocks/ | NewBlankState(), NewFromString(yaml), StateStoreMock (moq) |
pkg/whail | whailtest/ | FakeAPIClient, build scenarios, EventRecorder |
api/admin/v1 | mocks/ | AdminServiceClientMock (moq) |
api/agent/v1 | mocks/ | AgentServiceClientMock (moq) |
controlplane/agent | mocks/ | HostForwarderMock, OtelForwarderMock, RegistryMock (moq) |
controlplane/auth | mocks/ | IntrospectorMock (moq) |
controlplane/manager | mocks/ | ManagerMock (moq) |
controlplane/firewall | mocks/ | EgressRulesStoreMock, RouteIdentityStoreMock (moq) |
controlplane/firewall/ebpf | mocks/ | EBPFManagerMock (moq) |
internal/hostproxy | hostproxytest/ | MockManager |
internal/iostreams | Test() | iostreams.Test() → (*IOStreams, *bytes.Buffer, *bytes.Buffer, *bytes.Buffer) |
internal/storage | ValidateDirectories() | XDG directory collision detection |
The Agent Obligation
If a DAG node is missing test infrastructure, add it first.
This is not scope creep. A node without test infrastructure is an incomplete node — it blocks proper testing of everything downstream in the DAG.
┌─────────────────────────────────────────────────────────────┐
│ DECISION: Need to test component at tier N? │
├─────────────────────────────────────────────────────────────┤
│ For each dependency at tier N-1: │
│ ├─ Does it have a *test/ subpackage or test utils? │
│ ├─ Interface for the concrete type? │
│ ├─ Fake/mock/stub implementation? │
│ └─ Fixtures for common scenarios? │
├─────────────────────────────────────────────────────────────┤
│ ALL YES → Mock the entire chain, write your test │
│ ANY NO → STOP. Complete that node's test infra first. │
│ Then write your test. │
└─────────────────────────────────────────────────────────────┘Why This Compounds
Each node that gains test infrastructure:
- Enables all downstream nodes to mock it independently
- Every future test at any tier benefits
- The "incomplete node" case becomes rarer over time
The DAG fills in, and eventually any tier can mock/fake/stub the entire chain below it.
Anti-Patterns to Avoid
❌ Inline mocking: Creating ad-hoc mocks inside a test file instead of using/creating package test utils ❌ Workaround tests: Testing around missing infrastructure instead of adding it ❌ Copy-paste fakes: Duplicating fake implementations across test files ❌ Skipping DI: Directly instantiating concrete types instead of using Factory seams
✅ Pattern to follow: Use existing *test/ packages, or create them if missing
Isolated Test Environments (internal/testenv)
Unified XDG directory isolation with progressive options. Eliminates duplicated dir setup across config/mocks, project/mocks, and test/e2e/harness.
env := testenv.New(t) // dirs only
env := testenv.New(t, testenv.WithConfig()) // + real Config
env := testenv.New(t, testenv.WithProjectManager(nil)) // + real PM (implies Config)Higher-level helpers delegate here: configmocks.NewIsolatedTestConfig(t), projectmocks.NewTestProjectManager(t, gf), test/e2e/harness.NewIsolatedFS().
See internal/testenv/AGENTS.md for full API reference.
Directory Collision Detection (storage.ValidateDirectories)
Resolves all four XDG dirs and checks for path collisions (e.g., config and data pointing to the same path due to env var typos). Wire into app init for early detection.
Config Package Testing Guide (internal/config)
The config package exposes lightweight test doubles in internal/config/mocks/stubs.go. NewBlankConfig() and NewFromString(projectYAML, settingsYAML) return *ConfigMock (moq-generated) with every read Func field pre-wired to delegate to a real configImpl. Mutation goes through ProjectStore()/SettingsStore(), and those hand back the seam's real stores — which have no path options, so a Write() through them fails by design. Tests that mutate config use NewIsolatedTestConfig instead.
Import as:
configmocks "github.com/schmitthub/clawker/internal/config/mocks"Which helper to use
configmocks.NewBlankConfig()— default test double for consumers that don't care about specific config values. Returns*ConfigMockwith defaults.configmocks.NewFromString(projectYAML, settingsYAML)— test double with specific YAML values, NO defaults. Pass empty strings for schemas you don't care about. Returns*ConfigMock.configmocks.NewIsolatedTestConfig(t)— file-backed config (realstorage.Store) for tests that need a workingProjectStore()/SettingsStore()Set+Writeround-trip or env var overrides. ReturnsConfig.
Typical test mapping
- Defaults and typed getter behavior →
NewBlankConfig() - Specific YAML values for schema/parsing tests →
NewFromString(projectYAML, settingsYAML) - Mutation / persistence / env override tests →
NewIsolatedTestConfig(t) - Schema and node-validation errors → assert the error from
config.NewFromString(projectYAML, settingsYAML); validation runs inside the constructor
Focused commands
go test ./internal/config -v
go test ./internal/config -run TestSetProject -v
go test ./internal/config -run TestWriteProject -vPractical notes
- For tests asserting defaults/file values, clear
CLAWKER_*environment overrides first.
Project Package Test Doubles (internal/project/mocks/)
The project package exposes scenario-oriented doubles so dependents can choose the minimum coupling needed.
1) Pure mock manager (no config/git reads or writes)
Use projectmocks.NewMockProjectManager() when you only need interface-level behavior in unit tests.
import projectmocks "github.com/schmitthub/clawker/internal/project/mocks"
mgr := projectmocks.NewMockProjectManager()
mgr.GetFunc = func(_ context.Context, root string) (project.Project, error) {
return projectmocks.NewMockProject("demo", root), nil
}2) Mock project with identity
Use projectmocks.NewMockProject(name, repoPath) when your test needs a project with read accessors populated. Mutation methods return zero values.
proj := projectmocks.NewMockProject("demo", "/tmp/demo")
require.Equal(t, "demo", proj.Name())
require.Equal(t, "/tmp/demo", proj.RepoPath())3) Isolated writable config + real PM
Use projectmocks.NewTestProjectManager(t, gitFactory) when tests must exercise real Register/Remove/List round-trips with file-backed config.
- Backed by
testenv.New(t, testenv.WithProjectManager(gitFactory)) - Pass
nilfor gitFactory if worktree operations aren't needed
pm := projectmocks.NewTestProjectManager(t, nil)
_, err := pm.Register(context.Background(), "Demo", t.TempDir())
require.NoError(t, err)
entries, err := pm.List(context.Background())
require.NoError(t, err)
require.Len(t, entries, 1)
require.Equal(t, "Demo", entries[0].Name)Command Test Tiers
Command testing breaks into three tiers, each with distinct purpose and setup cost:
┌───────────────────┬────────────────────────────┬──────────────────────────────┐
│ TIER 1 │ TIER 2 │ TIER 3 │
│ Flag Parsing │ Integration │ Internal Function │
│ │ (Full Pipeline) │ (Direct Unit Tests) │
├───────────────────┼────────────────────────────┼──────────────────────────────┤
│ runF trapdoor │ nil runF → real execution │ Call domain function │
│ Intercepts opts │ Mock Docker client │ No Factory, no Cobra │
│ No run function │ IOStreams capture │ Just inputs → outputs │
│ No Docker mocks │ │ │
├───────────────────┼────────────────────────────┼──────────────────────────────┤
│ Tests that │ Tests that │ Tests that │
│ flags → Options │ flags + Docker → output │ inputs → Docker calls → │
│ mapping works │ works end-to-end │ results work correctly │
└───────────────────┴────────────────────────────┴──────────────────────────────┘Test File Organization
Each command verb has a single co-located test file:
cmd/<group>/<verb>/
├── <verb>.go # Options + NewCmd + run function
└── <verb>_test.go # All three tiers in one fileWithin the test file:
- Top:
runCommandhelper (if Tier 2 tests exist) - Middle: Tier 1 + Tier 2 test functions (named
TestVerb_*) - Bottom: Tier 3 table-driven tests (named
Test_verbLogic)
runF Trapdoor Pattern (Tier 1 Only)
The runF parameter on every NewCmd constructor intercepts the Options struct before the run function executes. Use this for Tier 1 (flag parsing) tests only — it validates that CLI flags map correctly to Options fields without executing any business logic. For full pipeline testing, use the Cobra+Factory Pattern below.
Tier 1 — Flag Capture Test
Intercepts the Options struct before the run function executes. Verifies CLI flags map correctly to Options fields.
func TestNewCmdStop_FlagParsing(t *testing.T) {
tio, _, _, _ := iostreams.Test()
f := &cmdutil.Factory{
IOStreams: tio,
Config: func() (config.Config, error) {
return configmocks.NewBlankConfig(), nil
},
}
var gotOpts *StopOptions
cmd := NewCmdStop(f, func(_ context.Context, opts *StopOptions) error {
gotOpts = opts
return nil
})
cmd.SetArgs([]string{"--force", "clawker.myapp.dev"})
cmd.SetIn(&bytes.Buffer{})
cmd.SetOut(&bytes.Buffer{})
cmd.SetErr(&bytes.Buffer{})
err := cmd.Execute()
require.NoError(t, err)
assert.True(t, gotOpts.Force)
assert.Equal(t, []string{"clawker.myapp.dev"}, gotOpts.Names)What this tests: flag registration, defaults, enum validation, mutual exclusion, required args, positional arg mapping.
What this does NOT test: Docker calls, output formatting, error handling in the run function.
Factory needs: minimal — often just IOStreams. Add Config if the command uses --agent flag or accesses project config in RunE.
Hybrid Injection Test
Uses runF to inject Pattern B deps while still calling the real run function:
cmd := NewCmdBuild(f, func(ctx context.Context, opts *BuildOptions) error {
opts.Builder = &mockBuilder{} // inject Pattern B dep
return buildRun(ctx, opts) // still calls real function
})This bypasses the nil-guard's real construction path while exercising the full run function logic.
Shared Test Helper Pattern
For Tier 2 (full pipeline) tests, define a private runCommand helper per command test file:
func runCommand(mockClient *docker.Client, isTTY bool, cli string) (*testCmdOut, error) {
tio, _, _, _ := iostreams.Test()
tio.SetStdoutTTY(isTTY)
tio.SetStdinTTY(isTTY)
tio.SetStderrTTY(isTTY)
factory := &cmdutil.Factory{
IOStreams: tio,
Client: func(_ context.Context) (*docker.Client, error) {
return mockClient, nil
},
Config: func() (config.Config, error) {
return configmocks.NewBlankConfig(), nil
},
}
cmd := NewCmdStop(factory, nil) // nil runF → full execution
argv, _ := shlex.Split(cli)
cmd.SetArgs(argv)
cmd.SetIn(&bytes.Buffer{})
cmd.SetOut(io.Discard)
cmd.SetErr(io.Discard)
_, err := cmd.ExecuteC()
return &testCmdOut{
OutBuf: out,
ErrBuf: errOut,
}, err
}Key design choices:
nilforrunF→ real run function executes the full pipeline- I/O capture via
iostreams.Test()→ accesstio,in,out,errOut - TTY toggling — tests both interactive and non-interactive paths
- Cobra output discarded (
io.Discard) — real output goes throughiostreams - Docker mock client injected via Factory closure
Cobra+Factory Pattern (Recommended for Command Tests)
The canonical pattern for Tier 2 (integration) command tests. Exercises the full CLI pipeline — cobra lifecycle, real flag parsing, real run function, and real docker-layer code through whail jail — without a Docker daemon.
When to Use
- Testing commands end-to-end without Docker daemon
- Verifying Docker API calls, output formatting, error handling
- All command tests (gomock fully removed from codebase)
How It Works
NewCmd(f, nil) — passing nil as runF means the real run function executes. The Factory is populated with faked closures that return test doubles:
// testFactory constructs a minimal *cmdutil.Factory for command-level testing.
func testFactory(t *testing.T, fake *mocks.FakeClient, sbmock *sockebridgemocks.MockManager) (*cmdutil.Factory, *iostreams.IOStreams) {
t.Helper()
tio, _, _, _ := iostreams.Test()
f := &cmdutil.Factory{
IOStreams: tio,
Client: func(_ context.Context) (*docker.Client, error) {
return fake.Client, nil
},
Config: func() (config.Config, error) {
return configmocks.NewBlankConfig(), nil
},
}
if sbmock != nil {
f.SocketBridge = func() socketbridge.SocketBridgeManager {
return sbmock
}
}
return f, tio
}Test Structure
func TestRunRun(t *testing.T) {
t.Run("detached mode prints container ID", func(t *testing.T) {
fake := mocks.NewFakeClient(configmocks.NewBlankConfig())
fake.SetupContainerCreate()
fake.SetupContainerStart()
f, tio := testFactory(t, fake)
cmd := NewCmdRun(f, nil) // nil runF → real run function
cmd.SetArgs([]string{"--detach", "alpine"})
cmd.SetIn(&bytes.Buffer{})
cmd.SetOut(out)
cmd.SetErr(errOut)
err := cmd.Execute()
require.NoError(t, err)
// Assert output
out := out.String()
require.Contains(t, out, "sha256:fakec")
// Assert Docker calls
fake.AssertCalled(t, "ContainerCreate")
fake.AssertCalled(t, "ContainerStart")
})
t.Run("container create failure returns error", func(t *testing.T) {
fake := mocks.NewFakeClient(configmocks.NewBlankConfig())
fake.FakeAPI.ContainerCreateFn = func(_ context.Context, _ moby.ContainerCreateOptions) (moby.ContainerCreateResult, error) {
return moby.ContainerCreateResult{}, fmt.Errorf("disk full")
}
f, tio := testFactory(t, fake)
cmd := NewCmdRun(f, nil)
cmd.SetArgs([]string{"--detach", "alpine"})
cmd.SetIn(&bytes.Buffer{})
cmd.SetOut(out)
cmd.SetErr(errOut)
err := cmd.Execute()
require.Error(t, err)
fake.AssertNotCalled(t, "ContainerStart")
})
}Why This Replaces FakeCli
The cobra+Factory pattern exercises the same pipeline FakeCli would test:
- Cobra lifecycle:
PersistentPreRunE→RunEchain runs naturally viacmd.Execute() - Real flag parsing: cobra parses flags,
Changed()works, mutual exclusion enforced - Real run function:
nilrunF meansrunRun(or equivalent) executes with all its logic - Real docker-layer code:
mocks.FakeClientcomposes throughwhail.Enginejail — label filtering, name generation, and middleware all run real code
FakeCli would only add: (1) command routing tests (cobra's responsibility) and (2) PersistentPreRunE chain tests (simple/stable). Neither justifies the maintenance cost of a CLI test shell.
Key Points
testFactoryis per-package — each command package creates its own suited to its dependencies- Reference implementation:
internal/cmd/container/run/run_test.go(TestRunRun) - Factory fields must include all closures the command's run function calls (Config, Client, etc.)
- Use
t.TempDir()forWorkDirto avoidos.Getwd()issues in tests
Which Tier to Use
┌───────────────────────────────┬─────────┬──────────────────┬─────────┐
│ What you're testing │ Tier 1 │ Tier 2 │ Tier 3 │
│ │ runF │ Cobra+Factory │ Direct │
├───────────────────────────────┼─────────┼──────────────────┼─────────┤
│ Flag default values │ ✓ │ │ │
│ Flag enum validation │ ✓ │ │ │
│ Mutual flag exclusion │ ✓ │ │ │
│ Required/positional args │ ✓ │ │ │
│ TTY vs non-TTY output │ │ ✓ │ │
│ Docker API call parameters │ │ ✓ │ ✓ │
│ Output formatting │ │ ✓ │ │
│ Error messages to user │ │ ✓ │ │
│ Container naming logic │ │ ✓ │ ✓ │
│ Data transformation │ │ │ ✓ │
│ Edge cases in domain logic │ │ │ ✓ │
└───────────────────────────────┴─────────┴──────────────────┴─────────┘Tier 2 uses the Cobra+Factory pattern — see section above for full details and templates.
E2E Container Testing via Harness
E2E tests run containers through the full CLI pipeline via Harness.Run, RunInContainer, ExecInContainer, and ExecInContainerAsRoot. These methods create a fresh Factory for each invocation, mirroring a real CLI process.
// Run a command inside a fresh container (auto-removed)
res := h.RunInContainer("dev", "curl", "-s", "https://api.anthropic.com")
require.NoError(t, res.Err, "stderr: %s", res.Stderr)
// Exec inside an existing container as the container user
res := h.ExecInContainer("dev", "cat", "/etc/resolv.conf")
// Exec as root
res := h.ExecInContainerAsRoot("dev", "cat", "/proc/self/cgroup")
// Run arbitrary CLI commands
res := h.Run("firewall", "status", "--json")
require.Equal(t, 0, res.ExitCode)Each RunResult provides ExitCode, Err, Stdout, Stderr, and the Factory used for that invocation.
Factory Testing
E2E Harness Factory (test/e2e/harness/factory.go)
harness.NewFactory(t, opts) constructs a *cmdutil.Factory with lazy singletons. All nouns share a single Config and Logger instance. Nil FactoryOptions fields use test fakes; set real constructors for integration tests.
// Returns: factory, inBuf, outBuf, errBuf
f, _, out, errOut := harness.NewFactory(t, &harness.FactoryOptions{
Config: config.NewConfig, // real config
Client: docker.NewClient, // real Docker client
ProjectManager: project.NewProjectManager,
})FactoryOptions fields (nil = test fake):
| Field | Signature | Default |
Config | func(...config.NewConfigOption) (config.Config, error) | configmocks.NewBlankConfig() |
Client | func(ctx, cfg, log, ...docker.ClientOption) (*docker.Client, error) | mocks.FakeClient |
ProjectManager | func(*logger.Logger, project.GitManagerFactory, string, project.Registry) (project.ProjectManager, error) | nil (no-op) |
GitManager | func(string) (*git.GitManager, error) | nil (no-op) |
HostProxy | func(cfg, log) (*hostproxy.Manager, error) | hostproxytest.MockManager |
SocketBridge | func(cfg, log) socketbridge.SocketBridgeManager | nil (no-op) |
UseRealAdminClient | bool | false (wires no-op AdminServiceClientMock) |
ControlPlane | func(ctx) (cpmanager.Manager, error) | nil (wires no-op ManagerMock) |
CLIState and HttpClient have no FactoryOptions field — the harness mirrors the real factory: state.New() resolves under the test's isolated XDG dirs, and HttpClient is the stdlib client.
Per-package testFactory Pattern
For command unit tests with fake Docker (no test/e2e/harness dependency):
func testFactory(t *testing.T, fake *mocks.FakeClient) (*cmdutil.Factory, *iostreams.IOStreams) {
tio, _, _, _ := iostreams.Test()
return &cmdutil.Factory{
IOStreams: tio,
Client: func(_ context.Context) (*docker.Client, error) {
return fake.Client, nil
},
// ... other fields
}, tio
}Golden File Testing
Golden file tests use GOLDEN_UPDATE=1 to regenerate expected outputs:
GOLDEN_UPDATE=1 go test ./pkg/whail/whailtest/... -run TestSeedRecordedScenarios -v # JSON testdata
make storage-golden # Storage merge golden (interactive)Docker Test Fakes (docker/mocks/)
FakeClient Architecture
mocks.FakeClient wraps a real *docker.Client backed by whailtest.FakeAPIClient:
type FakeClient struct {
Client *docker.Client // Real client to inject
FakeAPI *whailtest.FakeAPIClient // Function-field fake
Cfg config.Config // Config used for label keys
}Why this beats mocking: Docker-layer methods (ListContainers, FindContainerByAgent) run real code through whail's label-filtering jail. Tests catch actual integration bugs, not mock behavior.
Creating FakeClient
// Config is required (provides label keys for engine options)
fake := mocks.NewFakeClient(configmocks.NewBlankConfig())
// With specific config values
fake := mocks.NewFakeClient(cfg)Setup Helpers Reference
| Method | Signature | Purpose |
SetupContainerList | (containers ...container.Summary) | Configure container list results |
SetupFindContainer | (id string, fixture container.Summary) | Setup name-based lookup |
SetupImageExists | (exists bool) | Image existence check |
SetupImageTag | () | ImageTag success |
SetupImageList | (images ...whail.ImageSummary) | Image enumeration |
SetupContainerCreate | () | Container creation success |
SetupContainerStart | () | Container start success |
SetupVolumeExists | (name string, exists bool) | Volume lookup (empty name = wildcard) |
SetupNetworkExists | (name string, exists bool) | Network lookup (empty name = wildcard) |
SetupBuildKit | () *BuildKitCapture | BuildKit builder with capture |
BuildKit Testing
capture := fake.SetupBuildKit()
// Run code that builds images...
err := myBuildFunc(ctx, fake.Client, opts)
// Assert BuildKit was invoked correctly
assert.Equal(t, 1, capture.CallCount)
assert.Equal(t, "myimage:v1", capture.Opts.Tag)Fixtures Reference
| Function | Signature | Returns |
ContainerFixture | (project, agent, image string) | Exited container with clawker labels |
RunningContainerFixture | (project, agent string) | Running container, node:20-slim |
MinimalCreateOpts | () | Minimal ContainerCreateOptions |
MinimalStartOpts | (id string) | Minimal ContainerStartOptions |
ImageSummaryFixture | (ref string) | Image for list results |
BuildKitBuildOpts | () | BuildKit-enabled build options |
Error Simulation
// Simulate "not found" errors (satisfies errdefs.IsNotFound)
// Note: notFoundError is unexported — configure via SetupFindContainer(id, fixture) for not-found behavior
fake.FakeAPI.ImageInspectFn = func(ctx context.Context, ref string, opts ...client.ImageInspectOption) (types.ImageInspect, error) {
return types.ImageInspect{}, fmt.Errorf("image %s not found", ref)
}
// For proper not-found behavior, use SetupImageExists(false) which returns errdefs-compatible error
fake.SetupImageExists(false)Assertions
fake.AssertCalled(t, "ContainerCreate") // Method was called
fake.AssertNotCalled(t, "ImageBuild") // Method was not called
fake.AssertCalledN(t, "ContainerStart", 2) // Called exactly N times
fake.Reset() // Clear call logIn-Memory Test Utilities
The codebase provides in-memory implementations for testing without filesystem I/O.
config Test Stubs (mocks/stubs.go)
Config test helpers live in internal/config/mocks/:
import configmocks "github.com/schmitthub/clawker/internal/config/mocks"
// Default test double — in-memory *ConfigMock with defaults, Set/Write/Watch panic
cfg := configmocks.NewBlankConfig()
// Test double from YAML — in-memory *ConfigMock, Set/Write/Watch panic
cfg := configmocks.NewFromString(`build: { image: "alpine:3.20" }`, "")
// File-backed config for mutation tests
cfg := configmocks.NewIsolatedTestConfig(t)Factory wiring in tests:
f := &cmdutil.Factory{
IOStreams: tio,
Config: func() (config.Config, error) {
return configmocks.NewBlankConfig(), nil
},
}gittest.NewInMemoryGitManager
Creates an in-memory GitManager for testing git operations without touching the filesystem.
gitMgr := gittest.NewInMemoryGitManager()
// Currently provides Repository() method for go-git repo accessWhen to Use Which
| Need | Use |
| Default config for command tests | configmocks.NewBlankConfig() |
| Config with specific YAML values | configmocks.NewFromString(projectYAML, settingsYAML) |
Config needing a real store Set+Write round-trip | configmocks.NewIsolatedTestConfig(t) |
| Test git operations without filesystem | gittest.NewInMemoryGitManager() |
E2E Integration Test Patterns (test/e2e/)
Basic E2E Test
Uses the test/e2e/harness package with real constructors. NewIsolatedFS refuses to run while live agent containers exist. It creates isolated XDG dirs, links the real ~/.docker/cli-plugins into the isolated Docker config (so docker compose stays available), and registers cleanup. It does not build a clawker binary. TestMain in test/e2e/main_test.go makes the test binary act as the clawker CLI for host-proxy serve, because the host proxy manager starts its daemon from its own executable.
package e2e
func TestMyFeature(t *testing.T) {
h := &harness.Harness{
T: t,
Opts: &harness.FactoryOptions{
Config: config.NewConfig,
Client: docker.NewClient,
ProjectManager: project.NewProjectManager,
},
}
setup := h.NewIsolatedFS(nil)
// Write project config to isolated env
setup.WriteYAML(t, testenv.ProjectConfig, setup.ProjectDir, `
build:
image: "buildpack-deps:bookworm-scm"
`)
// Register project and build
regRes := h.Run("project", "register", "testproject")
require.NoError(t, regRes.Err, "register: %s", regRes.Stderr)
buildRes := h.Run("build")
require.NoError(t, buildRes.Err, "build: %s", buildRes.Stderr)
// Run container and assert
res := h.RunInContainer("dev", "echo", "hello")
require.NoError(t, res.Err, "run: %s", res.Stderr)
assert.Contains(t, res.Stdout, "hello")
}Firewall E2E Test Pattern
test/e2e/firewall_test.go keeps the real firewall wiring in one place. Copy these names exactly:
// firewallFactoryOpts: real Config/Docker/ProjectManager, production
// ControlPlane manager, production-identical AdminClient.
func firewallFactoryOpts(o *harness.FactoryOptions) {
o.Config = config.NewConfig
o.Client = docker.NewClient
o.ProjectManager = project.NewProjectManager
o.UseRealControlPlane = true
o.UseRealAdminClient = true
}
// fwSetup: isolated FS, CP removal, project config write, register, build.
func fwSetup(t *testing.T, h *harness.Harness, projectYAML string) *harness.SetupResult {
t.Helper()
setup := h.NewIsolatedFS(nil)
// The test CP must use this test environment's CA.
harness.EnsureNoControlPlane(t, 30*time.Second)
setup.WriteYAML(t, testenv.ProjectConfig, setup.ProjectDir, projectYAML)
regRes := h.Run("project", "register", "testproject")
require.NoError(t, regRes.Err, "register failed\nstdout: %s\nstderr: %s", regRes.Stdout, regRes.Stderr)
buildRes := h.Run("build")
require.NoError(t, buildRes.Err, "build failed\nstdout: %s\nstderr: %s", buildRes.Stdout, buildRes.Stderr)
return setup
}
func TestFirewall_BlockedDomain(t *testing.T) {
h := newFirewallHarness(t)
res := h.RunInContainer("firewall-test", "curl", "-s", "--max-time", "5", "https://example.com")
assert.NotNil(t, res.Err, "curl to blocked domain should fail")
}newFirewallHarness(t, requiredServices...)callsnewFirewallYAMLHarness(t, "\n", requiredServices...). UsenewFirewallYAMLHarnessfor a custom project config.newFirewallYAMLHarnesscallsharness.New(t, firewallFactoryOpts), registersh.RequireServicesWereRunningbeforefwSetup(cleanup is LIFO), then callsfwSetup.requiredServicesdefaults to"firewall", "controlplane". A test that stops the firewall gives only"controlplane". A test that also stops the CP givesfwNoServices.- A test that needs more wiring (for example, the host proxy) gives an additional mutator to
harness.New.
Table-Driven E2E Tests
func TestRunIntegration_ArbitraryCommand(t *testing.T) {
tests := []struct {
name string
args []string
wantOutput string
wantErr bool
}{
{
name: "echo command",
args: []string{"container", "run", "--rm", "--agent", "test", "@", "echo", "hello"},
wantOutput: "hello",
},
{
name: "command not found",
args: []string{"container", "run", "--rm", "--agent", "test", "@", "notacommand"},
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// ... harness setup + h.Run(tt.args...)
})
}
}Testing Both Invocation Patterns
All container commands should test BOTH flag-based and positional patterns:
func TestStopIntegration_BothPatterns(t *testing.T) {
tests := []struct {
name string
useFlag bool
}{
{"with agent flag", true},
{"with container name", false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var args []string
if tt.useFlag {
args = []string{"container", "stop", "--agent", "dev"}
} else {
args = []string{"container", "stop", "clawker.testproject.dev"}
}
// Execute and verify...
})
}
}Error Handling Examples
// BAD - silent failure
_, _ = client.ContainerRemove(ctx, id, true)
// GOOD - collect and report errors
if _, err := client.ContainerRemove(ctx, id, true); err != nil {
t.Logf("WARNING: cleanup failed: %v", err)
}
// BETTER - collect all errors
var errs []error
for _, c := range containers {
if _, err := client.ContainerRemove(ctx, c.ID, true); err != nil {
errs = append(errs, fmt.Errorf("remove %s: %w", c.ID[:12], err))
}
}
if len(errs) > 0 {
return errors.Join(errs...)
}Storage Oracle + Golden Test Strategy
The internal/storage package uses a defense-in-depth approach with two independent guards for merge correctness:
| Layer | How it works | What it catches |
| Oracle (randomized) | Computes expected merge from spec rules (~15 lines), independent of prod code. Runs every time with a new seed. | Any merge bug that manifests for the random placement |
| Golden (fixed seed) | Hardcoded struct literal blessed from known-correct state. No auto-update. | Any regression from the blessed baseline, including oracle bugs |
Key Design Decisions
| Decision | Rationale |
Deepest level forced to have both config.local.yaml and config.yaml | Guarantees filename priority is always exercised |
Main/local files have distinct names (level3-main vs level3-local) | Scalar assertions can distinguish which file won |
| Golden values are code, not files | Must be hand-edited to change — no accidental GOLDEN_UPDATE=1 sweep |
make storage-golden prints new values with interactive confirmation | Blocks CI — human must review and approve |
STORAGE_GOLDEN_BLESS env var is specific to this one test | No global sweep risk |
When to Use This Pattern
The oracle+golden dual-guard pattern is appropriate when:
- The system under test has combinatorial inputs (merge order, file placement, priority rules)
- A single deterministic test cannot cover the full space
- Regression protection and exploration serve different purposes
go test ./internal/storage -v # Runs both oracle + golden
go test ./internal/storage -run TestStore_WalkUpLayerMerge -v # Oracle only (randomized)
go test ./internal/storage -run TestStore_WalkUpGolden -v # Golden only (fixed baseline)
make storage-golden # Interactive golden updateCross-Phase Learnings (Testing Initiative)
Battle-tested insights from the multi-phase testing initiative (Phases 1-4a):
Moby API Quirks
client.Filtersismap[string]map[string]bool— label entries stored as"key=value": trueunder"label"keyFilters.Add()returns a new Filters (immutable) — must capture return valueContainerWaitreturnsContainerWaitResult(no error), containing Result/Error channelsImageInspectuses variadic options:ImageInspect(ctx, ref, ...ImageInspectOption)container.Summary.Stateiscontainer.ContainerState(string typedef) —assert.Equalrequiresstring()cast- Docker names always have leading
/in API responses ContainerInspectResultwraps response — labels atinspect.Container.Config.LabelsVolumeListAlldelegates toVolumeList— both returnVolumeListResultwith.ItemsFirewallSettings.Enableis*bool(settings.yaml global switch);FirewallConfig(clawker.yaml per-project) has noEnablefield;SecurityConfig.EnableHostProxyis*bool
FakeAPIClient Pattern
- Embeds nil
*client.Clientfor unexported moby interface methods — unoverridden methods panic (fail-loud) - Module path:
github.com/schmitthub/clawker - Container labels at
InspectResponse.Config.Labels, volume atVolume.Labels, network atNetwork.Labels, image atInspectResponse.Config.Labels(OCI ImageConfig)
docker/mocks (Composite Fake) Pattern
- Must use
dev.clawkerlabel prefix (notcom.whailtest) — docker-layer methods likeListContainerscallClawkerFilter()which filters bydev.clawker.managed; using test labels would cause zero results docker.Client.ImageExistscallsc.APIClient.ImageInspectdirectly (bypasses whail Engine jail) — theerrNotFoundtype must satisfyerrdefs.IsNotFoundviaNotFound()methodFindContainerByNameusesContainerList+ContainerInspect—SetupFindContainermust configure both Fn fields- No import cycles:
internal/docker/mocks->internal/docker+pkg/whail+pkg/whail/whailtestis clean
Cobra+Factory Test Pattern
NewCmd(f, nil)with faked Factory closures exercises full CLI pipeline (cobra lifecycle, real flag parsing, real run function, real docker-layer code through whail jail)- Default
NewFakeClientvolume/network inspect defaults sufficient forEnsureConfigVolumesandEnsureNetworkflows workspace.SetupMountshasWorkDirfield (empty-string fallback toos.Getwd()for backward compat)*copts.ContainerOptionsanonymous embedding promotes fields — must keep embedding syntax inreplace_symbol_body
Whail Jail Testing
jail_test.gomust usepackage whail_test(external) to avoid import cycle withwhailtest- Integration tests in
test/whail/usepackage whail_test(external) - Label override prevention: add
labels[e.managedLabelKey] = e.managedLabelValueAFTER final label merge (caller labels have highest precedence) ContainerStatsOneShotdelegates toAPIClient.ContainerStats— spy on"ContainerStats"not"ContainerStatsOneShot"
Context Window Management
- For multi-task initiatives across conversations, use the initiative skill to plan and complete one task at a time with a handoff prompt.
- Each task gets a fresh context window with self-contained handoff prompt providing all needed context
Pure Function Testing (internal/docker)
matchPatternhad a bug:**/*.extdidn't work (literal HasSuffix). Fixed withfilepath.Matchagainst basename when suffix contains wildcardsisNotFoundErrorchecks bothwhail.DockerError(viaerrors.As) and raw error stringsLoadIgnorePatternsreturns[]string{}(not nil) on file-not-found
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

