clawker / clawkerd
AGENTS.md59 starsChanged 12 days agoMITView on GitHub
Read the Control-plane safety section of controlplane/AGENTS.md before changing this package. Its failure rules (no panic, log.Fatal, or os.Exit on the boot or serve path; every long-lived goroutine recovers; subsystems degrade with event=<subsystem>_unavailable) apply here. Per-container agent daemon AND PID 1 init of the agent container. Owns: clawkerd runs as root inside the container. It does NOT drop its own privileges — privilege drop happens kernel-side in the spawned child between fork() and execve(). The supervisor stays root for: log…
Where files are headingNew AGENTS.md files have more rules about keeping secrets out of the code, and fewer about code style.
Of the AGENTS.md files we read, the 32,671 first seen in the last six months, against the older ones: the kind of rule that gained most share, and the kind that lost most. About files like this in general, not this one, and an estimate, since a kind is a pattern in the words.
In this repository17 other files it gives its agents.
- AGENTS.mdAGENTS.md
- controlplane/AGENTS.mdAGENTS.md
- docs/AGENTS.mdAGENTS.md
- test/AGENTS.mdAGENTS.md
- .github/copilot-instructions.mdCopilot instructions
- agent-filesSkill
- audit-memorySkill
- cli-outputSkill
- dev-checksSkill
- firewall-uatSkill
- initiativeSkill
- monitoring-checksSkill
- store-editorSkill
- test-hunterSkill
- wizard-commandsSkill
- writing-testsSkill
- code-reviewSkill
The file
We split this AGENTS.md into the rules it gives an agent
10 rules. About 0 are common, 2 are distinctive, and 8 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
8
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 3
Read the Control-plane safety section of controlplane/AGENTS.md before changing this package.
- Very specific: line 27
Read four bootstrap files from consts.BootstrapDir (/run/clawker/bootstrap):
- Very specific: line 36
Start listener on consts.DefaultClawkerdPort (:7700).
- Very specific: line 44
Wait for ctx.Done (SIGTERM/SIGINT), spawn.MainExited, listenerFatalCh, or cmdExitCh (command-requested exit):
- Very specific: line 95
Every ShellCommand dispatch emits two structured Info events:
- Very specific: line 123
No register gate: a restarted container skips Register (registry hit at Hello), so "wait for RegisterDone" would block forever.
- Distinctive: line 144
No proactive outbound dial beyond the CP-triggered Register handshake, the ExportTelemetry connection the OTLP relay holds, and the …
- Distinctive: line 145
No heartbeat — CP knows liveness via Docker events + dialer overseer events
- Very specific: line 146
No init-script execution — CP-driven Session.ShellCommand runs the post-init plan.
- Very specific: line 147
No reconnect logic — clawkerd is the SERVER.
≈ “Common” is an estimate: at least 1 in 5 of the 41,581 AGENTS.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
clawkerd
Read the Control-plane safety section of controlplane/AGENTS.md before changing this package. Its failure rules (no panic, log.Fatal, or os.Exit on the boot or serve path; every long-lived goroutine recovers; subsystems degrade with event=<subsystem>_unavailable) apply here.
Per-container agent daemon AND PID 1 init of the agent container. Owns:
- CP-dialed mTLS listener on
:7700(ClawkerdService.Session) — command dispatch surface for the entire container lifetime. - User CMD supervision — forks the container's user CMD (default
claude) as PID 1's only direct child on CP-dispatchedAgentReady, with privilege drop viaSysProcAttr.Credential. Forwards signals to the child's process group, reaps zombies (including reparented orphans), exits with the child's bash-convention exit code.
clawkerd runs as root inside the container. It does NOT drop its own privileges — privilege drop happens kernel-side in the spawned child between fork() and execve(). The supervisor stays root for: log rotation, /run/clawker/bootstrap reads, Wait4(-1, WNOHANG) orphan drain, and future privileged operations (eBPF coordination, cgroup writes).
Package Layout
Mirrors the controlplane binary split (thin cmd → internal entrypoint → daemon package):
clawkerd/(packageclawkerd) — the daemon code: listener, session, spawn, register, progress, user, recover, bootstrap (this package).Main/runimport it for every subsystem.clawkerd/embed/(packageclawkerdembed) —//go:embed assets/clawkerdexposes the compiled binary asBinaryforinternal/bundler. Lives in a separate package because the daemon binary is built FROM the code that this package would otherwise have to compile-then-embed (a build cycle) — same reasoncontrolplane/managerholds the clawkercp embed, notinternal/controlplane.internal/clawkerd/cmd.go(packageclawkerd) —Main()wrapsrun(ctx, log) (int, error); the supervisor orchestration. Importsclawkerdasdaemon.cmd/clawkerd/clawkerd.go(packagemain) —os.Exit(clawkerd.Main()), nothing else.
Role
CP is the host daemon; clawkerd is the per-container daemon. They communicate over the per-container gRPC listener on clawker-net (CP-dialed). The Session bidi-stream is the command dispatch channel. clawkerd has three outbound paths to CP's AgentService, all over its mTLS leaf: the CP-triggered Register handshake that writes the identity row (bearer-gated, one-time), the AgentService.ExportTelemetry connection the OTLP relay keeps open for the process lifetime, and the host action connection the host relay keeps open (both public scope, no bearer). Otherwise clawkerd only serves.
Boot Sequence
Read four bootstrap files from
consts.BootstrapDir(/run/clawker/bootstrap):cert.pem,key.pem— per-agent mTLS leaf (dual EKU:ClientAuth+ServerAuth)ca.pem— CLI CA cert (clawkerd's RootCA for verifying CP's client cert)assertion.jwt— CLI-signedclawker-agentHydraclient_assertionJWT; single-use, exchanged at Hydra for an access token on CP-dispatchedRegisterRequired, then used to callAgentService.Register(seeregister.go)
Files live in the container's writable layer (not tmpfs, not bind mount). The CLI streams a tar archive via
CopyToContainerbetweendocker createanddocker start. tmpfs can't be pre-populated this way (mounted at start, shadows prior writes). Permissions: parent dir 0700, files 0400. SeeWriteAgentBootstrapToContainerininternal/cmd/container/shared/agent_bootstrap.gofor the full tradeoff.- Resolve env:
CLAWKER_AGENT(required),CLAWKER_PROJECT(empty = global-scope agent, 2-segment naming),CLAWKER_USER(defaults toconsts.ContainerUser).CLAWKER_USERis resolved against/etc/passwdto fillSysProcAttr.Credentialfor the spawn child. Start listener on
consts.DefaultClawkerdPort(:7700). This is the entire inbound RPC surface.Then start the OTLP relay (
otlp_relay.go,StartTelemetryRelay) onconsts.AgentOTLPRelayAddr(127.0.0.1:4318). It degrades to nil withevent=otlp_relay_unavailablewhenCLAWKER_CP_AGENT_ADDRis unset, the leaf does not parse, or the bind fails; the daemon continues without telemetry.Then start the host relay (
host_relay.go,StartHostRelay) onconsts.AgentHostRelayAddr(127.0.0.1:18374). Same degrade contract (event=host_relay_unavailable); the daemon continues without host actions.- Build
spawnState(inspawn.go+spawn_unix.go) and aspawnEntryclosure that captures the resolvedExecUser+ argv. Thread the closure throughStartClawkerdListener→clawkerdServer→runSession→sessionas a constructor argument so a wiring bug fails loud at construction (nil-thunk rejected atStartClawkerdListener) rather than via a package-level mutable global. The entrypoint (internal/clawkerd/cmd.go) also passes thespawnState(spawn) down the same chain as thestate agentState(used for HelloAck andAgentInitialized); unlikespawnEntry/onFatal/requestExit,stateis intentionally nil-tolerant and is NOT rejected atStartClawkerdListener(test fixtures leave it unset). Spawn does NOT fire here —handleAgentReadyinvokes the closure when CP dispatchesAgentReadyas the terminal step of CP-driven boot. - Wait for
ctx.Done(SIGTERM/SIGINT),spawn.MainExited,listenerFatalCh, orcmdExitCh(command-requested exit):- On
ctx.Done:spawn.Stop(10s)forwards SIGTERM to the child pgroup, escalates to SIGKILL after grace, then blocks onMainExited. - On
spawn.MainExited: phase 1 reaped the user CMD; main proceeds to teardown. - On
cmdExitCh: a dispatchedShellCommandcarryingexit_on_non_zeroexited non-zero (or was signaled — clamped to a generic non-zero). The session signals therequestExitthunk with the mirrored code; main records it (exitRequested+requestedExitCode), tears down (the pre-spawn case has no child), and exits PID 1 with that code so the failing command's status surfaces to the user's terminal. This is the abort case that lets a fatal init step exit the container cleanly instead of parking forever with no child. Generic to the CP→clawkerd command service — init is its first consumer.
- On
- Teardown order is load-bearing:
clawkerdSrv.Stop()— force-close the listener. NOTGracefulStop: the user CMD has exited and CP holds the Session bidi stream open from its side, soGracefulStopwould hang waiting for the streaming RPC handler to return. In-flightShellCommandpipelines were already drained by the main-child-exit cascade BEFORE this point, so graceful drain is not needed.spawn.BeginOrphanDrain()— releases the reaper's phase 2 (Wait4(-1, WNOHANG)) so it can drain reparented orphans without racingsession.go'sc.Waitfor stage children.os.Exit(spawn.Wait())— bash-convention exit code (128+signumfor signaled child) so Dockerrestart: on-failurereads the right value.
Spawn Lifecycle (spawn.go + spawn_unix.go)
spawnState tracks the user CMD across its lifetime. Single-shot via CAS — a second Run call returns errAlreadySpawned (mapped to Done{0} by handleAgentReady for Session-reconnect idempotency).
Goroutines installed by Run (each defer recover()-wrapped per the resilience contract):
- Signal forwarder.
signal.Notify(...forwardableSignals())→unix.Kill(-childPgid, sig). Forwardable set excludes SIGCHLD (reaper handles), SIGURG (Go runtime preemption), and program-error signals (SIGFPE/SIGILL/SIGSEGV/SIGBUS/SIGABRT/SIGTRAP/SIGSYS — let those crash the supervisor rather than masking via forward). Filtering SIGURG is non-negotiable: forwarding it interferes with Go's goroutine preemption. - Reaper. Two-phase. Phase 1:
Wait4(mainPID, &ws, WNOHANG)only — neverWait4(-1)while the main child is alive, so concurrentexec.Cmd.Waitcalls insession.go'sShellCommandpipelines aren't stolen. Phase 2:Wait4(-1, &ws, WNOHANG)to drain reparented orphans, gated onorphanDrainChso callers (main()) canStopthe listener first. - Stop watchdog. Timer-driven SIGKILL escalation if the child pgroup hasn't drained within
Stop's grace window.
Phase-2 gate (BeginOrphanDrain / MainExited): the reaper holds at <-orphanDrainCh after phase 1. main() uses MainExited to trigger Stop, then calls BeginOrphanDrain to release phase 2. Tests with no concurrent exec.Cmd.Wait surface call BeginOrphanDrain immediately after Run. The gate is held by default — a forgetful caller hangs loudly on Wait/Done rather than silently racing concurrent c.Wait surfaces.
Privilege drop. SysProcAttr.Credential{Uid, Gid, Groups} populated from ResolveUser(CLAWKER_USER, /etc/passwd, /etc/group) (wraps github.com/moby/sys/user.GetExecUser, deliberately not GetExecUserPath — the latter silently swallows file-open failures). The kernel performs setgroups → setgid → setuid (in that order — setgroups MUST run while still root, before the setuid that drops privileges) between fork and exec; see Go's syscall/exec_linux.go. clawkerd's own goroutines stay root.
Exit-code mapping. mapWaitStatus (Wait4 path) and mapExitCode (*os.ProcessState path) return WEXITSTATUS for normal exit, 128+signum for signaled. Matches bash convention so restart: on-failure works.
HEALTHCHECK. /var/run/clawker/ready is touched by touchReadyFile immediately after cmd.Start() returns nil so the healthy transition lines up with the user CMD becoming a real process.
ClawkerdService Listener (CP->clawkerd)
The :7700 inbound listener (listener.go) has three guards before any handler executes:
- mTLS, RequireAndVerifyClientCert.
ClientCAs= clawker CA bundle. Server cert is the per-agent leaf with dual EKU (ServerAuthfor CP chain verify,ClientAuthfor future agent->CP dial). - CN pin.
pinPeerCNToCP(constant-time compare) rejects any peer whose CN is notconsts.ContainerCP. Prevents agent-to-agent privilege escalation via ShellCommand. - ClientAuth EKU assertion. Defense in depth: Go's TLS already enforces this for client certs, but the app-layer assertion documents the dependency so a refactor that loosens TLS config (e.g.
VerifyClientCertIfGiven) still fails closed.
Session Audit Log (load-bearing)
runSession emits two structured Info events per Session:
event=session_started—peer_cn+peer_thumbprinton every authenticated stream openevent=session_ended—peer_cn+durationvia defer when receiver loop returns
These are the audit trail for CP-driven command dispatch. Sessions are long-lived (server-streaming, agent lifetime). Operators MUST forward clawkerd logs to durable storage for compliance retention — no other surface captures "CP opened a command channel against this container".
ShellCommand Threat Surface
ShellCommand dispatches arbitrary argv with arbitrary uid/gid inside the container, and clawkerd runs as root. The CN-pinned mTLS listener (CP = sole authorized caller) is the entire trust boundary. No per-command argv allow-list, no policy gate, no syscall sandbox. Any compromise that lets a non-CP peer mint a ContainerCP-CN cert chained to the clawker CA grants root-equivalent code execution. Per-command policy gates are a v2 concern.
ShellCommand Audit Log (load-bearing)
Every ShellCommand dispatch emits two structured Info events:
event=shell_command_started(one per pipeline stage) — fullargv,cwd,uid,gid,timeout_seconds,command_id,stage_indexevent=shell_command_done(one per command) —duration,final_exit_code,timed_out,outcomeenum (completed/spawn_failed/timeout/incomplete)
Volume: N+1 lines for an N-stage pipeline (2-3 lines typical). incomplete outcome means runShellCommand returned via an unexpected path — treat as a clawkerd bug.
Hello Handler
The Command_Hello dispatch case replies HelloAck{Initialized, CmdRunning}, both read from the state agentState seam (session.go interface: Initialized() bool, MarkInitialized(), Spawned() bool; production impl = spawnState). This makes init/boot one-shot from CP's side: CP skips the one-time init plan once Initialized is true (the writable-layer marker exists) and skips the boot/spawn plan once CmdRunning (Spawned()) is true, instead of re-dispatching on every reconnect. Nil-tolerant — test fixtures leave state unset, so both fields report false.
AgentReady Handler
handleAgentReady is the terminal step of CP-driven boot. Calls s.spawnEntry (the thunk threaded through clawkerdServer → runSession → session from main()); replies Done{0} on success, Done{0} on errAlreadySpawned (reconnect idempotency), Error{IO_ERROR} on any other spawn failure or on a nil thunk (production rejects nil at StartClawkerdListener construction). spawnState.Run's CAS is the source of truth for "already spawned".
Host Relay (agent → CP host actions)
host_relay.go serves the HTTP paths the container scripts (host-open, git-credential-clawker, callback-forwarder) call, on loopback only; CLAWKER_HOST_PROXY points at it (consts.AgentHostRelayURL, set by the CLI only when the container has the host proxy grant). Each request becomes one AgentService host action over the bootstrap leaf: POST /open/url → OpenURL, POST /git/credential → GitCredential, POST /callback/register → RegisterCallback, GET /callback/{session}/data → GetCallbackData, DELETE /callback/{session} → DeleteCallback. The relay holds no policy: the CP decides every request (see controlplane/agent/AGENTS.md). Agents cannot reach the host proxy directly.
- JSON is camelCase (
sessionId,expiresAt,authUrl,ttlSeconds,receivedAt). Bodies are capped athostRelayMaxBody(1 MiB); one forward is bounded byhostRelayTimeout(90 s). - Verdict mapping:
Unavailable/DeadlineExceeded/Canceled→503;PermissionDenied/Unauthenticated→403;InvalidArgument→400;NotFound→404;ResourceExhausted→413; else500(loggedevent=host_relay_rejected). It reuses the OTLP relay'srelayHTTPStatusand adds theNotFoundrow. A gitFailedPrecondition(host helper failed) is200 {success:false}, the answer git expects for a failed lookup. An unknown git action is400without an RPC.
OTLP Relay (agent → CP telemetry)
otlp_relay.go serves the OTLP/HTTP paths the agent SDK exports to (POST /v1/{logs,metrics,traces}, application/x-protobuf, optional gzip) on loopback only. Each body is carried unchanged in one unary AgentService.ExportTelemetry call over a grpc.ClientConn dialed with the bootstrap leaf and no bearer (ExportTelemetry is public scope: mTLS plus the CP identity interceptor plus the registry thumbprint check in the CP handler). The relay holds no identity logic and never reads the batch.
- Caps: at most
relayMaxInflight(8) requests hold a body at once; over that the SDK gets503+Retry-Afterbefore any read, so a burst cannot exhaust PID1's memory. Body atagentv1.ExportTelemetryMaxBodyBytes(4 MiB) after gzip decoding (the compressed input is not capped separately, so a batch right at the cap is not refused for gzip framing), so a small compressed body cannot expand past what the CP listener accepts; one forward is bounded byrelayExportTimeout(15 s, longer than the CP gateway's own forward timeout so a slow collector surfaces as the CP's verdict). Reads are bounded byrelayReadHeaderTimeout(5 s) andrelayReadTimeout(10 s, the whole request, and idle keep-alive connections through theIdleTimeoutfallback), so a client that stalls an upload cannot hold a PID1 goroutine and its body buffer. - Verdict mapping (
relayHTTPStatus):Unavailable/DeadlineExceeded/Canceled→503+Retry-After(the SDK's bounded retry backs off);PermissionDenied/Unauthenticated→403;InvalidArgument→400;ResourceExhausted→413; anything else →500(the Node OTLP exporter retries 429/502/503/504 only). Client faults (404,405,415,413,400) are answered before any forward. - No register gate: a restarted container skips Register (registry hit at Hello), so "wait for RegisterDone" would block forever; the CP answers
PermissionDeniedfor an unregistered peer and the user CMD only spawns afterAgentReady, which follows Register. - Monitoring is optional: with the CP gateway breaker open the CP answers success and the relay returns
200; with the CP itself away the SDK sees503and gives up after its own retry budget. Rejections log at most once per minute (event=otlp_relay_rejected,event=otlp_relay_cp_unavailable). - Shutdown runs after the child exited (
relayShutdownGraceininternal/clawkerd/cmd.go) with a context that outlives the daemon's signal context.
AgentInitialized Handler
handleAgentInitialized is the terminal step of CP-driven init (distinct from AgentReady, the terminal step of boot). Like Hello, it runs synchronously on the receive loop. It calls state.MarkInitialized() (persists the writable-layer init marker — nil-tolerant) and replies Done{0}. This is what makes init one-time: the persisted marker is what Initialized() reads at the next Hello.
Resilience Contract
clawkerd is PID 1 of the agent container. A panic that escapes a goroutine kills PID 1 → container exits → restart: on-failure may retry but if the bug is deterministic the container restart-loops with no actionable signal. This is the same resilience contract as CP (see root CLAUDE.md's "CP crashing is a SECURITY incident, not an availability one" clarification — same shape applies here).
Hard rules for code on this path:
- No
panic(). Nolog.Fatal(). Noos.Exit()outside the post-spawn.Waitexit inrun's tail and the logger-init failure inmain. Constructors return(nil, error); runtime paths log structurally and degrade. - Every long-lived goroutine recovers. Signal forwarder, reaper, stop watchdog, gRPC
Serve, session sender, runShellCommand worker, drainOutput, register handler, AgentReady handler, stage reapers — all wrap withdefer recoverGoroutine(...)(or the equivalent inline pattern for session.go's stage reapers, which need the synthetic-StageExit emission too). The goroutine's panic-recovery path also closes anychan struct{}callers select on so nobody deadlocks.recoverGoroutinelives inrecover.go(no build tag) so listener.go and session.go share it with the unix-tagged spawnState goroutines in spawn_unix.go. - Subsystem failures degrade, never cascade. A broken spawn entry →
Error{IO_ERROR}Response, supervisor stays alive. A broken Session → cancelled Session ctx, listener stays up. A broken reaper goroutine →closeAllGatesreleasesMainExited/Donewaiters so main() can still exit cleanly. (SPAWN_FAILEDis reserved forrunShellCommandstage spawn failures.) - Every degraded path emits a structured log line. Operators see
event=<name>withcommand_id,pid,pgid,erroretc. Triage works from the log surface alone.
What It Does NOT Do
- No proactive outbound dial beyond the CP-triggered Register handshake, the ExportTelemetry connection the OTLP relay holds, and the AgentService connection the host relay holds for the host actions (each a lazy dial with gRPC-managed reconnect)
- No heartbeat — CP knows liveness via Docker events + dialer overseer events
- No init-script execution — CP-driven
Session.ShellCommandruns the post-init plan; clawkerd just dispatches - No reconnect logic — clawkerd is the SERVER; reconnect with backoff lives in
controlplane/agent/dialer.goon the CP side
Files
| File | Purpose |
bootstrap.go | ReadBootstrap reads the four bootstrap files (cert/key/ca/assertion) from consts.BootstrapDir into the in-memory bootstrap struct; missing/empty files fail loudly (a partial boot is a security regression). The supervisor orchestrator (Main/run) that consumes this lives in internal/clawkerd/cmd.go |
listener.go | CP→clawkerd inbound mTLS listener. buildListenerTLSConfig enforces RequireAndVerifyClientCert + dual-EKU server cert + chain validation; pinPeerCNToCP asserts peer is ContainerCP with ClientAuth EKU |
session.go | runSession per-stream owner: receive loop, sender goroutine, dispatch, ShellCommand pipeline (multi-stage exec, stdin/stdout/stderr fanout, signal forwarding, timeout watchdog, audit log). Defines the state agentState seam (Initialized/MarkInitialized/Spawned). dispatch's Command_Hello case replies HelloAck{Initialized, CmdRunning} from state; handleAgentInitialized runs on the receive loop, calls state.MarkInitialized(), replies Done{0}. handleAgentReady invokes the spawnEntry thunk threaded through the session struct from the entrypoint (internal/clawkerd/cmd.go) (no package-level mutable global). Every stage's stderr and the final stage's stdout share one combined write end (2>&1), so a single drainOutput streams the command's combined output to the caller as OutputChunk (always, any size — no accumulation buffer or cap) and echoes it live to the boot console (via progress.WriteOutput) when print_output is set; exit_on_non_zero + a non-zero exit runs Stop (flush the terminal Response) then signals the requestExit thunk (mirrored code). Both flags are generic to the command service — clawkerd makes no policy decision, the caller sets the flags |
spawn.go | Cross-platform pure logic: mapExitCode, envForUser, routeArgs, errAlreadySpawned, errEmptyArgv |
spawn_unix.go | //go:build unix — spawnState lifecycle: Run (fork+exec with privilege drop + Setpgid + ready-file touch), Wait, Stop, MainExited, BeginOrphanDrain, signal forwarder, two-phase reaper. buildSysProcAttr builds the *syscall.SysProcAttr (Setpgid + optional Credential) — extracted so the privilege-drop wiring is unit-testable without root. |
recover.go | Resilience-contract recoverGoroutine helper: structured-log + onPanic hook for every long-lived goroutine in clawkerd (no build tag — shared by spawn_unix's reaper/forwarder/watchdog AND listener.go's Serve AND session.go's sender/worker/drainer/register handler). |
progress.go | User-facing TTY progress reporter: two plain status lines per init step (Active form, then ✓/✗ Done) plus boot/closing banners. No animation — per-step shell scripts complete in milliseconds, below the threshold animation would be perceptible. Writes to os.Stdout (the attached TTY for the agent container) via TIOCGPGRP-detected isTTY which toggles ANSI color codes and the info-icon glyph (cyan ℹ on a TTY, [info] ASCII fallback off-TTY); per-step ✓/✗ glyphs are emitted unchanged in both modes. Wired by internal/clawkerd/cmd.go → StartClawkerdListener → clawkerdServer → runSession → session. Init step boundaries hooked in session.dispatch (Command_Shell with init- prefix → StartStep) and session.runSender (terminal Done/Error → EndStep, via settleInitStep, fired only after stream.Send succeeds); handleAgentReady calls Final immediately before spawn so the subsequent spawnEntry transfers the TTY foreground to the user CMD without visual collision. WriteOutput echoes raw captured command output to the same console under the shared mutex (suppressed once stopped, so a post-spawn command can't garble the user CMD's TTY); settleInitStep reads Done.final_exit_code and passes the result to EndStep, so a non-zero exit renders the red ✗ instead of the green ✓. |
user.go | ExecUser + ResolveUser wrapping github.com/moby/sys/user.GetExecUser (passwd snapshot read once into bytes; group file via explicit path reader) for name/name:group/uid/uid:gid spec parsing |
register.go | CP-triggered Register handshake: Hydra token exchange + AgentService.Register mTLS dial (buildDialTLSConfig is shared with the relay) |
host_relay.go | StartHostRelay orchestrator (degrade to nil), HostRelay loopback HTTP server → the AgentService host actions, HTTP verdict mapping |
otlp_relay.go | StartTelemetryRelay orchestrator (degrade to nil), OTLPRelay loopback OTLP/HTTP receiver → AgentService.ExportTelemetry, HTTP verdict mapping, rate-limited rejection log |
bootstrap_test.go | ReadBootstrap happy path, per-file missing variants, empty-file rejection |
listener_test.go | pinPeerCNToCP unit tests + runSession audit-log integration test (bufconn TLS) + bad-CN / no-cert / untrusted-CA / plain-TCP rejection |
progress_test.go | parseInitStep table tests + progressReporter output/mute/nil-safety |
recover_test.go | recoverGoroutine panic callback + structured-log verification |
register_test.go | registerCoordinator happy path, retry/serialization, Hydra consumption semantics; exchangeAssertion transport/HTTP-error/token-type variants |
session_test.go | Dispatch/command_id contract, dup-ID rejection, ShellCommand audit log, spawn-failure outcome, concurrent-pipeline race-detector, closePipeOnce dedup, routeSignal reaper-race filter, handleAgentReady happy/reconnect/spawn-fail/unwired/panic |
command_output_test.go | Combined stdout+stderr single-stream ordering (2>&1), print_output console-echo gating, exit_on_non_zero exit-code mirroring (including signal-clamp), settleInitStep ✓/✗ rendering by exit code |
spawn_test.go, spawn_unix_test.go, spawn_linux_test.go | spawn-state lifecycle (echo/sleep/false/exit-42), Stop signaling, double-Run idempotency, ready-file touch, descendant reap, signal-set composition, exit-code mapping |
user_test.go | ResolveUser happy paths (name/uid/name:group/uid:gid), empty-spec and not-found errors, missing passwd/group file errors |
Logging
Structured zerolog via internal/logger.New() writing to /var/log/clawker/clawkerd.log (50MB rotation, 7d retain, 3 backups). clawkerd as PID 1 inherits Docker's stdout/stderr — Go runtime panics land there but NOT the rotated log (lumberjack is unsafe for multi-writer access). Every log line carries agent=<name> + project=<slug> for multi-agent filterability.
Levels:
- ERROR — bootstrap-file read failure, listener bind failure,
otlp_relay_unavailable/host_relay_unavailable(leaf, dial, or bind failure),otlp_relay_serve_failed/host_relay_serve_failed, unrecoverable Serve return, spawn-goroutine panic recovery, agent-ready spawn failure - WARN —
otlp_relay_unavailable/host_relay_unavailable(no CP address),host_relay_rejected,otlp_relay_rejected/otlp_relay_cp_unavailable(rate-limited, one per minute), pipe-close failures during pipeline teardown, signal forward failures (non-ESRCH), reaper main-already-reaped fallback, Stop SIGTERM/SIGKILL forward failures - INFO — state transitions:
boot,clawkerd_listener_started,otlp_relay_ready,host_relay_ready,daemon_idle,session_started,session_ended,shell_command_started,shell_command_done,agent_initialized,agent_ready_spawned,agent_ready_already_spawned,spawn_started,spawn_main_reaped,main_child_exited,shutdown_signal_received,clawkerd_listener_stopping,clawkerd_listener_stopped,shutdown(when run() returns nil; logged at ERROR when run() returns non-nil) - DEBUG — orphan-reap events, signal-after-exit filter
The single allowed os.Stderr write is the logger init failure path in main.
Failure Model
- Deterministic pre-spawn config failure (missing
CLAWKER_AGENT,ResolveUserfails, bootstrap read fails) → exit 2 (exitCodeConfig). Distinct from transient exit 1 so an operator runningrestart: on-failure:max-retries=Ncan trip-and-stop on broken config instead of restart-looping. Unix tradition: 2 = config error. - Listener-bind failure → exit 1 (transient — port-in-use clears on restart).
- A dispatched
ShellCommandcarryingexit_on_non_zerothat exits non-zero (during init, before the user CMD spawns) → main exits PID 1 with the command's mirrored exit code (a signal-killed-1is clamped to a generic non-zero). This is how a fatal init step (e.g. a failingpost_init/pre_runhook) tears the container down cleanly with the failing code rather than parking forever unhealthy with no child. CP enforces aconsts.CPAgentKillGracegrace-then-SIGKILL backstop (Executor.KillAfterGrace) for a clawkerd that cannot self-exit. - SIGTERM before
AgentReady(no user CMD ever spawned) → main logsevent=shutdown_before_spawn(Info), skips theMainExitedwait, Stops the listener, returns exit 1.spawn.SpawnErr()==nilso theevent=shutdownline carries no error field — theshutdown_before_spawnline is the sole signal. Operators grep for this event when a container exits 1 immediately ondocker stopof an idle agent. - Once
spawn.Runsucceeds, exit code = bash-convention mapping of the user CMD's exit (WEXITSTATUSfor normal,128+signumfor signaled). Dockerrestart: on-failurereads this. - A clean SIGTERM that drains all goroutines and reaps the child = exit code carrying the child's signal exit (
128 + SIGTERM). - Reaper or signal-forwarder panic → recovery closes
MainExited/Done/orphanDrainChso main()'s teardown progresses; supervisor exits 1 iffinalWSwas never recorded.
Lifetime
Bootstrap material read once at boot, held in memory for process lifetime. The Hydra JWT is single-use: registerCoordinator (register.go) consumes it on the first CP-triggered Register and short-circuits subsequent dispatches. The container's writable layer dies on --rm or docker rm, bounding material lifetime to the container.
One-time init vs every-boot. consts.AgentInitializedMarkerPath (/var/lib/clawker/agent-initialized) is written by spawnState.MarkInitialized() on AgentInitialized and read by spawnState.Initialized() at Hello. It lives in the container writable layer (NOT a volume, NOT tmpfs): it survives docker stop/start, so a restart re-runs only the boot plan and CP skips the one-time init plan; it is reclaimed by docker rm, so a recreated container re-initializes. A marker mkdir/write failure is logged (WARN event=agent_initialized_marker_mkdir_failed / agent_initialized_marker_write_failed) and tolerated — re-running the idempotent init plan on the next restart beats killing the container over a marker write.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

