agentleFS
schmitthubschmitthub/clawker/clawkerd/AGENTS.md

clawker / clawkerd

AGENTS.md59 starsChanged 12 days agoMITView on GitHub

Read the Control-plane safety section of controlplane/AGENTS.md before changing this package. Its failure rules (no panic, log.Fatal, or os.Exit on the boot or serve path; every long-lived goroutine recovers; subsystems degrade with event=<subsystem>_unavailable) apply here. Per-container agent daemon AND PID 1 init of the agent container. Owns: clawkerd runs as root inside the container. It does NOT drop its own privileges — privilege drop happens kernel-side in the spawned child between fork() and execve(). The supervisor stays root for: log…

The file

We split this AGENTS.md into the rules it gives an agent

10 rules. About 0 are common, 2 are distinctive, and 8 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 8

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Read the Control-plane safety section of controlplane/AGENTS.md before changing this package.

    line 3
  2. Very specific:

    Read four bootstrap files from consts.BootstrapDir (/run/clawker/bootstrap):

    line 27
  3. Very specific:

    Start listener on consts.DefaultClawkerdPort (:7700).

    line 36
  4. Very specific:

    Wait for ctx.Done (SIGTERM/SIGINT), spawn.MainExited, listenerFatalCh, or cmdExitCh (command-requested exit):

    line 44
  5. Very specific:

    Every ShellCommand dispatch emits two structured Info events:

    line 95
  6. Very specific:

    No register gate: a restarted container skips Register (registry hit at Hello), so "wait for RegisterDone" would block forever.

    line 123
  7. Distinctive:

    No proactive outbound dial beyond the CP-triggered Register handshake, the ExportTelemetry connection the OTLP relay holds, and the …

    line 144
  8. Distinctive:

    No heartbeat — CP knows liveness via Docker events + dialer overseer events

    line 145
  9. Very specific:

    No init-script execution — CP-driven Session.ShellCommand runs the post-init plan.

    line 146
  10. Very specific:

    No reconnect logic — clawkerd is the SERVER.

    line 147

≈ “Common” is an estimate: at least 1 in 5 of the 41,581 AGENTS.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
AGENTS.md

clawkerd

Read the Control-plane safety section of controlplane/AGENTS.md before changing this package. Its failure rules (no panic, log.Fatal, or os.Exit on the boot or serve path; every long-lived goroutine recovers; subsystems degrade with event=<subsystem>_unavailable) apply here.

Per-container agent daemon AND PID 1 init of the agent container. Owns:

  1. CP-dialed mTLS listener on :7700 (ClawkerdService.Session) — command dispatch surface for the entire container lifetime.
  2. User CMD supervision — forks the container's user CMD (default claude) as PID 1's only direct child on CP-dispatched AgentReady, with privilege drop via SysProcAttr.Credential. Forwards signals to the child's process group, reaps zombies (including reparented orphans), exits with the child's bash-convention exit code.

clawkerd runs as root inside the container. It does NOT drop its own privileges — privilege drop happens kernel-side in the spawned child between fork() and execve(). The supervisor stays root for: log rotation, /run/clawker/bootstrap reads, Wait4(-1, WNOHANG) orphan drain, and future privileged operations (eBPF coordination, cgroup writes).

Package Layout

Mirrors the controlplane binary split (thin cmd → internal entrypoint → daemon package):

  • clawkerd/ (package clawkerd) — the daemon code: listener, session, spawn, register, progress, user, recover, bootstrap (this package). Main/run import it for every subsystem.
  • clawkerd/embed/ (package clawkerdembed) — //go:embed assets/clawkerd exposes the compiled binary as Binary for internal/bundler. Lives in a separate package because the daemon binary is built FROM the code that this package would otherwise have to compile-then-embed (a build cycle) — same reason controlplane/manager holds the clawkercp embed, not internal/controlplane.
  • internal/clawkerd/cmd.go (package clawkerd) — Main() wraps run(ctx, log) (int, error); the supervisor orchestration. Imports clawkerd as daemon.
  • cmd/clawkerd/clawkerd.go (package main) — os.Exit(clawkerd.Main()), nothing else.

Role

CP is the host daemon; clawkerd is the per-container daemon. They communicate over the per-container gRPC listener on clawker-net (CP-dialed). The Session bidi-stream is the command dispatch channel. clawkerd has three outbound paths to CP's AgentService, all over its mTLS leaf: the CP-triggered Register handshake that writes the identity row (bearer-gated, one-time), the AgentService.ExportTelemetry connection the OTLP relay keeps open for the process lifetime, and the host action connection the host relay keeps open (both public scope, no bearer). Otherwise clawkerd only serves.

Boot Sequence

  1. Read four bootstrap files from consts.BootstrapDir (/run/clawker/bootstrap):

    • cert.pem, key.pem — per-agent mTLS leaf (dual EKU: ClientAuth + ServerAuth)
    • ca.pem — CLI CA cert (clawkerd's RootCA for verifying CP's client cert)
    • assertion.jwt — CLI-signed clawker-agent Hydra client_assertion JWT; single-use, exchanged at Hydra for an access token on CP-dispatched RegisterRequired, then used to call AgentService.Register (see register.go)

    Files live in the container's writable layer (not tmpfs, not bind mount). The CLI streams a tar archive via CopyToContainer between docker create and docker start. tmpfs can't be pre-populated this way (mounted at start, shadows prior writes). Permissions: parent dir 0700, files 0400. See WriteAgentBootstrapToContainer in internal/cmd/container/shared/agent_bootstrap.go for the full tradeoff.

  2. Resolve env: CLAWKER_AGENT (required), CLAWKER_PROJECT (empty = global-scope agent, 2-segment naming), CLAWKER_USER (defaults to consts.ContainerUser). CLAWKER_USER is resolved against /etc/passwd to fill SysProcAttr.Credential for the spawn child.
  3. Start listener on consts.DefaultClawkerdPort (:7700). This is the entire inbound RPC surface.

    Then start the OTLP relay (otlp_relay.go, StartTelemetryRelay) on consts.AgentOTLPRelayAddr (127.0.0.1:4318). It degrades to nil with event=otlp_relay_unavailable when CLAWKER_CP_AGENT_ADDR is unset, the leaf does not parse, or the bind fails; the daemon continues without telemetry.

    Then start the host relay (host_relay.go, StartHostRelay) on consts.AgentHostRelayAddr (127.0.0.1:18374). Same degrade contract (event=host_relay_unavailable); the daemon continues without host actions.

  4. Build spawnState (in spawn.go + spawn_unix.go) and a spawnEntry closure that captures the resolved ExecUser + argv. Thread the closure through StartClawkerdListener → clawkerdServer → runSession → session as a constructor argument so a wiring bug fails loud at construction (nil-thunk rejected at StartClawkerdListener) rather than via a package-level mutable global. The entrypoint (internal/clawkerd/cmd.go) also passes the spawnState (spawn) down the same chain as the state agentState (used for HelloAck and AgentInitialized); unlike spawnEntry/onFatal/requestExit, state is intentionally nil-tolerant and is NOT rejected at StartClawkerdListener (test fixtures leave it unset). Spawn does NOT fire here — handleAgentReady invokes the closure when CP dispatches AgentReady as the terminal step of CP-driven boot.
  5. Wait for ctx.Done (SIGTERM/SIGINT), spawn.MainExited, listenerFatalCh, or cmdExitCh (command-requested exit):
    • On ctx.Done: spawn.Stop(10s) forwards SIGTERM to the child pgroup, escalates to SIGKILL after grace, then blocks on MainExited.
    • On spawn.MainExited: phase 1 reaped the user CMD; main proceeds to teardown.
    • On cmdExitCh: a dispatched ShellCommand carrying exit_on_non_zero exited non-zero (or was signaled — clamped to a generic non-zero). The session signals the requestExit thunk with the mirrored code; main records it (exitRequested + requestedExitCode), tears down (the pre-spawn case has no child), and exits PID 1 with that code so the failing command's status surfaces to the user's terminal. This is the abort case that lets a fatal init step exit the container cleanly instead of parking forever with no child. Generic to the CP→clawkerd command service — init is its first consumer.
  6. Teardown order is load-bearing:
    1. clawkerdSrv.Stop() — force-close the listener. NOT GracefulStop: the user CMD has exited and CP holds the Session bidi stream open from its side, so GracefulStop would hang waiting for the streaming RPC handler to return. In-flight ShellCommand pipelines were already drained by the main-child-exit cascade BEFORE this point, so graceful drain is not needed.
    2. spawn.BeginOrphanDrain() — releases the reaper's phase 2 (Wait4(-1, WNOHANG)) so it can drain reparented orphans without racing session.go's c.Wait for stage children.
    3. os.Exit(spawn.Wait()) — bash-convention exit code (128+signum for signaled child) so Docker restart: on-failure reads the right value.

Spawn Lifecycle (spawn.go + spawn_unix.go)

spawnState tracks the user CMD across its lifetime. Single-shot via CAS — a second Run call returns errAlreadySpawned (mapped to Done{0} by handleAgentReady for Session-reconnect idempotency).

Goroutines installed by Run (each defer recover()-wrapped per the resilience contract):

  • Signal forwarder. signal.Notify(...forwardableSignals()) → unix.Kill(-childPgid, sig). Forwardable set excludes SIGCHLD (reaper handles), SIGURG (Go runtime preemption), and program-error signals (SIGFPE/SIGILL/SIGSEGV/SIGBUS/SIGABRT/SIGTRAP/SIGSYS — let those crash the supervisor rather than masking via forward). Filtering SIGURG is non-negotiable: forwarding it interferes with Go's goroutine preemption.
  • Reaper. Two-phase. Phase 1: Wait4(mainPID, &ws, WNOHANG) only — never Wait4(-1) while the main child is alive, so concurrent exec.Cmd.Wait calls in session.go's ShellCommand pipelines aren't stolen. Phase 2: Wait4(-1, &ws, WNOHANG) to drain reparented orphans, gated on orphanDrainCh so callers (main()) can Stop the listener first.
  • Stop watchdog. Timer-driven SIGKILL escalation if the child pgroup hasn't drained within Stop's grace window.

Phase-2 gate (BeginOrphanDrain / MainExited): the reaper holds at <-orphanDrainCh after phase 1. main() uses MainExited to trigger Stop, then calls BeginOrphanDrain to release phase 2. Tests with no concurrent exec.Cmd.Wait surface call BeginOrphanDrain immediately after Run. The gate is held by default — a forgetful caller hangs loudly on Wait/Done rather than silently racing concurrent c.Wait surfaces.

Privilege drop. SysProcAttr.Credential{Uid, Gid, Groups} populated from ResolveUser(CLAWKER_USER, /etc/passwd, /etc/group) (wraps github.com/moby/sys/user.GetExecUser, deliberately not GetExecUserPath — the latter silently swallows file-open failures). The kernel performs setgroups → setgid → setuid (in that order — setgroups MUST run while still root, before the setuid that drops privileges) between fork and exec; see Go's syscall/exec_linux.go. clawkerd's own goroutines stay root.

Exit-code mapping. mapWaitStatus (Wait4 path) and mapExitCode (*os.ProcessState path) return WEXITSTATUS for normal exit, 128+signum for signaled. Matches bash convention so restart: on-failure works.

HEALTHCHECK. /var/run/clawker/ready is touched by touchReadyFile immediately after cmd.Start() returns nil so the healthy transition lines up with the user CMD becoming a real process.

ClawkerdService Listener (CP->clawkerd)

The :7700 inbound listener (listener.go) has three guards before any handler executes:

  1. mTLS, RequireAndVerifyClientCert. ClientCAs = clawker CA bundle. Server cert is the per-agent leaf with dual EKU (ServerAuth for CP chain verify, ClientAuth for future agent->CP dial).
  2. CN pin. pinPeerCNToCP (constant-time compare) rejects any peer whose CN is not consts.ContainerCP. Prevents agent-to-agent privilege escalation via ShellCommand.
  3. ClientAuth EKU assertion. Defense in depth: Go's TLS already enforces this for client certs, but the app-layer assertion documents the dependency so a refactor that loosens TLS config (e.g. VerifyClientCertIfGiven) still fails closed.

Session Audit Log (load-bearing)

runSession emits two structured Info events per Session:

  • event=session_started — peer_cn + peer_thumbprint on every authenticated stream open
  • event=session_ended — peer_cn + duration via defer when receiver loop returns

These are the audit trail for CP-driven command dispatch. Sessions are long-lived (server-streaming, agent lifetime). Operators MUST forward clawkerd logs to durable storage for compliance retention — no other surface captures "CP opened a command channel against this container".

ShellCommand Threat Surface

ShellCommand dispatches arbitrary argv with arbitrary uid/gid inside the container, and clawkerd runs as root. The CN-pinned mTLS listener (CP = sole authorized caller) is the entire trust boundary. No per-command argv allow-list, no policy gate, no syscall sandbox. Any compromise that lets a non-CP peer mint a ContainerCP-CN cert chained to the clawker CA grants root-equivalent code execution. Per-command policy gates are a v2 concern.

ShellCommand Audit Log (load-bearing)

Every ShellCommand dispatch emits two structured Info events:

  • event=shell_command_started (one per pipeline stage) — full argv, cwd, uid, gid, timeout_seconds, command_id, stage_index
  • event=shell_command_done (one per command) — duration, final_exit_code, timed_out, outcome enum (completed / spawn_failed / timeout / incomplete)

Volume: N+1 lines for an N-stage pipeline (2-3 lines typical). incomplete outcome means runShellCommand returned via an unexpected path — treat as a clawkerd bug.

Hello Handler

The Command_Hello dispatch case replies HelloAck{Initialized, CmdRunning}, both read from the state agentState seam (session.go interface: Initialized() bool, MarkInitialized(), Spawned() bool; production impl = spawnState). This makes init/boot one-shot from CP's side: CP skips the one-time init plan once Initialized is true (the writable-layer marker exists) and skips the boot/spawn plan once CmdRunning (Spawned()) is true, instead of re-dispatching on every reconnect. Nil-tolerant — test fixtures leave state unset, so both fields report false.

AgentReady Handler

handleAgentReady is the terminal step of CP-driven boot. Calls s.spawnEntry (the thunk threaded through clawkerdServer → runSession → session from main()); replies Done{0} on success, Done{0} on errAlreadySpawned (reconnect idempotency), Error{IO_ERROR} on any other spawn failure or on a nil thunk (production rejects nil at StartClawkerdListener construction). spawnState.Run's CAS is the source of truth for "already spawned".

Host Relay (agent → CP host actions)

host_relay.go serves the HTTP paths the container scripts (host-open, git-credential-clawker, callback-forwarder) call, on loopback only; CLAWKER_HOST_PROXY points at it (consts.AgentHostRelayURL, set by the CLI only when the container has the host proxy grant). Each request becomes one AgentService host action over the bootstrap leaf: POST /open/url → OpenURL, POST /git/credential → GitCredential, POST /callback/register → RegisterCallback, GET /callback/{session}/data → GetCallbackData, DELETE /callback/{session} → DeleteCallback. The relay holds no policy: the CP decides every request (see controlplane/agent/AGENTS.md). Agents cannot reach the host proxy directly.

  • JSON is camelCase (sessionId, expiresAt, authUrl, ttlSeconds, receivedAt). Bodies are capped at hostRelayMaxBody (1 MiB); one forward is bounded by hostRelayTimeout (90 s).
  • Verdict mapping: Unavailable/DeadlineExceeded/Canceled → 503; PermissionDenied/Unauthenticated → 403; InvalidArgument → 400; NotFound → 404; ResourceExhausted → 413; else 500 (logged event=host_relay_rejected). It reuses the OTLP relay's relayHTTPStatus and adds the NotFound row. A git FailedPrecondition (host helper failed) is 200 {success:false}, the answer git expects for a failed lookup. An unknown git action is 400 without an RPC.

OTLP Relay (agent → CP telemetry)

otlp_relay.go serves the OTLP/HTTP paths the agent SDK exports to (POST /v1/{logs,metrics,traces}, application/x-protobuf, optional gzip) on loopback only. Each body is carried unchanged in one unary AgentService.ExportTelemetry call over a grpc.ClientConn dialed with the bootstrap leaf and no bearer (ExportTelemetry is public scope: mTLS plus the CP identity interceptor plus the registry thumbprint check in the CP handler). The relay holds no identity logic and never reads the batch.

  • Caps: at most relayMaxInflight (8) requests hold a body at once; over that the SDK gets 503 + Retry-After before any read, so a burst cannot exhaust PID1's memory. Body at agentv1.ExportTelemetryMaxBodyBytes (4 MiB) after gzip decoding (the compressed input is not capped separately, so a batch right at the cap is not refused for gzip framing), so a small compressed body cannot expand past what the CP listener accepts; one forward is bounded by relayExportTimeout (15 s, longer than the CP gateway's own forward timeout so a slow collector surfaces as the CP's verdict). Reads are bounded by relayReadHeaderTimeout (5 s) and relayReadTimeout (10 s, the whole request, and idle keep-alive connections through the IdleTimeout fallback), so a client that stalls an upload cannot hold a PID1 goroutine and its body buffer.
  • Verdict mapping (relayHTTPStatus): Unavailable/DeadlineExceeded/Canceled → 503 + Retry-After (the SDK's bounded retry backs off); PermissionDenied/Unauthenticated → 403; InvalidArgument → 400; ResourceExhausted → 413; anything else → 500 (the Node OTLP exporter retries 429/502/503/504 only). Client faults (404, 405, 415, 413, 400) are answered before any forward.
  • No register gate: a restarted container skips Register (registry hit at Hello), so "wait for RegisterDone" would block forever; the CP answers PermissionDenied for an unregistered peer and the user CMD only spawns after AgentReady, which follows Register.
  • Monitoring is optional: with the CP gateway breaker open the CP answers success and the relay returns 200; with the CP itself away the SDK sees 503 and gives up after its own retry budget. Rejections log at most once per minute (event=otlp_relay_rejected, event=otlp_relay_cp_unavailable).
  • Shutdown runs after the child exited (relayShutdownGrace in internal/clawkerd/cmd.go) with a context that outlives the daemon's signal context.

AgentInitialized Handler

handleAgentInitialized is the terminal step of CP-driven init (distinct from AgentReady, the terminal step of boot). Like Hello, it runs synchronously on the receive loop. It calls state.MarkInitialized() (persists the writable-layer init marker — nil-tolerant) and replies Done{0}. This is what makes init one-time: the persisted marker is what Initialized() reads at the next Hello.

Resilience Contract

clawkerd is PID 1 of the agent container. A panic that escapes a goroutine kills PID 1 → container exits → restart: on-failure may retry but if the bug is deterministic the container restart-loops with no actionable signal. This is the same resilience contract as CP (see root CLAUDE.md's "CP crashing is a SECURITY incident, not an availability one" clarification — same shape applies here).

Hard rules for code on this path:

  1. No panic(). No log.Fatal(). No os.Exit() outside the post-spawn.Wait exit in run's tail and the logger-init failure in main. Constructors return (nil, error); runtime paths log structurally and degrade.
  2. Every long-lived goroutine recovers. Signal forwarder, reaper, stop watchdog, gRPC Serve, session sender, runShellCommand worker, drainOutput, register handler, AgentReady handler, stage reapers — all wrap with defer recoverGoroutine(...) (or the equivalent inline pattern for session.go's stage reapers, which need the synthetic-StageExit emission too). The goroutine's panic-recovery path also closes any chan struct{} callers select on so nobody deadlocks. recoverGoroutine lives in recover.go (no build tag) so listener.go and session.go share it with the unix-tagged spawnState goroutines in spawn_unix.go.
  3. Subsystem failures degrade, never cascade. A broken spawn entry → Error{IO_ERROR} Response, supervisor stays alive. A broken Session → cancelled Session ctx, listener stays up. A broken reaper goroutine → closeAllGates releases MainExited/Done waiters so main() can still exit cleanly. (SPAWN_FAILED is reserved for runShellCommand stage spawn failures.)
  4. Every degraded path emits a structured log line. Operators see event=<name> with command_id, pid, pgid, error etc. Triage works from the log surface alone.

What It Does NOT Do

  • No proactive outbound dial beyond the CP-triggered Register handshake, the ExportTelemetry connection the OTLP relay holds, and the AgentService connection the host relay holds for the host actions (each a lazy dial with gRPC-managed reconnect)
  • No heartbeat — CP knows liveness via Docker events + dialer overseer events
  • No init-script execution — CP-driven Session.ShellCommand runs the post-init plan; clawkerd just dispatches
  • No reconnect logic — clawkerd is the SERVER; reconnect with backoff lives in controlplane/agent/dialer.go on the CP side

Files

FilePurpose
bootstrap.goReadBootstrap reads the four bootstrap files (cert/key/ca/assertion) from consts.BootstrapDir into the in-memory bootstrap struct; missing/empty files fail loudly (a partial boot is a security regression). The supervisor orchestrator (Main/run) that consumes this lives in internal/clawkerd/cmd.go
listener.goCP→clawkerd inbound mTLS listener. buildListenerTLSConfig enforces RequireAndVerifyClientCert + dual-EKU server cert + chain validation; pinPeerCNToCP asserts peer is ContainerCP with ClientAuth EKU
session.gorunSession per-stream owner: receive loop, sender goroutine, dispatch, ShellCommand pipeline (multi-stage exec, stdin/stdout/stderr fanout, signal forwarding, timeout watchdog, audit log). Defines the state agentState seam (Initialized/MarkInitialized/Spawned). dispatch's Command_Hello case replies HelloAck{Initialized, CmdRunning} from state; handleAgentInitialized runs on the receive loop, calls state.MarkInitialized(), replies Done{0}. handleAgentReady invokes the spawnEntry thunk threaded through the session struct from the entrypoint (internal/clawkerd/cmd.go) (no package-level mutable global). Every stage's stderr and the final stage's stdout share one combined write end (2>&1), so a single drainOutput streams the command's combined output to the caller as OutputChunk (always, any size — no accumulation buffer or cap) and echoes it live to the boot console (via progress.WriteOutput) when print_output is set; exit_on_non_zero + a non-zero exit runs Stop (flush the terminal Response) then signals the requestExit thunk (mirrored code). Both flags are generic to the command service — clawkerd makes no policy decision, the caller sets the flags
spawn.goCross-platform pure logic: mapExitCode, envForUser, routeArgs, errAlreadySpawned, errEmptyArgv
spawn_unix.go//go:build unix — spawnState lifecycle: Run (fork+exec with privilege drop + Setpgid + ready-file touch), Wait, Stop, MainExited, BeginOrphanDrain, signal forwarder, two-phase reaper. buildSysProcAttr builds the *syscall.SysProcAttr (Setpgid + optional Credential) — extracted so the privilege-drop wiring is unit-testable without root.
recover.goResilience-contract recoverGoroutine helper: structured-log + onPanic hook for every long-lived goroutine in clawkerd (no build tag — shared by spawn_unix's reaper/forwarder/watchdog AND listener.go's Serve AND session.go's sender/worker/drainer/register handler).
progress.goUser-facing TTY progress reporter: two plain status lines per init step (Active form, then ✓/✗ Done) plus boot/closing banners. No animation — per-step shell scripts complete in milliseconds, below the threshold animation would be perceptible. Writes to os.Stdout (the attached TTY for the agent container) via TIOCGPGRP-detected isTTY which toggles ANSI color codes and the info-icon glyph (cyan ℹ on a TTY, [info] ASCII fallback off-TTY); per-step ✓/✗ glyphs are emitted unchanged in both modes. Wired by internal/clawkerd/cmd.go → StartClawkerdListener → clawkerdServer → runSession → session. Init step boundaries hooked in session.dispatch (Command_Shell with init- prefix → StartStep) and session.runSender (terminal Done/Error → EndStep, via settleInitStep, fired only after stream.Send succeeds); handleAgentReady calls Final immediately before spawn so the subsequent spawnEntry transfers the TTY foreground to the user CMD without visual collision. WriteOutput echoes raw captured command output to the same console under the shared mutex (suppressed once stopped, so a post-spawn command can't garble the user CMD's TTY); settleInitStep reads Done.final_exit_code and passes the result to EndStep, so a non-zero exit renders the red ✗ instead of the green ✓.
user.goExecUser + ResolveUser wrapping github.com/moby/sys/user.GetExecUser (passwd snapshot read once into bytes; group file via explicit path reader) for name/name:group/uid/uid:gid spec parsing
register.goCP-triggered Register handshake: Hydra token exchange + AgentService.Register mTLS dial (buildDialTLSConfig is shared with the relay)
host_relay.goStartHostRelay orchestrator (degrade to nil), HostRelay loopback HTTP server → the AgentService host actions, HTTP verdict mapping
otlp_relay.goStartTelemetryRelay orchestrator (degrade to nil), OTLPRelay loopback OTLP/HTTP receiver → AgentService.ExportTelemetry, HTTP verdict mapping, rate-limited rejection log
bootstrap_test.goReadBootstrap happy path, per-file missing variants, empty-file rejection
listener_test.gopinPeerCNToCP unit tests + runSession audit-log integration test (bufconn TLS) + bad-CN / no-cert / untrusted-CA / plain-TCP rejection
progress_test.goparseInitStep table tests + progressReporter output/mute/nil-safety
recover_test.gorecoverGoroutine panic callback + structured-log verification
register_test.goregisterCoordinator happy path, retry/serialization, Hydra consumption semantics; exchangeAssertion transport/HTTP-error/token-type variants
session_test.goDispatch/command_id contract, dup-ID rejection, ShellCommand audit log, spawn-failure outcome, concurrent-pipeline race-detector, closePipeOnce dedup, routeSignal reaper-race filter, handleAgentReady happy/reconnect/spawn-fail/unwired/panic
command_output_test.goCombined stdout+stderr single-stream ordering (2>&1), print_output console-echo gating, exit_on_non_zero exit-code mirroring (including signal-clamp), settleInitStep ✓/✗ rendering by exit code
spawn_test.go, spawn_unix_test.go, spawn_linux_test.gospawn-state lifecycle (echo/sleep/false/exit-42), Stop signaling, double-Run idempotency, ready-file touch, descendant reap, signal-set composition, exit-code mapping
user_test.goResolveUser happy paths (name/uid/name:group/uid:gid), empty-spec and not-found errors, missing passwd/group file errors

Logging

Structured zerolog via internal/logger.New() writing to /var/log/clawker/clawkerd.log (50MB rotation, 7d retain, 3 backups). clawkerd as PID 1 inherits Docker's stdout/stderr — Go runtime panics land there but NOT the rotated log (lumberjack is unsafe for multi-writer access). Every log line carries agent=<name> + project=<slug> for multi-agent filterability.

Levels:

  • ERROR — bootstrap-file read failure, listener bind failure, otlp_relay_unavailable / host_relay_unavailable (leaf, dial, or bind failure), otlp_relay_serve_failed / host_relay_serve_failed, unrecoverable Serve return, spawn-goroutine panic recovery, agent-ready spawn failure
  • WARN — otlp_relay_unavailable / host_relay_unavailable (no CP address), host_relay_rejected, otlp_relay_rejected / otlp_relay_cp_unavailable (rate-limited, one per minute), pipe-close failures during pipeline teardown, signal forward failures (non-ESRCH), reaper main-already-reaped fallback, Stop SIGTERM/SIGKILL forward failures
  • INFO — state transitions: boot, clawkerd_listener_started, otlp_relay_ready, host_relay_ready, daemon_idle, session_started, session_ended, shell_command_started, shell_command_done, agent_initialized, agent_ready_spawned, agent_ready_already_spawned, spawn_started, spawn_main_reaped, main_child_exited, shutdown_signal_received, clawkerd_listener_stopping, clawkerd_listener_stopped, shutdown (when run() returns nil; logged at ERROR when run() returns non-nil)
  • DEBUG — orphan-reap events, signal-after-exit filter

The single allowed os.Stderr write is the logger init failure path in main.

Failure Model

  • Deterministic pre-spawn config failure (missing CLAWKER_AGENT, ResolveUser fails, bootstrap read fails) → exit 2 (exitCodeConfig). Distinct from transient exit 1 so an operator running restart: on-failure:max-retries=N can trip-and-stop on broken config instead of restart-looping. Unix tradition: 2 = config error.
  • Listener-bind failure → exit 1 (transient — port-in-use clears on restart).
  • A dispatched ShellCommand carrying exit_on_non_zero that exits non-zero (during init, before the user CMD spawns) → main exits PID 1 with the command's mirrored exit code (a signal-killed -1 is clamped to a generic non-zero). This is how a fatal init step (e.g. a failing post_init/pre_run hook) tears the container down cleanly with the failing code rather than parking forever unhealthy with no child. CP enforces a consts.CPAgentKillGrace grace-then-SIGKILL backstop (Executor.KillAfterGrace) for a clawkerd that cannot self-exit.
  • SIGTERM before AgentReady (no user CMD ever spawned) → main logs event=shutdown_before_spawn (Info), skips the MainExited wait, Stops the listener, returns exit 1. spawn.SpawnErr()==nil so the event=shutdown line carries no error field — the shutdown_before_spawn line is the sole signal. Operators grep for this event when a container exits 1 immediately on docker stop of an idle agent.
  • Once spawn.Run succeeds, exit code = bash-convention mapping of the user CMD's exit (WEXITSTATUS for normal, 128+signum for signaled). Docker restart: on-failure reads this.
  • A clean SIGTERM that drains all goroutines and reaps the child = exit code carrying the child's signal exit (128 + SIGTERM).
  • Reaper or signal-forwarder panic → recovery closes MainExited/Done/orphanDrainCh so main()'s teardown progresses; supervisor exits 1 if finalWS was never recorded.

Lifetime

Bootstrap material read once at boot, held in memory for process lifetime. The Hydra JWT is single-use: registerCoordinator (register.go) consumes it on the first CP-triggered Register and short-circuits subsequent dispatches. The container's writable layer dies on --rm or docker rm, bounding material lifetime to the container.

One-time init vs every-boot. consts.AgentInitializedMarkerPath (/var/lib/clawker/agent-initialized) is written by spawnState.MarkInitialized() on AgentInitialized and read by spawnState.Initialized() at Hello. It lives in the container writable layer (NOT a volume, NOT tmpfs): it survives docker stop/start, so a restart re-runs only the boot plan and CP skips the one-time init plan; it is reclaimed by docker rm, so a recreated container re-initializes. A marker mkdir/write failure is logged (WARN event=agent_initialized_marker_mkdir_failed / agent_initialized_marker_write_failed) and tolerated — re-running the idempotent init plan on the next restart beats killing the container over a marker write.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.