clawker / controlplane
AGENTS.md59 starsChanged 12 days agoMITView on GitHub
The clawker control plane. A containerized, privileged, long-lived Go service that owns authoritative state for managed containers. The clawker-controlplane container runs cmd/clawkercp as PID 1, owns the firewall stack (Envoy + CoreDNS) and eBPF state, and serves the AdminService gRPC surface consumed by the CLI via f.AdminClient(ctx). "Domain" (DDD bounded context) is a way to think about the data-layer design — how a piece of state owns and projects its own data. It is NOT a thing in the running…
Where files are headingNew AGENTS.md files have more rules about keeping secrets out of the code, and fewer about code style.
Of the AGENTS.md files we read, the 32,671 first seen in the last six months, against the older ones: the kind of rule that gained most share, and the kind that lost most. About files like this in general, not this one, and an estimate, since a kind is a pattern in the words.
In this repository17 other files it gives its agents.
- AGENTS.mdAGENTS.md
- clawkerd/AGENTS.mdAGENTS.md
- docs/AGENTS.mdAGENTS.md
- test/AGENTS.mdAGENTS.md
- .github/copilot-instructions.mdCopilot instructions
- agent-filesSkill
- audit-memorySkill
- cli-outputSkill
- dev-checksSkill
- firewall-uatSkill
- initiativeSkill
- monitoring-checksSkill
- store-editorSkill
- test-hunterSkill
- wizard-commandsSkill
- writing-testsSkill
- code-reviewSkill
The file
We split this AGENTS.md into the rules it gives an agent
10 rules. About 0 are common, 4 are distinctive, and 6 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 9
NOT a name for a package, and NOT a vehicle you start, wire, or call.
- Distinctive: line 28
Read this section before changing CP startup, serving, their dependencies, or the CP-to-agent trust contract.
- Very specific: line 39
Do NOT gate non-firewall behavior on firewall.enable.
- Very specific: line 55
No new firewall rules can be applied (clawker firewall add writes to the rules file but Envoy/CoreDNS need CP to reload).
- Very specific: line 56
No bypass can be expired (clawker firewall bypass <duration> schedules a CP-side timer.
- Distinctive: line 57
No CP→clawkerd Session means no command dispatch, no observation of agent behavior, no containment commands available even if compromise is …
- Distinctive: line 68
Treat CP shutdown as a privileged operation: If you find yourself thinking "this should never happen, just panic," stop.
- Very specific: line 70
If you're tempted to write panic() in CP code, ask: "would this leave eBPF programs pinned with no supervisor?" If yes — you've just turned …
- Very specific: line 187
All ports from cfg.ControlPlaneSettings() (defaults via struct tags).
- Very specific: line 285
CP restart resilience: When the CP restarts, clawkerd's inbound :7700 listener stays up and the dialer re-establishes the Session once the …
≈ “Common” is an estimate: at least 1 in 5 of the 41,581 AGENTS.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Control Plane Package
The clawker control plane. A containerized, privileged, long-lived Go service that owns authoritative state for managed containers. The clawker-controlplane container runs cmd/clawkercp as PID 1, owns the firewall stack (Envoy + CoreDNS) and eBPF state, and serves the AdminService gRPC surface consumed by the CLI via f.AdminClient(ctx).
Naming — "domain" is data-layer design talk, never a runtime label
"Domain" (DDD bounded context) is a way to think about the data-layer design — how a piece of state owns and projects its own data. It is NOT a thing in the running system, NOT a name for a package, and NOT a vehicle you start, wire, or call.
The implementation names its sub-components for what they are: a Store, a Storage Repository — components that live inside a package. There is no "domain handler", no "agent domain", no "future domains embed alongside". There is the agent package with its Store; the firewall handler; concrete RPC handlers.
- It is the agent package, not "the agent domain". Packages are packages.
- Wire concrete things by their real names: the dialer (
agent.NewDialer), the watcher (NewAgentWatcher), the executor (NewExecutor), the registry subscriptions (agent.Start). No*Domainsymbol names. NostartAgentDomain-style blobs — a package is not a startable vehicle. - The CP wires those concrete constructors inline; it is not "an orchestrator calling domain wrappers". Don't oversell DDD/"orchestrator" terminology.
run()reads top-to-bottom through concrete calls. No numbered// Phase Ncomment scaffolding — a function that needs a comment table-of-contents is a god-function.
Control-plane safety
Read this section before changing CP startup, serving, their dependencies, or
the CP-to-agent trust contract. The failure rules apply to every package on the
CP boot or serve path, including packages outside this directory
(internal/controlplane/, cmd/clawkercp/, clawkerd/, internal/clawkerd/,
cmd/clawkerd/, and anything they import).
CP and firewall lifecycles
- CP is unconditional infrastructure. Auth (Hydra/Kratos/Oathkeeper), AdminService gRPC on
AdminPort, AgentService gRPC onAgentPort, agent registry, mTLS, OAuth2 — all running whenever any clawker container exists. CP boots viaManager.Start(controlplane/manager). No "disable CP" flag. CP owns the clawker network. - Firewall is one optional subsystem CP manages. Envoy + custom CoreDNS + eBPF egress enforcement. Toggled by
firewall.enableinsettings.yaml(NOTclawker.yaml). When disabled, CP/mTLS/registry/agent.Dialer/ListAgents continue to operate.
Do NOT gate non-firewall behavior on firewall.enable.
CP crashing is a SECURITY incident, not an availability one
This is the single most important invariant in the codebase. Read it before adding any failure path to CP code.
What happens when CP crashes (panic, log.Fatal, unrecovered goroutine):
- PID 1 exits. CP container goes down.
on-failurerestart policy retries 3×; if the bug is deterministic (most are), CP stays dead. - eBPF programs stay attached to cgroups. They're pinned under
/sys/fs/bpfand survive the CP container's death. Agent containers' egress traffic continues to be filtered by whatever rule set was loaded at the moment CP died. - The clean drain-to-zero path is skipped.
firewall.Stack.Stop()andebpfMgr.FlushDead()only run on intentional shutdown via the orchestrator. A panic skips both. eBPF state is now frozen and unsupervised, and a bypass in effect has no timer to end it. - Agent containers keep running. They have no awareness that their supervisor died. They keep serving their workloads.
- The user has no idea. They see agents running. They assume the firewall is enforcing — and it technically is, against the rules that happened to be loaded. They assume CP is observing — it isn't. They assume CP can dispatch containment — it can't.
The result:
- No new firewall rules can be applied (
clawker firewall addwrites to the rules file but Envoy/CoreDNS need CP to reload). - No bypass can be expired (
clawker firewall bypass <duration>schedules a CP-side timer; if CP died during a bypass, the bypass is now permanent until the user manually intervenes). - No CP→clawkerd Session means no command dispatch, no observation of agent behavior, no containment commands available even if compromise is detected.
- Agents are vulnerable to prompt injection, exfiltration, and lateral-movement attempts that CP would otherwise observe and contain. The user's mental model ("CP has them covered") is silently false.
The stack trace from a CP panic lands on os.Stderr → docker logs <cp>. It is NOT in the rotating ControlPlaneLogFile operators are wired to grep. It is NOT surfaced by clawker controlplane status (which only knows up/down). The user has to know to dig into raw docker logs to find it.
Hard rules for code on the CP boot/serve path (cmd/clawkercp/, internal/controlplane/, anything imported by them):
- No
panic(). Nolog.Fatal(). Noos.Exit()outside the orchestrator's intentional shutdown sequence. Constructors return(nil, error)(seeagent.NewDialer,agent.NewExecutor); main logs structurally and degrades. The only hard-exits permitted are: drain-to-zero clean exit (code 0), and the orchestrator's pre-SetReadystartup-gate failures (code 1) — these exit WITHOUT flushing eBPF, so any agents enrolled by a previous CP stay fail-closed (filtered against the old rule set) rather than fail-open. - Every long-lived goroutine recovers. Heartbeats, watchers, event handlers, RPC handlers — wrap with
defer func() { if r := recover(); r != nil { log.Error().Interface("panic", r)... } }(). The pub/sub stats heartbeat (pubsub.NewStatsHeartbeat, wired ininternal/controlplane/cmd.go) is the canonical template. One bad event must not take down the daemon and silently strand eBPF. - Subsystem failures degrade, never cascade. A broken Executor →
executor = nil; CP never dispatchesAgentReady, clawkerd-as-PID-1 never spawns the user CMD, and the container exits non-zero ondocker stop; the firewall, registry, AdminService, dialer all stay up. A broken dialer →dialer = nil; CP→clawkerd dispatch disabled; everything else stays up. The patterns ininternal/controlplane/cmd.go—wireExecutor(executor; emitsevent=agent_executor_unavailable) and theagent.NewDialer(...)block that degrades on error toevent=agent_dialer_unavailable— are the templates; copy either for any new subsystem. - Every degraded path emits a structured log line.
event=<subsystem>_unavailablewith component, error, downstream impact. Operator must be able to determine root cause AND blast radius from the structured log surface alone — they will not see panic stacks. - Treat CP shutdown as a privileged operation. If you find yourself thinking "this should never happen, just panic," stop. In CP that line of reasoning compromises the security boundary the user trusts to be intact. Return an error and let the orchestrator decide.
If you're tempted to write panic() in CP code, ask: "would this leave eBPF programs pinned with no supervisor?" If yes — you've just turned a logic bug into a silent firewall failure. Return an error instead.
Asymmetric trust: dialer permissive, listener strict
- clawkerd-side listener (server): STRICT.
clawkerd/listener.goenforces CP CN pin + Client-Auth EKU + CA chain at TLS layer. - CP-side dialer (client): PERMISSIVE.
controlplane/agent.Dialernever aborts on cert/identity grounds. Outcomes emitted as typed fields onSessionConnectedevents. Dial only fails on connectivity.
Why permissive: CP must reach clawkerd to issue containment commands even when certs are bad. Subscribers to SessionConnected enact policy; the dialer holds none.
Trust attestation: The CLI mints the agent certificate. CP writes the registry row through the Register handler. The dialer checks the peer certificate thumbprint against the row and emits the result on the bus. See the agent package for the identity contract.
Package-specific behavior below must preserve those requirements.
Responsibilities
- Authoritative eBPF management — the CP owns
ebpf.Manager.Load()lifetime for its process lifetime. BPF programs are loaded once at boot and stay live. The manager exposes read-only accessors for the netlogger pipeline:EventsRingbuf(),EventsDrops(),RatelimitDrops(),DNSCache()(all nil beforeLoad; callers MUST nil-check). - AdminService gRPC surface — the host CLI calls firewall/eBPF operations as typed gRPC over mTLS TCP with OAuth2 JWT authorization.
- Ory auth stack — Hydra (OAuth2), Oathkeeper (reverse proxy), Kratos (identity, placeholder for webui).
- Aggregate health reporting —
/healthzactively probes all 7 service ports before returning 200. - Per-decision eBPF egress event emission (netlogger) — drains BPF
events_ringbuf, enriches bycgroup_idvia the typed eBPF-enrolled pub/sub topic (enrolledTopic, subscribed by netlogger), ships OTLP log records (service.name=ebpf-egress) to the trusted-infra OTLP receiver via theotel.NewOtelLoggerProviderfactory (controlplane/otel). Degraded paths emitevent=netlogger_unavailableand leave firewall enforcement untouched.
Auth (Hydra introspection + mTLS + JWT)
The auth stack uses Ory Hydra as the OAuth2 provider (replaces the earlier custom OIDC provider):
| Layer | Purpose | Implementation |
| mTLS over TCP | Authenticate the channel; server cert + CLI client cert both signed by CLI CA | Server: RequireAndVerifyClientCert + ClientCAs from bind-mounted CA. Client: LoadClientCert() in controlplane/adminclient/dial.go |
| Hydra OAuth2 | Issue JWTs via client_credentials + private_key_jwt (ES256) | Hydra subprocess with in-memory DSN |
| gRPC AuthInterceptor | Validate bearer tokens via Hydra introspection (RFC 7662); enforce per-method scopes | controlplane/auth/authz.go — HydraIntrospector calls POST /admin/oauth2/introspect |
CLI auth flow: CLI presents mTLS client cert (signed by CLI CA) during TLS handshake → server verifies against CA → CLI signs a JWT assertion with its ES256 private key → POST to Hydra /oauth2/token (plain TLS, separate config) with client_credentials grant + client_assertion → Hydra validates signature against registered JWKS → returns access token (JWT) → CLI sends bearer token on gRPC calls → CP's AuthInterceptor introspects token via Hydra admin API → grants/denies based on scope.
Two TLS configs on CLI side (controlplane/adminclient/dial.go): tokenTLSCfg (plain TLS for Hydra token endpoint) and grpcTLSCfg (mTLS with client cert for AdminService). This pattern scales to future agent clients that will have their own CA-signed certs.
Failure mode: Fail-closed. Any error (network, introspection failure, unmapped method) returns codes.Unauthenticated.
Subpackages
controlplane/ has no root .go files — it is purely the parent of the CP's bounded-context subpackages. Each row is a package directory under controlplane/.
| Subpackage | Purpose |
pubsub/ | Generic, dumb in-memory pub/sub pipe — Topic[T]/Event[T] (the typed bus), NewStatsHeartbeat. Zero imports of any CP sibling; recover-per-delivery so a panicking subscriber can't strand eBPF. |
dockerevents/ | Docker-event bounded context: feeder.go (sole DockerEvent producer), dispatch/reconcile of purpose=agent container lifecycle onto the typed topic. |
agent/ | Agent bounded context — sqlite registry, in-memory worldview repository, CP→clawkerd dialer (agent.NewDialer), NewAgentWatcher, NewExecutor, IdentityInterceptor. See controlplane/agent/AGENTS.md. |
server/ | gRPC composition: NewAdminServer(fw, agents, recovery, log) (adminv1.AdminServiceServer, error) (server.go) + NewGRPCStack(GRPCDeps) (*GRPCStack, error) (grpc_stack.go) — builds both listeners (admin + agent), wires interceptors, registers services. |
auth/ | Ory auth stack: AuthInterceptor/HydraIntrospector (authz.go), RegisterCLIClient/RegisterAgentClient/RegisterHostProxyClient (hydra_client.go), WriteOryConfigs (ory_configs.go), Ory subprocess bringup (ory_stack.go). Mocks in auth/mocks/. |
subprocess/ | SubprocessManager + NewSubprocessManager — Ory subprocess lifecycle (start, health, crash detection, reverse-order shutdown). |
hostgateway/ | Gateway — the HostProxyService server on the admin listener and the CP side of the AgentService host actions. The host proxy dials out and holds one Connect stream (CN pinned to consts.HostProxyClientName; the newest stream replaces the old one); each action goes down it with an id and a deadline. No connected host proxy → ErrUnavailable. Owns callback-session ownership (session → container); ErrNotOwner/ErrDenied/ErrInvalidRequest/ErrNotFound/ErrActionFailed/ErrUnavailable. Built with hostgateway.New in internal/controlplane/cmd.go. |
otel/ | NewOtelLoggerProvider(OtelClientOptions) (*sdklog.LoggerProvider, error) (otelclient.go) — generic per-subsystem OTel log-provider factory pushing OTLP/gRPC over mTLS to the trusted-infra receiver. |
firewall/ | Envoy + CoreDNS + eBPF egress enforcement; firewall.Handler (the 12 firewall RPCs), firewall.Stack, Envoy/CoreDNS config generation, and the ebpf/ subtree (loader + netlogger). See controlplane/firewall/AGENTS.md. |
manager/ | Host-side CP lifecycle. ensureRunning/Stop/CPRunning (bootstrap.go), BuildCPContainerConfig (cp_container.go), Manager interface (Start/Stop/IsRunning/ProbeHealthz) + NewManager (manager.go) — Start is the idempotent bringup; a boot the CP cannot finish alone surfaces as *CPSOSError for the CLI bootstrap verbs to assist (internal/cmd/controlplane/shared). Also the //go:embed of clawkercp + ebpf-manager + the host-side bpffs-delegate (embed_cp.go/embed_ebpf.go/embed_bpffs.go). Replaces the former cpboot/. See controlplane/manager/AGENTS.md. |
adminclient/ | Clients for the admin listener (dial.go): Dial (the CLI's AdminService client), DialHostProxy (the host proxy's connection for HostProxyService: its own leaf, its own Hydra client and scope, keepalive), ProbeCPTime, the two TLS configs (token-endpoint plain TLS vs gRPC mTLS), token source (client ID + scope; drops a cached token the CP rejects as Unauthenticated, so a restarted CP gets a new one). AwaitReady (ready.go) polls /healthz while it reports not_ready, so a CLI process does not send admin RPCs into the ready gate of a CP another process is booting. |
infracerts/ | Trusted-infra (OTLP/monitoring) mTLS cert material. See controlplane/infracerts/AGENTS.md. |
otelcerts/ | OTel client cert provisioning. See controlplane/otelcerts/AGENTS.md. |
sdscerts/ | Dedicated Envoy→CP SDS client identity (envoy-sds-client leaf, own material dir + readiness gate — deliberately NOT the telemetry lane's certs). See controlplane/sdscerts/AGENTS.md. |
AdminService composition
controlplane/server/server.go exposes the unexported adminServer type that embeds *firewall.Handler (and, in future branches, additional RPC handlers). Method promotion produces the AdminServiceServer surface. server.NewAdminServer(fw, agents, recovery, log) (adminv1.AdminServiceServer, error) is the composition constructor — it returns an error (e.g. ErrNilRegistry, ErrNilRecovery) rather than panicking, per the CP no-crash contract. It is composed into the gRPC stack by server.NewGRPCStack (controlplane/server/grpc_stack.go), which buildGRPCStack in internal/controlplane/cmd.go calls to build and serve both listeners.
The 12 firewall RPCs live in controlplane/firewall/handler.go — see controlplane/firewall/AGENTS.md for the per-RPC table. Future handlers (Monitor, Clawkerd) embed alongside. HostProxyService is not embedded: hostgateway.Gateway serves it on the admin listener. The the <Subsystem><Action>[<Object>] proto naming convention prevents method-name collisions.
All RPCs require the uniform admin scope (INV-B2-009) with one deliberate exception: GetSystemTime is mapped to the public scope (consts.ScopePublic) in AdminMethodScopes() — no bearer token, mTLS client cert still required at the listener — because it bootstraps the token exchange itself. WatchSOS is admin-scoped like the rest (recoverable failures only happen after the Ory stack is up — anything earlier exits 1 — so the CLI can always mint a token by the time there is something to watch): it is a server stream the CLI holds open while the CP boots, carrying an error or nothing — a delivered SOS is a startup failure the CLI can assist with while the CP stays alive waiting, a clean end-of-stream means resolved or nothing to report; the SOS.kind enum (SOSKind) is the CLI's dispatch discriminator — it switches on kind and never parses message, and an unknown kind (older CLI, newer CP) surfaces message as the error rather than hanging; unrecoverable failures never appear — the CP exits non-zero as usual (see the recovery queue in internal/controlplane/cmd.go). The ready-gate exemption list (readyGateExemptMethods in controlplane/server/grpc_stack.go) is the public set plus WatchSOS: the admin listener serves from construction, and the ready-gate interceptor rejects every non-exempt RPC with codes.FailedPrecondition until SetReady. An empty or unmapped scope fails closed (deny) — public is the explicit ScopePublic sentinel, never the zero value. Per-method scope diversification beyond this is intentionally not used — see Spec §8.
Startup Sequence (run() in internal/controlplane/cmd.go)
run() reads top-to-bottom in two halves — CONSTRUCTION (objects only, no side effects) then the STARTUP FLOW (ordered method calls that create the side effects). BPF pins, subprocesses, and firewall state are flow steps, never construction side effects.
bootLogging— trusted-lane OTel certs + the file/OTEL logger (degradedos.Stderrfallback whenNewfails).- Config + signal-aware contexts —
config.NewConfig,signalCtx(SIGTERM/SIGINT),watcherCtx(long-lived workers),subprocess.NewSubprocessManager,NewControlPlane. auth.NewOryStack— CONSTRUCTION only: builds the single CA pool + CA TLS surface fromcaCertPathup front. No subprocess starts here — Ory bringup is flow step 10.buildEnforcement— CONSTRUCTION only: Docker client +firewall.Stack+ thefirewall.EgressRulesStore+ the sharedfirewall.CAStore+ theRouteIdentityStore-backedIdentityAllocator+ebpf.NewManager. No BPF state is created here;ebpfLoadFlow(flow step 10) is the step that loads. Returns the joined cleanup, safe on an unloaded manager.buildTopics— the typed pub/sub topics (dockerTopic,agentTopic,enrolledTopic); one topic per payload type, the generic audit hook self-attaches inNewTopic.buildAgentInfra— agent sqlite registry +MobyPeerLookup+ContainerLister+ the in-memoryagent.Repository(worldview) with its agent-event and docker-event subscriptions wired.buildOtelGateway— the agent telemetry gateway (controlplane/otelgateway.Gateway): one lazy mTLS client connection to the collector'sotlp/agentsreceiver atotel-collector:<consts.MonitoringOtelAgentIngestPort>(4320) (over the internalclawker-telemetrynetwork) with thecpinfra leaf. NOT a gate: a missing issuer or constructor failure degrades to a nilagent.OtelForwarderwithevent=otel_gateway_unavailable(ExportTelemetry answersUnavailable; the clawkerd relay retries); success logsevent=otel_gateway_ready. The monitoring stack is optional, so the gateway carries a breaker: three consecutive forward failures open it (event=otel_gateway_tripped), batches are then dropped with a success result so the agent SDK stops retrying, one probe per minute reaches the collector, and a probe success closes it (event=otel_gateway_recovered) with no CP restart.buildGRPCStack— firewallActionQueue+fwhandler.Handler(holds publish-onlyenrolledTopic) + the admin (cp.AdminPort, mTLS + ready gate +adminAuth, which sendsHostProxyServicecalls to the host proxy interceptor (scopehostproxy:serve, client ID pinned) and all others to the admin interceptor; keepalive enforcement for the host proxy stream) and agent (cp.AgentPort, clawker-net only, agent-scope AuthInterceptor, then a process-wideExportTelemetryadmission cap of 16 in-flight calls, chained ahead ofagent.IdentityInterceptor) gRPC listeners; both listeners are BOUND here and the agent listener starts serving (ServeAgent) — boot-time clawkerd dial-back/registration needs it. The admin surface hosts the 12 firewall RPCs +ListAgents+ the bootstrap RPCs (the publicGetSystemTime, the admin-scoped ready-gate-exemptWatchSOS), plusHostProxyService.Connect(the host proxy's outbound stream, served byhostgateway.Gateway).IdentityInterceptorruns a universal three-stage gate (CN pin toconsts.ContainerClawkerd→ peer-IP→purpose=agentcontainer resolution readingdev.clawker.{project,agent}labels → constant-timeAgentFullNamevsurn:clawker:agent:URI SAN compare). The agent surface hostsRegister(bearer-gated),ExportTelemetry(public scope; the handler adds the registry thumbprint check and forwards through the step-7 gateway), and the five host actions (public scope; thumbprint check plus the container's label grants, thenhostgateway.Gateway, which sends the action down the host proxy stream — seecontrolplane/agent/AGENTS.md). CP→clawkerd dispatch is the OUTBOUND dialer, not this listener — seecontrolplane/agent/AGENTS.mdand asymmetric trust.- Serve the boot surfaces —
SetAdminServingCheck, thengrpcStack.ServeAdmin(serveFailed)MID-BOOT: the ready-gate-exempt bootstrap RPCs answer while the flow below runs (WatchSOSis the CLI's window into a boot waiting for assistance), and the ready-gate interceptor rejects every other admin RPC withcodes.FailedPreconditionuntilSetReady— no rule mutation is accepted mid-boot. ThenstartHealthz, which answers 503not_readyuntil the flow completes. - STARTUP FLOW — ordered side-effect steps, each a pre-
SetReadygate (exit 1, no eBPF flush, enrolled agents stay fail-closed). Any errorrun()returns is logged to the CP log file asevent=cp_exit_errorbefore exit (Mainalso prints it to stderr, which is lost when the container is removed):
oryStack.Start— writes Ory configs, starts Kratos + Hydra + Oathkeeper subprocesses, waits healthy, registers the CLI + agent Hydra clients (both with the CLI JWK) and the host proxy Hydra client (RegisterHostProxyClient, scopehostproxy:serve, with the host proxy JWK atconsts.CPHostProxyPubKeyPath); thenSetServiceProbesinstalls the aggregate/healthzprobes.ebpfLoadFlow(aControlPlanemethod — the recovery protocol is orchestrator state management) —ebpfMgr.Load()+CleanupStaleBypass(INV-B2-013). On the default deployment this is the whole story. The permission-denied load is the ONE recoverable failure (rootless Docker) and the only thing that engages any delegation machinery (delegateBPFFSFlow): kernel gate (ebpf.CheckKernelSupport, 6.9 floor — too old is terminal, no SOS),ebpf.OpenForDelegation(fsopen; the superblock's owning userns is stamped from this process), publish ONCE on the orchestrator's recovery queue (PublishRecovery— delivered to every connectedWatchSOSstream, and to any that connects later), then serve the filesystem context on the handoff socket bounded byconsts.CPSOSIdleTTL(30s — a CP nobody is listening to shuts down rather than waiting forever). Once the helper acks,ClearRecoveryends every watcher stream cleanly and the flow returnserrBPFFSDelegated—Mainmaps it to a CLEAN exit (code 0, so the on-failure restart policy does not resurrect a process whose mount namespace predates the delegated filesystem), and the CLI'sManager.Startretry recreates the CP container onto the delegated path when the source changed (the bpffs-source drift gate) or simply restarts it when the helper replaced a stale mount at the same path — Docker re-establishes bind mounts at every container start. No in-process retry exists by design: cilium/ebpf caches its token decision on the first BPF syscall. Every other load failure is unrecoverable and exits 1 as before.startSDSServer— the firewall on-demand certificate SDS listener (Envoy ↔ CP, clawker-net only,ControlPlaneSettings.SDSPort). Serves per-SNI MITM leaves to Envoy's wildcard-chain certificate selector. mTLS narrowed by aVerifyConnectionSAN pin to the dedicatedconsts.EnvoySDSClientNameleaf (provisioned bycontrolplane/sdscerts, constructed inrun()and threaded throughbuildEnforcementinto the stack; the telemetry lane'senvoy-otel-clientleaf is refused). NOT a gate: any failure degrades withevent=sds_unavailable/event=sds_certs_unavailable(wildcard multi-label handshakes fall back to static certs; everything else stays up). Runs before the firewall gate so Envoy can fetch secrets from its first boot. The deferred stop waits up tosdsStopTimeout(defaultShutdownWait) for streams to close, then logsevent=sds_graceful_stop_timeoutand callsgrpc.Server.Stop.firewallBringupGate— whenfirewall.enable(settings.yaml) is true, runsFirewallInitsynchronously BEFORESetReadyso a green/healthzmeans "everything the settings enable is enforcing". A failure FAILS startup (loggedevent=firewall_bringup_failed, bounded byconsts.FirewallStackBringupTimeout). Whenfirewall.enableis false it runsebpfMgr.FlushAll()instead (event=firewall_disabled_state_cleared): the drain'sFlushDeadkept live agents enrolled against a stopped Envoy/CoreDNS, and with the firewall off no CP would re-enroll them. A flush failure degrades (event=firewall_disabled_flush_failed), it never gates startup. Caveat: re-enrollment events published by this gate precede netlogger construction (step 12), so netlogger's label cache stays cold for agents that outlived the previous CP until the next FirewallInit/FirewallEnable — telemetry enrichment only, enforcement unaffected.orchestrator.SetReady()— the ready gate flips; the already-serving admin surface accepts everything from here;/healthzcan now go green.
startFeeder— thedockereventsfeeder, sole producer ofDockerEventonto its typed topic.startWorkers— the long-lived observability workers: thepubsub.NewStatsHeartbeat, thenetlogger.Service(subscribesenrolledTopicto hydrate its label cache; degrades tonetloggerSvc=nilwithevent=netlogger_unavailableon any chain failure), and thedns_cacheGC goroutine (event=dns_gc_*, escalatesdns_gc_degradedafterdnsGCDegradedThresholdconsecutive reclaim-failures). All run onwatcherCtx.- Agent watcher +
startAgentDialer—agent.NewAgentWatcher(drain-to-zero trigger; its goroutine recovers panics into a terminal shutdown error,event=agent_watcher_panic) plus the executor, CP→clawkerd dialer, and agent-axis subscriptions (§3.4 degrade contract). - Serve + drain — the select waits on signal / drain-to-zero / subprocess crash / serve failure, then runs the drain callback (
actionQueue.Close()→hostGateway.Close()(ends the host proxy stream so GracefulStop does not wait on it) →grpcStack.GracefulStop()→handler.CancelAllBypassTimers()→firewall.Stack.Stop()→netloggerSvc.Stop→stopDNSGC()→ebpfMgr.FlushDead(), INV-B2-007) exactly once (sync.Once).FlushDeadkeeps the eBPF state of live cgroups and removes the state of dead cgroups and every bypass flag, so an agent that outlives the CP stays enrolled against the stopped Envoy and CoreDNS and fails closed until the next CP re-enrolls it. E2E:TestFirewall_ControlPlaneDownFailsClosed. EVERY arm drains — the subprocess-crash and serve-failure arms (event=cp_subprocess_crashed/event=cp_serve_failed) run the callback before returning their error, since a bare return would exit PID 1 with eBPF pinned and unsupervised. Drain-to-zero and signal exit 0 (theon-failurerestart policy does NOT retrigger); a crash or serve failure exits 1 after draining.
Aggregate Health (internal/controlplane/cmd.go)
The ControlPlane orchestrator type (NewControlPlane, SetReady, HealthzHandler, SetServiceProbes) manages the /healthz endpoint:
- Before ready: returns 503
- After ready: actively probes all 7 service ports on every request:
- Hydra public (TLS), Hydra admin (TLS)
- Kratos public (TLS), Kratos admin (TLS)
- Oathkeeper proxy (TLS), Oathkeeper API (TLS)
- gRPC admin (raw TCP plus a serving check —
GRPCStack.AdminServing, installed viaSetAdminServingCheck)
- Returns 200 only when ALL probes succeed
The admin listener socket is bound at gRPC stack construction, so a bare TCP
dial completes out of the accept backlog whether or not anything is in
Serve. The grpc-admin probe therefore gates its dial on the stack's serving
signal (GRPCStack.AdminServing): the dial proves reachability, the signal
proves liveness. The check fails closed until it is installed (right after
buildGRPCStack, before ServeAdmin). ServeAdmin runs mid-boot — the ready
gate in the interceptor chain, not late serving, is what keeps non-public RPCs
out until SetReady.
Container Config (controlplane/manager/cp_container.go)
func BuildCPContainerConfig(cfg config.Config, opts CPContainerOpts) (*CPContainerConfig, error)All ports from cfg.ControlPlaneSettings() (defaults via struct tags). Published to 127.0.0.1 only:
| Published Port | Purpose |
| AdminPort (7443) | gRPC AdminService |
| HydraPublicPort (4444) | OAuth2 token endpoint |
| OathkeeperPort (4456) | HTTP reverse proxy (future webui) |
| HealthPort (7080) | /healthz endpoint |
Not published: Hydra admin (4445), Kratos ports, Oathkeeper API — internal-only (127.0.0.1 bind inside container).
Key mounts: config dir (RO), CA cert (RO), CLI public JWK (RO), host proxy public JWK → consts.CPHostProxyPubKeyPath (RO, for the host proxy Hydra client), server TLS cert+key (RO), FirewallDataSubdir → /var/lib/clawker/firewall (RW — egress rules, Envoy/CoreDNS configs, MITM CA), /sys/fs/cgroup (RO), the BPF filesystem → /sys/fs/bpf (RW, no propagation options; source chosen by resolveBPFFSSource — the host's own /sys/fs/bpf by default, clawker's delegated bpffs at BPFFSSubdir when one exists after a rootless heal; the choice is stamped as consts.LabelCPBPFFSSource for the recreate drift gate and injected as consts.EnvHostBPFFSSource for the CoreDNS sibling), logs dir.
Networks: the CP joins clawker-net with a static IP and the internal clawker-telemetry network (consts.TelemetryNetwork, ensured by EnsureTelemetryNetwork before create) with a dynamic address. The telemetry network carries the trusted lane to the collector. The CP container has no ExtraHosts entry: the host proxy dials out to the CP, so the CP does not look up the host gateway.
Restart policy: on-failure with MaximumRetryCount=3. A clean drain-to-zero exit (code 0 from AgentWatcher) does NOT retrigger the policy.
Invariant INV-B1-006: CLI private signing key is NEVER mounted into the container.
Capabilities: BPF, SYS_ADMIN (for eBPF program attachment).
eBPF Subsystem
The eBPF subsystem lives at firewall/ebpf/ — see firewall/ebpf/CLAUDE.md for full reference. Key surface consumed by CP core:
ebpf.Manager— concrete loader.Load()runs once at CP startup;CleanupStaleBypassruns beforeSetReady(INV-B2-013);FlushDeadruns in the drain (INV-B2-007);FlushAllruns only onfirewall down(FirewallRemove).ebpf.EBPFManagerinterface — consumed byfirewall.Handler. Methods:Install,Remove,Enable,Disable,SyncRoutes,FlushAll.ebpf.Route+ebpf.BPFContainerConfig— shared types used byinternal/dnsbpfandcontrolplane/firewall. Route identities are allocated byfirewall.IdentityAllocator(sticky, persisted), never derived by hashing.ebpf.Manager.EventsRingbuf()/EventsDrops()/RatelimitDrops()/DNSCache()— read-only map accessors consumed bynetlogger.Servicefor the per-decision egress event pipeline. All return nil beforeLoad()— callers MUST nil-check.
Ory Config Generation (controlplane/auth/ory_configs.go)
func WriteOryConfigs(cp config.ControlPlaneSettings, hydraSecret string) errorGenerates /etc/clawker/{hydra,kratos,oathkeeper}.yaml:
- Hydra: in-memory DSN, JWT access tokens, admin at
127.0.0.1:4445(internal-only), public at0.0.0.0:4444, 1h access token TTL - Kratos: in-memory DSN,
127.0.0.1:4480(placeholder for future webui identity) - Oathkeeper: HTTP reverse proxy at
0.0.0.0:4455(placeholder for future webui auth), API at127.0.0.1:4456
Subprocess Management (controlplane/subprocess/subprocess.go)
type SubprocessManager struct { ... }
func (sm *SubprocessManager) Start(name string, cmd *exec.Cmd) error
func (sm *SubprocessManager) WaitHealthy(ctx context.Context, name string, check HealthCheck) error
func (sm *SubprocessManager) CrashChan() <-chan error
func (sm *SubprocessManager) Shutdown(timeout time.Duration)Manages Ory service lifecycle. Crash reporting via channel. Shutdown sends SIGTERM then SIGKILL, reverse start order.
Test seam overview
EBPFManagerinterface —controlplane/firewall/ebpf/mocks/EBPFManagerMockfor firewall handler tests.Introspectorinterface —controlplane/auth/mocks/IntrospectorMockfor authz tests (no real Hydra).cpmanager.Managerinterface —cpmanagermocks.ManagerMock(controlplane/manager/mocks) for break-glasscontrolplane up/down/statusCLI tests.adminv1.AdminServiceClient—api/admin/v1/mocks.AdminServiceClientMockfor CLI tests that speak to the AdminService.firewall.ContainerResolver— handler-side injectable Docker lookup (seecontrolplane/firewall/AGENTS.md).firewall.EgressRulesStore/firewall.RouteIdentityStore— the two store-backed firewall domain facades; moq mocks incontrolplane/firewall/mocks/. CP wiring (buildEnforcement) threads the interfaces, never astorage.Store. Firewall's own tests use real stores instead.agent.OtelForwarder— moq-generatedOtelForwarderMock(controlplane/agent/mocks/otel_forwarder_mock.go) for the ExportTelemetry handler tests.agent.HostForwarder— moq-generatedHostForwarderMock(controlplane/agent/mocks/host_forwarder_mock.go) for the host action handler tests (no realhostgateway.Gateway).agent.Registry— moq-generatedRegistryMock(incontrolplane/agent/mocks/registry_mock.go) forIdentityInterceptor,ListAgents, and the dialer-side classification tests that need a deterministic snapshot independent of dockerevents wiring.
Test coverage
Tests for CP entrypoint helpers belong in internal/controlplane/cmd_helpers_test.go, including SDS listener shutdown and client SAN checks. Tests for the SDS service belong in controlplane/firewall/.
| File | Invariants | What |
controlplane/auth/authz_test.go | INV-B1-011 | Token validation, scope enforcement, unmapped method denial, Hydra introspection mock |
controlplane/auth/grpc_mtls_test.go | — | mTLS connection acceptance (valid cert), rejection (no cert), rejection (no TLS) |
controlplane/manager/container_config_test.go | INV-B1-005, 006, 008, 009, 015, 017, 018, 020 | Port bindings, mounts, labels, private key exclusion (signing + client), config-driven ports |
internal/controlplane/cmd_test.go | INV-B1-010, 013 | IsReady()/SetReady() atomic gate; /healthz 503→200 transition across the single ready boundary |
controlplane/firewall/ebpf/manager_test.go | — | Link cleanup, map schema detection, Install/Remove/Enable/Disable, SyncRoutes, DNS cache GC |
controlplane/subprocess/subprocess_test.go | — | Start/WaitHealthy, crash detection, SIGTERM/SIGKILL shutdown |
controlplane/manager/ebpf_regression_test.go | — | eBPF package regressions (no kernel required) |
Package imports
The CP daemon core is the internal/controlplane orchestrator plus these controlplane/* subpackages.
Uses: internal/config, internal/consts, internal/docker, internal/logger, controlplane/firewall, controlplane/firewall/ebpf, controlplane/otelgateway (+ go.opentelemetry.io/proto/otlp/collector/{logs,metrics,trace}/v1 through it), controlplane/hostgateway, api/admin/v1, api/agent/v1, api/hostproxy/v1, google.golang.org/grpc, github.com/cilium/ebpf, github.com/moby/moby/api/types/{mount,network}, go.opentelemetry.io/otel, go.opentelemetry.io/otel/log, go.opentelemetry.io/otel/sdk/log, go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc (the last four via controlplane/otel's NewOtelLoggerProvider).
Used by: cmd/clawkercp/ (daemon entrypoint → internal/controlplane), internal/cmd/controlplane/ (break-glass up/down/status → controlplane/manager), internal/cmd/factory/ (controlplane/adminclient + ControlPlane Factory closures), internal/cmd/firewall/ (AdminService consumers via f.AdminClient; controlplane/manager for stack up/down/status), internal/cmd/container/* (BootstrapServicesPostStart → controlplane/manager), controlplane/firewall/ebpf/netlogger (consumes controlplane/otel's NewOtelLoggerProvider), internal/dnsbpf (reuses ebpf types), internal/auth (cert paths).
No circular dependencies.
What's deferred
- Kratos active usage — running as subprocess placeholder. Lights up with webui.
- Oathkeeper active routing — running with empty rules. Lights up with webui HTTP auth.
- Per-method scopes beyond
admin— finer-grained scopes (webui:read, etc.) would add entries toAdminMethodScopes()inapi/admin/v1/admin.go(typedadminv1.AdminScope). INV-B2-009 mandates a uniformadminscope across all AdminService methods except the public bootstrap RPCGetSystemTime(consts.ScopePublic). The agent listener's scope vocabulary lives inAgentMethodScopes()inapi/agent/v1/agent.go(typedagentv1.AgentScope) and holds 7 entries:Register→ScopeSelfRegister, andExportTelemetryplus the five host actions (OpenURL,GitCredential,RegisterCallback,GetCallbackData,DeleteCallback) →consts.ScopePublic. clawkerd spends its single-use Hydra assertion onRegister, so the public methods use mTLS,IdentityInterceptor, and the registry thumbprint check as the gate.
Known limitations (deferred to cp-restart-resilience)
- CP restart resilience. When the CP restarts, clawkerd's inbound
:7700listener stays up and the dialer re-establishes the Session once the new CP boots and reaches the container. The agent registry sqlite DB persists across CP restarts so identity is preserved without re-bootstrapping.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

