agentleFS
schmitthubschmitthub/clawker/.agents/skills/monitoring-checks/SKILL.md

monitoring-checks

Skill59 starsChanged 12 days agoMITView on GitHub

Use when changing or verifying the monitoring stack: OpenTelemetry collector, Prometheus metric labels, OpenSearch indices and event schemas, dashboards, and the verification workflow.

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 3 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    When modifying templates, datasource, workspace, or saved objects:

    line 148
  2. Very specific:

    Run make test to ensure template tests pass

    line 150
  3. Distinctive:

    After changes to the telemetry relay, the CP gateway, or a collector lane, run the relay e2e test on the host:

    line 155
  4. Very specific:

    Verify saved objects landed in the Clawker workspace at http://localhost:5601.

    line 160
  5. Distinctive:

    CRITICAL: Only query this URL when you need to verify schema changes or discover new event types/fields.

    line 165

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

monitoring-checks

Package contract and runtime UAT: internal/monitor/AGENTS.md.

Backend: logs in OpenSearch — six indices claude-code (Claude Code OTLP, forwarded by the CP on the gateway lane: receiver otlp/agents, mTLS, ingest_source=cp_gateway), clawker-cli (host CLI OTLP push, untrusted port), clawkercp (mTLS-gated CP push), clawker-envoy (firewall data-plane access logs, mTLS-gated), clawker-coredns (firewall DNS query logs, mTLS-gated), and clawker-ebpf-egress (eBPF per-decision egress events from netlogger, service.name=ebpf-egress, mTLS-gated). Cross-index queries use pattern clawkercp,claude-code,clawker-cli,clawker-envoy,clawker-coredns,clawker-ebpf-egress. Traces in OpenSearch SS4O dataset traces / namespace clawker (Claude Code beta export gated on CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1, both env vars baked into the image). Metrics in Prometheus. UIs: OpenSearch Dashboards (:5601) for logs+traces, Prometheus (:9090) for metrics. Resource attributes are written FLAT at resource.* (so resource.service.name, resource.project, resource.agent) by the OTel opensearchexporter (SS4O shape). The envelope-normalize ingest pipeline only mirrors resource.service.name → resource.attributes.service.name (the one path OSD Explore's default log columns hard-code) — other resource attrs are NOT duplicated under resource.attributes.*. Prefer the flat resource.<k> path for queries. Event content lives under attributes.* (attributes.event.name, attributes.tool_name).

Stack ships preconfigured — clawker-opensearch-bootstrap (one-shot compose service, see internal/monitor/templates/opensearch-bootstrap/) applies component + index templates, the default ISM retention policy, the clawker_prometheus direct-query datasource, an OSD Clawker workspace with features: ["use-case-all"], and Dashboards saved objects (index patterns for every log index + the preconfigured Claude Code dashboard with KPI strip and filter controls) on every monitor up. After the import, bootstrap writes each index pattern's field list from the live mapping: OpenSearch Dashboards 3.9 no longer fetches it at load time, and a pattern without fields makes every field-naming visualization fail with "Could not locate that index-pattern-field". The visualization surface grows via the NDJSON-export-and-bake workflow into internal/monitor/templates/opensearch-bootstrap/saved-objects/clawker.ndjson.

Prometheus Metric Labels — type is rewritten to kind

The OTel collector's transform/metrics processor unconditionally renames any datapoint attribute named type to kind before metrics reach the Prometheus exporter. This is a workaround for a bug in the OpenSearch SQL plugin's experimental direct-query Prometheus connector (currently pinned at OpenSearch 3.9.0 — see OpenSearchImage / OpenSearchDashboardsImage in internal/monitor/templates.go).

What it means when querying:

MetricUpstream Claude Code labelStored label in our Prom
claude_code_token_usage_tokens_totaltype=input/output/cacheRead/cacheCreationkind=input/output/cacheRead/cacheCreation
claude_code_active_time_seconds_totaltype=cli/userkind=cli/user
claude_code_lines_of_code_count_totaltype=added/removedkind=added/removed

Cross-reference with Claude Code's monitoring docs accordingly — the values are unchanged, only the label key differs.

Why: ExecuteDirectQueryActionResponse.parseResult (direct-query/src/main/java/org/opensearch/sql/directquery/transport/model/ExecuteDirectQueryActionResponse.java) uses a rawResult.contains("\"type\":") substring check to decide whether to wrap the Prom response with the Jackson polymorphic discriminator at the JSON root. Any Prom label literally named type flips that check false-positive, the wrap is skipped, and Jackson fails with MismatchedInputException: missing type id property 'type'. The OSD Explore "Metrics" UI is the path that breaks; PPL (source = clawker_prometheus.<metric>) and the native Prom UI at :9090 take separate paths and are unaffected.

Removal criteria: when the pinned OpenSearch / OpenSearch Dashboards image is bumped, re-read parseResult in the file above. If the substring check has been replaced (e.g. with objectMapper.readTree(rawResult).has("type")) or the wrap is unconditional, drop the two rename statements in otel-config.yaml.tmpl's transform/metrics block. No upstream issue tracks this bug at the time of writing; closest neighbor is opensearch-project/sql#5251 (a different scalar-shape deserialization bug in the same PrometheusResult class).

Complete Event Schemas

The field tables below use the flat names emitted by Claude Code OTLP records (service_name, event_name, tool_name, etc.). When querying OpenSearch, map event_name → attributes.event.name, service_name → resource.service.name, project/agent → resource.project/resource.agent, etc. Resource attrs land FLAT under resource.* (canonical); only resource.service.name is mirrored to resource.attributes.service.name by the envelope-normalize final pipeline so OSD Explore default columns render. Event-time fields stay under attributes.*.

Validated against live data as of 2026-02-13 (Claude Code v2.1.42). Subject to change — consider valid first before querying for updates.

Common OTEL envelope fields (present on ALL events)

service_name, service_version, scope_name, scope_version, detected_level, observed_timestamp, host_arch, os_type, os_version, terminal_type, organization_id, user_account_uuid, user_email, user_id

Common clawker-injected fields (from OTEL_RESOURCE_ATTRIBUTES)

project, agent, session_id

CP gateway stamp: for records on the gateway lane (claude-code), the CP sets project, agent, and container_id from the verified agent identity. It deletes the client values of these keys before it adds its own values. It also deletes client keys under the reserved clawker. prefix (controlplane/otelgateway/gateway.go). The CP and the collector (resource/agent_gateway) set ingest_source=cp_gateway on these records. The collector sets ingest_source=untrusted_otlp on records from the untrusted otlp port (sender-declared, forgeable).

netlogger exception: for clawker-ebpf-egress records (service.name=ebpf-egress), project and agent are NOT carried via OTEL_RESOURCE_ATTRIBUTES on a sender. The CP-side netlogger pipeline derives them from the target container's dev.clawker.{project,agent} Docker labels via LabelCache enrichment, keyed on the BPF-attested cgroup_id. They land as event-time attributes on the record (attributes.project, attributes.agent), not under resource.*.

Common event fields

event_name, event_timestamp, event_sequence

claude_code.tool_result

Logged when a tool completes.

FieldTypeDescription
tool_namestringTool name (e.g., "Bash", "Read", "Edit", "mcp_tool")
successstring"true" or "false"
duration_msnumeric stringExecution time in milliseconds
decision_typestring"accept" or "reject"
decision_sourcestring"config", "user_permanent", "user_temporary", "user_abort", "user_reject"
tool_result_size_bytesnumeric stringSize of tool result
errorstringError message (only present when success=false)
tool_parametersJSON stringTool-specific params (requires OTEL_LOG_TOOL_DETAILS=1). Bash: {bash_command, full_command, timeout, description, sandbox}. MCP: {mcp_server_name, mcp_tool_name}. Skill: {skill_name}
mcp_server_scopestringPresent for mcp_tool calls

MCP tool name resolution: tool_name is "mcp_tool" for all MCP calls — the actual tool name is inside the tool_parameters JSON string as mcp_tool_name. tool_parameters is a flat string attribute (not pre-parsed by the indexer); extract mcp_tool_name in the query before aggregation.

Field naming discrepancy: Live event data uses decision_type/decision_source, official docs use decision/source. Dashboard queries use the live field names.

claude_code.tool_decision

Logged when a tool permission decision is made.

FieldTypeDescription
tool_namestringTool name
decisionstring"accept" or "reject"
sourcestring"config", "user_permanent", "user_temporary", "user_abort", "user_reject"

claude_code.api_request

Logged for each API call.

FieldTypeDescription
modelstringModel ID (e.g., "claude-opus-4-6")
cost_usdnumeric stringEstimated cost in USD
duration_msnumeric stringRequest duration
input_tokensnumeric stringInput tokens
output_tokensnumeric stringOutput tokens
cache_read_tokensnumeric stringTokens read from cache
cache_creation_tokensnumeric stringTokens for cache creation
speedstringe.g., "normal" (not in official docs, present in live data)

claude_code.api_error

Logged when API request fails.

FieldTypeDescription
modelstringModel ID
errorstringError message
status_codestringHTTP status code (may be "undefined")
duration_msnumeric stringRequest duration
attemptnumeric stringRetry attempt number
speedstringe.g., "normal"

claude_code.user_prompt

Logged when user submits a prompt.

FieldTypeDescription
prompt_lengthnumeric stringLength of prompt
promptstringPrompt content (requires OTEL_LOG_USER_PROMPTS=1, redacted otherwise)

ebpf.egress

Logged once per BPF egress decision (per-cgroup rate-limited by ratelimit_state). Source: controlplane/firewall/ebpf/netlogger. Resource: service.name=ebpf-egress. Instrumentation scope: clawker.netlogger. Lands in the clawker-ebpf-egress OpenSearch index on the mTLS-gated otlp/infra lane.

Emission contract: every field below is written on every record by default — empty strings and zero numbers ship verbatim. Three attributes are omitted when their source value is absent: dst_ip (when Event.DstIP is invalid), dst_port (when Event.NoDst is true), dst_host (when Event.Domain is empty). Operators partition cleanly via _exists_:attributes.<key> / NOT _exists_:attributes.<key> in OSD. Adding or removing an attribute is a contract change.

AttributeTypeSource
event.namestringper-emit-site via Event.EmitSite.EventName() — ebpf.egress.{connect,sendmsg,sock_create}. The OS OTLP exporter does not project LogRecord.event_name into the SS4O document; netlogger emits event.name as an attribute too so OSD can filter by it. SetEventName is kept for consumers that honor the OTLP field (e.g. Loki).
actionstringEvent.Verdict.String() (allowed / denied / bypassed) — internal Verdict type stays kernel-accurate; emitted attribute is action for cross-lane parity (CoreDNS + Envoy emit same field name; aligns with ECS / OCSF / Cloudflare / AWS conventions).
container_idstringEvent.ContainerID (empty on LabelCache miss)
agentstringEvent.Agent — derived from the container's dev.clawker.agent label by LabelCache enrichment
projectstringEvent.Project — derived from the container's dev.clawker.project label by LabelCache enrichment
cgroup_idstringstrconv.FormatUint(Event.CgroupID, 10) — opaque kernel identifier; emitted as string so the OS index template maps it as keyword (group/filter dimension) instead of long (metric). Sending a JSON number to a keyword field is officially supported via numeric→string coercion but operator UIs treat numerics as metrics by default, which is wrong for ID-shaped fields.
bpf_ts_nsint64Event.BPFTsNs (raw bpf_ktime_get_ns)
dst_ipstringEvent.DstIP.String(). Omitted when !Event.DstIP.IsValid() (sock_create with no_dst=true; defensive guard against an unset address).
dst_portstringstrconv.FormatUint(uint64(Event.DstPort), 10) — opaque port identifier; emitted as string for the same reason as cgroup_id (keyword dimension, not metric). OSD formats numeric fields with thousands separators ("4,318") which is wrong for an ID-shaped axis. Omitted when Event.NoDst is true (sock_create has no destination port).
l4_protostringstream / dgram / raw (kernel SOCK_* type decoded by l4ProtoString)
l4_proto_codeintraw SOCK code (resilient to renames)
ipv6boolnative IPv6
ipv4_mappedbool::ffff:x.x.x.x
no_dstboolEvent.NoDst — sock_create event with no destination
dst_hoststringEvent.Domain populated via ReverseDNSMap.Lookup(Event.Identity). Omitted when Event.Domain is empty (direct-IP connect, domain outside firewall rules, stale dnsbpf entry); operators filter via NOT _exists_:attributes.dst_host.
identitystringstrconv.FormatUint(uint64(Event.Identity), 10) — CP-allocated route identity for the resolved domain. Emitted as string for the same keyword-mapping rationale as cgroup_id / dst_port. Operators use it to correlate userspace records with BPF dns_cache / route_map entries when dst_host is empty (direct-IP connect, rule removed mid-flight, stale dnsbpf entry).

Verification Workflow

When modifying templates, datasource, workspace, or saved objects:

  1. Validate JSON syntax (parse the template output)
  2. Run make test to ensure template tests pass
  3. Rebuild the binary and re-bootstrap the monitoring stack. Always use --volumes: index templates, ISM, datasources, workspace, and saved objects all apply at bootstrap time and are bound to the OpenSearch volume — a plain monitor down keeps the old state.
    make clawker && clawker monitor init --force && clawker monitor down --volumes && clawker monitor up
  4. After changes to the telemetry relay, the CP gateway, or a collector lane, run the relay e2e test on the host:
    go test ./test/e2e -run TestTelemetryRelay_E2E -count=1
    Source: test/e2e/telemetry_relay_internal_test.go.
  5. Verify saved objects landed in the Clawker workspace at http://localhost:5601. For dashboards, open Discover/Explore against the relevant index pattern to confirm fields resolve before adjusting visualizations.

Reference

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.