monitoring-checks
Skill59 starsChanged 12 days agoMITView on GitHub
Use when changing or verifying the monitoring stack: OpenTelemetry collector, Prometheus metric labels, OpenSearch indices and event schemas, dashboards, and the verification workflow.
In this repository17 other files it gives its agents.
- AGENTS.mdAGENTS.md
- clawkerd/AGENTS.mdAGENTS.md
- controlplane/AGENTS.mdAGENTS.md
- docs/AGENTS.mdAGENTS.md
- test/AGENTS.mdAGENTS.md
- .github/copilot-instructions.mdCopilot instructions
- agent-filesSkill
- audit-memorySkill
- cli-outputSkill
- dev-checksSkill
- firewall-uatSkill
- initiativeSkill
- store-editorSkill
- test-hunterSkill
- wizard-commandsSkill
- writing-testsSkill
- code-reviewSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 3 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 148
When modifying templates, datasource, workspace, or saved objects:
- Very specific: line 150
Run make test to ensure template tests pass
- Distinctive: line 155
After changes to the telemetry relay, the CP gateway, or a collector lane, run the relay e2e test on the host:
- Very specific: line 160
Verify saved objects landed in the Clawker workspace at http://localhost:5601.
- Distinctive: line 165
CRITICAL: Only query this URL when you need to verify schema changes or discover new event types/fields.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
monitoring-checks
Package contract and runtime UAT: internal/monitor/AGENTS.md.
Backend: logs in OpenSearch — six indices
claude-code(Claude Code OTLP, forwarded by the CP on the gateway lane: receiverotlp/agents, mTLS,ingest_source=cp_gateway),clawker-cli(host CLI OTLP push, untrusted port),clawkercp(mTLS-gated CP push),clawker-envoy(firewall data-plane access logs, mTLS-gated),clawker-coredns(firewall DNS query logs, mTLS-gated), andclawker-ebpf-egress(eBPF per-decision egress events from netlogger,service.name=ebpf-egress, mTLS-gated). Cross-index queries use patternclawkercp,claude-code,clawker-cli,clawker-envoy,clawker-coredns,clawker-ebpf-egress. Traces in OpenSearch SS4O datasettraces/ namespaceclawker(Claude Code beta export gated onCLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1, both env vars baked into the image). Metrics in Prometheus. UIs: OpenSearch Dashboards (:5601) for logs+traces, Prometheus (:9090) for metrics. Resource attributes are written FLAT atresource.*(soresource.service.name,resource.project,resource.agent) by the OTelopensearchexporter(SS4O shape). Theenvelope-normalizeingest pipeline only mirrorsresource.service.name→resource.attributes.service.name(the one path OSD Explore's default log columns hard-code) — other resource attrs are NOT duplicated underresource.attributes.*. Prefer the flatresource.<k>path for queries. Event content lives underattributes.*(attributes.event.name,attributes.tool_name).Stack ships preconfigured —
clawker-opensearch-bootstrap(one-shot compose service, seeinternal/monitor/templates/opensearch-bootstrap/) applies component + index templates, the default ISM retention policy, theclawker_prometheusdirect-query datasource, an OSDClawkerworkspace withfeatures: ["use-case-all"], and Dashboards saved objects (index patterns for every log index + the preconfiguredClaude Codedashboard with KPI strip and filter controls) on everymonitor up. After the import, bootstrap writes each index pattern's field list from the live mapping: OpenSearch Dashboards 3.9 no longer fetches it at load time, and a pattern withoutfieldsmakes every field-naming visualization fail with "Could not locate that index-pattern-field". The visualization surface grows via the NDJSON-export-and-bake workflow intointernal/monitor/templates/opensearch-bootstrap/saved-objects/clawker.ndjson.
Prometheus Metric Labels — type is rewritten to kind
The OTel collector's transform/metrics processor unconditionally renames any datapoint attribute named type to kind before metrics reach the Prometheus exporter. This is a workaround for a bug in the OpenSearch SQL plugin's experimental direct-query Prometheus connector (currently pinned at OpenSearch 3.9.0 — see OpenSearchImage / OpenSearchDashboardsImage in internal/monitor/templates.go).
What it means when querying:
| Metric | Upstream Claude Code label | Stored label in our Prom |
claude_code_token_usage_tokens_total | type=input/output/cacheRead/cacheCreation | kind=input/output/cacheRead/cacheCreation |
claude_code_active_time_seconds_total | type=cli/user | kind=cli/user |
claude_code_lines_of_code_count_total | type=added/removed | kind=added/removed |
Cross-reference with Claude Code's monitoring docs accordingly — the values are unchanged, only the label key differs.
Why: ExecuteDirectQueryActionResponse.parseResult (direct-query/src/main/java/org/opensearch/sql/directquery/transport/model/ExecuteDirectQueryActionResponse.java) uses a rawResult.contains("\"type\":") substring check to decide whether to wrap the Prom response with the Jackson polymorphic discriminator at the JSON root. Any Prom label literally named type flips that check false-positive, the wrap is skipped, and Jackson fails with MismatchedInputException: missing type id property 'type'. The OSD Explore "Metrics" UI is the path that breaks; PPL (source = clawker_prometheus.<metric>) and the native Prom UI at :9090 take separate paths and are unaffected.
Removal criteria: when the pinned OpenSearch / OpenSearch Dashboards image is bumped, re-read parseResult in the file above. If the substring check has been replaced (e.g. with objectMapper.readTree(rawResult).has("type")) or the wrap is unconditional, drop the two rename statements in otel-config.yaml.tmpl's transform/metrics block. No upstream issue tracks this bug at the time of writing; closest neighbor is opensearch-project/sql#5251 (a different scalar-shape deserialization bug in the same PrometheusResult class).
Complete Event Schemas
The field tables below use the flat names emitted by Claude Code OTLP records (
service_name,event_name,tool_name, etc.). When querying OpenSearch, mapevent_name→attributes.event.name,service_name→resource.service.name,project/agent→resource.project/resource.agent, etc. Resource attrs land FLAT underresource.*(canonical); onlyresource.service.nameis mirrored toresource.attributes.service.nameby theenvelope-normalizefinal pipeline so OSD Explore default columns render. Event-time fields stay underattributes.*.
Validated against live data as of 2026-02-13 (Claude Code v2.1.42). Subject to change — consider valid first before querying for updates.
Common OTEL envelope fields (present on ALL events)
service_name, service_version, scope_name, scope_version, detected_level, observed_timestamp, host_arch, os_type, os_version, terminal_type, organization_id, user_account_uuid, user_email, user_id
Common clawker-injected fields (from OTEL_RESOURCE_ATTRIBUTES)
project, agent, session_id
CP gateway stamp: for records on the gateway lane (
claude-code), the CP setsproject,agent, andcontainer_idfrom the verified agent identity. It deletes the client values of these keys before it adds its own values. It also deletes client keys under the reservedclawker.prefix (controlplane/otelgateway/gateway.go). The CP and the collector (resource/agent_gateway) setingest_source=cp_gatewayon these records. The collector setsingest_source=untrusted_otlpon records from the untrustedotlpport (sender-declared, forgeable).
netlogger exception: for
clawker-ebpf-egressrecords (service.name=ebpf-egress),projectandagentare NOT carried viaOTEL_RESOURCE_ATTRIBUTESon a sender. The CP-side netlogger pipeline derives them from the target container'sdev.clawker.{project,agent}Docker labels viaLabelCacheenrichment, keyed on the BPF-attestedcgroup_id. They land as event-time attributes on the record (attributes.project,attributes.agent), not underresource.*.
Common event fields
event_name, event_timestamp, event_sequence
claude_code.tool_result
Logged when a tool completes.
| Field | Type | Description |
tool_name | string | Tool name (e.g., "Bash", "Read", "Edit", "mcp_tool") |
success | string | "true" or "false" |
duration_ms | numeric string | Execution time in milliseconds |
decision_type | string | "accept" or "reject" |
decision_source | string | "config", "user_permanent", "user_temporary", "user_abort", "user_reject" |
tool_result_size_bytes | numeric string | Size of tool result |
error | string | Error message (only present when success=false) |
tool_parameters | JSON string | Tool-specific params (requires OTEL_LOG_TOOL_DETAILS=1). Bash: {bash_command, full_command, timeout, description, sandbox}. MCP: {mcp_server_name, mcp_tool_name}. Skill: {skill_name} |
mcp_server_scope | string | Present for mcp_tool calls |
MCP tool name resolution:
tool_nameis"mcp_tool"for all MCP calls — the actual tool name is inside thetool_parametersJSON string asmcp_tool_name.tool_parametersis a flat string attribute (not pre-parsed by the indexer); extractmcp_tool_namein the query before aggregation.
Field naming discrepancy: Live event data uses
decision_type/decision_source, official docs usedecision/source. Dashboard queries use the live field names.
claude_code.tool_decision
Logged when a tool permission decision is made.
| Field | Type | Description |
tool_name | string | Tool name |
decision | string | "accept" or "reject" |
source | string | "config", "user_permanent", "user_temporary", "user_abort", "user_reject" |
claude_code.api_request
Logged for each API call.
| Field | Type | Description |
model | string | Model ID (e.g., "claude-opus-4-6") |
cost_usd | numeric string | Estimated cost in USD |
duration_ms | numeric string | Request duration |
input_tokens | numeric string | Input tokens |
output_tokens | numeric string | Output tokens |
cache_read_tokens | numeric string | Tokens read from cache |
cache_creation_tokens | numeric string | Tokens for cache creation |
speed | string | e.g., "normal" (not in official docs, present in live data) |
claude_code.api_error
Logged when API request fails.
| Field | Type | Description |
model | string | Model ID |
error | string | Error message |
status_code | string | HTTP status code (may be "undefined") |
duration_ms | numeric string | Request duration |
attempt | numeric string | Retry attempt number |
speed | string | e.g., "normal" |
claude_code.user_prompt
Logged when user submits a prompt.
| Field | Type | Description |
prompt_length | numeric string | Length of prompt |
prompt | string | Prompt content (requires OTEL_LOG_USER_PROMPTS=1, redacted otherwise) |
ebpf.egress
Logged once per BPF egress decision (per-cgroup rate-limited by ratelimit_state). Source: controlplane/firewall/ebpf/netlogger. Resource: service.name=ebpf-egress. Instrumentation scope: clawker.netlogger. Lands in the clawker-ebpf-egress OpenSearch index on the mTLS-gated otlp/infra lane.
Emission contract: every field below is written on every record by default — empty strings and zero numbers ship verbatim. Three attributes are omitted when their source value is absent: dst_ip (when Event.DstIP is invalid), dst_port (when Event.NoDst is true), dst_host (when Event.Domain is empty). Operators partition cleanly via _exists_:attributes.<key> / NOT _exists_:attributes.<key> in OSD. Adding or removing an attribute is a contract change.
| Attribute | Type | Source |
event.name | string | per-emit-site via Event.EmitSite.EventName() — ebpf.egress.{connect,sendmsg,sock_create}. The OS OTLP exporter does not project LogRecord.event_name into the SS4O document; netlogger emits event.name as an attribute too so OSD can filter by it. SetEventName is kept for consumers that honor the OTLP field (e.g. Loki). |
action | string | Event.Verdict.String() (allowed / denied / bypassed) — internal Verdict type stays kernel-accurate; emitted attribute is action for cross-lane parity (CoreDNS + Envoy emit same field name; aligns with ECS / OCSF / Cloudflare / AWS conventions). |
container_id | string | Event.ContainerID (empty on LabelCache miss) |
agent | string | Event.Agent — derived from the container's dev.clawker.agent label by LabelCache enrichment |
project | string | Event.Project — derived from the container's dev.clawker.project label by LabelCache enrichment |
cgroup_id | string | strconv.FormatUint(Event.CgroupID, 10) — opaque kernel identifier; emitted as string so the OS index template maps it as keyword (group/filter dimension) instead of long (metric). Sending a JSON number to a keyword field is officially supported via numeric→string coercion but operator UIs treat numerics as metrics by default, which is wrong for ID-shaped fields. |
bpf_ts_ns | int64 | Event.BPFTsNs (raw bpf_ktime_get_ns) |
dst_ip | string | Event.DstIP.String(). Omitted when !Event.DstIP.IsValid() (sock_create with no_dst=true; defensive guard against an unset address). |
dst_port | string | strconv.FormatUint(uint64(Event.DstPort), 10) — opaque port identifier; emitted as string for the same reason as cgroup_id (keyword dimension, not metric). OSD formats numeric fields with thousands separators ("4,318") which is wrong for an ID-shaped axis. Omitted when Event.NoDst is true (sock_create has no destination port). |
l4_proto | string | stream / dgram / raw (kernel SOCK_* type decoded by l4ProtoString) |
l4_proto_code | int | raw SOCK code (resilient to renames) |
ipv6 | bool | native IPv6 |
ipv4_mapped | bool | ::ffff:x.x.x.x |
no_dst | bool | Event.NoDst — sock_create event with no destination |
dst_host | string | Event.Domain populated via ReverseDNSMap.Lookup(Event.Identity). Omitted when Event.Domain is empty (direct-IP connect, domain outside firewall rules, stale dnsbpf entry); operators filter via NOT _exists_:attributes.dst_host. |
identity | string | strconv.FormatUint(uint64(Event.Identity), 10) — CP-allocated route identity for the resolved domain. Emitted as string for the same keyword-mapping rationale as cgroup_id / dst_port. Operators use it to correlate userspace records with BPF dns_cache / route_map entries when dst_host is empty (direct-IP connect, rule removed mid-flight, stale dnsbpf entry). |
Verification Workflow
When modifying templates, datasource, workspace, or saved objects:
- Validate JSON syntax (parse the template output)
- Run
make testto ensure template tests pass - Rebuild the binary and re-bootstrap the monitoring stack. Always use
--volumes: index templates, ISM, datasources, workspace, and saved objects all apply at bootstrap time and are bound to the OpenSearch volume — a plainmonitor downkeeps the old state.make clawker && clawker monitor init --force && clawker monitor down --volumes && clawker monitor up - After changes to the telemetry relay, the CP gateway, or a collector lane, run the relay e2e test on the host:
Source:go test ./test/e2e -run TestTelemetryRelay_E2E -count=1test/e2e/telemetry_relay_internal_test.go. - Verify saved objects landed in the
Clawkerworkspace athttp://localhost:5601. For dashboards, open Discover/Explore against the relevant index pattern to confirm fields resolve before adjusting visualizations.
Reference
- Official Claude Code monitoring docs: https://code.claude.com/docs/en/monitoring-usage.md
- CRITICAL: Only query this URL when you need to verify schema changes or discover new event types/fields. Do not fetch it routinely.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

