observability-queries
Skill0 starsChanged 6 months agoMITView on GitHub
Canonical LogsQL, PromQL, and Jaeger-compatible trace queries against the harness Victoria stack. Use when investigating a bug via logs, metrics, traces, building an evidence bundle, or wiring a new alert. Includes copy-pasteable curl examples and gotchas: severity, not level; case-sensitive enum; mandatory `| fields` projection.
In this repository24 other files it gives its agents.
- AGENTS.mdAGENTS.md
- architectureSkill
- before-after-evidenceSkill
- chrome-devtools-deepSkill
- ci-optimizationSkill
- configurationSkill
- continuous-deliverySkill
- dependenciesSkill
- developer-experienceSkill
- documentationSkill
- drySkill
- environmentsSkill
- error-handlingSkill
- loggingSkill
- orchestrating-a-vps-agent-swarmSkill
- playwright-debugSkill
- principles-and-patternsSkill
- purpose-and-scopeSkill
- running-experimentsSkill
- securitySkill
- software-complexitySkill
- testingSkill
- typesSkill
- versioningSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 3 are distinctive, and 6 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 11
Run from the repository root after just bootstrap:
- Very specific: line 29
If just is not available in an automation context, use the slot binary directly:
- Very specific: line 36
Do not use pnpm harness inspect in this template.
- Very specific: line 42
LogsQL has no
|~regex-pipe operator: Use field:/regex/ or bare "word" for text match. - Distinctive: line 81
Returns: every log line emitted under one trace_id.
- Very specific: line 130
Add , service.name to the sum by (...) group to get per-service p95.
- Distinctive: line 159
Run Q3 to fetch the log lines for that trace_id.
- Distinctive: line 160
Run Q1 filtered to that service to pull surrounding ERROR lines for context.
- Very specific: line 161
Run Q6 and open the trace in the Jaeger-compatible UI at http://localhost:${VT_PORT}/select/jaeger/...
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Observability queries: the 6 shapes you'll actually use
Discover stack identity and ports
Run from the repository root after just bootstrap:
just stack inspect
eval "$(just stack ports)"
ISO=$(just stack inspect | awk -F: '/Iso key:/ {gsub(/^ +/, "", $2); print $2}')
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:${OTEL_OTLP_PORT}"just stack inspect is the human-readable status entrypoint. just stack ports prints shell-safe KEY=value lines for the current worktree, including:
VL_PORT=...
VM_PORT=...
VT_PORT=...
OTEL_OTLP_PORT=...If just is not available in an automation context, use the slot binary directly:
eval "$(harness/stack/bin/stack ports)"
ISO=$(harness/stack/bin/stack inspect | awk -F: '/Iso key:/ {gsub(/^ +/, "", $2); print $2}')Do not use pnpm harness inspect in this template. That lab command is not a root package script here. Host-run apps should set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:${OTEL_OTLP_PORT} before emitting telemetry.
Universal gotchas
severity, notlevel. Filteringlevel:errorsilently returns empty.severityis case-sensitive. Pino emitsINFOandERRORuppercase. Matchseverity:ERRORor use the unionseverity:(ERROR OR error)to handle mixed sources.- LogsQL has no
|~regex-pipe operator. Usefield:/regex/or bare"word"for text match. - Always project with
| fields .... Without it each log returns much larger rows. The projection is mandatory for any query touching more than 10 rows. - OTEL to Prom attribute conversion: dots become underscores.
service.namein logs becomesservice_namein PromQL labels. Metric names keep dots in VictoriaMetrics, for examplehttp.server.duration_bucket, and must be selected as{__name__="http.server.duration_bucket"}. | fieldsdrops_streamand_stream_id. If you need full context for one line, re-query that_timewithout projection.
Q1: LogsQL error scan with | fields projection
Returns: recent error log lines, projected to 4 fields.
curl -sG "http://localhost:${VL_PORT}/select/logsql/query" --data-urlencode \
"query={harness.iso_key=\"$ISO\"} severity:ERROR | fields _time, service.name, severity, _msg | limit 20"Sample response, one NDJSON result per line:
{"_time":"2026-05-14T09:09:10.084033844Z","service.name":"api-cpp","severity":"ERROR","_msg":"migration error: SQL execution timeout"}Q2: LogsQL structured-field filter by service.name
Returns: logs scoped to one service. Use for "what was API doing at T?"
curl -sG "http://localhost:${VL_PORT}/select/logsql/query" --data-urlencode \
"query={harness.iso_key=\"$ISO\",service.name=\"api\"} | fields _time, severity, _msg | limit 20"Sample:
{"_time":"2026-05-14T09:09:12.912Z","severity":"info","_msg":"Server listening at http://0.0.0.0:3000"}The stream selector {service.name="api"} is cheaper than the post-filter _stream:{service.name="api"} because it filters at the index.
Q3: LogsQL by traceID
Returns: every log line emitted under one trace_id. This is the critical step in trace to logs correlation.
TRACE_ID=d17a6f60131d4c23162495a8e4ae5bc3
curl -sG "http://localhost:${VL_PORT}/select/logsql/query" --data-urlencode \
"query={harness.iso_key=\"$ISO\"} trace_id:$TRACE_ID | fields _time, severity, _msg, req.url, res.statusCode | limit 50"Sample:
{"_time":"2026-05-14T16:48:00.408Z","trace_id":"d17a6f60131d4c23162495a8e4ae5bc3","_msg":"incoming request"}
{"_time":"2026-05-14T16:48:00.409Z","trace_id":"d17a6f60131d4c23162495a8e4ae5bc3","_msg":"request completed"}Get the trace_id from a failed Playwright response: response.request().headers()['traceparent'], then take the second hex segment.
Q4: PromQL error rate
Returns: instant rate of requests grouped by HTTP status over the last 5 minutes.
curl -sG "http://localhost:${VM_PORT}/api/v1/query" --data-urlencode \
'query=sum by (http.status_code) (rate({__name__="http.server.duration_count"}[5m]))'Sample:
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"http.status_code":"200"},"value":[1778778648,"0.199"]}]}}The metric name http.server.duration_count keeps OTEL dots, so select it through {__name__="..."}. Raw http.server.duration_count is a PromQL syntax error.
Q5: PromQL latency p95
Returns: 95th-percentile HTTP server latency over the last 5 minutes.
curl -sG "http://localhost:${VM_PORT}/api/v1/query" --data-urlencode \
'query=histogram_quantile(0.95, sum by (le) (rate({__name__="http.server.duration_bucket"}[5m])))'Sample:
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1778778648,"4.749"]}]}}Add , service.name to the sum by (...) group to get per-service p95. Values are in milliseconds for the OTEL HTTP semantic convention default.
Q6: Jaeger-compatible recent traces by service name
Returns: the N most recent traces emitted by a given service.
curl -sG "http://localhost:${VT_PORT}/select/jaeger/api/traces" \
--data-urlencode "service=api" --data-urlencode "limit=10"For one specific trace:
curl -s "http://localhost:${VT_PORT}/select/jaeger/api/traces/$TRACE_ID"List services first if unsure:
curl -s "http://localhost:${VT_PORT}/select/jaeger/api/services"VictoriaTraces exposes the Jaeger HTTP API, not raw TraceQL HTTP. This is by design. The compose template uses VictoriaTraces and the OTEL collector exports to /api/v2/spans.
Composed workflow: failed Playwright response to root-cause log
- Playwright catches a
requestfailedevent or 5xxresponse. - Pull
traceparentfromresponse.request().headers(); the trace_id is segment 2. - Run Q3 to fetch the log lines for that trace_id.
- Run Q1 filtered to that service to pull surrounding ERROR lines for context.
- Run Q6 and open the trace in the Jaeger-compatible UI at
http://localhost:${VT_PORT}/select/jaeger/...to see span timing.
See .claude/skills/before-after-evidence/SKILL.md for the full trace-correlation recipe.
Verification
These query shapes were ported from the lab observability skill and adapted to this template's stack-manager entrypoints. For an executable end-to-end smoke against this template, run:
just observability-smokeDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

