datadog-metrics-investigator
Skill1 starsChanged 37 days agoMITView on GitHub
Trigger: "investigate metric X spike", "why did Y metric change", "analyse Z metric by tag", "is metric W healthy", saturation, capacity or Cloud Cost metrics. Also the metrics step of incident triage. Read-only; report goes to ops/datadog/triage/.
In this repository13 other files it gives its agents.
- AGENTS.mdAGENTS.md
- capture-insightsSkill
- create-runbookSkill
- datadog-logs-analyzerSkill
- datadog-report-publisherSkill
- datadog-service-healthSkill
- datadog-triageSkill
- k8s-error-analyzerSkill
- k8s-optimizer-costSkill
- meeting-notes-enhanceSkill
- runSkill
- second-opinionSkill
- stack-updaterSkill
The file
We split this skill into the rules it gives an agent
11 rules. About 0 are common, 4 are distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 10
This skill is read-only against Datadog: All output lives exclusively in the Obsidian vault under {{DATADOG_TRIAGE_DIR}}/.
- Very specific: line 19
If you find yourself about to call any of the above: stop, re-read this block, and instead construct the equivalent metric-explorer URL …
- Distinctive: line 44
Do NOT use this skill for:
- Distinctive: line 83
If the user gave a fuzzy name:
- Very specific: line 118
Call get_datadog_metric with multiple queries in one call (the tool supports it):
- Very specific: line 137
If the delta on any series is > 50% → flag as "significant" in findings.
- Distinctive: line 154
If a covering dashboard already exists (Step 5), prefer linking to the dashboard URL too — the user lands on a curated view rather than a …
- Very specific: line 172
If the metric comes from a service (e.g.
- Distinctive: line 226
Search events for deploys in that window.
- Very specific: line 232
High cardinality group_by: if by {pod_name} returns hundreds of series, cap to top 10 with top(..., 10, 'max', 'desc') formula.
- Very specific: line 235
Metric-explorer URL too long: if group_by cardinality is high and the encoded query exceeds practical URL length, drop the by {…} clause …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Datadog Metrics Investigator
Hard Constraints — READ-ONLY in Datadog
This skill is read-only against Datadog. All output lives exclusively in the Obsidian vault under {{DATADOG_TRIAGE_DIR}}/. Datadog itself is never written to.
You MUST NOT call any of the following Datadog MCP tools, regardless of "just one widget" reasoning or caller request:
create_datadog_notebook— never. This skill does not create notebooks.edit_datadog_notebook— never as part of triage.upsert_datadog_dashboard— never. Permanent dashboards are managed by humans.get_widget,validate_dashboard_widget,ask_widget_expert,get_widget_reference— never. Findings are surfaced as live Datadog metric-explorer / dashboard deep-links inside the Markdown report, not as embedded widgets.
If you find yourself about to call any of the above: stop, re-read this block, and instead construct the equivalent metric-explorer URL (https://app.datadoghq.com/metric/explorer?query=<urlencoded>) and put it in the Evidence table.
Metric-centric investigation skill. Given a metric name (or a fuzzy reference), this skill:
- Locates the metric and inspects its metadata, tags, and related assets.
- Queries timeseries across multiple slicings (by host, service, env, tenant, etc.).
- Compares current window to baseline to quantify the regression.
- Produces a triage report under
{{DATADOG_TRIAGE_DIR}}/with live Datadog UI deep-links embedded for every finding (metric explorer, dashboards, monitors). Knowledge lives in the Obsidian vault; live data is one click away in Datadog.
See docs/triage-architecture.md.
When to Use
Trigger when the user asks to:
- "Why did
<metric>spike at<time>?" - "Investigate
<metric>in staging / prod" - "Break down
<metric>by<tag>" - "Is
<metric>above its baseline?" - "Capacity / saturation check on
<resource>via<metric>" - "How much does
<cloud-cost-metric>show for<tag>?" (uses Cloud Cost Management) - As the metrics-scoped step of
datadog-triage.
Do NOT use this skill for:
- APM latency/error investigation tied to a specific service → use
datadog-service-health(which internally uses span aggregates, not metrics). - Log-pattern investigation → use
datadog-logs-analyzer. - Building permanent dashboards → that work belongs in the Datadog UI by a human; this skill (and the whole triage suite) is read-only against Datadog.
Required MCP
Datadog MCP server (plugin-datadog-datadog) with:
search_datadog_metrics— discover the exact metric name and list candidates.get_datadog_metric_context— metadata, unit, type, available tags, related assets. Setuse_cloud_cost=truefor Cloud Cost metrics.get_datadog_metric— timeseries queries with formulas & functions, multiple queries per call.search_datadog_dashboards— find existing dashboards that already track this metric (avoid re-inventing).search_datadog_monitors— check whether a monitor covers this metric.datadog-report-publisher(delegated skill) for final publication.
Widget-building tools (
get_widget,ask_widget_expert,get_widget_reference,validate_dashboard_widget) are forbidden by the Hard Constraints block above. The publisher writes a Markdown-only artefact with live metric-explorer deep-links and does not embed widgets in a Datadog notebook.
Inputs
Collect from the user; ask only if missing:
- metric (required): full name (
aws.rds.cpuutilization) or fuzzy hint (rds cpu). If fuzzy, callsearch_datadog_metricsfirst and confirm. - time_window (required):
15m,1h,4h,24h,7d. Default1h. - scope_filters (optional): tag filters like
{env: staging, cluster: eks-data}to narrow the query. Applied as{tag:value}in DD query syntax. - group_by (optional): list of tags to slice the metric by (e.g.
[host],[service],[env, cluster]). If omitted, default to[](overall line) and the top 2 high-cardinality tags from metadata. - baseline_window (optional, default
previous equal-length window): for before/after comparison. - incident_id (optional): pass through.
- is_cloud_cost (optional, default
false): setuse_cloud_cost=trueonget_datadog_metric_contextif the metric belongs to Cloud Cost Management.
Workflow
Step 0 — Init audit trail
Track every MCP call (tool, args summary, purpose, timestamp). Include all DD query expressions verbatim so the investigation is reproducible.
Step 1 — Resolve the metric
If the user gave a fuzzy name:
search_datadog_metrics(query="<hint>", limit=15)Pick the best candidate. If multiple are plausible, ask the user. Do not guess silently.
Step 2 — Inspect metric context
get_datadog_metric_context(
metric=<metric>,
use_cloud_cost=<bool>
)Record:
type(gauge | count | rate | distribution) — drives how you aggregate.unit— include in every chart title so numbers are meaningful.- Available tags / dimensions — the shortlist for
group_by. - Integration / source — tells you where to look for related context (AWS, k8s, custom).
- Related assets (dashboards, monitors) — reuse them instead of rebuilding.
Aggregation rule of thumb based on type:
| Metric type | Default aggregation in queries |
| gauge | avg (or max for saturation) |
| count | sum |
| rate | avg |
| distribution | p95 (p99 for tail analysis) |
Step 3 — Query timeseries (overall + sliced)
Call get_datadog_metric with multiple queries in one call (the tool supports it):
- Overall line:
<agg>:<metric>{<scope_filters>} - Sliced by primary group_by:
<agg>:<metric>{<scope_filters>} by {<group_by[0]>}— returns top series. - Formula for % change vs baseline (if helpful): use Datadog's
anomalies(...)or a manualhour_before(...)formula.
Example queries:
avg:kubernetes.cpu.usage.total{env:staging, cluster:eks-data}
avg:kubernetes.cpu.usage.total{env:staging, cluster:eks-data} by {kube_deployment}For the same window, also fetch the baseline window (same expression, shifted back by time_window) so you can compute:
- Current mean / p95 / max.
- Baseline mean / p95 / max.
- Delta and % change.
If the delta on any series is > 50% → flag as "significant" in findings. If > 100% → flag as "major". Derive the report severity from the Severity Guidance table, not from these flags.
Step 4 — Build a Datadog metric-explorer deep-link for each informative query
For the 2–3 most informative queries, construct a metric-explorer URL the user can click from the MD:
https://app.datadoghq.com/metric/explorer?query=<urlencoded query>&from_ts=<window_start_ms>&to_ts=<window_end_ms>URL-encode : (%3A), { (%7B), } (%7D), and spaces (%20). Example:
- Query:
max:kubernetes.memory.usage_pct{env:data-prod,kube_deployment:data-ai-mcp-data-fever-external} by {image_tag} - URL:
https://app.datadoghq.com/metric/explorer?query=max%3Akubernetes.memory.usage_pct%7Benv%3Adata-prod%2Ckube_deployment%3Adata-ai-mcp-data-fever-external%7D+by+%7Bimage_tag%7D
These URLs go in the Evidence table and in the curated datadog_links header list. They render the live chart with one click — no widget needed.
If a covering dashboard already exists (Step 5), prefer linking to the dashboard URL too — the user lands on a curated view rather than a raw explorer.
Per the Hard Constraints block, you must not call
get_widget/validate_dashboard_widget/ask_widget_expert/get_widget_referencehere. The metric-explorer URL is the deliverable, not a Datadog widget.
Step 5 — Reuse existing dashboards / monitors
search_datadog_dashboards(query="<metric fragment or owning team>")
search_datadog_monitors(query="<metric fragment>")Capture:
- Any dashboard already tracking this metric → include its link in the report so the user can open it.
- Whether a monitor covers the metric; if not → log an observability gap in P2 recommendations.
Step 6 — Correlate with services (optional)
If the metric comes from a service (e.g. a custom app.latency.custom), cross-check:
search_datadog_services(query="<service inferred from tags>")And suggest running datadog-service-health as a follow-up if the metric suggests application-level issues.
Step 7 — Synthesise findings
Payload for datadog-report-publisher:
scope:{ env (if single), service (if single), resource: <metric-name>, severity, incident_id }findings:Executive Summary: brief. Current state, delta vs baseline, top offender slice(s), probable driver (load / capacity / downstream / config change), monitor coverage status.Evidence: table (query expression → current value → baseline value → delta % → metric-explorer URL or dashboard URL). Every row must have a clickable Datadog deep-link.Timeline: when the regression began, correlated events if any (fromsearch_datadog_events).Root Cause: if clear; otherwise "hypothesis: X — see follow-ups".Recommendations:- P1: immediate mitigation (scale up, rollback, shed load).
- P2: fix the driver, add monitor if missing.
- P3: capacity planning, SLO review, architectural changes.
Query Audit Trail: every metric query verbatim with its DD query expression — the most important debugging aid.
datadog_links: 3–6 curated live URLs for the MD header — typically the metric explorer for the dominant query, the metric explorer for the by-group_bybreakdown, and any covering dashboard / monitor URL discovered in Step 5.
Step 8 — Publish
Delegate to datadog-report-publisher. The publisher writes a single Markdown artefact under {{DATADOG_TRIAGE_DIR}}/ with all evidence rows linking back to the live metric explorer / dashboards. Knowledge lives in the Obsidian vault; the embedded UI deep-links keep the user one click away from live charts.
Filename example: 2026-04-21-metric-kubernetes-cpu-usage-total-staging.md.
Severity Guidance
| Severity | Trigger |
critical | Prod saturation metric > 95% for > 10m, OR metric tied to a SEV-1/2 incident, OR > 5× baseline with user impact |
error | > 2× baseline in prod or > 5× in staging, or crossing a known SLO threshold |
warn | > 50% deviation from baseline but within healthy operating range |
info | Metric inspected as routine check; no regression |
Example Usage
User: "Why did kubernetes.memory.usage spike in the data-prod cluster around 14:00 today?"
Agent flow:
- Resolve:
search_datadog_metrics(query="kubernetes.memory.usage")→ confirm exact name. - Context:
get_datadog_metric_context(metric="kubernetes.memory.usage")→ gauge, unit bytes, tags includekube_deployment, pod_name, cluster. - Time window: user said "around 14:00 today" → query 13:00–15:00 today vs 13:00–15:00 yesterday as baseline.
- Queries (one
get_datadog_metriccall with multiple):- Overall:
avg:kubernetes.memory.usage{cluster:data-prod} - Sliced:
avg:kubernetes.memory.usage{cluster:data-prod} by {kube_deployment}
- Overall:
- Build a metric-explorer URL for the overall query and one for the
by {kube_deployment}slice; capture both for the Evidence table and thedatadog_linksheader. - Find covering dashboards/monitors and add their URLs.
- Search events for deploys in that window.
- Build findings → publish (Markdown-only).
Edge Cases
- Metric name not found:
search_datadog_metricsreturns nothing → stop, report to user with suggestions from the top of the fuzzy search. - High cardinality group_by: if
by {pod_name}returns hundreds of series, cap to top 10 withtop(..., 10, 'max', 'desc')formula; document the truncation. - Distribution metrics: avg-of-distribution is usually not what you want; default to
p95:and note in the report. - Cloud Cost: remember
use_cloud_cost=trueonget_datadog_metric_context; these metrics have different aggregations (usuallysumover time windows). - Metric-explorer URL too long: if
group_bycardinality is high and the encoded query exceeds practical URL length, drop theby {…}clause from the link and keep the breakdown in the Evidence table as text — the link still opens the right starting point.
Best Practices
- Every query must state its aggregation explicitly (
avg:/sum:/p95:). Implicit aggregation is a common source of misreading data. - Baseline comparison is mandatory — one number is meaningless without a reference.
- Top-N slicing early — an overall line hides the offender; slice by the highest-cardinality tag from metadata.
- Reuse existing dashboards — link them in the report so the user can continue interactively.
- Unit in every chart title / Evidence row — "CPU 0.85" is not interpretable; "CPU cores (avg) 0.85" is.
- Every Evidence row has a clickable URL — metric explorer, dashboard, or monitor. No row should be a dead end.
- Frozen artefact — a published triage note's evidence and analysis do not change; the only permitted edits are
status:and a section appended at the bottom (Resolution, Promoted to runbook), per docs/triage-architecture.md. Follow-ups get new notes.
Dependencies
Required:
- Datadog MCP server
plugin-datadog-datadog. datadog-report-publisherskill installed.
Optional:
datadog-service-health(for service-scoped follow-up).create-runbook(when the same regression repeats).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

