Datadog Monitors

Use this skill to design, debug, and troubleshoot Datadog monitor evaluation logic, query semantics, and alert state management.

When to Use

• When writing Pulumi, Terraform, or API payloads to create or update Datadog monitors.
• To debug Datadog alerts that are stuck in a "ghost" or frozen state.
• When reducing noise and false positives in infrastructure and application monitoring.

When Not to Use

• For installing the actual Datadog Agent on a host (use datadog-agent instead).
• When querying raw log or trace telemetry directly (use datadog-api or datadog-mcp instead).
• If the user simply wants a link to the Datadog UI without altering query definitions.

Common Pitfalls

• Missing Evaluation Timeouts: Forgetting to set a timeoutH on multidimensional alerts, resulting in "ghost alerts" when a pod or host terminates unexpectedly.
• Ignoring Loop Devices: Failing to exclude device_name:loop* in disk monitors, causing a flood of false alerts for 100% full SquashFS mounts.
• Scope Creep: Grouping a monitor by too many high-cardinality tags, causing massive evaluation lag and delayed alerting.

Common Query Issues & Operational Fixes

1. Frozen Multidimensional Alerts (Ghost Alerts)

Problem: Monitors grouping by transient dimensions (e.g., workloads, short-lived hosts, or distinct network domains) remain stuck in an Alert state forever after the underlying entity disappears or stops sending metrics.

Why: Datadog does not auto-resolve an alert group if no new data arrives to bring the metric back below the threshold.

Solution: Enable automatic resolution after a period of missing data using the timeout parameter (in Pulumi: timeoutH) to instruct the engine to drop the alert. Example: timeoutH: 1 auto-resolves if no data is seen for 1 hour.

2. Disk Usage / Forecast False Positives

Problem: Disk monitors trigger on system mounts that are intended to be 100% full, leading to significant alerting noise.

Why: Linux loop devices (used by snap packages) are read-only SquashFS mounts and inherently have no free space. Temporary mount points like tmpfs can also occasionally falsely trigger forecast horizons.

Solution: Filter these devices out directly in the monitor query tag scope. Example: avg:system.disk.in_use{!device_name:loop*,!device_name:tmpfs} by {host,device_name}

3. Missing NPM / Specialized Event Permissions

Problem: When investigating Network Performance Monitoring (NPM) or specialized Formula/Event-based monitors via scripts or MCP, the Datadog API returns Unauthorized or empty datasets.

Solution: Application Keys need explicitly elevated scopes for these datasets. Cloud Network Monitoring requires scopes like network_connections_read and network_health_insights_read, while broader query evaluation relies on timeseries_query and events_read.

Core Principles

• Always filter out known system noise explicitly in the query scope to prevent evaluation bloat.
• Understand the difference between on_missing_data: default (what to do during normal evaluation) and explicit timeframe timeouts (what to do when an alerting entity vanishes).
• When a monitor seems "stuck," inspect the specific tag groupings using the UI or the API (group_states=all) to understand exactly what tags are frozen.

Related Skills

• datadog-pulumi: For mapping these monitor definition principles to Pulumi YAML.
• datadog-api: For raw API state extraction (e.g., checking group states directly to find ghost tags).
• datadog-mcp: For using Model Context Protocol to query underlying telemetry and build queries.