anomaly-detection

Core Principles

1. Define normal first — You cannot detect anomalies without a clear model of what "normal" looks like. Start by characterizing the baseline.
2. Context over thresholds — A value that's anomalous in one context is expected in another. Day-of-week, seasonality, and deployment events all shift what's normal.
3. False positives are expensive — An alerting system that cries wolf gets ignored. Optimize for precision (low false positives) before recall (catching everything).
4. Anomaly ≠ problem — An anomaly is a statistical observation, not an automatic incident. Always pair detection with investigation.
5. Simple methods first — Z-scores and IQR catch most anomalies. Don't reach for Isolation Forests until simple methods have failed.
6. Stationarity matters — Most statistical methods assume your data doesn't change character over time. If it does, you need methods that handle trend, seasonality, or concept drift.

Methodology Reference

Tier 1: Statistical Methods (Start Here)

Use these when data is well-understood, roughly stationary, and you can define "normal" numerically.

When to use Tier 1: You have a single metric, the distribution is understandable, and you can eyeball what "normal" looks like. Most operational alerting starts here.

Tier 2: Time Series Methods

Use when your data has trend, seasonality, or autocorrelation that invalidates simple statistical bounds.

When to use Tier 2: Your metric has daily, weekly, or seasonal patterns. A flat threshold would fire every Monday morning or every holiday.

Tier 3: Machine Learning Methods

Use when the data is multivariate, the patterns are complex, or statistical methods can't capture what "normal" means.

When to use Tier 3: Simple methods produce too many false positives, the data is multivariate, or "normal" is a complex region in feature space — not a simple range.

Selection Strategy

Follow this decision tree to choose a method:

1. Is the data univariate (single metric)?
   ├── Yes → Is there seasonality or trend?
   │         ├── No  → Tier 1: Z-Score, IQR, or Control Chart
   │         └── Yes → Tier 2: STL + Residual, SARIMA, or Prophet
   └── No (multivariate) → 
       Is labeled anomaly data available?
       ├── Yes → Supervised classifier (beyond this skill's scope)
       └── No  → Tier 3: Isolation Forest, LOF, or Autoencoder

2. How much data is available?
   ├── < 100 points   → Tier 1 only (not enough for ML)
   ├── 100–10,000     → Tier 1 or Tier 2
   └── > 10,000       → Any tier is viable

3. What's the cost of a false positive vs. false negative?
   ├── FP is expensive (alert fatigue, manual investigation)
       → Favor higher thresholds, conservative methods (IQR, Grubbs')
   └── FN is expensive (missed fraud, missed outage)
       → Favor lower thresholds, ensemble methods

Process

Step 1: Characterize the Data

Before selecting a method, understand what you're working with:

## Data Profile

**Source**: <e.g., "Prometheus metric: api_request_latency_p99">
**Type**: <univariate | multivariate> — <time series | tabular | event stream>
**Volume**: <row/point count, time range>
**Frequency**: <per-second | per-minute | daily | irregular>
**Distribution**: <roughly normal | skewed | heavy-tailed | multimodal | unknown>
**Stationarity**: <stationary | trend | seasonal | both | unknown>
**Known patterns**: <e.g., "higher on weekdays, spikes during deploys">
**Labeled anomalies available?**: <yes (n examples) | no>

Step 2: Define What "Anomalous" Means

An anomaly is domain-specific. Define it before applying any method:

## Anomaly Definition

**Point anomaly**: A single observation that deviates significantly
  → Example: "latency > 500ms when baseline is 50ms"

**Contextual anomaly**: Normal in one context, abnormal in another
  → Example: "1000 rps is normal at 2pm, anomalous at 3am"

**Collective anomaly**: A sequence of observations that together are unusual
  → Example: "5 consecutive requests from the same IP to /admin"

**Detection goal**: <Which type(s) are we looking for?>
**Action on detection**: <Alert | Log | Auto-remediate | Queue for investigation>

Step 3: Establish the Baseline

Build a model of "normal" before looking for deviations:

• For statistical methods: Compute mean, median, std dev, percentiles on a clean reference period (no known incidents)
• For time series methods: Decompose into trend + seasonality + residual on historical data
• For ML methods: Train on a dataset of known-good observations

## Baseline

**Reference period**: <date range used for baseline>
**Exclusions**: <removed known incidents, maintenance windows, deploys>
**Statistics**: mean=<X>, median=<X>, std=<X>, p50=<X>, p95=<X>, p99=<X>
**Seasonality**: <daily cycle? weekly? none?>

Step 4: Select and Apply the Method

Choose a method from the tiers above using the selection strategy. Document the choice:

## Method Selection

**Method**: <name>
**Tier**: <1 | 2 | 3>
**Why this method**: <reasoning tied to data profile>
**Parameters**: <e.g., "Z-score threshold = 3.0", "IQR multiplier = 1.5", "Isolation Forest contamination = 0.01">
**Alternatives considered**: <what else was viable and why it wasn't chosen>

Step 5: Evaluate Results

Every detection method needs calibration. Check these before trusting results:

## Evaluation

**Flagged anomalies**: <count>
**Manually verified (sample of N)**: <true positive rate>
**Known incidents caught**: <X of Y>
**False positive pattern**: <e.g., "flags every deploy; needs deploy-window exclusion">
**Recommended threshold adjustment**: <raise / lower / add context filter>

Step 6: Operationalize (If Building a Pipeline)

For ongoing detection (not one-time analysis):

## Operational Design

**Detection frequency**: <real-time | every 5 min | hourly | daily>
**Baseline refresh**: <static | rolling window of N days | retrain weekly>
**Alert routing**: <PagerDuty | Slack channel | log only>
**Suppression rules**: <e.g., "suppress during deploy windows", "deduplicate within 30 min">
**Escalation**: <when does a detected anomaly become an incident?>
**Dashboard**: <link to Grafana / CloudWatch / custom dashboard>

Output Format

Every anomaly detection analysis should produce:

1. Data Profile — What the data looks like
2. Anomaly Definition — What counts as anomalous in this domain
3. Baseline — What "normal" looks like, with statistics
4. Method Selection — Which method, why, and with what parameters
5. Results — What was flagged, with evaluation of accuracy
6. Operational Design — How to run this ongoing (if applicable)

Combining Methods

Real-world pipelines often layer methods for better results:

Rule of thumb: Don't combine methods to increase sensitivity — combine them to increase precision (fewer false positives).

Tooling Reference

Use the statistics skill for assumption validation (normality tests, distribution checks) before applying Tier 1 methods.

Common Pitfalls