Nexus Observability — API Failure Correlation Engine

Systematically correlate failures across distributed services to identify causal chains, blast radius, and the origin service — before proposing any fix.

Compatibility

• Supporting files: checklists/investigation-checklist.md, anti-patterns/common-mistakes.md, validation/output-validation.md
• Required tools: Read, Bash, Grep
• Optional tools: WebSearch (vendor-specific error lookups)
• Hands off to: nexus:debugging once origin service is identified; the roadmap-planner agent after fix is agreed upon

Core Principle

Never assign blame before building the full cross-service timeline. A cascade always has one origin — fixing a victim while the cause is live means the failure recurs.

Workflow

Step 1 — Context Acquisition

Collect before reading any logs. Require items 1–5 minimum; ask for all in one message:

Step 2 — Timeline Construction

Align all timestamps to UTC. Use first occurrence per service, not peak. Trust trace timestamps over log timestamps when they differ. Note data gaps explicitly — they are blind spots, not clean windows.

| Time (UTC) | Service     | Event                              | Error Rate |
|------------|-------------|------------------------------------|------------|
| T+00:00    | database    | connection pool at 95% utilization | 0%         |
| T+00:47    | database    | connection pool exhausted          | 0%         |
| T+01:03    | api-gateway | upstream timeouts begin            | 2%         |
| T+01:15    | auth-service| DB queries failing, JWT slow       | 8%         |
| T+01:22    | user-service| auth check timeouts                | 34%        |
| T+01:31    | api-gateway | circuit breaker opens on auth      | 78%        |

Step 3 — Identify the Origin Service

The origin has the earliest elevated error signal. Confirm with all three tests:

State the origin as: "[Service X] is the origin. Its [metric] saturated at [T+XX:XX], [N]s before the first downstream error in [Service Y]."

If two candidates remain ambiguous, state both — do not guess.

Step 4 — Map Dependency Graph and Failure Mode

Classify using this table:

Draw the dependency graph and mark each node: ORIGIN, PROPAGATOR, or VICTIM.

[database] ← [auth-service] ← [api-gateway] ← [user-service]
                              ↑
                          [billing-service]

Step 5 — Root Cause + Blast Radius

Root cause format (required):

"[Service X] failed because [mechanism] when [trigger], causing [failure mode] that propagated to [affected services], with [N]% of [user-facing operation] requests failing by [T+XX:XX]."

Blast radius table:

| Service         | Role        | Peak Error Rate | User Impact            |
|-----------------|-------------|-----------------|------------------------|
| database        | ORIGIN      | 100% timeouts   | Internal only          |
| auth-service    | PROPAGATOR  | 45%             | Login failures         |
| api-gateway     | PROPAGATOR  | 78%             | All authenticated reqs |
| user-service    | VICTIM      | 34%             | Profile page errors    |

Fix table (one action per service):

| Service      | Immediate Action                          | Prevention                          |
|--------------|-------------------------------------------|-------------------------------------|
| database     | Increase pool limit; kill idle conns      | Alert at 80% pool utilization       |
| auth-service | Add circuit breaker on DB calls           | Cache JWT verify results for 30s    |
| api-gateway  | Tune circuit breaker thresholds           | Add fallback for auth timeout       |

Signal Patterns

Specialization Notes

Service meshes (Istio/Linkerd): Check control plane health first — a sick control plane degrades all data plane services simultaneously. 503 upstream connect error and upstream reset before response started are cascade signatures. Envoy proxy logs often contain the causal chain before application logs do.

Databases (PostgreSQL/MySQL): Check max_connections/pg_stat_activity — connection exhaustion is the most common DB-originated cascade. Check slow query log (one long query can block all connection slots). Check replication lag — replica failures appear before primary errors.

Caches (Redis/Memcached): Check rejected_connections — full connection pool causes DB fallback storms. Check memory/evictions — maxmemory-policy evictions cause cache misses that look like cache failure. Check slowlog — an expensive command (LRANGE, SMEMBERS) blocks all other operations.

API gateways/load balancers: Check backend health check logs — misconfigured health checks mark healthy backends as down. Check connection timeout vs. read timeout mismatch — causes partial failure modes that are hard to correlate.

Output Contract

Every investigation closes with this report. Fill every field; write none if not applicable — never leave blank:

## Failure Correlation Report

**Outcome:**          [one-line summary]
**Failure Mode:**     [Linear cascade | Fan-out | Independent | Thundering herd | Brownout]
**Origin Service:**   [service name + specific metric/event that triggered failure]
**Trigger:**          [what caused the origin service to fail]
**Timeline:**         [inline table T+00:00 to resolution]
**Causal Chain:**     [A → B → C with failure mode at each hop]
**Blast Radius:**     [table: service | role | peak error rate | user impact]
**Fix — Immediate:**  [per-service immediate actions]
**Fix — Prevention:** [per-service prevention measures]
**Blind Spots:**      [services or time windows with no data]
**Next Step:**        [one clear action for the user]

Anti-Patterns

Read anti-patterns/common-mistakes.md before starting any investigation.

• Never blame a downstream service without proving its upstream was healthy at failure time.
• Never construct a timeline from summarized logs — raw timestamps only.
• Never propose a fix before completing Steps 1–5.
• Never skip the dependency graph — fan-out cascades look like independent failures without it.
• Never mark complete if Blind Spots is empty when data is missing — state what is unknown.
• Never conflate correlation with causation — simultaneous spikes do not imply causation.