Dark Factory — Observability Standard (the agents' eyes)

Overview

Observability is the sensory apparatus the agents use to build, deploy, and test — not optional garnish. Emitting telemetry is necessary but not sufficient: a :9090/metrics endpoint nobody can see is not observability. This standard makes the consumable surface a named, verified deliverable.

Rule of thumb: if an agent cannot answer "did scenario TS-0x run, and where did it succeed or fail?" by looking at a dashboard or running one query, the system is not observable yet — no matter how many metrics it emits.

The three legs — Emit → Surface → Act

Emission is platform default (deltas only). Surface is product-specific and must be delivered + verified — that is the gap this standard closes.

Required baseline dashboard set (the floor)

Flow/saga (end-to-end path of each PO scenario) · Throughput · Tiered failure log filtered by the log level field ("level":"ERROR"), not a substring (?i)error (substring matching floods false positives from name collisions) · Correlation/trace lookup (a $correlationId variable pulling every log + span across services) · DLQ + queue depth · Latency (p50/p95/p99) · System health. Panels keyed to PO scenarios.

The acceptance bar — "verified rendering live data"

A dashboard that POSTs valid JSON but shows "No data" is not delivered. Every view reachable at a known URL; every panel renders live data (against the datasource, not just schema-valid); $correlationId returns real cross-service results; every alert rule evaluates. Verification is by observation, not assertion.

Anti-patterns

• Emit-only — /metrics exists, no dashboard renders it → QA can't capture evidence.
• Schema-valid, no data — dashboard JSON loads but every panel shows "No data".
• Substring failure filters — (?i)error instead of the level field.
• Hand-clicked dashboards — not in version control as code → lost on cluster rebuild.
• Human-only access — reachable by a human in prod, but the QA/Dev agent has no programmatic path in dev/test.