Agent Observability

Instrument before scaling, not after. A system with no telemetry has no maintenance, and a system with no maintenance degrades on a predictable schedule.

When to use

• Before any agent goes from one user to several
• Nobody can answer how something is performing
• Quality complaints arrive with no data behind them
• Before a scale-up decision

Inputs

• Reads: workspace/agents/<name>/brief.md for the success threshold, workspace/evals/log/eval-log.jsonl
• Needs from user: where logs can be written, and who reads the weekly review

Workflow

1. Log the run, not just the output

Every run writes one line. Append-only.

{"ts":"<iso>", "agent":"<name>", "version":"<semver>", "user":"<id>",
 "input_summary":"<short>", "tools_called":[...], "tool_spend":n,
 "duration_s":n, "guardrail_trips":[...], "eval_verdict":"<if judged>",
 "outcome":"kept|edited|discarded"}

outcome is the field that matters and the one that requires a human to supply it. Everything else is automatic and less informative. Whether the user kept the output, edited it, or threw it away is the only direct measure of usefulness, and it has to be captured at the moment of use.

Make it a single keystroke. A three-field feedback form gets filled in for a week and then never again.

2. Track the six numbers

3. Alert on shape, not just on failure

The dangerous failures do not throw errors.

The last one is the one people forget. A monitoring system that stops reporting looks exactly like a healthy quiet week.

4. Diagnose degradation in the right order

When quality drops, check in this sequence. It runs cheapest first and catches the common cases early.

1. Did the inputs change?          data freshness, source changes, a broken sync
2. Did the context go stale?       positioning, proof points, plays out of date
3. Did the spec drift?             run eval-calibration
4. Did usage change?               people using it for something it was not scoped for
5. Did the model or tooling change? last, because it is the least likely and the least fixable

Teams reliably start at step 5 and skip steps 1 and 2, which are the actual cause most of the time.

5. Run the weekly review

Fifteen minutes, one page, a named reader.

AGENT TELEMETRY: week of <date>

<agent>   WAU n/n   runs n   correction n%   cost/run n   trips n%
<agent>   ...

Alerts:            <anything tripped>
Corrections made:  <written to the context portfolio, not to chat>
Decision:          <continue | fix <specific thing> | retire>

Corrections get written to the context portfolio. A correction made in a review meeting and not written to a file is lost, and the same correction will be made again next month.

6. Retire on the kill criterion

The brief named a kill criterion. At the review date, apply it. An unused agent is not neutral, because its description competes for selection with the ones that work.

Output

• Writes: workspace/evals/telemetry-spec.md, workspace/evals/log/runs.jsonl, the weekly review
• Prints: the six numbers, any alerts, and the decision

Rules & quality bar

• Instrument before scaling. Not after the first complaint
• The outcome field is captured from the human, in one keystroke
• Append-only logs, including failures
• Alert on shape and on silence, not only on errors
• Diagnose inputs and context before blaming the model. In that order
• Corrections write to the context portfolio
• A named reader for the weekly review, or it stops happening
• Apply the kill criterion on schedule

Related skills

• Requires: agent-rollout for the intended-user denominator
• Pairs with: eval-loop, eval-calibration, credit-and-cost-control
• See also: docs/metrics.md, docs/field-notes.md story 11