LLM observability

An LLM app fails in ways that logs and error rates do not show. The request returns 200, the answer is wrong, the agent looped nine times, and the bill doubled. To see these you need a trace of every model call and tool call, token counts on each, and a way to connect the trace to quality checks. This skill covers how to record that data, what to measure, and what to keep out of it.

Core Principles

What a trace looks like

invoke_agent support-bot                     (one user request)
  chat model-x                               (model call 1: plans, picks a tool)
  execute_tool search_orders                 (tool call)
  chat model-x                               (model call 2: reads result)
  execute_tool issue_refund                  (tool call)
  chat model-x                               (model call 3: final answer)

Span types to emit:

Parent-child links matter. A tool span nested under the agent span lets you see which step was slow or failed. Put the application's own fields on the run span: feature name, prompt version, release version, a hashed user or tenant id, experiment arm.

Details on the OpenTelemetry conventions and the metrics they define: otel-genai.

Standards status

• The OpenTelemetry GenAI semantic conventions have Development status. They are not stable. Operation names, attribute names, and metric names have changed and can change again.
• They now live in the open-telemetry/semantic-conventions-genai repository. The older pages on opentelemetry.io say they moved.
• Pin your instrumentation library version. Read its changelog before upgrading. Isolate GenAI attribute names in one module or collector config so a rename is a one-place change.
• Your backend may map several conventions (GenAI, OpenInference, vendor-specific). Check that token and cost fields land where you expect after you send a test trace.

What to measure

Use percentiles for latency and per-run totals for cost. A mean hides the 5 percent of runs that loop and cost ten times the average. Details: cost-and-latency.

Cost and latency levers

Prompt caching and batching are the two big levers. Both are measurable from traces.

Provider price multipliers and rules differ by model and change over time. Read the current provider docs for the model in use, see cost-and-latency.

Sampling

Head sampling decides when the trace starts and cannot know whether the run will fail or be slow. Tail sampling decides after the trace is complete. For LLM apps use tail sampling in a collector: keep every trace with an error, high latency, high cost, a human flag, or negative feedback, and a random slice of the rest as a baseline. Compute rates and cost totals from metrics that see all traffic, so sampling does not skew them. See privacy-and-sampling.

Privacy

• Message content, system prompts, tool arguments, and tool results are sensitive. The conventions mark message content as opt-in. Leave it off by default and turn it on per environment.
• Redact before data leaves your process or at a collector you control: emails, phone numbers, names, account numbers, secrets, tokens.
• Hash user identifiers. Never use a raw email as a trace attribute.
• Set retention shorter for content than for metadata. Restrict who can open traces.
• Remember the backend is a data processor. Pick its region and contract to match your data rules.

Full checklist in privacy-and-sampling. For wider security concerns, use security.

Backends

Most LLM observability tools accept OTLP, so you can instrument once and choose a backend later, or send to two through a collector.

Endpoints, headers, and regions change. Copy them from each vendor's current docs. Details and the eval loop: backends-and-eval-loop.

Linking traces to evals

1. Give every production trace a stable id and a release and prompt version.
2. Sample traces for human review and add confirmed failures to the regression dataset, with the trace id as the source.
3. Run the eval suite with the same instrumentation, tagged as an eval run with a case id, so offline and production data share one shape.
4. Attach scores (human labels, judge verdicts, user feedback) to traces and filter by them.
5. Compare prompt versions on the same metrics: cost, latency, steps, score.

Quality checklist

• One trace per request, a span per model call and tool call, parent links correct
• Input, output, cached, and reasoning tokens recorded on each model call
• Cost computed from a versioned price table, grouped by feature and prompt version
• Time to first token and total latency tracked as percentiles
• Agent runs record step count, tool call count, and stop reason, with a step limit
• Message content capture is off by default, redaction runs before export
• User ids hashed, retention and access rules written down
• Tail sampling keeps errors, slow, costly, and flagged traces
• GenAI attribute names confined to one place, instrumentation version pinned
• Test trace sent to the backend and the token and cost fields checked
• Production traces feed the eval dataset on a schedule

Reference Files

Integration with Other Skills