Tracing & Observability

Core Question

What happened, and why, at every layer of this request?

Events say what happened. Spans say the context in which it happened. If you only emit events without spans, you lose the "why." tracing unifies both into one structured, async-aware API.

Error → Design Question

Quick Decisions

The Async Span Problem

Never use span.enter() in async code. The guard is tied to the current thread — when the future yields at an .await and resumes on a different thread, the span context is wrong.

// Bad: guard held across await
async fn handle(req: Request) -> Response {
    let span = info_span!("handle_request");
    let _guard = span.enter();
    let data = fetch_data().await; // span context lost here
    process(data)
}

// Good: #[instrument] handles async spans correctly
#[instrument(skip(req))]
async fn handle(req: Request) -> Response {
    let data = fetch_data().await;
    process(data)
}

// Good: .instrument() for dynamic span names
async fn handle(req: Request) -> Response {
    let span = info_span!("handle_request", method = %req.method());
    async {
        let data = fetch_data().await;
        process(data)
    }
    .instrument(span)
    .await
}

Spawned tasks also need explicit span propagation:

let span = info_span!("background_job", job_id = %id);
tokio::spawn(
    async move { run_job().await }.instrument(span)
);

Log Levels

If you would wake someone at 3am for it, it is error. If you would mention it in a standup, it is info. Everything else is debug or trace.

Usage Scenarios

Scenario 1: "I'm starting a new Axum service and need logging" → Set up tracing_subscriber with fmt::layer() for dev and json() for production. Add TraceLayer to your router for per-request spans. Use #[instrument] on handlers. See references/setup.md for init patterns.

Scenario 2: "I have logs but can't tell which belong to the same request" → Extract or generate a request ID in middleware, add it as a span field. All events within that span tree carry the ID. See references/instrumentation.md for correlation patterns.

Scenario 3: "I need traces in Jaeger for production but plain text locally" → Use a layered subscriber: fmt::layer() for local, tracing-opentelemetry for production. Switch via config. See references/production.md for OpenTelemetry setup.

Reference Files

Cross-References