Trace Analysis

Analyze distributed traces in Grafana Tempo using span attributes from packages/api/typescript/src/shared/utils/Tracing.ts.

When to Use

• Debugging production errors or unexpected behavior
• Investigating slow API responses
• Analyzing request flows across services
• Finding error patterns in specific endpoints

When NOT to Use

• Local development debugging (use console logs or debugger)
• Investigating frontend-only issues
• Database performance issues (use Grafana dashboards instead)

Available Span Attributes

Class/Component

Method

Result

Error

TraceQL Queries

Find Errors

// All errors in a module
{ class.module = "appointment" && status = error }

// Specific error type
{ method.error.type = "ValidationError" }

// Errors containing message
{ method.error.message =~ ".*not found.*" }

Performance Analysis

// Slow operations (>1000ms)
{ method.duration.ms > 1000 }

// Slow repository calls
{ class.component = "repository" && method.duration.ms > 500 }

// Slow use cases in specific module
{ class.component = "usecase" && class.module = "user" && method.duration.ms > 2000 }

Filter by Component Layer

// All controller calls
{ class.component = "controller" }

// Repository operations for a module
{ class.component = "repository" && class.module = "appointment" }

// Event handlers
{ class.component = "handler" }

Find Specific Operations

// Specific class and method
{ class.name = "CreateUserUseCase" && method.name = "execute" }

// All save operations
{ method.name = "save" }

// Search by input content
{ method.input.json =~ ".*email@example.com.*" }

Trace Flow Analysis

// Full request trace for a module
{ class.module = "appointment" } | select(class.name, method.name, method.duration.ms)

// Compare async vs sync methods
{ method.type = "async" && class.component = "service" }

Debugging Workflows

Investigating a Failed Request

1. Find the error span:

   { status = error && class.module = "<module>" }

2. Check error details in span attributes:
   • method.error.type - What error class
   • method.error.message - Error description
   • method.error.stack - Stack trace
3. Look at input that caused the error:
   • method.input.json - Arguments passed

Performance Investigation

1. Find slow operations:

   { method.duration.ms > 1000 }

2. Check call hierarchy in trace view:
   • Which component is slowest?
   • Which method within that component?
3. Analyze pattern:

   { class.name = "SlowClass" } | rate() by (method.name)

Request Flow Analysis

1. Get full trace for a request
2. Check component order: router → middleware → controller → service → repository
3. Verify data transformation via method.input.json and method.output.json

Grafana Dashboard Queries

Error Rate by Module

sum(rate(traces_spanmetrics_calls_total{status="STATUS_CODE_ERROR"}[5m])) by (class_module)

P95 Latency by Component

histogram_quantile(0.95, sum(rate(traces_spanmetrics_latency_bucket[5m])) by (le, class_component))

Throughput by Use Case

sum(rate(traces_spanmetrics_calls_total{class_component="usecase"}[5m])) by (class_name)

Debugging Process

Step 1: Identify the Symptom

Determine what you're investigating: error responses (5xx), slow requests, missing data, or unexpected behavior.

Step 2: Choose Query Strategy

Step 3: Analyze Spans

Look for: error status codes, long durations between spans, gaps in the trace, and unexpected span attributes.

Step 4: Identify Root Cause

Read span attributes (input, output, error.message) to understand what went wrong.

Complete Debugging Example

Problem: API returns 500 on POST /appointments

// Step 1: Search error spans
{ resource.service.name = "template-api" } && status = error && span.http.route = "POST /appointments"

// Step 2: Found error span with attributes:
//   error.message: "DOCTOR_NOT_AVAILABLE"
//   input: { doctorId: "abc-123", date: "2024-03-15" }
//   module: "service"

// Step 3: Root cause — doctor availability not checked before scheduling
// Fix: Add availability validation in CreateAppointment use case

Common Issues

Missing Traces

• Check OTEL_COLLECTOR_TRACE_URL in Config
• Verify startTelemetry() called at startup
• Check if class is resolved via DI container (auto-traced via autoTrace)

Truncated Data

• method.input.json and method.output.json are truncated at 10000 characters
• Shows [truncated] suffix when exceeded
• Check method.result.keys for object structure

Unserializable Data

• Shows [unserializable] for circular references or special objects
• Common with streams, Buffers, complex class instances

References

• packages/api/typescript/src/shared/utils/Tracing.ts - Tracing implementation
• packages/api/typescript/src/shared/utils/Config.ts - OTEL configuration