Go Troubleshooting

Compatibility: Baseline Go 1.27 (see COMPATIBILITY.md). runtime/trace.FlightRecorder Go 1.25+; debug.SetCrashOutput Go 1.23+; testing/synctest Go 1.25+; the goroutineleak profile needs a Go 1.27 toolchain; GOTRACEBACK, GODEBUG, net/http/pprof, and go tool pprof are long-standing.

Find the mechanism that explains the observed failure. Treat the ticket's proposed cause as a hypothesis; tie conclusions to an affected request, data, code version, and a check that could disprove them.

Resource Routing

• references/TICKET-INVESTIGATION.md - Read for tickets, regressions, tenant-specific failures, environment differences, or an incomplete report.
• references/DATA-FLOW-TRACING.md - Read for wrong/missing results or a failure crossing layers.
• references/DIAGNOSTIC-TOOLS.md - Read before capturing profiles, stacks, or traces, using GODEBUG/GOTRACEBACK, or choosing pprof, dlv, or the race detector.
• references/SYMPTOM-CATALOG.md - Read for unclear runtime/build symptoms or edits that do not affect behavior.

Scope and Starting Evidence

• Investigate / explain / review only: inspect available artifacts, reproduce within the authorized environment, and report findings. Do not edit source, change configuration/data, deploy, or post/close a ticket.
• Investigate and fix: continue through a minimal correction and regression verification once evidence supports the mechanism. Routine implementation choices do not require another approval. Keep unrelated findings out of the diff.
• A pasted ticket or the available repository is enough to start. If a resource is unavailable, name the missing evidence and continue independent work.
• Preserve the distinction between observed behavior, the intended contract, and the reporter's interpretation. If the contract is ambiguous in a way that changes the fix, ask a focused question in the same message as the next capture and keep collecting; end the turn on it only when no capture is left.

Investigation Loop

1. Pin the failure. Record expected vs actual, a concrete input/request ID, time, affected tenant/role/data, frequency, and scope. Shrink a reproduction without deleting the condition that triggers it (concurrency, load, or state). Once it fails, remove one remaining input, caller, config value, data row, or step at a time; keep a piece only when dropping it makes the failure stop.

2. Match the environment. Establish the running image/revision and relevant effective config, flags, schema/migrations, and dependencies. The checkout is not evidence of deployed code. Compare a working case with the same input. For a regression, name the last working revision or deployment and diff what changed up to the first failing one: source, go.mod, effective config, schema. When the diff is too wide to read, bisect it; exit 125 skips a revision that does not build or lacks the test, which a bare go test would call bad or good:

   git bisect run sh -c 'go build ./pkg || exit 125; go test -list "^TestName$" ./pkg | grep -qx TestName || exit 125; go test -count=1 -run "^TestName$" ./pkg'

   Keep a test written for this investigation uncommitted, in its own new file: an untracked file stays at every revision bisect checks out; one committed only at HEAD is absent from all of them and every step is skipped. Compare dependency versions with go version -m on both binaries, not on the checkout.

3. Locate the first divergence. Follow the relevant execution/data path or capture the artifact below. List the callers of the diverging function; each caller is a path the same input may take. Inspect values at boundaries before changing code. If no reproduction exists, use historical evidence or propose the smallest targeted capture; do not guess a patch or instrument everything. Tag every temporary log with one prefix for the investigation, such as [DEBUG-a4f2], so removing them is a single search.
4. Test a hypothesis. State mechanism, supporting evidence, and an experiment with different predicted outcomes if it is right or wrong. When more than one cause is still plausible, write down each one's prediction before testing any, then run the cheapest check that separates the likeliest from the rest. Vary one factor at a time. Record the observed result separately from the proposed check. For nondeterministic behavior, predict allowed outcomes or an invariant; a single repeat choosing a different row/order need not refute the mechanism.
5. Reassess. A failed hypothesis narrows the search. Repeated failed patches call for revisiting assumptions and boundaries, not another speculative fix or an arbitrary attempt-count approval stop. Track plausible alternatives. When each fix moves the failure to a new place, the shared state or ownership behind those places is the finding: name it and route the restructuring to go-code-refactor instead of patching the next site.
6. Finish in scope. Investigation ends with the evidence report below. An authorized fix corrects the responsible boundary, adds regression coverage where a test can reach the original mechanism, and runs the applicable repository verification through go-linting. Either way, no temporary log stays behind: a search for the investigation's debug prefix matches nothing.

Stop Signals

Each of these is a patch on the symptom and means the mechanism is still unknown. Return to step 3 before writing it:

• recover() around the panicking call, or a nil check at the faulting line before the nil's origin is known.
• A longer time.Sleep, a wider -timeout, or a retry loop in a flaky test.
• A mutex or channel added without naming the two racing accesses.
• A retry, restart, cache flush, flag change, or GOMEMLIMIT that makes the ticket go quiet.
• A re-run with -count=1 or a cleared cache that passes once.
• "It is probably X" followed by an edit, with no check that could disprove X.

Any of these can be the right mitigation once the cause is recorded as unknown. It is not a fix, and the report says which of the two it is.

Live Capture Boundaries

• SIGQUIT normally prints stacks and exits a Go process. Use an existing admin dump first; termination is only an option within authorized scope.
• Use access-controlled debug listeners. Dumps/profiles can expose internal details; retain only relevant sanitized evidence, including in ticket drafts.
• Profiling overhead varies by workload and capture type. Bound captures, check available headroom, and avoid concurrent profiles that distort results. Enabling endpoints, verbose logging, or attaching a stopping debugger changes service behavior; do not treat these as passive reads.
• runtime/trace.FlightRecorder (Go 1.25+) retains recent trace data when already enabled, allowing a snapshot after a spike. Capture still has a cost.
• Before a requested restart of the failing process, take the goroutine dump, heap profile, and flight-recorder snapshot it allows: the restart destroys the only evidence of the state.

Fix and Regression Proof

Correct the earliest responsible boundary with the smallest attributable change. Use a shared function when its contract is wrong for all affected callers; otherwise fix the caller that violates its own contract. Do not change correct callers merely to centralize a ticket's special case. Preserve neighboring behavior outside the defect: a default for a missing input must not silently redefine explicit empty, zero, negative, or invalid inputs without evidence.

The regression should fail for the original mechanism and pass after the fix. Write that failing test before the fix when it can reach the original mechanism: at the call site that failed, or at the shared function whose contract the fix corrects. A test that cannot — one caller when the bug needs several, or a unit that cannot rebuild the chain — does not lock the mechanism down: report that gap instead of adding the test. Cover the trigger and a nearby working case (for example, colliding IDs across tenants). After the fix, re-run the reproduction from before it was shrunk, when one exists. For nondeterminism, report runs/failures and seed or workload details; a lower failure rate is not proof of elimination. Preserve user changes: use an isolated copy/worktree or a reversible local comparison, never stash/discard the user's work by default. Run required checks through go-linting, reusing applicable unchanged evidence.

Investigation Result

Lead with confirmed, probable, or unresolved, and the mechanism or remaining question. Include only what makes it assessable:

• Expected/actual behavior and the affected version/environment.
• Evidence chain with request IDs, source locations, query/profile observations; distinguish supplied artifacts, static deductions, and executed checks.
• Disproved hypotheses, relevant alternatives, and the next discriminating check if unresolved. Missing access or a non-reproduction is not proof of absence.
• In fix mode: change, regression before/after (including the reproduction from before it was shrunk, when one exists), and required-check results. When no test can reach the original mechanism, say so. Report unrun/unavailable checks honestly; a proposed patch is not a verified fix.

An investigation can be complete with an unresolved cause if the evidence limit and next useful check are explicit. Report length follows go-style-core; no mandatory dossier for a one-line bug. Ticket publication requires the user's requested scope.

Symptom → First Capture

Choose a capture to distinguish hypotheses. An expensive profile or the full suite is not the default first step for an incorrect JSON response.

Reading Runtime Evidence

Panics

Start at the panic message and relevant application frame, then trace inputs backward. A faulting frame is the failure site; the cause can be upstream. Printed argument words can be incomplete or stale under optimization; confirm suspected nil values against the matching source or debugger.

• concurrent map writes indicates unsupported concurrent access. A send on closed channel panic also happens in sequential code: inspect send/close ownership instead of assuming a data race was detected.
• created by identifies a goroutine's origin, not the cause of its failure. runtime.goexit is a normal goroutine exit frame, not proof its parent caused a panic. Preserve the full panic/defer chain before adding recovery logic.
• Go 1.27 prints runtime/pprof goroutine labels in traceback headers — goroutine 19 [chan receive] {request_id: "abc-123"} — which attributes a stack to its request or job, for modules whose go directive is 1.27 or later (GODEBUG=tracebacklabels=1 enables it for older directives). Missing labels prove nothing: pprof.Do restores the previous set in a defer, so the panicking goroutine's own labels are usually gone by the time its trace prints, while the other goroutines in a GOTRACEBACK=all dump still carry theirs.

Hangs and Goroutine Leaks

For an existing, authorized admin listener:

curl -fsS --max-time 10 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=2' > goroutines.txt
curl -fsS --max-time 10 'http://127.0.0.1:6060/debug/pprof/goroutineleak?debug=1' > leaks.txt

The goroutineleak profile replaces the count-comparison ritual for the cases it covers: it lists goroutines blocked on a primitive that can never unblock again, decided by GC reachability rather than by inference. What it reports is a leak. What it omits may still be one — a goroutine waiting on a channel a live object could theoretically still use is reachable, so an empty profile narrows the search instead of ending it. It follows the toolchain that built the binary, not the go directive: Go 1.27 toolchain (1.26 needs GOEXPERIMENT=goroutineleakprofile; without it the endpoint answers 404 Unknown profile). Debug levels, the forced GC cycle each read runs, and the nil-guarded in-process form are in DIAGNOSTIC-TOOLS.md.

A blocked goroutine can be an idle worker. Compare equivalent workload windows and expected lifetimes. Use full stacks to find the creator and wait partners; on the goroutine profile debug=1 groups stacks with counts, while debug=2 helps inspect individual waits. Two goroutines in Lock do not prove lock-order inversion: establish the ownership/wait cycle. A missing ctx.Done() matters when no other termination path satisfies the caller's lifetime. For a local hanging test, go test -timeout 30s ./pkg yields stacks when it times out.

Memory and Resources

• Heap inuse_space shows sampled retained allocations; alloc_space shows cumulative allocation activity. Compare the same workload and GC phase. An allocation stack does not identify every reference retaining an object.
• A growing map may need bounded retention; first check its intended lifetime and workload.
• RSS growth without heap growth calls for runtime stack/span metrics and native/cgo memory evidence. Do not infer a Go heap leak from RSS alone.
• For file/connection leaks, correlate growing descriptor/pool usage with ownership and missing closes; route cleanup to go-defensive or go-database.

Flaky Tests

Start with the affected package/test and preserve the failing seed and environment:

go test -count=20 -failfast -run '^TestName$' ./pkg   # focused repeat
for i in 1 2 3 4 5; do go test -count=1 -shuffle=on -v ./pkg || break; done   # order dependence; a new seed per run

-shuffle reorders top-level tests, so it does nothing when -run matches one. Choose the repetition count from observed frequency and cost; add -race for concurrent access. A clean race run only covers executed paths. If the test fails only with the package, replay the printed seed with -shuffle=<seed> -v, then bisect the tests that ran before it with -run, reading the order -v prints each time, until the polluting pair remains. Inspect ports, time, temp files, and cleanup. testing/synctest (Go 1.25+) can make supported concurrent time-based tests repeatable; it is not a universal replacement for real I/O or external systems.

Related Skills

• go-resilience: retries, overload, fallback.
• go-database: query, tenant-filter, pool fixes.
• go-http: request, response, client, server fixes.
• go-concurrency and go-context: races, leaks, deadlocks.
• go-data-structures and go-defensive: aliasing, cleanup.
• go-performance: a located bottleneck.
• go-testing: the regression test.
• go-security: debug endpoint exposure; go-linting: the gate.