Debugging rocprofiler-systems

Collaborative debugging for rocprofiler-systems issues. Work WITH the user through systematic triage, not autonomously.

When to Use

• Crashes or segfaults when profiling with rocprof-sys
• Missing, incorrect, or corrupted profiling output
• Instrumentation failures (Dyninst, symbol issues)
• Multiprocess (fork/MPI) or multithreaded app problems
• GPU/ROCm tracing or sampling failures
• Signal handling conflicts between app and profiler
• Profiling overhead issues

When NOT to Use

• Quick usage questions -> ask
• Root cause known, planning a fix -> planning-bugfix
• Exploring code without a bug -> exploration-explore-code

Workflow

digraph debugging {
    "User reports issue" [shape=doublecircle];
    "Gather context" [shape=box];
    "Classify problem" [shape=diamond];
    "Enable debug logging" [shape=box];
    "Logs reveal root cause?" [shape=diamond];
    "Conclude" [shape=doublecircle];
    "Choose escalation tool" [shape=diamond];
    "Run tool with user" [shape=box];
    "Root cause found?" [shape=diamond];
    "Deep dive: source + minimal repro" [shape=box];

    "User reports issue" -> "Gather context";
    "Gather context" -> "Classify problem";
    "Classify problem" -> "Enable debug logging";
    "Enable debug logging" -> "Logs reveal root cause?";
    "Logs reveal root cause?" -> "Conclude" [label="yes"];
    "Logs reveal root cause?" -> "Choose escalation tool" [label="no"];
    "Choose escalation tool" -> "Run tool with user";
    "Run tool with user" -> "Root cause found?";
    "Root cause found?" -> "Conclude" [label="yes"];
    "Root cause found?" -> "Deep dive: source + minimal repro" [label="no"];
    "Deep dive: source + minimal repro" -> "Conclude";
}

Phase 0: Gather Context

Use AskUserQuestion to collect:

Problem Classification

Phase 1: Log Analysis (Always Start Here)

Enable Debug Logging

export ROCPROFSYS_LOG_LEVEL=debug   # trace|debug|info|warn|error|critical|off
export ROCPROFSYS_VERBOSE=3
export ROCPROFSYS_LOG_FILE=/tmp/rocprof-sys-debug.log

# For instrumentation issues
rocprof-sys-instrument -vvv -- ./app

Check Configuration

rocprof-sys-avail -G /tmp/current-config.cfg --all
env | grep ROCPROFSYS

Log Patterns

Decision point: Present log findings to user. If root cause is clear, go to Phase 4. Otherwise, escalate.

Phase 2: Tool Escalation

Ask user before using any of these tools. Explain WHY the chosen tool fits.

Tool Selection

rocprof-sys-specific gdb Tips

Standard gdb usage applies. These are rocprof-sys-specific considerations:

• Look for rocprof frames in backtrace to distinguish profiler vs app crash
• For fork/MPI: set follow-fork-mode child or attach to specific rank PID
• For threads: thread apply all bt to see all thread states
• Useful breakpoints: rocprofiler_systems_v1::configure, fork_gotcha::audit

rocprof-sys-specific strace Tips

• Always use -f for threaded apps, -ff -o prefix for multiprocess
• Watch for open("/dev/kfd") failures (GPU access)
• Repeated futex(FUTEX_WAIT) suggests deadlock in profiler
• For MPI: mpirun -n 4 strace -ff -o /tmp/strace-mpi rocprof-sys-run -- ./app

Phase 3: Deep Dive

If Phase 2 narrows but doesn't resolve, inspect source or create minimal repro.

Source inspection: Use exploration-explore-code to navigate the rocprof-sys source when needed.

Minimal Reproduction

1. Strip app to smallest failing case
2. Isolate rocprof-sys mode (run vs instrument vs sample)
3. Remove optional features one at a time (GPU, MPI, sampling)
4. Test with simplest preset (--balanced)

Phase 4: Conclude

Summarize: problem, root cause, evidence, tools used, resolution.

If code fix needed, suggest planning-bugfix.

Known Gotchas

Common Mistakes

Integration