adamant-debugging

Three debugging layers, in escalation order: interactive GDB (a failing test in front of you), post-mortem triage (a crash already happened -- decode what the flight software left behind), and target/compiler-level (the bug only exists on the cross target). The framework already builds debuggable artifacts and ships crash-reporting components -- the skill is knowing which level you are at and the Ada-specific moves at each.

Quick Start (debug a failing unit test)

From the host, build with admt test <component>/test as usual; gdb itself must run inside the activated container shell -- enter one with admt env login (the redo/gdb lines below are what you run there).

cd src/components/<component_name>/test
redo test                        # builds build/bin/Linux_Test/test.elf
gdb build/bin/Linux_Test/test.elf

catch exception          # break wherever any Ada exception is raised
run                      # runs the whole AUnit suite
bt                       # on catch: raise point <- test proc <- AUnit caller

The same session runs headless for reproducible, pasteable evidence -- the form to prefer when the finding goes in a report:

gdb -batch -ex 'catch exception' -ex run -ex bt -ex 'info locals' \
    build/bin/Linux_Test/test.elf

(-ex 'break file.adb:N' and print commands chain the same way; note catch assert hooks GNAT's assert-failure routine -- pragma Assert / Ada.Assertions / SPARK contracts under -gnata -- so it does NOT fire on AUnit/Adamant test assertions, which raise their own Assertion_Error as an ordinary exception. catch exception DOES stop at those; catch assert does not.)

Diagnosis standard: a root cause is demonstrated, not inferred. Reading source produces a hypothesis; the diagnosis is complete only when the mechanism has been shown at runtime -- the actual values at the defect (a catchpoint stop, a breakpoint print, a watchpoint transition). This is what separates "the bug is probably X" from a report a reviewer can act on, and it routinely falsifies convincing source-reading theories.

No flags, no rebuild-for-debug: Linux test binaries are always compiled -O0 -g -gnata -gnatVa and never stripped, so source-level debugging works on the artifact redo test already built.

Build targets: which artifacts are debuggable

Debug readiness is decided by the build TARGET, not a flag:

• There is no production Linux mode -- Linux is a rename of Linux_Debug.
• Bareboard ELFs keep symbols in every mode, so a production ELF loads in gdb -- but for honest stepping build the -O0 variant: redo build/bin/<Platform>_Debug/main.elf.
• DEBUG=1 redo ... is build verbosity (prints gprbuild/gcc command lines), not debug symbols -- the target already provides those.
• Executables land at <src_dir>/build/bin/<TARGET>/<test|main>.elf; test dirs force the _Test suffix automatically via env.py. redo targets lists every target.

Ada-specific GDB moves

1. Set breakpoints by file:line, not Ada name -- the expression form (break My_Package.My_Proc) frequently fails to resolve:

   break <unit_name>-implementation.adb:326

   For symbolic breakpoints use the linker name (Ada mangles with double underscores): nm test.elf | grep -i <proc_name> then break on <unit>__implementation__<proc>.

2. catch exception is the failing-test power tool -- -gnata means a failing pragma Assert raises ADA.ASSERTIONS.ASSERTION_ERROR; the catchpoint stops at the raise with the full stack:

   catch exception                    # all Ada exceptions
   catch exception unhandled          # only ones that would kill the test
   catch assert                       # only failed assertions

3. SPARK contracts are assertions at runtime. On Debug/Test targets (checks on), a failing SPARK pre/postcondition raises like a pragma Assert -- catch exception/catch assert stops at it the same way. Static proof is a separate GNATprove analyze target, not part of the gdb loop (see adamant-formal-verification); do not confuse a proof failure (compile/analyze time) with a runtime contract violation (what you debug here).
4. No single-test filter -- the AUnit harness runs the whole suite; break in the specific Test_* procedure and continue past the rest.
5. Validity checks change what you see -- -gnatVa + Initialize_Scalars means uninitialized scalars hold recognizable invalid patterns; an extreme value in info locals usually means "never assigned", not "corrupted".

Debugging environment

• gdb ships with the GNAT toolchain and is on PATH only in the activated environment (container login via the project's env tooling); a bare docker exec shell will not find it.
• DWARF records build-environment paths. Inside the container everything matches; a host gdb works on Linux-target binaries if launched from the source dir (cwd resolution) or with set substitute-path <container_prefix> <host_checkout>.
• Rebuild before debugging -- "Source file is more recent than executable" means line numbers are lying.
• Assembly binaries on Linux: same story without the _Test suffix -- redo build/bin/Linux/main.elf, then gdb it (redo run = build + run).

Cross targets: Renode and JTAG

Cross ELFs cannot execute on the host. Attach a cross gdb (riscv32-elf-gdb, arm-eabi-gdb) to a GDB server:

• Renode: the project's .resc loads the ELF (sysbus LoadELF) and starts machine StartGdbServer 3333; attach with the ELF as symbols only -- no load (Renode already loaded it):

  <cross>-gdb build/bin/<Platform>/main.elf -ex 'target remote :3333'

  Ctrl-C halts the live target; monitor <cmd> passes to the Renode console. Projects wrap this in redo renode / debug_renode.sh scripts beside the assembly or test_renode dirs -- discover them before hand-rolling. Server in a different container: attach via host.docker.internal:3333.

• JTAG/SWD hardware: the probe's GDB server runs on the host (USB); gdb attaches (target [extended-]remote host.docker.internal:<port> from a container, localhost:<port> on host) and owns the bring-up -- script it (gdb -x bringup.gdb). Server variants differ in monitor syntax:
  • OpenOCD (open example: adamant_example's Raspberry Pi Pico -- src/assembly/pico/main/{debug.sh,config.gdb,program.sh} and its README): server openocd -f interface/cmsis-dap.cfg -f target/rp2040.cfg, port 3333; bring-up monitor reset init, load, continue; one-shot flash-without-debug via openocd ... -c "program <elf> verify reset exit".
  • SEGGER J-Link: port 2331 typical; bring-up monitor reset, monitor halt, file <elf>, load (needs a RAM-loader link), then break/continue. Start the server with its all-interfaces flag or a containerized gdb cannot reach it. A serial/UART cable beside the probe carries the console/telemetry channel -- it is the capture path, not the debug transport.

Post-mortem: crashes and the Last Chance Handler

When an exception escapes on the flight target, GNAT's __gnat_last_chance_handler hook fires and Adamant's LCH pattern captures a Packed_Exception_Occurrence (exception name, message, and a stack-trace address array) and broadcasts it as a telemetry packet until watchdog reset. Framework components involved (all open, in src/components/):

Triage rules (each learned from a real incident -- details and the full runbook in references/post-mortem-and-lch.md):

• A hung test suite IS a crash until proven otherwise. The LCH stops telemetry; downstream waits spin forever, so the failure never reports. Check whether housekeeping sequence counts froze.
• Symbolize against the same build. Generate a symbol table at build time (<cross>-nm -C -n main.elf > main.nm); resolve trace addresses by nearest-symbol-below, or <cross>-addr2line -f -C -e main.elf <addr> for file:line. A symbol file from any other build resolves plausible garbage.
• The crashed target is still attachable: Renode's GDB server works while the CPU spins in the LCH -- capture the exception occurrence and registers via a batch gdb script, no reproduction needed.
• Interrupt-context traps record only ONE frame (the unwinder cannot cross the trap handler); recover the real chain by re-running paused with a hardware breakpoint at the LCH entry.

scripts/symbolize_traceback.py (in this skill) resolves pasted traceback addresses against an nm dump or ELF -- see the reference for usage.

Target/compiler-level debugging

"Works on Linux, traps on the flight target" is a class, not a fluke: undefined behavior is target-dependent, so cross-run the unit tests on the flight target (redo test_renode where the project provides it) rather than trusting host green. The known pitfall families -- alignment-erroneous overlays, scalar-storage-order codegen bugs, per-flag workarounds pinned in gpr files, and the post-rebase stale-generated-code signatures -- are in references/target-pitfalls.md with their proof techniques (bit-pattern analysis, disassembly around the faulting PC, RISC-V trap-register decoding).

Project tooling discovery

Closed-source projects commonly ship debug helpers the framework does not: symbolizers, per-task stack maps, UART/log capture scripts, scripted gdb bring-ups. Before hand-rolling, look in the project's ground tooling and assembly directories:

ls gnd/bin/ | grep -iE 'stack|trace|decode|symbol|log'
ls src/assembly/<name>/main/*.sh src/assembly/<name>/main/*.gdb

Prefer the project's own scripts -- they encode target-specific details (ports, ELF paths, register sets) this skill keeps generic.

Checklist

1. Pick the layer: failing test in hand -> interactive gdb; crash/hang already happened -> post-mortem; target-only misbehavior -> pitfalls ref
2. Rebuild so symbols match source; pick the -O0 target if stepping
3. Interactive: catch exception before run; breakpoints by file:line. A hung/stalled/parked task, a mystery write, or a packed-layout mismatch means read references/gdb-advanced-inspection.md -- generic gdb gets you a backtrace; the reference has the Adamant-specific moves (for a hang, thread apply all bt + mapping the parked thread to its component by the tick-handler frame -- info tasks is unreliable for Adamant tasks; see the reference), so read it rather than improvising
4. Cross: server first (Renode/JTAG), then cross gdb; load only on JTAG
5. Post-mortem: capture the LCH output (telemetry, NV dump via last_chance_manager, UART log, or gdb attach to the spinning target)
6. Symbolize addresses against the SAME build's nm dump or ELF
7. Suite hang: check for frozen housekeeping counters before debugging the test script
8. Target-only bug: reproduce under Renode; a PROGRAM_ERROR raise with no message usually means a runtime CHECK fired, not a trap (catch the exception or the __gnat_rcheck_* call -- see the pitfalls reference); decode trap registers only for genuine traps, and read CSRs through the emulator monitor if the GDB stub does not expose them

Common Errors

1. break Package.Proc "not defined" -- use file:line or the mangled pkg__proc name from nm.
2. Breakpoint never hits / lines mismatch -- stale binary; gdb warned "Source file is more recent than executable"; rebuild. More generally, a reproduction or nm/objdump signature that contradicts the source means stale objects until proven otherwise: redo clean_all (not a shallow redo clean) at the component, rebuild, re-verify -- see the stale-objects section of references/target-pitfalls.md.
3. gdb: command not found in the container -- bare shell; enter the activated environment (gdb lives in the toolchain, not /usr/bin).
4. No source listing on host gdb -- DWARF has container paths; cd to the source dir or set substitute-path.
5. Stepping jumps around on a bareboard ELF -- attached to an -O2 image; build and load the _Debug variant.
6. load fails/hangs against Renode -- Renode already LoadELFed; attach with symbols only (load belongs to the JTAG flow).
7. Expected a debug build from DEBUG=1 -- that is build verbosity; the target selects switches, and tests/Linux are already debug.
8. Cannot reach the GDB server from the container -- server listening on another host/container's localhost; use host.docker.internal:<port> and all-interfaces server flags.
9. Symbolized trace names look wrong / nonsensical -- symbol file and ELF are from different builds; regenerate the nm dump from the exact ELF that was running.
10. Suite "hangs" with no failure -- likely an LCH crash upstream: telemetry died, waits spin. Triage as a crash (post-mortem reference), not as a slow test.
11. Single-frame stack trace from an interrupt trap -- expected; the unwinder stops at the trap boundary. Use the paused-emulation + hardware-breakpoint recovery in the post-mortem reference.
12. Assertions "fixed" by disabling them -- never ship Assertion_Policy (Ignore) to hide an LCH: the code then runs on the bad input and produces garbage instead of a diagnosable crash. Fix the dependency (fetch + early-return on non-Success).

Improving this skill in the field

Everything in this skill was demonstrated on a real program, and field use has more than once falsified content that was merely plausible. Hold the skill itself to the same bar, while the evidence is still in front of you:

• When a debugging task goes beyond what is written here -- a failure family, tool behavior, target quirk, or workaround this skill does not cover, or a documented move that did not work as described -- capture it. Silently working around a gap leaves it for the next debugger.
• New content meets the diagnosis standard: demonstrated, not inferred. Record what you observed (the command, the output, the failure signature and its resolution), never what you believe should happen. If you falsified a claim here, correct it and state what you observed instead -- a wrong confident claim is worse than a gap.
• Keep it generic. No project names, component or symbol names, memory addresses, platform files, or naming motifs from the program you were debugging -- placeholders only (<addr>, <component>, "the project's LCH"). Project-specific guidance belongs in that project's own skill, not here.
• Extend, don't duplicate. Search this file and the references for an existing section on the topic first and sharpen it in place; put quick moves here and depth in the layer-appropriate reference, and add a routing cue (References blurb, selector row) so the new content is findable.
• If your task does not permit editing skills (some validation and review tasks forbid it), put the proposed change in your report verbatim -- the gap, the observed evidence, and suggested wording -- so a maintainer can fold it in. Version control of skill edits belongs to the orchestrator or maintainer either way.

References

• references/gdb-advanced-inspection.md -- watchpoints (who mutates this?), Ada task/thread triage (info tasks, thread apply all bt, live attach), packed-record vs raw-memory views, conditional/scripted stops, signals vs cross-target traps, core-file handoff, Renode monitor passthrough. Read when breakpoints + catchpoints are not enough to demonstrate a mechanism.
• references/post-mortem-and-lch.md -- LCH anatomy and wire format, capture paths (telemetry / NV dump / UART / gdb attach), symbolization pipeline + script usage (including extracting a trace from a raw binary capture), interrupt-context trap recovery, RISC-V trap register decoding, crash root-cause checklist, suite-hang signatures, and the headless-emulator bench-operations runbook (launch form, port hygiene, kill patterns). Read when a crash or hang already happened, or before any scripted emulator session.
• references/target-pitfalls.md -- works-on-Linux-traps-on-target families with reproducers and proof techniques; compiler-bug workaround flag pinning; post-rebase stale-codegen signature table; runtime debug-variant builds; monitor components for resource debugging. Read when behavior differs between host and flight target, or after a rebase produces baffling compile errors.
• scripts/symbolize_traceback.py -- stdlib-only resolver: traceback addresses + (nm dump | ELF + binutils prefix) -> symbol+offset and optional file:line.