How is the time distributed? Where is the bottleneck? What is not the problem? Every finding gets chased to a file:line.

torch-profile

• Profile the script at the current commit with torch.profiler.
• Add the profiler hooks yourself when the script has none.
• What you produce is a diagnosis, not a patch. Those hooks are the only edit you make: nothing here changes what the job computes or how long it takes, and nothing here measures a fix.

① capture                ② views                ③ findings
   one plain run +    →     how the wall      →    ranked, anchored,
   a profiled window        actually splits        costed, handed off

① Capture

Two runs.

First, unprofiled: torch.profiler adds overhead, so take the honest clock with the cheap APIs instead, such as torch.cuda.Event(enable_timing=True) and torch.cuda.max_memory_allocated(). This is also the only run that sees startup, which no profiling window covers: dataset preprocessing, checkpoint loading, first-step compilation, and anything else paid once. Time every step, and save both that series and those startup numbers under commits/<sha>/.

Then profiled, over about 3 steady-state steps, with the window chosen from the step-time series the first run produced. Steps 3-4-5 is a default, not a law: if records/execution_schedule.json marks a periodic phase, the window must not straddle it, and if step time trends, the window has to sit in the regime the run actually spends its steps in. Traces go under commits/<sha>/trace/, which workspace.py candidate <ws> <sha> creates.

② Views

One profiled step can hold a million events, and nobody reads a million events. A view is a reshaping that makes one kind of anomaly stand out and hides the rest. tools/view/ loads the traces into a store, then renders all five from it, one numbered file each:

export PYTHONPATH=<plugin root>   # you are standing in the target, not here
python -m tools.view load   <ws>/commits/<sha>/trace/*.json -o <ws>/commits/<sha>/store.db
python -m tools.view render <ws>/commits/<sha>/store.db -o <ws>/commits/<sha>/views --capacity-gb <GB>

load wants rank<N> in each trace filename. torch.profiler's own export name has none, so name them at capture time or the rank becomes sort order and every per-rank row below is labelled with a rank that may not have produced it. tools/view/common.py documents the schema, and each view is also a module with a render(con) if you want one on its own.

1. Runtime decomposition (1_runtime_decomposition.txt). wall = busy + idle, per step and per rank, with the spread across steps and the slowest rank. Read it first: it forks the whole diagnosis into waiting against busy but inefficient.
2. Lost time, ranked (2_lost_time.txt). Every gap where nothing ran and every run of tiny kernels, ordered by what it costs in wall clock, each stamped with its offset from the step start so the same one lines up across steps. Each one arrives with what the machine was doing instead, which you then name as an entry in records/execution_schedule.json rather than as a symptom.
3. Kernel table (3_kernel_table.txt). Cumulative time bucketed into attention, GEMM, communication, element-wise and transfer, with bytes moved and achieved bandwidth wherever the event carries a size. Copies are in here on purpose: a device-to-host copy counts as GPU-busy, so it sits in no gap, and it disappears from every view at once if this one drops it.
4. Overlap and exposure (4_overlap_exposure.txt). Per collective, how much hid behind compute and how much was exposed. Only the exposed part lengthens the step, so a large communication share that is fully hidden is not a finding.
5. Memory over time (5_memory.txt). Allocated and reserved, the peak, and the headroom per GPU. Nothing else says whether a fix that spends memory is available at all. Pass --capacity-gb or the headroom column comes back unknown.

Read all five before you conclude anything. They are where to start rather than the limit: the store is an ordinary database, so query it directly for whatever this particular job makes you suspect.

③ Findings

Every finding carries all of these, or it is not one yet:

Include the negative results. Half of what a reader needs is the list of things that looked expensive and were not.

Write it to commits/<sha>/report.md. Inside optimize, that report is what the loop reads to decide whether the candidate is kept.

Output contract

Three sections, in this order, and nothing may appear in section 3 that was not established in sections 1 and 2:

1. Where the time went. How this run divides into working and waiting, for the whole job and for a single step.
2. Findings. Ranked by how much wall clock each one actually costs, meaning the part not already hidden behind other work.
3. Prescription. Directions, each with a what-if bound stated as a ceiling, the effort it takes, whether numerics would change, and whether the code is the reader's to edit. No predicted numbers for fixes you did not run. Whoever runs the experiment writes the prediction.