Workflow AGENTS.md

Context for AI coding assistants working in the workflow/ directory. See root AGENTS.md for project overview, CLI usage, and architecture.

Processing Modes: Notebooks vs SLURM

KINTSUGI has two distinct processing modes with different resource utilization strategies:

Notebook Processing (Interactive):

• GPU is required - fails loudly if unavailable
• User is present to diagnose and fix GPU issues
• Quality parameters are non-negotiable (e.g., BaSiC iterations=500)
• See gpu-quality-priority skill for details

SLURM Processing (Headless — GPU-Only Scheduling):

• All cycles route through GPU slots — no CPU fallback
• Overflow cycles queue in SLURM until a GPU slot frees up
• Measured speedups (CX_19-003, 9x7, 4ch, 20z): stitch 25x, decon 5x, EDF 15x (~13x full cycle)
• Snakemake -j is set to total GPU slots (not total slots) to limit concurrent submissions
• Per-cycle pipeline (stitch→decon→edf) enables cycles to flow through freed GPU slots
• See gpu-only-scheduling skill for the performance data behind this decision

Resource Pool Calculation (Multi-Account Architecture)

KINTSUGI calculates available GPU slots from all non-blocked SLURM accounts:

• GPU slots per account = QOS gres/gpu limit (via sacctmgr)
• CPU slots are still detected but not used for scheduling (CPU is 5-25x slower per step — see gpu-only-scheduling skill)
• Blocked accounts (Apr 8 2026): brusko, clive — hard-coded in BLOCKED_ACCOUNTS frozenset in hpc.py. clive was blocked because its QOS pool was throttled to 1 GPU / 312.5 GB / 40 CPUs and is regularly saturated by other group members on the shared investment QOS.
• Active account: maigan only (gpu_slots: 2, cpu_slots: 8)

Total Concurrent = sum(GPU slots) across all unblocked accounts

Live-aware routing (Apr 8 2026): kintsugi workflow run queries detect_live_multi_account() and forwards per-account gpu_avail/cpu_avail/mem_avail_gb to Snakemake via --config live_accounts=<json>. The Snakefile's _build_cycle_assignment, _registration_assignment, and _qc_cpu_assignment use the live data to skip accounts saturated by other users on the same QOS investment pool. Hard-fails if every account has zero live availability instead of queueing forever. Falls back to static gpu_slots for bare snakemake invocations. See the live-aware-account-routing skill for the full implementation.

How GPU-Only Scheduling Works

1. detect_multi_account_resources() / detect_live_multi_account() query sacctmgr show associations for all user accounts (filtering burst -b accounts and blocked accounts)
2. Each account's GPU slots are calculated from its QOS limits and current usage (live mode also subtracts other users' consumption)
3. _build_cycle_assignment() pre-assigns ALL cycles to GPU, round-robin across accounts proportional to live GPU slot counts (when injected) or static gpu_slots (fallback)
4. Overflow cycles (beyond total GPU slots) still get GPU assignments — they queue in SLURM until a GPU slot frees up
5. Snakemake -j = total GPU slots, so concurrent SLURM submissions match available GPUs
6. Per-cycle pipeline (stitch→decon→edf) means freed GPU slots are immediately used by the next stage

Why no CPU fallback: Measured per-step (CX_19-003, 9x7, 4ch, 20z): stitch GPU ~8 min vs CPU ~200 min (25x), decon ~12 vs ~60 min (5x), EDF ~1.5 vs ~22 min (15x). Full cycle ~22 min GPU vs ~282 min CPU. Queuing for GPU is always faster.

Recognizing noisy-neighbor failures:

Snakemake Workflow (Alternative to submit.sh)

KINTSUGI includes a Snakemake-based workflow as an alternative to submit.sh for SLURM batch processing. Both produce files in the same data/processed/ tree but should not run simultaneously on the same project.

Setup and usage:

kintsugi workflow config .    # Generate workflow/config.yaml + Snakefile
kintsugi workflow check .     # Show live per-account availability
kintsugi workflow run .       # Submit via Snakemake (auto-sets -j)

Key design decisions:

Config format (workflow/config.yaml, post-clive-block):

resources:
  accounts:
    - name: maigan
      partition_gpu: "hpg-b200,hpg-turin"
      partition_cpu: hpg-default
      gpu_slots: 2
      cpu_slots: 8
  total_gpu_slots: 2
  total_cpu_slots: 8
  total_slots: 10
  cpu_time_multiplier: 5
  cpu_cpus_per_task: 8

Cycle assignment example (9 cycles, maigan 2 GPU slots):

• Cycles 1-2 → maigan GPU (fills both slots)
• Cycles 3-9 → overflow: round-robin maigan GPU (queue in SLURM until a slot frees up)
• All cycles run on GPU; pipeline pressure shows as PENDING (QOSGrpGRES) until slots drain

Lambda resource routing (per-rule in Snakefile):

resources:
    slurm_partition=lambda wc: _assign(wc)["partition"],
    slurm_account=lambda wc: _assign(wc)["account"],
    gres=lambda wc: "gpu:1" if _is_gpu(wc) else "",
    runtime=lambda wc: base_time * (1 if _is_gpu(wc) else CPU_TIME_MULT),

GPU resource naming: Use gres="gpu:1" (maps to --gres=gpu:1). Do NOT use gpus=1 (maps to --gpus=1) or gpu=1 — both trigger SLURM_TRES_PER_TASK conflicts on SLURM >= 24.11.

SLURM profile (workflow/profiles/slurm/config.yaml):

• precommand: "module load conda && conda activate KINTSUGI" — activates the existing conda environment on compute nodes so that conda-installed packages (cupy, torch, etc.) are importable inside srun jobs

SLURM_TRES_PER_TASK fix (critical for SLURM >= 24.11):

• SLURM 24.11+ sets SLURM_TRES_PER_TASK in GPU job environments, which conflicts with the jobstep plugin's srun call
• Error: srun: fatal: SLURM_TRES_PER_TASK is mutually exclusive with --ntasks-per-gpu
• Fix: patched snakemake_executor_plugin_slurm_jobstep/__init__.py to unset this variable in __post_init__(): os.environ.pop("SLURM_TRES_PER_TASK", None)
• This patch must be re-applied after any pip upgrade of snakemake-executor-plugin-slurm-jobstep

Cycle directory resolution: _resolve_raw_cycle_dir() handles cyc001_reg001_*, cyc001, cyc01, and Cyc01 naming conventions at DAG creation time. Accepts int or str (Snakemake CLI --config passes strings).

workflow config behavior: Always overwrites the Snakefile and profiles (so pipeline logic and SLURM precommand updates propagate); only copies scripts if they don't already exist. This means deployed scripts can become stale — if you update a script in workflow/scripts/ (e.g., qc_report.py), you must manually copy it to existing project workflows or delete the old copy and re-run workflow config.

QC Report Rules (aggregate — added 2026-02-13):

The workflow includes 5 aggregate QC rules that produce rich QC reports (summary heatmaps, z-plane profiles, cross-stage comparison PDFs, registration overlays, signal isolation pages) after each processing stage completes.

Key design decisions:

• Aggregate (not per-cycle): QC runs once across ALL cycles for cross-cycle comparison heatmaps
• Single dispatch script: scripts/qc_report.py dispatches to the correct function based on snakemake.params.stage (stitch/decon/edf/registration use Kprocess; signal_isolation uses kintsugi.signal.isolation_qc.generate_qc_pages())
• Cross-stage comparison: Each stage loads the previous stage's cached stats pickle (cache/stitch_stats.pkl → decon_stats.pkl → edf_stats.pkl)
• Headless matplotlib: matplotlib.use("Agg") is set before any Kprocess import to prevent display issues on compute nodes
• GPU with CPU fallback: Same pattern as processing scripts — tries CuPy, falls back to CPU
• Resources: 64 GB RAM, 120 min, 1 GPU on first account (reuses _REG_ASSIGN from registration)
• rule all includes QC: all_qc_sentinels() is included alongside registration sentinel in the default target
• rule qc standalone target: Run snakemake qc to generate only QC reports without reprocessing

snakemake -n qc           # Dry run — show QC DAG
snakemake qc --profile profiles/slurm -j 1  # Run QC only

Config resources (workflow/config.yaml):

resources:
  mem_qc: 64000              # 64 GB for stitch/decon/edf QC
  time_qc: 120               # 2 hours
  mem_qc_registration: 16000 # 16 GB (no GPU, pyvips streaming)
  time_qc_registration: 30   # 30 min

Registration Rule (aggregate — added 2026-02-13):

Registration (multi-cycle alignment via VALIS) is Rule 4 in the pipeline. Unlike stitch/decon/edf which process one cycle each, registration processes ALL cycles in a single SLURM job.

Full pipeline: stitch → deconvolve → edf (per-cycle, pipelined) → registration (all cycles) → signal_isolation (all channels) + qc_stitch/qc_decon/qc_edf (aggregate after each stage) + qc_registration + qc_signal_isolation

Key design decisions:

• No {cycle} wildcard: Registration is a single job, not per-cycle. Uses static _REG_ASSIGN dict instead of _assign(wc) lambdas
• _registration_assignment(): Prefers first account with gpu_slots > 0 for VggFD; falls back to CPU with OrbFD
• Single-cycle handling: If len(CYCLES) < 2, copies EDF → registered without alignment (no VALIS needed)
• Error handling: On VALIS failure, raises the exception (fails loudly). No fallback copy of EDF images — silent failures mask registration problems
• Skip-existing: All-or-nothing — checks if ALL registered files exist for ALL cycles before skipping
• Slide matching: Builds cycle_to_slide_name dict from DAPI filenames for robust slide_dict lookup
• Non-rigid architecture (Feb 2026): Three bugs were fixed: (1) serial_non_rigid.py:438 passed params=init_kwargs instead of **init_kwargs — all smoothing params silently ignored, (2) coarse NR at 1024px poisoned alignment — max_non_rigid_registration_dim_px defaults to DEFAULT_MAX_PROCESSED_IMG_SIZE = 1024, not DEFAULT_MAX_NON_RIGID_REG_SIZE = 3000, (3) Valis init didn't pass non_rigid_registrar_cls. Current approach: coarse NR is disabled (non_rigid_registrar_cls=None explicit in Valis init), non-rigid runs exclusively via register_micro() at 4096px with tuned smoothing. register_micro() handles None bk_dxdy (no prior coarse NR) by creating zero displacement fields. The earlier "rigid-only is best" conclusion (11/12 datasets worse) was based on broken non-rigid code — the skill has been updated.
• Three VALIS dimension parameters: max_image_dim_px (saved output size), max_processed_image_dim_px (rigid feature detection), max_non_rigid_registration_dim_px (coarse NR field) — all default to 1024. Conflating them causes silent quality loss.
• imgs_ordered=True: Prevents VALIS from reordering cycles by feature similarity. CODEX cycles must stay in sequential acquisition order.
• align_to_reference=True: All cycles align directly to cycle 1, preventing error accumulation from serial chain alignment (cyc N → cyc N-1 → ... → cyc 1)

Dimension Normalization (added Feb 2026):

EDF images can have inconsistent dimensions across cycles (e.g., 7472x12662 vs 7479x12660) due to minor stitching variations. VALIS requires all input images to have identical dimensions — mismatched sizes cause registration failures or silent misalignment.

• normalize_dimensions: true in config triggers automatic padding of all input images to (max_h, max_w) across all cycles
• Images are padded (not cropped) to preserve all tissue content, using zero-fill on the right/bottom edges
• Padded images are written to a temporary staging directory: kintsugi_reg_staging_XXXXXXXX (8-character random suffix)
• The staging directory is automatically cleaned up after registration completes (success or failure)
• This is the default behavior (registration.normalize_dimensions: true); set to false only if all input images are guaranteed to have identical dimensions
• The staging directory is created alongside the EDF output directory, not inside it, to avoid polluting the processed data tree

Validated Registration Results (Feb 2026):

Batch re-registration complete: all 26 registered datasets processed with tuned non-rigid (Feb 2026).

This confirms the params unpacking bug fix (serial_non_rigid.py:438, params=init_kwargs → **init_kwargs) was the root cause of previously poor non-rigid registration quality. The earlier "rigid-only is best" conclusion (11/12 datasets worse with non-rigid) was based on broken code where all smoothing parameters were silently ignored.

Registration config (workflow/config.yaml):

registration:
  reference_cycle: 1          # Cycle used as fixed reference
  reference_channel: 1        # Channel for feature detection (usually DAPI)
  max_image_dim: 4096         # Rigid feature detection resolution (was 2048)
  rigid_only: false           # Non-rigid enabled with tuned smoothing
  feature_detector: "VggFD"   # "VggFD" (GPU, better) or "OrbFD" (CPU-only)
  imgs_ordered: true          # Keep sequential cycle order (don't reorder by similarity)
  align_to_reference: true    # All cycles align directly to reference (not serial chain)
  normalize_dimensions: true  # Pad images to uniform (max_h, max_w) via temp staging dir
  non_rigid:
    max_dim: 4096             # Displacement field resolution for register_micro() only
    smoothing_method: "gauss" # Only "gauss" and None work ("inpaint"/"regularize" are broken)
    sigma_ratio: 0.01         # sigma = 0.01 * 4096 = ~41px (was 0.05 = 205px, extreme over-smoothing)
    n_grid_pts: 50            # No-op with "gauss" (only used by broken "regularize"/"inpaint")
    fold_penalty: 1.0e-6      # No-op with "gauss" (only used by broken "regularize")

resources:
  mem_registration: 64000     # 64 GB (sufficient for non-rigid at 4096px with 13 cycles)
  time_registration: 120      # 2 hours
  cpu_mem_registration: 64000 # Same for CPU (loads one image per cycle, not z-stacks)

VALIS metrics: D values are at processing resolution, not original. Use rTRE = D / max(processed_shape) for cross-experiment comparison at different max_image_dim settings. Average metrics are meaningless — poor registration is localized; use spatial NCC heatmaps or targeted overlays.

Non-rigid parameter tuning guide (sigma_pixels = sigma_ratio * max_dim):

• If over-smoothing (aligned but blurry): decrease sigma_ratio to 0.005 (20px at 4096)
• If under-correcting (still misaligned): increase max_dim to 8192 or decrease sigma_ratio to 0.005
• If over-warping (tissue distorted): increase sigma_ratio to 0.02 (82px at 4096)
• Do NOT use smoothing_method: "regularize" — it's broken in VALIS

Stale Snakemake Coordinators (operational hazard):

When a Snakemake coordinator process is running in the background (e.g., inside tmux) and it detects a cancelled SLURM job, it may automatically resubmit the failed rule. This can create duplicate registration jobs writing to the same output directory, leading to corrupted results or race conditions.

• Before manual resubmission: Always check squeue -u $USER for duplicate jobs targeting the same dataset
• Kill stale coordinators: Before re-running snakemake --profile profiles/slurm, ensure no previous Snakemake process is still alive (ps aux | grep snakemake)
• Symptom: Multiple SLURM jobs with the same name (e.g., two registration jobs for the same project) running simultaneously
• Prevention: Use tmux to manage Snakemake sessions and explicitly cancel (Ctrl-C) before restarting

Batch Re-Registration (/blue/maigan/smith6jt/reregister_all.sh):

Wave-based parallel execution across multi-account GPU pool:

• Waves of 5 projects (3 clive GPUs + 2 maigan GPUs), wait for wave completion before next
• Account distribution: reorders resources.accounts list in each project's config.yaml so _registration_assignment() picks the target account
• Targets only registered/.snakemake_complete + qc_plots/.snakemake_complete_registration — avoids GPU QC rules (qc_stitch/qc_decon/qc_edf) that would compete for GPU slots
• Idempotent: skips projects with non_rigid_smoothing=gauss in registration sentinel
• Phase 1: updates config, copies scripts/Snakefile, cleans old data + staging dirs + stale locks
• Phase 2: launches snakemake per project as background processes, waits per wave

Race condition hazard: If old SLURM registration jobs (from killed coordinators) are still running when Phase 1 cleans data, they can write back stale results. Cancel old jobs before relaunching.

Per-channel skip-existing: Wrapper scripts (stitch.py, deconvolve.py, edf.py) check per-channel completion inside jobs to resume interrupted cycles. Sentinel files include skipped=N in logs. Dynamic worker counts read from snakemake.resources.cpus_per_task.

Boundary QC + Multi-Z Fallback (Apr 2026): After computing the stitch model from the reference z-plane (n_zplanes // 2, CH1/DAPI), workflow/scripts/stitch.py runs kintsugi.qc.boundary_check.check_boundary_quality() on the stitched reference image. The metric is sharpness ratio (Laplacian variance): sharpness(overlap_region) / sharpness(adjacent_non_overlap). A healthy boundary has ratio ~1.0; a misaligned boundary has ghosted/blurred features in the overlap → ratio < 0.7 (default threshold 0.65). If any boundaries are flagged, the script tries alternate z-planes from select_alternate_zplanes(ref, n, 4) (quarter/three-quarter/top/bottom), running stitch_images() on each and picking the complete model with the fewest + least severe flagged boundaries. The chosen model is saved as result_df.pkl and the reference z-plane is re-stitched. Decision log written to data/processed/stitched/cyc{NN}/CH1/boundary_qc.json. Config keys: boundary_qc_enabled (default true), boundary_qc_threshold (default 0.65), boundary_qc_max_alternates (default 4). Overhead: ~100ms when no boundaries fail (common case); ~30-60s per alternate z-plane (rare). Tests: tests/test_boundary_check.py.

Snakemake replaces submit.sh orchestration/dependency wiring; keeps Python wrapper scripts (workflow/scripts/*.py). Run one system or the other, not both.

Batch Processing (Multi-Dataset)

KINTSUGI supports automated batch processing of multiple CODEX datasets via the kintsugi workflow batch CLI command. This replaces the old bash scripts (run_all_workflows.sh, process_remaining.sh) with a proper Python CLI that validates GPU resources, prevents silent CPU fallback, and tracks progress.

CLI command (preferred):

kintsugi workflow batch /path/to/KINTSUGI_Projects           # All eligible datasets
kintsugi workflow batch /path/to/KINTSUGI_Projects --dry-run  # Preview eligible
kintsugi workflow batch . -d CX_19-004                        # Single dataset
kintsugi workflow batch . -p 2 --detach                       # Background, 2 concurrent
kintsugi workflow batch . --force                             # Reprocess completed
kintsugi workflow stop /path/to/KINTSUGI_Projects             # Stop background batch

Eligibility: project has workflow/config.yaml + data/raw/.staged, and is missing data/processed/signal_isolated/.snakemake_complete (unless --force). Projects are processed sequentially by default (all share 5 GPU slots). Use --parallel 2 to split GPU budget across concurrent datasets.

NEVER write custom batch scripts — always use kintsugi workflow batch or kintsugi workflow run. Custom scripts bypass GPU validation, SLURM profile detection, and sentinel tracking.

Legacy scripts (in /blue/maigan/smith6jt/, historical reference only):

• dataset_manifest.csv - Central registry of all 47 datasets (34 spleen/LN + 13 thymus; source paths, parameters, sizes)
• setup_all_projects.sh - Creates KINTSUGI projects for all datasets in manifest
• configure_all_workflows.sh - Generates Snakemake configs for each project
• stage_datasets.sh - Submits SLURM rsync jobs to copy raw data from orange to blue (wave-based)
• stage_datasets_globus.py - Globus-based staging for thymus datasets from PATH lab SMB share
• cleanup_datasets.sh - Verifies EDF outputs + QC sentinels, prompts for QC review, deletes intermediates and raw data (--force skips prompt)

Pipeline lifecycle:

1. setup_all_projects.sh      Create project dirs + copy metadata from orange
2. configure_all_workflows.sh Generate Snakemake configs for all projects
3. stage_datasets.sh 5        Stage next wave of raw data (orange → blue)
4. kintsugi workflow batch .   Process all staged datasets (GPU-validated)
5. cleanup_datasets.sh        Verify outputs + QC, prompt review, delete intermediates + raw
6. Repeat 3-5 for next wave

Why sequential by default: All datasets share 5 GPU slots — parallel Snakemake instances cause contention. The --detach flag survives SSH disconnects. Re-runs auto-skip completed datasets via sentinel files + per-channel skip-existing.

Snakemake lock recovery: If a coordinator dies (tmux disconnect, OOM kill), the lock file persists and blocks re-runs with LockException. Fix: cd /path/to/project/workflow && snakemake --unlock --profile profiles/slurm. Always kill stale coordinators (ps aux | grep snakemake) before unlocking.

Storage: /orange/maigan/ (long-term, ~9.7 TB spleen/LN), PATH lab SMB (thymus, ~2.4 TB via Globus), /blue/maigan/ (processing workspace). Peak blue usage per dataset: ~3x raw size.

CRITICAL rsync rule: Never use bash glob */ on rsync source — it flattens cycle directories. Use trailing / and let rsync handle the tree: rsync -av '${orange_path}/' '${RAW_DIR}/' --include='*/' --include='*.tif' --exclude='*'

Globus staging (thymus datasets from PATH lab): Uses stage_datasets_globus.py with PATH lab GCP endpoint → HiPerGator. Transfer log: /blue/maigan/smith6jt/globus_transfers.json. Requires module load globus and session consent. See stage_datasets_globus.py docstring for endpoint IDs and path quirks.

Sentinel files:

Cleanup QC Guard (added 2026-02-16): cleanup_datasets.sh checks all 4 QC sentinels (qc_plots/.snakemake_complete_{stitch,decon,edf,registration}) before proceeding. If any are missing, the dataset is skipped with "QC not complete". If all are present, the user is prompted to confirm QC review before deletion. Use --force to skip the interactive prompt (for re-runs after initial review). In --dry-run mode, shows what would be prompted without blocking.

./cleanup_datasets.sh              # Interactive: checks QC, prompts before each dataset
./cleanup_datasets.sh --force      # Skip prompts (after initial QC review)
./cleanup_datasets.sh --dry-run    # Preview without deleting

Phases: Discovery → Setup/Staging → Validation → SLURM batch (incl. signal isolation) → QC Review → Cleanup → Segmentation

Sentinel validation for batch-processed projects (Feb 2026): Projects processed by legacy batch scripts (outside Snakemake) have signal_isolation_manifest.json and TIF output but no .snakemake_complete sentinel — making them invisible to kintsugi workflow batch. Use scripts/create_si_sentinels.py to validate output (manifest JSON parse, TIF existence per channel, file size > 0) and create sentinels:

python scripts/create_si_sentinels.py ../KINTSUGI_Projects --dry-run  # Preview
python scripts/create_si_sentinels.py ../KINTSUGI_Projects            # Create sentinels

25 projects validated and promoted this way (Feb 25, 2026). All had quality scores 0.70-0.86.

Stale script deployment hazard: workflow config only copies scripts to projects if they don't already exist. When updating workflow/scripts/*.py (e.g., adding signal_isolation support to qc_report.py), existing project copies remain stale. Fix: bulk-copy ALL scripts to every project that has a workflow/scripts/ directory:

for proj_scripts in ../KINTSUGI_Projects/*/workflow/scripts/; do
    [ -d "$proj_scripts" ] || continue
    for script in deconvolve.py edf.py log_utils.py qc_report.py registration.py \
                  signal_isolation.py spillover_correction.py stitch.py vessel3d.py; do
        cp workflow/scripts/"$script" "$proj_scripts/$script"
    done
done

Also update Snakefiles for projects with outdated workflow architecture:

cp workflow/Snakefile /path/to/project/workflow/Snakefile

Verify with MD5 comparison loop — should report 0 stale across all scripts. Last bulk remediation: Feb 27 2026 (35 projects had stale log_utils.py, 15 had stale registration.py, 6 had stale Snakefiles).

Stale Snakemake lock recovery: If batch processing stalls, check for stale lock files across all projects:

find ../KINTSUGI_Projects -path '*/.snakemake/locks/*' -type f

Safe to remove locks when no snakemake processes are running (ps aux | grep snakemake) and no jobs are queued (squeue -u $USER). Remove both workflow/.snakemake/locks/ and project-root .snakemake/locks/.

Processing status (Feb 27, 2026):