HPC Deucalion

Operational knowledge for running AEGIS-KG on the Deucalion supercomputer. Deucalion is a shared HPC cluster: it uses SLURM for scheduling, Lmod for software modules, has login nodes (interactive, lightweight) and compute nodes (batch, dedicated), and exposes tiered storage. Most local-workstation assumptions (free Docker, persistent background services, free port use) do not apply.

Progressive disclosure: SKILL.md (this file) gives the operational rules and quick-start. Detailed topics live in references/. Copy-pasteable job scripts live in examples/. The companion tutorial for humans lives in docs/deucalion/README.md.

When to Activate

Activate this skill when any of the following is true:

• The user mentions "deucalion", "hpc", "supercomputer", "cluster", "slurm", "sbatch", "srun", "squeue", "login node", "compute node", "gpu partition", "module load", "$SCRATCH", "$WORK", or "walltime".
• The user asks to run eval/ETL/agent workloads at scale on a shared cluster.
• A workflow needs Neo4j or Ollama as long-running services (these must go on compute nodes, not login nodes).
• A planned change involves pip-installing heavy packages, moving large CSVs, or running a job longer than a few minutes.

Staged launch process (MANDATORY, 2026-09-04)

Do NOT submit a job to Deucalion until each previous stage is green.

Why staging matters — confirmed wins (2026-09-02):

• 4 scouts FAILED because Ollama crashed at 5 min (port 11434 collision on same node).
• 3 jobs FAILED with empty logs (pre-cluster update of run-full-m-aegis.sbatch).
• 5 jobs FAILED because model tags were wrong (cache miss + no egress).
• 2 jobs COMPLETED but produced 0 docs because run_all aborted before OUTPUT stage (patched upstream as CORR-114).

Each class is preventable. Stage 0 catches about half before cluster, the others between stages 1 and 2.

Hard Rules (Non-Negotiable)

These apply to any Deucalion work, regardless of task. Violating any of them is a STOP and report to user.

Standard facts — learned the hard way, do not relitigate

These are facts that have bitten us in real sessions. Treat them as known facts from now on; if a new situation contradicts them, gather evidence and surface it rather than re-deriving.

Ollama model tags (must match the exact library name)

Trap to avoid: short-form tags (nemotron3.5:30b, granite:30b, ornith-1.5:9b) silently trigger ollama pull — which fails because the cluster has no egress to https://registry.ollama.ai from compute nodes. Result: minutes wasted, job aborts. Verify the cache directory before sbatch, not the tag string alone. Use examples/gpu-pick.sh <tag> to also pick the right partition.

Two Ollama binaries — pick the right one for the model

0.31.1 cannot resolve modern manifests (granite4.2:30b fails with unknown model architecture). When in doubt, use 0.32.13 from graphify-methodology/.

Parallel jobs (proven pattern, 2026-09-02)

• Across nodes: safe. Multiple scouts on different nodes each get their own GPU and port 11434 — confirmed with JOBs 1869099/100/101. Each lives on its own --gres=gpu:1.
• Same node: 2 jobs collide on 11434 (Ollama server). Resource contention shows up as Connection refused in the 2nd job and the warm-up fails. Mitigation: check squeue -o "%.6R" (Reason column); if both jobs say the same node and it isn't yet fully routed, cancel and submit again.
• Walltime budgets (as of 2026-09-02, battle-tested):
  • Phase 1B scout (4 LLM calls + Doc 05 only): 30 min.
  • Phase 1+2+3 scout-full (full pipeline, 9 docs + xlsx): 1 h 30.
  • Full run-all (warm-up + pipeline + bulk output): 8 h.

Stale .pyc cache after a code sync (proven 2026-09-02)

When you sync updated Python files to Deucalion (via scp, rsync, or tar extract) the cluster's _archive/.../markdown_parser.py is fresh but the corresponding __pycache__/*.pyc is stale. Python happily loads the old bytecode, finds ImportError: cannot import name 'X', and the job aborts ~1 minute after warm-up with a confusing trace.

Always include these in sbatch scripts before python is invoked:

echo "=== wiping stale .pyc + __pycache__ (defense-in-depth) ==="
find "$PROJ/src" -name "__pycache__" -type d -exec rm -rf {} + 2>/dev/null || true
find "$PROJ/src" -name "*.pyc" -delete 2>/dev/null || true
PYTHONDONTWRITEBYTECODE=1
export PYTHONDONTWRITEBYTECODE

The byte-code-suppression flag is also useful for --gres shared GPU runs where two Python processes might write to the same __pycache__ directory simultaneously and corrupt each other.

scp and $HOME quota

• The user's $HOME on Deucalion has a tight quota (a few GB). scp ... ~/ often fails with Disk quota exceeded. Stage through /tmp (per-job, auto-cleared): scp foo.py user@login:/tmp/, then ssh user@login 'cp /tmp/foo.py <dest>'.
• Two scp calls with the same basename to the same /tmp/ overwrite each other (no subdir mapping). Use unique basenames, or combine into a single tar -cz first.
• Always chmod +x after the copy if the destination is a script.

Job-name uniqueness — squeue is unreadable if everything is aegis_scout

The --job-name directive is pre-evaluated; you cannot use $1 in it. For multiple parallel scouts, pass the sanitised model tag via the sbatch CLI:

sbatch -J "aegis_<sanitised-tag>_full" <script> <model:tag>

where <sanitised-tag> is the model tag with : and . replaced by _. Use the wrapper scripts/scouts/scout-full-submit.sh to print the exact line — otherwise each scout ends up with the same aegis_scout_full job-name and squeue becomes a wall of duplicates.

Egress

Compute nodes cannot reach registry.ollama.ai (HTTPS timeout). New models must be pulled on the login node with the graphify 0.32.13 binary in /projects/F202512235CPCAA1/graphify-methodology, then the manifests land in the shared /projects/F202512235CPCAA1/CyberMetric_Deucalion/ollama_data/models/ cache and are visible to compute nodes.

Monitoring jobs

• squeue -u $USER — quick status (only running/pending; finished jobs drop off after a few minutes).
• sacct -j <JOBID> --format=JobID,State,ExitCode,Elapsed,MaxRSS — post-mortem state and resource use.
• scontrol show job <JOBID> — full live state (good for diagnostics like Reason, StdOut path, NodeList).
• cat /projects/<...>/<project>/logs/scout_runs/scout_<tag>_<JOB>.log for the per-run structured log (preferred over slurm-*.out).
• cat /projects/<...>/<project>/slurm-<script>-<JOB>.out for stdout/stderr (slurm-managed, slow to appear).

Path conventions for AEGIS-Phase-1

• Active project dir: /projects/F202512235CPCAA1/CyberMetric_Deucalion/aegis-phase1. (The old ~/aegis-kg path is the CORR-057 eval layout; superseded.)
• Output dirs (created by --output):
  • output/run_<model>_<JOB> for full run-alls.
  • output/scout_<model>_<JOB> for Phase 1B-only scouts.
  • output/scout_full_<JOB> for pipeline-complete scouts.
• Run logs: logs/scout_runs/scout_<tag>_<JOB>.log.
• Per-model JSONL traces: logs/phase1/<model>/v2/pipeline_<model>.log.

Quick-Start (5 Steps)

1. Access. SSH to the login node:

   ssh -i /home/epmq-cyber/.ssh/id_ed25519 paulinho@login.deucalion.macc.fccn.pt

   If 2FA is required, follow the cluster's MFA flow (typically TOTP + SSH key).
2. Clone the repo (code lives in ~/aegis-kg).

   # Deucalion has no $WORK. Code in ~/aegis-kg.
   cd ~
   # HTTPS git clone fails (needs auth). Use tar+scp from workstation:
   #   on workstation: tar --exclude='.venv' --exclude='__pycache__' -czf aegis-kg.tgz aegis-kg/
   #   scp aegis-kg.tgz paulinho@login.deucalion.macc.fccn.pt:~
   #   on cluster: tar xzf aegis-kg.tgz

3. Load the modules and create the venv (interactively, on a login node is OK for this step — no heavy compute). Modules need login shell. If module load fails, use bash -lc 'python ...'. venv needs --system-site-packages (see references/04). See references/04-environment-setup.md.
4. Submit a scout first, then the heavy work via sbatch.
   • examples/scout.sbatch — 5 min CPU job that prints a full environment report
   • examples/test-ollama-gpu.sbatch — real run with Ollama on GPU + Langfuse Cloud
   • examples/test-ollama-cpu.sbatch — CPU fallback when no GPU available
   • examples/neo4j-server.sbatch — start Neo4j on a compute node
   • examples/ollama-gpu.sbatch — start Ollama (with GPU) on a compute node
   • examples/eval-batch.sbatch — run the eval harness in batch
   • examples/run-etl.sbatch — run the ETL pipeline
5. Monitor with squeue -u $USER and read job logs from slurm-<JOBID>.out. See references/07-slurm-jobs.md for full inspection/debugging.

Ollama on Deucalion

Ollama is pre-installed at /projects/F202512235CPCAA1/CyberMetric_Deucalion/bin/ollama. Use this path — it's faster and more stable than the module.

# In your sbatch scripts (or login shell):
export PATH="/projects/F202512235CPCAA1/CyberMetric_Deucalion/bin:$PATH"
export LD_LIBRARY_PATH="/projects/F202512235CPCAA1/CyberMetric_Deucalion/lib/ollama:$LD_LIBRARY_PATH"
export OLLAMA_MODELS="/projects/F202512235CPCAA1/CyberMetric_Deucalion/ollama_data/models"

Models are pre-loaded in the shared cache above. To add new models, run examples/download-models.sbatch (a separate sbatch job on normal-a100-40). Do NOT ollama pull inside AEGIS-KG jobs — compute nodes have spotty egress to registry.ollama.ai.

vLLM on Deucalion (AEGIS-KG CORR-110, 2026-09-02, unverified)

For models that the local Ollama 0.31.1 daemon refuses to load (Qwen3.8-Flash-Next, qwen3.8:27b, anything outside the known-arch list), or when you want the exact HF model id with no tag drift, use vLLM. The aegis-phase1 pipeline talks to any OpenAI-compatible HTTP server through ChatOpenAICompat (no new project dep — only httpx).

# Pilot sbatch (single-job, TP=8 on dev-a100-80 for Qwen3.8-Flash-Next):
sbatch /projects/F202512235CPCAA1/CyberMetric_Deucalion/aegis-phase1/examples/deucalion/scout-vllm-qwen-flash-next-aegis.sbatch
# Override default scope (run-all → phase-1b for cheap scout):
sbatch --export=AEGIS_SCOPE=phase-1b \
       /projects/F202512235CPCAA1/CyberMetric_Deucalion/aegis-phase1/examples/deucalion/scout-vllm-qwen-flash-next-aegis.sbatch

Prereqs: weights pre-staged on Lustre under $BD/models/hf/Qwen_Qwen3.8-Flash-Next/ (~360 GB — no egress from compute nodes); a venv with vllm at $AEGIS_VLLM_VENV (default /projects/F202512235CPCAA1/cybermetric-vllm-cache/venv — the sibling project's working install, but verify it still works after a cluster change). The pilot sbatch defaults to TP=2 on A100-40 (sized for the original gemma-4-31B-it pilot); you MUST override --gres=gpu:a100:8 and -p dev-a100-80 before submitting for Qwen3.8-Flash-Next (125B+ params ≈ 360 GB BF16 — won't fit otherwise). Recipe details + patterns A/B/C + TP=N sizing + likely gotchas: see references/11-running-vllm.md. The branch carrying the wiring is feature/aegis-p1-corr-110-vllm-provider in aegis-phase1. No successful cluster run logged yet — the section above is the documented recipe, not a verified pattern.

Ollama version split (2026-09-01, battle-tested)

There are two Ollama binaries on the cluster and they are NOT interchangeable:

Pull recipe (login node, spotty egress): serve 0.32.13 locally, pull one model at a time with timeout 600, verify the manifest appeared under ollama_data/models/manifests/registry.ollama.ai/library/<model>/. Exact tag names matter — check the Ollama library page first (granite4.2:30b exists; granite4:30b does not; nemotron-3.5-lightning:30b, muse-glimmer:30b|latest, ornith:9b all pulled successfully 2026-09-01).

AEGIS Phase 1 scout recipe (2026-09-01, battle-tested)

The aegis-phase1 project runs model scouts (Phase 1B only: 4 LLM calls) on dev-a100-80. Use the generic sbatch aegis-phase1/examples/deucalion/scout-bench-m-aegis.sbatch — pass the model as a positional argument, never via --export (flaky):

# from the aegis-phase1 dir on the login node:
sbatch examples/deucalion/scout-bench-m-aegis.sbatch granite4.2:30b

Rules encoded in that sbatch (do not relearn the hard way):

1. Sequential, never parallel. One Ollama per node. Three 30B scouts in parallel → Connection refused / Server disconnected on all of them. Submit, wait for SUCCESS, then submit the next.
2. Logs to NFS, not compute-node scratch: --out/--err point to $PROJ/slurm-scout-bench-m-<JOBID>.{out,err} and a run log to $PROJ/logs/scout_runs/scout_<model>_<JOBID>.log. Scratch logs are unrecoverable after the job ends.
3. Warm-up with retries before python. 5 attempts, 180s timeout each; abort before wasting the pipeline run if the model won't load.
4. Walltime: budget ≥2× the slowest previous scout (qwen3.8 = 18 min, ornith:9b = 5:36, 30B models ≈ 15-25 min). 30 min is the working default for Phase 1B scouts on 1× A100-80.
5. Active project dir: /projects/F202512235CPCAA1/CyberMetric_Deucalion/aegis-phase1 (NOT ~/aegis-kg — that is the old CORR-057 eval layout).
6. Monitor: squeue -u $USER + sacct -X -j <JOBID> -n -P -o "JobID,State,ExitCode,Elapsed". Success = STATUS: SUCCESS in the run log AND rationale_by_reg has 2 entries in stderr.
7. Post-run: scp Doc 05 to the local mirror <workspace>/Deucalion/results/<model>_<jobid>/, then evaluate: PYTHONPATH=src python3 scripts/eval/generate_report.py --run-dir <mirror_dir> --preproc preproc_out --output-{dir,md,json} ... --use-parser-gate (see execution/reports/EVAL_PROTOCOL.md for the judge pass).

Operational Cheat Sheet

Topic Index — Read on Demand

Templates (Copy-Paste)

How This Skill Coordinates with Others

• neo4j-verify — invoke before any Neo4j work (HPC or not). Port rules still apply (7688/7475).
• etl-runner — invoke when running ETL. On HPC, the execution part of ETL must go through sbatch (see examples/run-etl.sbatch).
• eval-runner — invoke when running eval. On HPC, prefer batch mode with examples/eval-batch.sbatch instead of interactive runs.
• python-best-practices — still required for any Python change. venv location differs (per-project inside ~/aegis-kg instead of ~/shared-venv).
• agents-md-writer — invoke when updating docs/deucalion/ or AGENTS.md sections related to Deucalion.

Async Workflow on Deucalion

Running on HPC is asynchronous: you submit, the scheduler decides when your job runs, and you collect results later. The full cycle is:

1. Workstation: git push your changes
2. Login node: cd ~/aegis-kg && git pull, edit .aegis_env if needed
3. Login node: sbatch scout.sbatch to verify environment
4. Login node: sbatch test-ollama-gpu.sbatch (or -cpu)
5. Login node: squeue -u $USER + tail -f slurm-<JOBID>.out to monitor
6. Job ends: results land in ~/aegis-kg/results/<JOBID>/
7. Workstation: rsync results back, inspect, iterate

If the GPU queue is long, see the queue strategy in references/07-slurm-jobs.md and the human-oriented docs/deucalion/08-hpc-workflow.md in the consuming project.

Stop Conditions

Stop and report to the user if:

• The user gives an ambiguous account/partition/QoS — wrong flags = job rejected.
• A required module is missing on the cluster (module avail shows nothing).
• The job hits a quota error — do not retry blindly.
• The user asks to "make Neo4j persistent across jobs" — this requires site-specific setup (e.g., a reserved service node). Confirm before attempting.
• A sbatch script will run >24h without checkpointing — design needs review.

Placeholders

All site-specific values are real. No placeholders to substitute.

Error playbook (2026-09-02 + 2026-09-04 lessons)

Run grep -ho "<token>" ~/.zcode/cli/rollout/*.jsonl | sort | uniq -c | sort -rn on the prior sessions to count these if you're unsure. Recurring errors with their cause and the resolver:

Companion: Human-Oriented Workflow Doc

The consuming project ships docs/deucalion/08-hpc-workflow.md which is the step-by-step async cycle (workstation → login → sbatch → wait → collect → iterate) and the queue-handling strategies. Point users there for the narrative view; this SKILL.md is the operational reference.