SSH GPU/HPC Execution

Capability Routing

This is a first-layer entry skill. Keep it loaded as the user-facing route; when a request needs a specialized capability below, resolve the debuffer repo root from .debuffer_skills/installed-skills-codex.txt (repo_root) when available, read the referenced library SKILL.md, then follow that skill. Do not copy the whole library skill into this file.

• /system-profile: read ../library/experiments/system-profile/SKILL.md.
• /training-check: read ../library/experiments/training-check/SKILL.md.
• /run-experiment: read ../library/experiments/run-experiment/SKILL.md.
• /analyze-results: read ../library/experiments/analyze-results/SKILL.md.
• /hpc-formal-run-gate: read ../library/experiments/hpc-formal-run-gate/SKILL.md.

Use this skill for AutoDL or other SSH GPU/HPC machines. Determine the machine type first: AutoDL-managed, shared HPC with a scheduler, or a standalone shared server. Use provider defaults only for the matching type; otherwise use the project contract or SSH alias supplied by the owner.

For an AutoDL-managed instance, read references/autodl-hpc.md. For a non-AutoDL, shared, non-Slurm, or advisor-provided machine, read references/shared-hpc-project-contract.md and instantiate assets/HPC_PROJECT_CONTRACT.template.md in the project. Do not load the AutoDL provider guide merely because the skill name starts with autodl.

A user- or organization-supplied machine policy overrides provider defaults in this skill; if two safety rules conflict, apply the stricter rule. For any nontrivial remote environment setup or rebuild, also read ../shared-references/compute-env-contract.md and use an environment specification, hash ledger, and smoke witness before declaring the machine ready.

Direct SSH Contract

• AutoDL console commands apply only to AutoDL-managed instances. For another machine, use only the alias, key path, host, port, and user supplied by the owner; never infer them.
• In AutoDL mode only, start from the exact SSH command shown in the console, for example: ssh -p <PORT> root@connect.<REGION>.seetacloud.com.
• All connection and repository examples in this skill are templates. Replace <PORT>, <REGION>, <project_slug>, <owner>, <private_repo>, <repo_name>, and <branch> only with values supplied for the current machine and repository. Never reuse values from an earlier project or infer missing values.
• Treat the host, port, username, and password as session-specific secrets. Never commit them, place the password in a command block, or put the password in a runbook. The user enters a password interactively, or configures an AutoDL public key through the provider console.
• Store the connection as placeholders or an SSH config alias only after the user has created it locally. Do not invent a region, port, host, or alias.
• Use the same approved alias/endpoint for ssh, scp, and monitoring. Do not silently fall back to an old server alias. Shutdown verification applies only where shutdown is allowed by the machine contract.
• The backslash sometimes shown before @ is escaping from a display layer; the shell command is root@host.
• In AutoDL mode, distinguish login credentials from the machine-generated GitHub Deploy Key. Neither should be regenerated merely because an SSH session ended. Reuse the existing key after a normal restart of the same instance and rotate only after destruction, loss, revocation, or an explicit request.
• In AutoDL mode only, route private GitHub setup through the reference's six-step machine-key workflow. For another machine, follow the project contract's server-generated Deploy Key procedure.

Customized Pack Defaults

• Prefer command preparation over direct SSH execution. If the user has not explicitly approved remote execution, output the exact command block using the configured alias or provider endpoint and wait for pasted results.
• Read-only inspection may proceed without a second approval only when the active machine contract explicitly allows it. Environment setup, writes, downloads, smoke runs, and formal execution still require scoped approval.
• Keep local work lightweight: repo edits, tests, lint, config validation, manifest updates, and result audit. Heavy training belongs on AutoDL/HPC.
• Do not implement fully autonomous SSH workflows. SSH remains a manual or explicitly approved boundary. Shutdown is always a separate action.
• Keep runbooks concise. Merge repeated operational notes into the project's HPC_PROJECT_CONTRACT.md or one provider runbook instead of generating a new long Markdown file for every attempt.
• Read ../shared-references/autosci-lite-patterns.md for the pilot gate and negative-memory rules. Formal suites require a passed pilot or an explicit waiver recorded in the runbook.

Workflow

1. Classify the current phase: bootstrap, data staging, setup, preflight, smoke, pilot gate, formal approval, formal execution, result download, or local audit.
2. Confirm the repo has the expected contracts before remote execution:
   • docs/runbooks/HPC_PROJECT_CONTRACT.md or an equivalent machine contract
   • a project setup command or scripts/hpc/setup.sh
   • a machine preflight command or scripts/hpc/preflight.py
   • a smoke wrapper or scripts/hpc/run_smoke.sh
   • scripts/run_experiment_suite.py
   • scripts/analysis/audit_run_bundles.py
   • experiments/suites/smoke.yaml
   • data/DATA_MANIFEST.md when large uploaded data exists
   • a concise machine runbook or project contract
3. Treat target-machine smoke as engineering validation only. Do not present smoke metrics as paper evidence or formal results.
4. Prepare a clean local package before touching the remote machine: syntax checks, suite dry-run, manifest generation, job-count inspection, then git commit and git push. Do not use the server as a development workspace.
5. Keep code sync Git-based when possible: commit and push locally, then use the project alias/endpoint to run git pull --ff-only on the server. If GitHub access from the server is unavailable, use narrowly scoped scp/rsync transfers; never copy a local private key to the server.
6. Always use the Python/environment path specified by the project contract. Use a placeholder plus the contract path in generic command blocks. The /root/miniconda3 example belongs only to an AutoDL instance that actually has that environment.
7. Network policy comes from the machine contract. /etc/network_turbo and AutoDL mirrors apply only in AutoDL mode; on another machine use only verified reachable endpoints and approved mirrors.
8. Run setup, preflight, dry-run, smoke, formal dry-run, and formal manifest generation before any formal execution. If setup changes Python/CUDA/packages/weights, update the compute environment ledger first and rerun the smoke witness.
9. Run or review a bounded pilot gate before formal suites. Pilot output stays under experiments/runs/pilot/... or another raw run path and is not paper evidence.
10. Require explicit user approval before enabling or running any formal suite. Dry-run formal suites first.
11. Launch long formal work only in a detached screen/tmux session, or the contract's process manager, with logs and PID metadata. AutoDL shutdown protocol applies only to an owned AutoDL instance; shared HPC is never shut down unless its owner explicitly requests it.
12. Download raw run folders back under experiments/runs/...; do not copy them into experiments/results/ until local audit and result-to-claim scripts have passed.

Key Rules

• Do not hand-edit server code for formal experiments. Change locally, commit, push, then pull fast-forward on the target machine.
• Do not use AutoDL paths, package environments, network accelerators, or shutdown commands on another machine. Use the active project contract.
• Do not reboot or power off a shared HPC machine unless its owner explicitly requests that exact action.
• Do not use pkill, killall, broad process-name matching, or terminate a PID whose project, command, and ownership have not been verified.
• Do not exceed the machine's account-wide GPU, CPU, RAM, storage, or concurrency budget. Record the selected GPU UUIDs and resource envelope before launch.
• Do not start a long formal run directly from the SSH foreground shell.
• Do not auto-shutdown on failure. AutoDL success-only shutdown applies only in AutoDL mode.
• Do not treat a normal stop/restart of the same instance as key loss.
• In AutoDL mode, do not replace an existing working GitHub Deploy Key without first checking /root/.ssh/ and the effective SSH configuration.
• Do not call shutdown successful because the remote command returned; success requires a subsequent SSH connection attempt to fail.
• Do not automate shutdown verification with a plaintext password or sshpass. Batch verification requires a preconfigured key or agent.
• Do not upload the whole local repository with FileZilla/SFTP, and never sync .git/.
• Do not rely on unapproved runtime downloads. Use the network endpoints and mirrors named in the project contract, and verify revision, size, and hashes.
• Do not write smoke output into experiments/results/.
• Do not treat pilot output as paper evidence; use it only to decide whether formal runs are safe to launch.
• Do not edit a frozen protocol after seeing smoke or formal outputs unless making an explicit protocol revision.
• Stop at a failed gate and report the failed command, artifact path, and required fix.
• If pilot or smoke fails, update experiments/NEGATIVE_RESULTS.md with the run id, artifact path, failure reason, and decision.

Coordination

If the repo needs structural changes before HPC work, use the research-repo architecture conventions first: reusable code in src/, thin entrypoints in scripts/, suites in experiments/suites/, raw outputs in experiments/runs/, and curated outputs only in experiments/results/.