Remote GPU Experiments

Adapt the execution strategy to the actual host, repository, and user request.

Requirements

• Inspect the relevant local code and remote environment before making changes.
• Detect the host's runtime, GPU tooling, storage layout, and job-control mechanism instead of assuming them.
• Preserve existing local and remote data. Verify synchronization targets before using options that remove or overwrite files.
• Validate launch-critical credentials and configuration from the path the remote worker will read after transfer. Check formatting and permissions without exposing secrets, and make a minimal authenticated request through the same integration when applicable; a source-side check cannot detect transfer corruption.
• For long-running jobs, use a durable execution mechanism that outlives the SSH session, and retain useful logs.
• After launching work, provide the commands needed to inspect, attach to, and stop it.
• Avoid installing packages or rebuilding the remote environment unless necessary and within the requested scope.
• When comparing variants, keep conditions comparable and report the results as a comparison.
• Detect the local shell and toolchain as well as the remote one. A macOS controller has zsh and BSD tools, so GNU-only flags such as find -printf fail locally.
• Run anything longer than one line on a remote host or container as a script file: write it locally, copy it over, and run it with ssh host bash <file> or docker exec <container> bash <file>. Do not nest quoting across the local shell, ssh, docker exec, and bash -c or inline heredocs.
• Guard Python multiprocessing entry points, such as inference engines that spawn workers, with if __name__ == "__main__": in probe scripts.
• Check and report the exit status of every verification step. A failed or skipped check blocks the claim it supports; do not infer success from partial output.

Tool Selection

• Use ssh for remote inspection and command execution.
• Use rsync for repeated directory transfers; use scp, sftp, or an equivalent tool for simple one-off transfers.
• Use an available persistent session or process manager such as tmux. If the host is a managed cluster, beware of the cluster's scheduler and job submission system. Follow system/project-specific guidelines.
• Use diagnostics appropriate to the host, such as scheduler status commands, nvidia-smi, or rocm-smi.
• Capture stdout and stderr in durable logs, and use an appropriate transfer tool to retrieve results.

Host References

Read references/autodl-preset.md only when working with the user's autodl SSH target.