Code Reproducer Skill

Purpose

Automate the full reproduction of a research paper's experiments on remote GPU servers. This skill reads the analysis report from code-analyzer and executes the reproduction plan via mcp-ssh.

Architecture

code-analyzer (analysis)  →  code-reproducer (this skill)  →  mcp-ssh (SSH)
  └── code_analysis.json       └── Reads analysis report        └── SSH connection
      ├── Framework                  ├── Environment setup          ├── Command execution
      ├── Training scripts           ├── Training execution         ├── File upload/download
      ├── Configs                    ├── Progress monitoring        ├── tmux sessions
      └── Reproduction plan          └── Result downloading         └── Persistent sessions

Prerequisites

• code-analyzer must have been run first (produces code_analysis.json)
• mcp-ssh must be installed and configured in the MCP client
  • Install: git clone https://github.com/shuakami/mcp-ssh.git && cd mcp-ssh && npm install && npm run build
  • Config: Add to your MCP config (Cursor/Claude Desktop mcp.json)
  • Docs: https://github.com/shuakami/mcp-ssh

When to Use

• After paper-downloader has cloned source code
• User asks to "reproduce", "train", "run experiments", or "replicate results"
• User wants to execute code on a remote GPU server

Workflow

Phase 1: Analyze Repository (LOCAL) — via code-analyzer

Run the code-analyzer skill first:

python code-analyzer/analyze.py workspace/<paper>/code/<repo>/ -o workspace/<paper>/code_analysis.json

This produces a comprehensive JSON report containing framework detection, AST call graph, training loop analysis, reproducibility score, and a step-by-step reproduction plan. See code-analyzer/SKILL.md for full details.

Phase 2: Setup Server (via mcp-ssh)

First-time setup — ask the user these questions:

1. "What is your GPU server's SSH address and port?"
2. "What is your username?"
3. "Do you use SSH key or password authentication?"
4. "Does your server need a jump/bastion host?"

Then use mcp-ssh to:

1. Create SSH connection: ask mcp-ssh to connect to the server
2. Test connection: run echo "Connected" and nvidia-smi
3. Create a tmux session for persistence: tmux new-session -d -s repro

Phase 3: Upload Code (via mcp-ssh)

1. Create remote workspace: mkdir -p ~/reproduce/<project>
2. Upload code via mcp-ssh file upload tool
3. Verify: ls -la ~/reproduce/<project>/

Phase 4: Setup Environment (via mcp-ssh)

Based on code_analysis.json, execute the appropriate commands IN the tmux session:

For conda + requirements.txt:

tmux send-keys -t repro "conda create -n repro_<project> python=3.10 -y" Enter
# Wait for completion, then:
tmux send-keys -t repro "conda activate repro_<project>" Enter
tmux send-keys -t repro "cd ~/reproduce/<project> && pip install -r requirements.txt" Enter

For environment.yml:

tmux send-keys -t repro "conda env create -f ~/reproduce/<project>/environment.yml -n repro_<project>" Enter

For Docker:

tmux send-keys -t repro "cd ~/reproduce/<project> && docker build -t repro ." Enter

After setup, verify:

# For PyTorch
python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, GPUs: {torch.cuda.device_count()}')"

# For TensorFlow
python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

Phase 5: Run Training (via mcp-ssh + tmux)

[!IMPORTANT] Always use tmux for training. This ensures the training continues even if the SSH session drops.

1. Start training in the tmux session:

tmux send-keys -t repro "cd ~/reproduce/<project>" Enter
tmux send-keys -t repro "python <training_script> <args> 2>&1 | tee training.log" Enter

2. The training command comes from code_analysis.json → reproduction_plan:
   • First priority: README training commands
   • Second: detected training scripts (with confidence scores)
   • Third: ask the user

Phase 6: Monitor Training (periodic)

[!TIP] For long training runs: After starting training, wait/sleep for a reasonable interval (e.g., 10-30 minutes for quick tests, 1-2 hours for full training), then check progress.

Check progress via mcp-ssh:

# Get latest log lines
tmux capture-pane -t repro -p | tail -20

# Or check the log file
tail -20 ~/reproduce/<project>/training.log

# Check if process is still running
ps aux | grep python | grep train

# Check GPU usage
nvidia-smi

What to look for:

• Loss decreasing → training is working
• OOM errors → reduce batch size
• CUDA errors → check CUDA version compatibility
• NaN loss → check learning rate, data preprocessing
• Import errors → missing dependency, install it

Error recovery: If training fails:

1. Read the error message
2. Diagnose and fix (install missing package, adjust hyperparameters)
3. Restart in the same tmux session

Phase 7: Download Results (via mcp-ssh)

After training completes:

1. Check what was generated:

find ~/reproduce/<project> -name "*.pt" -o -name "*.pth" -o -name "*.ckpt" -o -name "*.png" -o -name "*.csv" -o -name "*.log" | head -30

2. Download result files via mcp-ssh to local:
   • Model checkpoints
   • Training logs
   • Generated images/figures
   • Evaluation metrics
3. Save to workspace/<paper>/results/

Phase 8: Report to User

Tell the user:

• ✅ Training completed (or ❌ failed with reason)
• Duration
• Final metrics (loss, accuracy, FID, etc.)
• Files downloaded
• Next steps (run result-analyzer for comparison with paper)

Dependencies

• code-analyzer skill (for repository analysis)
• mcp-ssh for SSH operations (https://github.com/shuakami/mcp-ssh)

Common Pitfalls and Solutions