OSMO LeRobot Training

Submit, monitor, analyze, and evaluate LeRobot behavioral cloning training workflows on the OSMO platform. Covers the full lifecycle: job submission, log streaming, Azure ML metric retrieval, training summary generation, and post-training inference evaluation.

Read the skill file .github/skills/osmo-lerobot-training/SKILL.md for parameter defaults, GPU configuration, and training duration estimates. Read references/DEFAULTS.md for known datasets, GPU profiles, and Azure environment auto-resolution.

Prerequisites

Authentication must be configured before any OSMO or Azure ML operations:

az login
osmo login <service-url> --method dev --username guest

Quick Start

Train from Azure Blob Storage (typical production flow)

scripts/submit-osmo-lerobot-training.sh \
  -d my-robot-dataset \
  --from-blob \
  --storage-account mystorageaccount \
  --blob-prefix my-robot-dataset \
  --no-val-split \
  --steps 100000 \
  --batch-size 32 \
  --learning-rate 1e-4 \
  --save-freq 10000 \
  -j my-robot-act-train \
  --experiment-name my-robot-training \
  -r my-robot-act-model

Train from HuggingFace Hub

scripts/submit-osmo-lerobot-training.sh -d lerobot/aloha_sim_insertion_human

Run Continuous Eval During Training (preferred)

Start the background poller immediately after submitting training. It watches AzureML for new checkpoint versions and submits an inference job per version automatically, stopping when training reaches a terminal state.

# Launch in the background — runs until training completes
nohup scripts/poll-and-eval-checkpoints.sh \
  --model-name my-robot-act-model \
  --training-workflow-id lerobot-training-32 \
  --blob-prefix my-robot-dataset \
  --job-prefix my-robot-eval \
  --experiment-name my-robot-inference \
  --poll-interval 60 \
  --max-concurrent 2 \
  > /tmp/my-robot-eval.log 2>&1 & disown

# Monitor the poller
tail -f /tmp/my-robot-eval.log

The poller caps concurrent inference workflows at --max-concurrent (default 2) to avoid cluster saturation. Submitted versions are tracked in /tmp/<model-name>-submitted-versions.txt.

Run a Single Inference Job

# OSMO inference (GPU, evaluates against the same dataset)
scripts/submit-osmo-lerobot-inference.sh \
  --from-aml-model \
  --model-name my-robot-act-model \
  --model-version 3 \
  --from-blob-dataset \
  --storage-account mystorageaccount \
  --blob-prefix my-robot-dataset \
  --mlflow-enable \
  --eval-episodes 10 \
  -j my-robot-eval \
  --experiment-name my-robot-inference

# Local inference (CPU/MPS, for quick validation)
python scripts/run-local-lerobot-inference.py \
  --model-name my-robot-act-model \
  --model-version 3 \
  --dataset-dir /path/to/local/dataset \
  --episodes 5 \
  --output-dir outputs/local-eval \
  --device cpu

Post-Submission Browser Monitoring

After every successful training or inference submission, open the OSMO workflow page in VS Code's SimpleBrowser so the user can track progress and access logs directly.

Steps:

1. Capture the workflow ID from the submission output (the line Workflow ID - <id>).
2. Build the URL from the service URL used for osmo login: <service-url>/workflows/<workflow-id>. Don't rely on the Workflow Overview line, which can show an in-cluster address that a browser can't reach.
3. Open it with the open_browser_page tool (VS Code SimpleBrowser).
4. Tell the user that the Logs tab on that page streams live output per task (e.g., lerobot-train, lerobot-infer).

Example after a training submission:

Workflow ID - lerobot-training-31
Workflow Overview - <service-url>/workflows/lerobot-training-31

Open: <service-url>/workflows/lerobot-training-31

Example after an inference submission:

Workflow ID - lerobot-inference-20
Workflow Overview - <service-url>/workflows/lerobot-inference-20

Open: <service-url>/workflows/lerobot-inference-20

The page has a Logs tab with per-task log streams. For training, select the lerobot-train task. For inference, select the lerobot-infer task. Use the OSMO CLI (osmo workflow logs <id> -t <task> -n 100) as a fallback when the browser is not reachable.

Azure ML Portal Monitoring (Playwright)

After submitting a training job, and whenever the background eval poller reports a new inference job, open the Azure ML portal with Playwright to view live metrics and trajectory plots. Use mcp_playwright_browser_navigate, mcp_playwright_browser_snapshot, mcp_playwright_browser_click, and mcp_playwright_browser_take_screenshot.

Training Metrics — Open Immediately After Submission

After the training job is submitted, navigate to the training experiment page and open the Metrics tab:

1. Construct the experiment URL from Azure environment variables in scripts/.env:

   https://ml.azure.com/experiments/{experiment_name}?wsid=/subscriptions/{AZURE_SUBSCRIPTION_ID}/resourceGroups/{AZURE_RESOURCE_GROUP}/providers/Microsoft.MachineLearningServices/workspaces/{AZUREML_WORKSPACE_NAME}

2. Call mcp_playwright_browser_navigate with that URL.
3. Call mcp_playwright_browser_snapshot to confirm the page loaded and identify the latest run row in the table.
4. Click the first (most recent) run link.
5. On the run detail page, call mcp_playwright_browser_snapshot to locate the Metrics tab.
6. Click Metrics.
7. Call mcp_playwright_browser_take_screenshot and show the live training curves to the user.

Key metrics to surface: train/loss, train/learning_rate (confirm 1e-04, not 1e-05), train/grad_norm, gpu_percent.

Refresh by calling mcp_playwright_browser_navigate again on the same URL at any time.

See references/REFERENCE.md for exact click paths, tab selectors, and screenshot guidance.

Inference / Eval Plots — Open When Poller Submits a Job

While the background eval poller is running, monitor the poller log and navigate to Azure ML to view trajectory plots as each inference job completes:

1. Tail the poller log to detect a new inference submission:

   tail -n 30 /tmp/<model-name>-eval.log | grep -E "Submitting|Workflow ID"

2. Construct the inference experiment URL using the --experiment-name passed to the poller:

   https://ml.azure.com/experiments/{inference_experiment_name}?wsid=/subscriptions/{AZURE_SUBSCRIPTION_ID}/resourceGroups/{AZURE_RESOURCE_GROUP}/providers/Microsoft.MachineLearningServices/workspaces/{AZUREML_WORKSPACE_NAME}

3. Call mcp_playwright_browser_navigate with that URL.
4. Call mcp_playwright_browser_snapshot to identify the latest run row (most recently submitted checkpoint eval).
5. Click that run.
6. On the run detail page, click the Images tab.
7. Call mcp_playwright_browser_take_screenshot and show the trajectory plots to the user.

The Images tab contains per-episode trajectory plots logged by the inference job (episode_NNN_trajectory.png and eval_summary.png). They appear after the OSMO inference workflow reaches completed status. If images are not yet present, check osmo workflow query <inference-workflow-id> and wait for completed.

Parameters Reference

Training Submission Parameters

Inference Submission Parameters

Continuous Evaluation Parameters (poll-and-eval-checkpoints.sh)

GPU Configuration Guidelines

Azure ML Context

Resolved from CLI flags > environment variables > Terraform outputs:

Training Completion Estimation

Estimate training duration based on dataset and configuration:

Checkpoints are registered to AzureML at every --save-freq interval. Jobs may be evicted on spot GPU instances — checkpoints already registered remain available for inference even if the job is interrupted.

OSMO CLI Reference

See references/REFERENCE.md for full CLI and SDK documentation.

osmo workflow query <workflow-id>
osmo workflow logs <workflow-id> -n 100
osmo workflow logs <workflow-id> --error
osmo workflow list
osmo workflow cancel <workflow-id>

Checkpoint Poller Commands

# Start continuous eval loop in background
nohup scripts/poll-and-eval-checkpoints.sh \
  --model-name <model-name> \
  --training-workflow-id <workflow-id> \
  --blob-prefix <dataset-blob-prefix> \
  > /tmp/<model-name>-eval.log 2>&1 & disown

# Monitor poller
tail -f /tmp/<model-name>-eval.log

# Check which versions have been submitted
cat /tmp/<model-name>-submitted-versions.txt

# Stop the poller early
pkill -f poll-and-eval-checkpoints

Key Metrics Logged

Troubleshooting

See references/REFERENCE.md for detailed debugging commands.

Brought to you by microsoft/physical-ai-toolchain