Dataviewer Skill

Launch and interact with the Dataset Analysis Tool, a full-stack application for analyzing and annotating robotic training data from episode-based datasets. Complete requested operations with persistence read-back; job acceptance is not inference completion or label application.

Prerequisites

The backend virtual environment and repository-root npm workspace dependencies are auto-created on first launch by start.sh.

Launch and Connect Workflow

Follow these steps when a launch is required and authorized. Inspect existing terminals/task output first and reuse healthy services with matching configuration. Read-only inspection does not authorize restarts, inference or dataset writes. Use VS Code tasks for an already configured development session; use the launcher for requested overrides and readiness checks.

When a caller or workflow supplies the URL of a Dataviewer it already runs, such as the instance the Sim Workspace Command Center opens for a workspace, skip Step 1 and open that URL in Step 2. Start start.sh only when no running instance is supplied, so one dataset folder never gets a second Dataviewer.

Step 1 — Start the app

Launch start.sh as a background terminal process. The script prints [OK] Both services are running and the URLs when both services are healthy.

Use ./start.sh --check to check installed launch prerequisites without starting or installing services. The launcher binds both services to loopback, requires the selected frontend port and returns failure when either service fails readiness.

cd data-management/viewer && ./start.sh

With a custom dataset path:

cd data-management/viewer && ./start.sh --data-dir /path/to/datasets

Step 2 — Open the app in the browser

After confirming both services are running (look for [OK] Backend is healthy in terminal output), or when a running URL was supplied, open the frontend with the integrated browser's open_browser_page tool:

open_browser_page("http://localhost:5173")

SimpleBrowser is the visual interface for the user when that provider is available. Configured headless Playwright automation runs in the background. Native browser tools can operate a shared page directly; separate browser sessions share persisted backend state, not necessarily unsaved drafts. Use the supplied URL, or substitute a non-default FRONTEND_PORT for 5173.

Step 3 — Load browser automation tools

Use the host's tool search to discover available browser capabilities before calling deferred tools. Native tools include read_page, run_playwright_code, navigate_page, click_element, type_in_page and screenshot_page. Use those when available; an equivalent native read does not require MCP reconfiguration.

Drive the UI with whichever browser tool family the host provides:

For the Playwright MCP background workflow, configure headless mode so it does not open a separate browser window. Its visual feedback goes through SimpleBrowser (Step 2). A Playwright MCP server configuration in .vscode/mcp.json can use:

// .vscode/mcp.json
{
  "servers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest", "--headless"]
    }
  }
}

[!IMPORTANT] Use --headless for the MCP background workflow. Reconfiguring or installing a browser provider requires approval; do not change it merely to obtain inspection evidence.

Search by capability using the host's available tool-search schema, for example:

Find browser tools to read an accessible page snapshot, navigate, click, type and capture screenshots.

If no suitable provider is available, report the limit and guide manual inspection. When the user chooses the configured MCP provider, they can run MCP: Start Server then select playwright in the Command Palette. An empty tool search alone does not prove that a server is stopped.

Step 4 — Interact with the UI

Both tool families see the same backend state, so API-driven changes (labels, annotations) appear in both.

Take a fresh accessible snapshot before click/type actions and refresh refs after content changes. Use the Frontend UI Structure below to orient; forced DOM clicks are not keyboard accessibility evidence.

Quick Start

Start the dataviewer with the default dataset path:

cd data-management/viewer && ./start.sh

Start with a custom dataset path:

cd data-management/viewer && ./start.sh --data-dir /path/to/datasets

Parameters Reference

Dataset Path Configuration

The DATA_DIR environment variable controls which datasets are visible in the app. Each subdirectory under this path is treated as a separate dataset_id.

Use one of these methods to set DATA_DIR.

Launch argument

Use this method for workflow handoff and ad-hoc sessions:

./start.sh --data-dir /path/to/datasets

Backend environment file

Use this method only when a persistent local default is requested:

DATA_DIR=/path/to/datasets

Session environment

Use this method for a shell-scoped override:

export DATA_DIR=/path/to/datasets
cd data-management/viewer && ./start.sh

Persist a path in backend/.env only when the user explicitly requests a local default across restarts. Do not mutate .env for a profile-bound workflow handoff. The workflow owns the viewer child and passes the manifest's exact dataset parent through --data-dir.

When accepted-dataset.json exists, verify the dataset capabilities before browser inspection. Require the returned dataset ID, output adapter/version, viewer adapter, profile ID/hash, capture-provenance hash, export-validation hash, capture features, and sensor roles to match the descriptor.

Read the accepted-dataset contract reference for the descriptor schema, artifact integrity checks, and invalid-contract behavior.

Architecture

data-management/viewer/
├── start.sh              # Orchestrator: launches backend + frontend
├── backend/
│   ├── .env              # DATA_DIR and test config
│   ├── pyproject.toml    # Python dependencies (uv)
│   └── src/api/
│       ├── main.py       # FastAPI app, CORS, router registration
│       ├── routers/      # REST endpoints: datasets, annotations, labels, export, detection, analysis
│       ├── routes/       # AI analysis routes
│       ├── services/     # Business logic and dataset service
│       ├── models/       # Pydantic models
│       └── storage/      # Persistence layer
├── frontend/
│   ├── vite.config.ts    # Dev server + API proxy to :8000
│   └── src/
│       ├── App.tsx       # Root: dataset selector, episode list, annotation workspace
│       ├── api/          # HTTP client and typed API functions
│       ├── components/   # UI components (annotation, dashboard, episode viewer, export)
│       ├── hooks/        # React Query hooks for datasets, episodes, annotations
│       ├── stores/       # Zustand stores for episode and dataset state
│       └── types/        # TypeScript type definitions

API Reference

Core Endpoints

Label Endpoints

VLM-as-Judge Endpoints

[!NOTE] Mounted only when VLM_JUDGE_ENABLED=true. Dataset capabilities advertise this state, and the frontend does not request an episode judge status while disabled.

POST submission returns HTTP 202 with a durable job summary, Location and Retry-After, not a finished result. Use a stable Idempotency-Key for retries of the same payload; changed requests need a new key. Request-body instruction overrides are rejected with 422: save instruction edits first and bind the resolved snapshot.

Poll the returned location, respecting Retry-After, until succeeded, partial, failed or cancelled. Paginate targets and separately retrieve canonical episode evidence before interpreting outcomes. Job success can contain an inconclusive model assessment; acceptance or a cache badge alone is not success. The evidence's result payload contains outcome votes/confidence, progress/VOC, milestones and failure mode, with run/result and saved-input identity on its canonical record.

Acceptance response

The HTTP 202 body is a job summary. This illustrative subset shows acceptance, not a completed judgment:

{
  "id": "00000000000000000000000000000001",
  "dataset_id": "leisaac-pick-orange",
  "status": "queued",
  "total": 1,
  "judged": 0,
  "applied": 0,
  "errors": 0
}

id identifies the durable job to poll. total counts selected targets, judged counts successful judgments, applied counts applied label contributions and errors counts failed judgments. Inspect application status/errors separately; judged evidence is not proof that labels were applied.

Completed judgment fields

After terminal status, GET /api/judge/results?dataset_id={id}&episode_index={idx} returns a paginated {items, total} response. Each evidence item wraps result with run_id, result_id, saved input, runtime config, config_revision, applicability and applied. The following example is the composite JudgeResult inside items[].result, not the POST response (snake_case on the wire, camelCased by the frontend client):

{
  "episode_id": "leisaac-pick-orange/episode_000007",
  "instruction": "Grab orange and place into plate",
  "judge_model": "Qwen/Qwen3-VL-4B-Instruct",
  "prompt_version": "outcome-mcq-v1+gvl-process-v1+milestones-v1+failuremode-v1",
  "n_frames": 12,
  "outcome_success": true,
  "outcome_confidence": 0.83,
  "outcome_n_valid_votes": 3,
  "progress_per_frame": [0, 9, 18, 27, 36, 45, 55, 64, 73, 82, 91, 100],
  "voc": 0.92,
  "milestones": [
    {
      "name": "approach_object",
      "completed": true,
      "frame_range": "0-3",
      "evidence": "Gripper moves toward the orange in the sampled frames."
    }
  ],
  "failure_mode": null
}

The canonical result does not contain cached: read that flag from GET episode cache status or the job-status target. Cache status can also expose an optional result, but it is not the canonical evidence/history response. Check evidence applicability (current, stale or withdrawn) and saved-input identity before using an outcome.

Read the backend environment example for runtime settings, VLM_JUDGE_JOB_DIR, VLM_JUDGE_CAPACITY, capacity scope and storage permissions. An empty VLM_JUDGE_CACHE_DIR disables only the fallback cache, not viewer per-dataset/snapshot caches. The Storage and identity section distinguishes persistence from cache and export.

Annotation Endpoints

Export and Analysis Endpoints

When an episode's files can't be read, GET /api/datasets/{id}/episodes/{idx}, detect, annotations/auto and export/preview return HTTP 500 with {"code": "EPISODE_LOAD_FAILED"}, and the backend log has the cause. A failed export returns "error": "Export failed", or a complete event with "success": false from the stream; the backend log has the reason.

Annotation Workflow

Annotation combines API calls for efficiency with Playwright UI interaction for verification. Use the API for bulk operations and the UI for visual review and spot-checking.

Annotation surfaces

The annotation panel exposes three structured surfaces in addition to free-form labels:

Multi-camera selection

The annotation workspace supports multiple selected playback cameras. Read available controls from the current episode; selection is reconciled when media sources change. Video and frame extraction follow their selected sources. Judge views are separate saved-input configuration: playback selection does not prove which views a job evaluated.

Language instruction (VLA annotation)

The LanguageInstructionWidget writes a structured payload through PUT /api/datasets/{id}/episodes/{idx}/annotations:

When a dataset task description is available the widget seeds the instruction with source = template; otherwise it creates a blank instruction with source = human. The source dropdown allows changing the value at any time. A LeRobot export can write the most recently saved instruction as LeRobot rows: the instruction and paraphrases as task_aug, and the subtask instructions as one numbered plan row at the first frame.

Step 1 — Analyze trajectory data

Fetch episode trajectory data from the API to determine labels programmatically:

curl -s "http://localhost:8000/api/datasets/{dataset_id}/episodes/{idx}" | python3 -c "
import sys, json
d = json.load(sys.stdin)
traj = d['trajectory_data']  # List of frames with joint_positions and timestamps
print(f'Frames: {len(traj)}')
print(f'First joint positions: {traj[0]["joint_positions"][:8]}')
print(f'Last joint positions: {traj[-1]["joint_positions"][:8]}')
"

Episode trajectory data is a list of frame dictionaries, each containing:

The meta field of the episode response contains index, length, task_index, and has_annotations.

Step 2 — Determine labels from trajectory

Analyze gripper and joint data at multiple time points to classify episodes. Check the midpoint first, then 25% and 75% for episodes where grasp actions happen earlier or later:

Resolve gripper channels and units from the dataset's declared features and verified capture profile before assigning grasp labels. The example below assumes a verified 16-joint bimanual layout; do not reuse its offsets for single-arm datasets.

# Example: check grip values at multiple points for robust classification
for pct in [25, 50, 75]:
    idx = int(len(traj) * pct / 100)
    jp = traj[idx]["joint_positions"]
    right_grip = jp[7]  # Right arm gripper index
    left_grip = jp[15]  # Left arm gripper index

[!IMPORTANT] Some episodes have late or early grasp actions, so checking only the midpoint may yield UNKNOWN results. Always check multiple time points (25%, 50%, 75%) and the minimum grip value across the full trajectory for robust classification.

Step 3 — Apply labels via API

Successful PUT requests persist immediately. Label mutations require the latest dataset-label ETag in If-Match, or If-None-Match: * when the GET response has no ETag yet. Stop and reconcile after HTTP 412; do not overwrite another writer's changes.

These examples use the launcher's loopback-only development mode with DATAVIEWER_AUTH_DISABLED=true. Authenticated servers also require authentication and CSRF headers; the frontend supplies them automatically.

curl -i -fsS "http://localhost:8000/api/datasets/{dataset_id}/labels"
curl -fsS -X PUT "http://localhost:8000/api/datasets/{dataset_id}/episodes/{idx}/labels" \
  -H "Content-Type: application/json" \
  -H 'If-Match: "<ETag from the GET response>"' \
  -d '{"labels": ["RIGHT", "SUCCESS"]}'

For bulk annotation, loop over episodes in a script:

from __future__ import annotations

import json
from urllib.request import Request, urlopen


def annotate(dataset_id: str, episode_idx: int, labels: list[str]) -> dict[str, object]:
    dataset_url = f"http://localhost:8000/api/datasets/{dataset_id}"
    with urlopen(f"{dataset_url}/labels") as response:
        etag = response.headers.get("ETag")
    headers = {"Content-Type": "application/json"}
    headers["If-Match" if etag else "If-None-Match"] = etag or "*"
    data = json.dumps({"labels": labels}).encode()
    req = Request(
        f"{dataset_url}/episodes/{episode_idx}/labels",
        data=data,
        method="PUT",
        headers=headers,
    )
    with urlopen(req) as response:
        return json.load(response)

Step 4 — Verify persisted labels

Read back the saved labels after applying changes:

curl -fsS "http://localhost:8000/api/datasets/{dataset_id}/labels"

The optional POST /labels/save endpoint confirms persistence and also requires a current revision precondition. It is not needed after a successful PUT.

Label storage on disk

In local storage mode, successful label mutations write inside the dataset's meta/ directory:

{DATA_DIR}/{dataset_id}/meta/episode_labels.json

For example, with the default dataset path:

datasets/ur10e_episodes/meta/episode_labels.json

File structure:

{
  "dataset_id": "ur10e_episodes",
  "available_labels": ["SUCCESS", "FAILURE", "PARTIAL", "LEFT", "RIGHT"],
  "episodes": {
    "0": ["LEFT", "SUCCESS"],
    "1": ["RIGHT", "SUCCESS"]
  }
}

When clearing labels is explicitly requested, send an empty labels array through the episode-label PUT endpoint for each selected episode. Use the latest revision precondition for every write and verify the result with GET. Do not overwrite label files behind a running server.

Step 5 — Verify in UI with Playwright

After applying labels via API, refresh the browser and verify using Playwright:

1. Preserve unsaved drafts before refreshing. Navigate to the app using navigate_page or browser_navigate on the configured port.
2. Wait for the episode list using the available provider and take a fresh accessible snapshot.
3. Take a screenshot to confirm labels appear in the sidebar.
4. Use label filter buttons in the sidebar to verify counts match expectations.
5. Click individual episodes and scroll to the "Episode Labels" section to verify correct labels are applied.

Step 6 — Interactive annotation via UI

For individual episode review or correction:

1. Click an episode button in the sidebar using the current snapshot.
2. Scroll to "Episode Labels" through the provider or the agent's scrollIntoView example.
3. Toggle requested labels (SUCCESS, FAILURE, PARTIAL or custom labels); clicking a selected label removes it.
4. Click "Save Episode", wait for acknowledgment and independently GET the saved resources. Resolve partial saves and HTTP 412 before continuing.
5. Use the separate "Next Episode" control after persistence is verified. Retain unrelated and unmounted drafts.

Before judging, check saved-input readiness for every selected target and validation sample under the current source and principal. A clean active episode does not establish readiness elsewhere. Failed refreshes retain drafts and do not authorize work from stale acknowledgments. Interpret metadata warnings by source; HTTP 200, local-origin listings or provider availability do not prove Blob metadata synchronization.

The Edit Tools trajectory editor adjusts state channels per frame, labelled with the dataset's own channel names, and the trajectory plot previews each adjustment. Adjustments belong to the recorded frame under the playhead; after frames are removed or inserted, the editor names that recorded frame, and an inserted frame can't be adjusted directly. HDF5 exports keep the recorded joint positions as data/qpos and add the adjustments beside them as data/qpos_adjusted, with data/qpos_adjusted_mask marking the edited rows and the adjustment list in the episode's .meta.json. Velocities, actions and the other exported arrays stay as recorded.

LeRobot v3.0 sources whose camera features are non-depth videos, including sim captures, export to a new LeRobot v3.0 dataset at the output path; datasets with image features or depth video are refused. That path must be new or empty and outside the source; an existing empty directory, such as a mount point, is kept. The export locks the directory with a hidden .dataviewer-export.lock file and stages in a hidden .dataviewer-export.partial directory, so a second export to the same directory fails while the first runs. If the backend stops mid-export, or an export fails and can't remove what it moved, the next export to that directory cleans up first. It removes the earlier export's staging and the moved files that still match the identities that export recorded, and keeps a dataset whose move finished. Anything else stays, and exports there fail until it's removed. The cleanup never follows links and ignores a record whose paths don't describe a staged dataset. On Windows it leaves the earlier export's files, so remove them, including .dataviewer-export.partial, before exporting there again. LeRobot exports need a filesystem with file locking; on one without, the export fails and can leave an empty .dataviewer-export.lock. The export keeps observation.state and every other supported recorded feature, and adds adjustments as adjusted.observation.state with adjusted.observation.state_mask. The adjusted. prefix keeps both out of LeRobot policy inputs. Removing or inserting frames renumbers frame_index and timestamp. dataviewer-export.json maps every output frame to its source frame and records the edits and remapped subtasks. Subtasks shrink to the frames that survive the edits, and a LeRobot export also writes each one as a LeRobot subtask row in language_persistent, starting at its first output frame. LeRobot keeps a subtask active until the next one starts, so frames between two subtasks read as the earlier one; dataviewer-export.json keeps the exact ranges. Opening an export shows its subtasks in the editor: LeRobot rows run until the next one starts, and HDF5 .subtasks.json files keep exact ranges. Re-exporting keeps unchanged subtasks as recorded, writes changed ones in their place, and removes them when all are deleted; a saved draft takes precedence. dataviewer-export.json records subtasks as null when the export kept the recorded ones and [] when it removed them. Recorded language annotations move with the edited frames, apart from rows the exported subtasks or language instructions replace. Clearing Include subtasks as LeRobot subtask annotations in the export dialog leaves your subtask changes out; recorded annotations are still exported. For HDF5 the option reads Include subtask metadata, and clearing it still carries a recorded .subtasks.json forward. A LeRobot export has language columns when its source has them or when an exported episode gets subtask or language-instruction rows. Include language instructions as LeRobot task phrasings and plan, on by default for LeRobot sources, writes each episode's most recently saved language instruction as task_aug and plan rows that replace the recorded ones; dataviewer-export.json records whose instruction was used. API exports through /export and /export/stream include them too, unless the request body sets "includeLanguageInstructions": false. Each export is a separate dataset. LeRobot merges datasets only when their features match, so a cropped or adjusted export won't merge with an unedited one.

Frontend UI Structure

The React app has these key areas for browser automation:

Use fresh roles/names, not fixed header selectors or heading-based clicks. Observe disabled, denied and checking-access states without enabling features merely to make controls appear. Check affected keyboard navigation, focus restoration, live-region announcements and narrow-layout reflow; visual inspection alone is not accessibility acceptance.

Troubleshooting

VLM-as-Judge Workflow

The VLM judge scores each episode with an outcome MCQ (success/fail with N-sample self-consistency), process reward (per-frame progress + Spearman VOC), milestone decomposition and failure-mode attribution. Viewer and CLI share evaluator code, not automatically storage or identity.

Storage and identity

CLI local defaults use the dataset parent's .curation/judge; viewer local defaults use DATA_DIR/.curation/judge. Azure viewer jobs use the dataset container. Keep durable stores on persistent writable storage separate from scratch cache.

Cross-client reuse requires matching canonical dataset/source, principal scope, saved author/revision/snapshot, media identity/windows and runtime configuration (model revision, prompt, views and scoring settings), plus access to the same canonical evidence store. Prove matching run/result and snapshot identity through a CLI-to-viewer evidence read. Matching model names, a cache badge or JSONL filename is insufficient.

Enable the judge

Pass the judge settings through the launcher's child environment. Store them in backend/.env only when persistent defaults are explicitly requested:

DATA_DIR=/abs/path/to/datasets
DATAVIEWER_AUTH_DISABLED=true

VLM_JUDGE_ENABLED=true
VLM_JUDGE_BACKEND=echo                        # echo | qwen3-vl | openai-compat
VLM_JUDGE_MODEL_ID=Qwen/Qwen3-VL-4B-Instruct
VLM_JUDGE_N_FRAMES=12
VLM_JUDGE_CACHE_DIR=outputs/vlm-judge/cache

[!IMPORTANT] Restart only an owned, authorized backend after changing launch settings. Code reload does not refresh environment variables. Require vlm_judge_enabled from capabilities before submission; do not enable the judge for read-only inspection. Approve downloads and remote transmission of frames/saved instructions before inference.

Backends at a glance

UI workflow (Trajectory tab)

1. Open or reuse the dataviewer page on the configured port.
2. Choose a dataset, select an episode and switch to Trajectory.
3. Expand Episode Analysis and locate Judge assessment. A "VLM Judge" heading may instead identify an unavailable state.
4. Observe checking, disabled and denied states. If authorized and ready, save edits, verify read-back and resolve the saved snapshot before clicking Run judge.
5. Track the durable job to terminal status, then read canonical evidence/applicability before interpreting the outcome badge, progress, VOC, milestones and failure mode.
6. Force fresh bypasses inference cache, not saved-input or approval checks. Judge-only evidence does not apply labels.

Browser UI verification

Use a fresh read_page or browser_snapshot, then click current accessible controls. With run_playwright_code, a locator-based pattern is:

const assessment = page.getByRole('region', { name: 'Judge assessment' });
await assessment.scrollIntoViewIfNeeded();
await assessment.getByRole('button', { name: 'Run judge', exact: true }).click();

Use that pattern only when submission is authorized and the control is available. Do not force clicks or use an unavailable-state heading to find the action. After terminal status and evidence read-back, wait for the evidence's outcome text through the provider:

Wait for SUCCESS, FAILURE or Inconclusive, matching the retrieved evidence.

Direct API calls (curl / Python)

Authenticated POST requires authentication and CSRF headers; the frontend hook handles those. Keep credentials/cookies outside Git and model-visible output. The example below uses loopback-only development with DATAVIEWER_AUTH_DISABLED=true, one actual inventory ID and an approved saved instruction:

API=http://localhost:8000
DATASET_ID='<dataset-id>'
EPISODE_ID='<actual-episode-id>'
EPISODE_URL="$API/api/datasets/$DATASET_ID/episodes/$EPISODE_ID/judge"
SNAPSHOT_ID=$(curl -fsS "$EPISODE_URL/snapshot" | jq -r .snapshot_id)
REQUEST_ID='<stable-request-id>'
curl -fsS "$EPISODE_URL" | jq
curl -i -fsS -X POST "$EPISODE_URL" \
  -H 'Content-Type: application/json' -H "Idempotency-Key: $REQUEST_ID" \
  -d "$(jq -n --arg snapshot "$SNAPSHOT_ID" '{snapshot_id: $snapshot, force: false}')"

Read Location and Retry-After from acceptance, poll that location through the client's polling facility until terminal and retrieve canonical evidence separately:

JOB_ID='<accepted-job-id>'
curl -fsS "$API/api/judge/jobs/$JOB_ID" | jq
curl -fsS --get "$API/api/judge/results" \
  --data-urlencode "dataset_id=$DATASET_ID" \
  --data-urlencode "episode_index=$EPISODE_ID" | jq

Paginate status/results as needed. 404 indicates missing/inaccessible resources; 422 indicates invalid input including instruction overrides; 409 requires saved-state/configuration reconciliation. Never report 202 as inference success.

Single-episode CLI smoke (no dataviewer needed)

Use the already-restored viewer backend environment with its judge profile. --no-sync avoids installing dependencies. Set DATASET to an approved isolated fixture, EPISODE_ID to one actual inventory ID, and JOB_DIR/OUTPUT to isolated paths. Commands write durable state and exports; do not test on user datasets without permission.

uv run --project data-management/viewer/backend --no-sync python -m physical_ai_curation.run \
  --dataset "$DATASET" --indices "$EPISODE_ID" \
  --single --mode judge --backend echo --n-frames 6 \
  --job-dir "$JOB_DIR" --request-id "$REQUEST_ID" --output "$OUTPUT"

Inspect terminal status and use --operation results --episode-index "$EPISODE_ID" with matching dataset/principal/store arguments to retrieve evidence. Echo proves wiring, not model quality or GPU readiness. Never issue repeated single submissions to bypass dataset approval.

Bulk evaluation via the CLI

1. Enumerate actual target and sample IDs from dataset metadata, not a count-derived range. HTTP users can paginate /api/judge/episodes, retaining its inventory revision across pages.
2. Save human sample annotations. Resolve saved annotation_author_id, annotation_revision and snapshot_id with the matching saved-input resolver. Use viewer snapshot references for CLI only after proving source/principal identity equivalence. Store a JSON object keyed by actual sample episode ID; each value contains exactly those three non-empty fields. Drafts or guessed revisions are not valid references.
3. Define SAMPLE_IDS/TARGET_IDS as comma-separated actual IDs, SAMPLE_REFERENCES as the saved JSON path and PRINCIPAL_ID as the matching scope. Keep shared runtime arguments unchanged throughout sample, approval and batch. For real inference, replace echo with the approved backend and pinned model/runtime before sampling; an echo approval does not approve another runtime.

JUDGE_ARGS=(--dataset "$DATASET" --job-dir "$JOB_DIR" \
  --principal-scope-id "$PRINCIPAL_ID" --backend echo --n-frames 6)
uv run --project data-management/viewer/backend --no-sync python -m physical_ai_curation.run "${JUDGE_ARGS[@]}" \
  --mode sample --indices "$SAMPLE_IDS" --sample-references "$SAMPLE_REFERENCES" \
  --request-id "$SAMPLE_REQUEST_ID" --output "$SAMPLE_OUTPUT"

Read the returned sample job ID, terminal status and evidence for each sample. Compare with saved human outcomes and review disagreements/inconclusive evidence. Only after explicit review, approve the sample job. Add --acknowledge-exceptions only when the reviewer explicitly accepts those exceptions.

uv run --project data-management/viewer/backend --no-sync python -m physical_ai_curation.run "${JUDGE_ARGS[@]}" \
  --operation approve --job-id "$SAMPLE_JOB_ID"

Read the returned approval id and submit the selected batch. Judge mode produces evidence only; --mode judge-and-label additionally requires explicit label-write authority.

uv run --project data-management/viewer/backend --no-sync python -m physical_ai_curation.run "${JUDGE_ARGS[@]}" \
  --mode judge --indices "$TARGET_IDS" --approval-id "$APPROVAL_ID" \
  --request-id "$BATCH_REQUEST_ID" --output "$BATCH_OUTPUT"

If approval is stale, stop, refresh saved inputs/configuration, rerun samples and review a new approval. Do not force or fan out single submissions. HTTP follows the same sequence: sample POST /api/judge/jobs with episode_indices, samples and snapshot_ids, review evidence, POST /jobs/{id}/approve, then submit judge/judge-and-label with approval_id and current snapshots. Dataset submissions require Idempotency-Key.

Dataset wrappers and the standalone HTTP launcher live under data-management/curation/scripts/. They are conveniences, not approval bypasses. Inspect their accepted lifecycle arguments before use; fall back to the generic CLI when a wrapper cannot carry the required selection, references or approval. Policy-rollout inference stays under evaluation/vlm_judge/scripts/ and does not expose curation operations.

Status, cancellation and restart recovery

Use the same dataset/principal/job root with --operation status --job-id, --operation cancel --job-id or --operation retry --job-id. Status describes saved work, not worker health. Cancellation fences queued/running work and queued application; saved evidence remains. Retry queues eligible failed/cancelled targets in partial, failed or cancelled jobs; the worker revalidates saved inputs/approval during execution. Withdrawn jobs cannot be retried.

Submit/retry normally runs a local worker until terminal and exports available results. --detach returns acceptance only and requires a matching --operation worker process. Client interruption is not cancellation. After restart, inspect the existing job before resubmitting; retain durable storage and allow worker lease recovery rather than deleting jobs. Workers sharing inference capacity must use the same store, capacity limit and scope.

Apply or withdraw AI labels

After reviewing evidence and obtaining label-write authority, use HTTP /jobs/{id}/apply (optionally selected episode_indices) or CLI --operation apply --job-id. Application has separate counts/errors and conditional persistence: verify saved labels and provenance independently. Judge-only success is not applied-label success.

For requested withdrawal, obtain CLI --operation reset-preview or HTTP /resets/preview; review affected AI-applied labels and conflicts, then confirm that exact preview with --operation reset-confirm --preview-id or HTTP /resets. Read reset-status and use reset-retry for eligible failures. A stale/changed preview requires a new preview and confirmation.

Withdrawal selectively removes AI-applied values while preserving human edits and retained evidence/history; do not clear all labels or delete cache/job files as a substitute.

Verification and reporting

Use isolated echo cases for single acceptance/polling/evidence, approved batch, stale approval rejection and preview-bound withdrawal preserving human edits/history. Prove CLI-to-viewer reuse with matching canonical evidence identity. Source inspection alone does not claim these cases executed.

For behavior changes, follow scoped viewer guidance: failing-first tests, focused grouped development checks and configured gates after coherent changes. Use frozen dependencies and isolated storage, without changing user .env or restarting user-owned services for evidence.

Report terminal judged/applied/error counts, evidence applicability, runtime/model/prompt identity and output paths. Separate unit/static, real-HTTP, browser collection, native browser execution, simulation, assistive-technology and target GPU evidence. Missing prerequisites remain explicit gaps; prior passing counts or browser collection do not close unexecuted acceptance gates.

Brought to you by physical-ai-toolchain