NVIDIA Physical AI Datasets (Hugging Face)

Purpose

Help the agent find, evaluate, and download an NVIDIA PhysicalAI-* dataset on Hugging Face that fits the user's downstream task — autonomous-vehicle reconstruction, robotics manipulation, GR00T post-training, spatial-intelligence research, grasping, or sim-ready content — and then hand off to the companion skill (ncore, nurec, asset-harvester, harmonizer) or upstream NVIDIA tool that actually consumes it.

Use this skill when: the user asks "is there an NVIDIA dataset for X?", "where do I get NCore / NuRec / GR00T sample data?", or is shopping the Hugging Face NVIDIA org for PhysicalAI-* collections.

Do NOT use this skill when:

• The user already knows the dataset and just wants to run a pipeline — jump straight to the consuming skill.
• The user needs a non-NVIDIA dataset (Waymo, nuScenes, KITTI, …) — this catalog is NVIDIA-only.
• The user wants to train Cosmos / GR00T / Isaac Sim itself — that's the upstream tool's job, not this catalog's.

Overview

Catalog of NVIDIA's open Physical AI dataset family on Hugging Face. Pick by task (Section 2 § lookup table) or family (Sections 3–10). Every entry lists: dataset path, size, format, license, gating, and the downstream tool that consumes it.

Source of truth: https://huggingface.co/nvidia (filter PhysicalAI-) and the curated Physical AI collection. When upstream cards drift, re-check the HF page; this skill mirrors the cards as of Apr 2026.

Recommendation handoff

Every recommendation, including robotics/manipulation datasets, must state:

• The exact repository and selected subset, format, and license from its current card.
• Whether it is public or gated, with browser-acceptance/token steps only when needed.
• A bounded download command or file-selection procedure and an estimated storage budget.
• A concrete consuming tool/workflow and the next action with that data.

Verify a companion skill is installed before routing to it; otherwise link to the upstream tool's actual documentation. For a Physical AI AV benchmark on a new Brev instance, prefer physical-ai-av-benchmark when installed. For raw Physical AI AV reconstruction on an existing host, use the source-built workflow documented by nurec; for pretrained scene USDZ files, use its direct rendering workflow.

Prerequisites

• git, git-lfs, and huggingface_hub[cli] on PATH. That is all you need for the public majority of these datasets — no account, no token.

• Only for the gated minority (the PhysicalAI-Autonomous-Vehicles* family in particular):

  • a HuggingFace account with that dataset's card opened in a browser at least once and its gating checkbox accepted;
  • a HuggingFace user access token exported as HF_TOKEN (create at https://huggingface.co/settings/tokens).

  Check the dataset's own Gating row before assuming either is required.

• Storage room sized to the dataset you're pulling (see the per-row size column; some are < 1 GB, the AV dataset is 133 TB — always pre-filter with --include or physical_ai_av).

Verifying secrets safely

Always check token presence with hf auth whoami or a length-only shell test; never write ad-hoc bash that interpolates HF_TOKEN values. The common one-liner

# BAD — leaks the secret to the terminal when the variable is set
echo "HF_TOKEN: ${HF_TOKEN:+yes}${HF_TOKEN:-no}"

prints yes<token-value> whenever HF_TOKEN is set, because ${VAR:-no} only falls back to "no" when VAR is empty — when set it expands to $VAR. Use one of these instead:

hf auth whoami                              # confirms the token without echoing it
test -n "$HF_TOKEN" && echo "HF_TOKEN: set (${#HF_TOKEN} chars)" || echo "HF_TOKEN: missing"

Rotate any token you suspect was echoed at https://huggingface.co/settings/tokens.

Table of Contents

1. Common download recipe — HF auth, gating, CLI.
2. Filtered AV download recipe — default hyperion_8.1 filter for PhysicalAI-Autonomous-Vehicles raw pulls.
3. Pick a dataset by task — fast lookup table.
4. Autonomous Vehicles — 5 datasets.
5. Robotics — Manipulation — 6 datasets.
6. Robotics — GR00T — 7 datasets.
7. Robotics — mindmap — 4 datasets.
8. Robotics — NuRec / Sim-Ready scenes — 2 datasets.
9. Robotics — Healthcare — 1 dataset.
10. Robotics — Grasping — 1 dataset.
11. Robotics — Physical / material properties — 2 datasets.
12. Spatial Intelligence + SimReady scenes — 5 datasets.
13. License decision tree — what you can do with each.
14. Cross-skill usage map — which skill consumes which dataset.

Common download recipe

All NVIDIA Physical AI datasets live on huggingface.co/datasets/nvidia/.... Most are public; only a minority are gated (see each entry's License / Gating rows).

sudo apt -y install git git-lfs
git lfs install

uv tool install -U "huggingface_hub[cli]"   # or: pip install --upgrade "huggingface_hub[cli]"

# Only needed for the GATED minority (the AV family in particular). Most
# PhysicalAI-* repos are public and pull with no token at all -- do not log in
# reflexively.
#   export HF_TOKEN=...   # then `hf download` picks it up from the environment

Most PhysicalAI-* repos are public — a plain hf download works with no token at all. A token is only needed for the gated minority.

For a gated dataset, the token must:

1. Belong to a HF account that has logged in to that dataset page in a browser at least once.
2. Have accepted the license / terms-of-use checkbox the dataset shows (re-accept if it has been re-gated — common for AV).

If a download 404s or returns HTML, check whether that specific dataset is gated rather than assuming every one is.

Three download patterns:

# REQUIRED: pick a dataset from the catalog below. There is deliberately no
# default -- the previous one was GR00T-X-Embodiment-Sim, a 1.91 TB collection,
# which is not something to pull by accident.
: "${DATASET:?set DATASET to a PhysicalAI-* name from the catalog below}"

# Whole dataset -- ONLY for datasets the catalog lists as small/medium. Check
# the Size row first; for anything multi-hundred-GB use the --include form.
hf download "nvidia/${DATASET:?}" --repo-type dataset --local-dir "./${DATASET:?}"

# Sub-folder only (recommended for large multi-task collections)
hf download nvidia/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim \
  --repo-type dataset \
  --include "gr1_arms_only.CanSort/**" \
  --local-dir ./gr00t_dataset

# Sparse-checkout via git-LFS (if you want incremental git-style work)
SUBFOLDER=gr1_arms_only.CanSort
git clone --filter=blob:none --no-checkout "https://huggingface.co/datasets/nvidia/${DATASET:?}"
cd "${DATASET:?}"
git sparse-checkout init --cone
git sparse-checkout set "${SUBFOLDER:?}"
git checkout main

Special-case downloaders:

• PhysicalAI-Autonomous-Vehicles (133 TB) — use the official Python toolkit pip install physical_ai_av (NVlabs/physical_ai_av) to filter by sensor / country / split before downloading; otherwise you will pull TBs you don't need.

  Default policy: pre-filter to platform_class == 'hyperion_8.1' before any bulk pull. The dataset mixes two sensor rigs (hyperion_8 and hyperion_8.1). NCore's PAI converter accepts both (tools/data_converter/pai/converter.py), but the downstream NuRec / Asset-Harvester chain is only validated on hyperion_8.1. Roughly half of the 306k clips fall outside that platform, so the filter typically cuts the working set (and download size) in half.

  Apply this filter by default in any download script you generate. Only skip the filter when the user explicitly asks for it (e.g. "download all platforms", "include hyperion_8", "don't filter by platform") or when they hand you a specific clip UUID — in that single-clip case, download_clip_features(clip_id=...) is already scoped and no platform filter is needed.

  Canonical filtered recipe (see § Filtered AV download recipe below for an end-to-end example):

  from physical_ai_av import PhysicalAIAVDatasetInterface
  
  dataset = PhysicalAIAVDatasetInterface()
  dataset.download_metadata()
  dc = dataset.metadata['data_collection']
  hyperion_81_clip_ids = dc[dc['platform_class'] == 'hyperion_8.1'].index.tolist()

• PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams (3 TB) — use the upstream download.py (nv-tlabs/Cosmos-Drive-Dreams) with --file_types {hdmap,lidar,synthetic} to select layers.
• PhysicalAI-SpatialIntelligence-Lyra-SDG (25 TB) — hf download --local-dir lyra_dataset/tar; untar each tar yourself.
• Spatial-Intelligence-Warehouse — chunked TAR-GZs need a manual loop after download (script provided in the upstream card).

For dataset filtering / preview: NVIDIA's Cosmos Dataset Search (CDS) lets you query a 41K subset of the AV dataset semantically before downloading.

Filtered AV download recipe

Always use this recipe (or a derivative of it) when the user asks to download raw clips from PhysicalAI-Autonomous-Vehicles. The hyperion_8.1 filter is the default; it matches every downstream skill (ncore, nurec, asset-harvester, harmonizer).

When to skip the platform filter:

1. The user explicitly opts out — e.g. "download all platforms", "include hyperion_8 too", "ignore platform_class", or asks for a dataset-wide statistic. In that case, drop the platform mask and warn them that NuRec / NCore tooling will not work on the hyperion_8 clips.
2. The user gave you a specific clip UUID. A single-clip download_clip_features(clip_id=...) is already scoped, and filtering by platform_class for one clip is pointless.

Recipe (defaults: filter on; user may layer extra masks like country or sensor presence on top):

from physical_ai_av import PhysicalAIAVDatasetInterface

dataset = PhysicalAIAVDatasetInterface()

dataset.download_metadata()
dc = dataset.metadata['data_collection']

clip_mask = dc['platform_class'] == 'hyperion_8.1'

# Optional extra masks (only add when the user asked for them):
# clip_mask &= dc['country'] == 'US'
# sp = dataset.metadata['feature_presence']   # 26.03+; was 'sensor_presence' in 25.10
# clip_mask &= sp['lidar_top_360fov']

clip_ids = dc[clip_mask].index.tolist()
print(f"Downloading {len(clip_ids)} hyperion_8.1 clips")

dataset.download_clip_features(
    clip_id=clip_ids,
    features=["camera_front_wide_120fov", "lidar_top_360fov", "egomotion"],
    max_workers=8,
)

Single-clip fast path (no platform filter — the UUID is already specific):

dataset.download_clip_features(
    clip_id="<paste-clip-uuid>",
    features=["camera_front_wide_120fov", "lidar_top_360fov", "egomotion"],
)

Explicit opt-out (only when the user asked for it):

clip_ids = dc.index.tolist()   # NO platform filter — all 306k clips
# WARNING: NCore converts both platforms, but only the hyperion_8.1
# subset is validated downstream (NuRec / Asset-Harvester).

Pick a dataset by task

Autonomous Vehicles

PhysicalAI-Autonomous-Vehicles

The flagship real-world AV dataset.

PhysicalAI-Autonomous-Vehicles-NCore

Curated NCore V4 subset of the above.

PhysicalAI-Autonomous-Vehicles-NuRec

Pre-built NuRec dynamic neural reconstructions ready for IsaacSim / CARLA.

PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams

Cosmos-Transfer-style synthetic + HD-map labels for diverse weather.

PhysicalAI-Autonomous-Vehicle-Cosmos-Synthetic

Robotics — Manipulation

All in LeRobot v2.x format unless noted, generated in IsaacSim with task-and-motion planning + scene_synthesizer procedural scenes + CuRobo motion generation.

PhysicalAI-Robotics-Manipulation-Kitchen

Bimanual Kinova Gen3 in procedurally-generated kitchens.

PhysicalAI-Robotics-Manipulation-Objects

Same kitchen environment, bimanual Kinova; pick / place bench / place cabinet.

PhysicalAI-Robotics-Manipulation-SingleArm

Franka Panda tabletop, procedurally generated.

PhysicalAI-Robotics-Manipulation-Augmented

Mimic-generated Franka cube-stacking, plus Cosmos-Transfer1 visual augmentation.

PhysicalAI-Robotics-Manipulation-Kitchen-Demos

Massive human-teleop dataset on Franka + Omron mobile base.

PhysicalAI-Robotics-Manipulation-Objects-Kitchen-MJCF

The MuJoCo XML assets that back the Kitchen-Demos environment.

Robotics — GR00T

GR00T = NVIDIA's generalist humanoid foundation model line. Most data is sim-generated for post-training; eval / real-robot supplements are small.

PhysicalAI-Robotics-GR00T-X-Embodiment-Sim

Largest GR00T post-training corpus.

PhysicalAI-GR00T-Tuned-Tasks

Two industrial post-training task families.

PhysicalAI-Robotics-GR00T-Teleop-G1

Real-robot Unitree G1 fruit pick-and-place.

PhysicalAI-Robotics-GR00T-Teleop-Sim

Simulated GR1 tabletop teleop.

PhysicalAI-Robotics-GR00T-Teleop-GR1

DreamDojo pretraining corpus — large-scale human egocentric video.

PhysicalAI-Robotics-GR00T-GR1

Lab-recorded GR1-T2 third-person video.

PhysicalAI-Robotics-GR00T-Eval

GR00T eval initial-state frames.

Robotics — mindmap

Spatial-memory benchmark from nvidia-isaac/nvblox_mindmap. Each dataset is one task with the same multimodal layout (RGB-D + camera intrinsics/poses + nvblox vertex features in .zst + robot state). All four are CC-BY-NC-4.0 (research only). Models trained: nvidia/PhysicalAI-Robotics-mindmap-Checkpoints.

Provided datasets ship 10 mindmap-formatted demos for storage reasons; regenerate the full 200/1000/250 with the upstream data-generation docs.

Paper: https://arxiv.org/abs/2509.20297. Codebase: https://github.com/nvidia-isaac/nvblox_mindmap.

Robotics — NuRec / Sim-Ready scenes

PhysicalAI-Robotics-NuRec

Indoor 3DGUT scenes for IsaacSim AMR simulation.

PhysicalAI-NuRec-PPISP

Photometric-variation benchmark for radiance-field methods.

Robotics — Healthcare

PhysicalAI-Robotics-Open-H-Embodiment

Surgical / ultrasound robotics multi-embodiment corpus.

Robotics — Grasping

PhysicalAI-Robotics-GraspGen

Sim2Real grasping at scale.

For Franka Panda, use grasp_data/franka_panda/ and splits/franka_panda/. Before selecting a shard, list the files in the chosen subset on the Hub and confirm the exact path; download only the user-selected shard or split.

Robotics — Physical / material properties

VoMP = Volumetric Mechanical Properties.

PhysicalAI-Robotics-PhysicalAssets-VoMP

PhysicalAI-Robotics-PhysicalAssets-VoMP-Eval

Spatial Intelligence + SimReady scenes

PhysicalAI-SimReady-Warehouse-01

OpenUSD warehouse scene + asset library.

PhysicalAI-DigitalCousin-Assets

Companion-asset library for the GR1 tabletop sim environments.

PhysicalAI-SmartSpaces

Multi-camera tracking + 3D box benchmark (AI City Challenge).

PhysicalAI-Spatial-Intelligence-Warehouse

VLM-style spatial QA in warehouses.

PhysicalAI-SpatialIntelligence-Lyra-SDG

GEN3C-derived multi-view 3D + 4D training data for nv-tlabs/lyra.

(See also) PhysicalAI-NuRec-PPISP

Listed in § 7 but is also a spatial-intelligence radiance-field benchmark.

License decision tree

Quick rule of thumb:

1. If Robotics-Manipulation-* and not mindmap / DigitalCousin-Assets / Objects → CC-BY-4.0 commercial OK. (…-Manipulation-Objects is NVIDIA OneWay Noncommercial.)
2. If mindmap-* → research only (NC).
3. If Autonomous-Vehicles* → AV License only, gated.
4. Everything else → check the card.

Cross-skill usage map

The first column names the skill that consumes each dataset; the second lists other consumers that have no skill.

Verify

After downloading any dataset:

LOCAL_DIR=./my-downloaded-dataset
CLIP_UUID=02a6f330-5ab0-4e92-99d4-d19e406952f4

ls "${LOCAL_DIR:?}"
du -sh "${LOCAL_DIR:?}"          # confirm size matches the table above (within 10 %)

# For LeRobot v2.x:
python -c "from lerobot.datasets.lerobot_dataset import LeRobotDataset; \
           d = LeRobotDataset(repo_id='nvidia/<the-dataset>', root='${LOCAL_DIR:?}'); print(d.meta.info)"

# For NCore V4 (.zarr.itar):
# ncore_vis is not a wheel entry point -- run it from an ncore checkout.
# Bind loopback: its Viser server is unauthenticated.
bazel run //tools/ncore_vis -- \
    --host=127.0.0.1 \
    v4 --component-group="${LOCAL_DIR:?}/clips/${CLIP_UUID:?}/pai_${CLIP_UUID:?}.json"

# For USDZ scenes:
# load the .usdz in Isaac Sim 5.1 — File -> Open

GREEN when:

• File counts and total size match the dataset card to within ~10 %.
• For gated datasets, the download did not silently terminate at the license-agreement page (re-hf auth login if the first chunk is HTML).
• For LeRobot datasets, meta/info.json parses and the episode count matches the card.

Troubleshooting

• Repo gated. Cannot access … 401 — open the dataset URL in a browser, click Agree on the license / contact-info form, then retry with the same token. Tokens don't get auto-refreshed when a new agreement appears (re-accept after major version bumps).
• First chunk is 5 KB of HTML — same cause as above.
• Download hangs indefinitely on AV (133 TB) — you almost certainly don't want the whole thing. Use physical_ai_av to filter platform_class == 'hyperion_8.1' (default — see Filtered AV download recipe), plus any sensor / country / split mask, BEFORE pulling.
• AV clip downloaded but downstream NCore / NuRec / Asset-Harvester rejects it — double-check data_collection['platform_class'] for that clip; only hyperion_8.1 is validated. If it's hyperion_8, either swap to a hyperion_8.1 clip or accept that the downstream reconstruction tooling will not work.
• Cosmos-Drive-Dreams 3 TB on a small disk — pass --file_types synthetic (700 GB) or --file_types hdmap (small) to the official download.py.
• Lyra-SDG 25 TB out of disk — hf download --include "tar/static_*" --exclude "tar/dynamic_*" to take just the 3D half (or vice-versa).
• Cosmos-Synthetic looks empty (2.59 kB) — it's a pointer page redirecting to Cosmos-Drive-Dreams. Use that.
• mindmap dataset only has 10 demos when the card says 200 / 1000 — expected; regenerate the rest from the HDF5 via the upstream mindmap data-generation docs.
• AV license expired (12 months) — re-accept on the HF page; this is by-design per the License Agreement section 7.

Limitations

• Catalogue-only. This skill does not actually run any pipeline — it routes the agent to the consuming tool once the right dataset is chosen.
• NVIDIA-only. Non-NVIDIA datasets (Waymo, nuScenes, KITTI, PandaSet, …) are out of scope; the ncore skill covers how to convert those into NCore V4 manually.
• Gating is enforced by Hugging Face, not by this skill. A minority of datasets — the PhysicalAI-Autonomous-Vehicles* family in particular — require both license acceptance on the HF page and a valid HF_TOKEN, and those acceptances expire (AV is annual) and must be re-clicked. The four mindmap repos are public and ungated; they are CC-BY-NC, which restricts use, not access. Do not assume a dataset is gated because it is non-commercial.
• Footprint can be enormous. PhysicalAI-Autonomous-Vehicles is ~133 TB; Lyra-SDG ~25 TB; Cosmos-Drive-Dreams ~3 TB. Always check free disk and use --include / --exclude filters before kicking off a full pull.
• The catalogue drifts. NVIDIA ships new PhysicalAI-* datasets regularly. If a dataset isn't in this skill, browse https://huggingface.co/nvidia?search_datasets=PhysicalAI- and consider opening a PR to update Section 2 / the relevant family.
• No GPU / runtime checks here. Storage and HF auth are the only prerequisites this skill validates; per-dataset compute needs belong to the consuming tool.

Source links

• HF org (filter PhysicalAI-): https://huggingface.co/nvidia
• Curated collection (29 items): https://huggingface.co/collections/nvidia/physical-ai
• Cosmos Dataset Search (semantic preview of AV): https://build.nvidia.com/nvidia/cosmos-dataset-search
• AV Python toolkit: https://github.com/NVlabs/physical_ai_av
• NCore: https://github.com/NVIDIA/ncore
• Cosmos-Drive-Dreams downloader: https://github.com/nv-tlabs/Cosmos-Drive-Dreams/blob/main/scripts/download.py
• Lyra repo: https://github.com/nv-tlabs/lyra
• mindmap repo: https://github.com/nvidia-isaac/nvblox_mindmap
• Isaac-GR00T repo: https://github.com/NVIDIA/Isaac-GR00T
• DreamDojo project: https://dreamdojo-world.github.io/
• AI City Challenge eval: https://eval.aicitychallenge.org/