pai-benchmark-policy
Skill8 starsChanged 3 days agoApache-2.0View on GitHub
Benchmarks a trained Physical AI Studio policy in a simulation gym and reports success metrics. Use when running physicalai benchmark, selecting a benchmark config for a checkpoint or export, tuning episode/env settings, recording videos, or interpreting results.json and results.csv. DO NOT USE FOR implementing a new Benchmark class in Studio source.
In this repository42 other files it gives its agents.
- agent-harness-installSkill
- anomalib-adding-a-datamoduleSkill
- anomalib-adding-a-modelSkill
- anomalib-benchmarkingSkill
- anomalib-tiled-ensembleSkill
- anomalib-trainingSkill
- chatqna-docker-deploySkill
- chatqna-helm-deploySkill
- dlsps-userSkill
- dlstreamer-coding-agentSkill
- edgemart-agent-mcpSkill
- edgemart-stack-deploySkill
- edgemart-store-servicesSkill
- getitune-discovering-modelsSkill
- getitune-exporting-a-modelSkill
- getitune-optimizing-a-modelSkill
- getitune-preparing-datasetsSkill
- getitune-running-inferenceSkill
- getitune-training-a-modelSkill
- geti-using-the-pipelineSkill
- manufacturing-ai-app-builderSkill
- manufacturing-app-recipeSkill
- metro-ai-app-builderSkill
- metro-ai-app-recipeSkill
- model-download-userSkill
- multimodal-dataprep-userSkill
- multimodal-embedding-serving-userSkill
- openedge-cliSkill
- pai-configure-inferenceSkill
- pai-export-policySkill
- pai-load-policySkill
- pai-run-policySkill
- pai-train-policySkill
- pai-work-with-datasetsSkill
- scenescape-setupSkill
- time-series-analytics-userSkill
- uav-mission-compute-sdkSkill
- vss-deploy-helmSkill
- vss-deploySkill
- vss-search-indexSkill
- vss-summarize-videoSkill
- skill-nameSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 13
Use this path for notebooks, tests, custom scripts, or direct library integrations.
- Very specific: line 62
Choose API or CLI deliberately: Use the Python API for code-level tasks.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Benchmarking a Studio Policy
Benchmarking evaluates a trained policy by rolling it out in a gym and scoring success. PushTBenchmark and LiberoBenchmark are available through the Python API, alongside BenchmarkResults and TaskResult. Use an installed physicalai-train package or the Studio benchmark examples for CLI configs.
Python API invocation
Use this path for notebooks, tests, custom scripts, or direct library integrations.
from physicalai.benchmark.gyms import PushTBenchmark
from physicalai.policies import ACT
policy = ACT.load_from_checkpoint("experiments/act/version_0/checkpoints/last.ckpt")
benchmark = PushTBenchmark(num_episodes=1)
results = benchmark.evaluate(policy)
print(results.summary())
results.to_json("results/benchmark/results.json")
results.to_csv("results/benchmark/results.csv")For exported artifacts, load the Runtime-facing model first:
from physicalai.benchmark.gyms import PushTBenchmark
from physicalai.inference import InferenceModel
model = InferenceModel("./exports/act_policy")
results = PushTBenchmark(num_episodes=1).evaluate(model)CLI invocation
physicalai benchmark \
--config <benchmark-config.yaml> \
--policy physicalai.policies.ACT \
--ckpt_path experiments/act/version_0/checkpoints/last.ckpt \
--output_dir ./results/benchmark--policy— policy class path.--ckpt_path— a.ckptor an export directory.--config— a benchmark config copied into your project from the Studio examples, selecting theBenchmarkclass and its settings.--output_dir— defaults to./results/benchmark.
Override benchmark settings on the CLI, e.g. --benchmark.num_episodes 10 --benchmark.num_envs 8.
Output
- Prints
results.summary()to stdout. - Writes
results.jsonandresults.csvinto--output_dir. - Optional video via config
video_dir+record_mode(all|failures|successes|none).
Workflow
- Choose API or CLI deliberately. Use the Python API for code-level tasks; use CLI for config/docs/entry-point tasks.
- Done when: the selected path matches the user's requested surface area.
- Confirm the policy loads from the checkpoint/export before a full sweep:
physicalai benchmark --config <benchmark-config.yaml> --policy <ClassPath> --ckpt_path <path> --benchmark.num_episodes 1- Done when: one episode runs end-to-end and a summary prints.
- Run the full benchmark with the intended episode/env counts.
- Done when:
results.jsonandresults.csvare written and the success metric is populated.
- Done when:
- Interpret results via
BenchmarkResults/TaskResultfields; compare against a baseline checkpoint on the same config. - Record videos for qualitative review when a task regresses (
record_mode: failures).
Required checks
- The policy runs from both a
.ckptand an export dir if both are supported paths. - The Python API path (
Benchmark(...).evaluate(...)) and CLI wrapper agree on supported inputs for the run. - Success/episode metrics are populated (not zero/NaN by accident) and reproducible across runs.
- Env/episode counts match hardware; large
num_envsfits memory. - Install the optional gym extra needed by the chosen environment before running it.
Related skills
pai-train-policy— to produce the checkpoint being benchmarked.pai-export-policy— when benchmarking an exported artifact for deployment parity.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

