agentleFS
open-edge-platformopen-edge-platform/skills/.agents/skills/pai-benchmark-policy/SKILL.md

pai-benchmark-policy

Skill8 starsChanged 3 days agoApache-2.0View on GitHub

Benchmarks a trained Physical AI Studio policy in a simulation gym and reports success metrics. Use when running physicalai benchmark, selecting a benchmark config for a checkpoint or export, tuning episode/env settings, recording videos, or interpreting results.json and results.csv. DO NOT USE FOR implementing a new Benchmark class in Studio source.

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use this path for notebooks, tests, custom scripts, or direct library integrations.

    line 13
  2. Very specific:

    Choose API or CLI deliberately: Use the Python API for code-level tasks.

    line 62

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Benchmarking a Studio Policy

Benchmarking evaluates a trained policy by rolling it out in a gym and scoring success. PushTBenchmark and LiberoBenchmark are available through the Python API, alongside BenchmarkResults and TaskResult. Use an installed physicalai-train package or the Studio benchmark examples for CLI configs.

Python API invocation

Use this path for notebooks, tests, custom scripts, or direct library integrations.

from physicalai.benchmark.gyms import PushTBenchmark
from physicalai.policies import ACT

policy = ACT.load_from_checkpoint("experiments/act/version_0/checkpoints/last.ckpt")
benchmark = PushTBenchmark(num_episodes=1)
results = benchmark.evaluate(policy)
print(results.summary())
results.to_json("results/benchmark/results.json")
results.to_csv("results/benchmark/results.csv")

For exported artifacts, load the Runtime-facing model first:

from physicalai.benchmark.gyms import PushTBenchmark
from physicalai.inference import InferenceModel

model = InferenceModel("./exports/act_policy")
results = PushTBenchmark(num_episodes=1).evaluate(model)

CLI invocation

physicalai benchmark \
  --config <benchmark-config.yaml> \
  --policy physicalai.policies.ACT \
  --ckpt_path experiments/act/version_0/checkpoints/last.ckpt \
  --output_dir ./results/benchmark
  • --policy — policy class path.
  • --ckpt_path — a .ckpt or an export directory.
  • --config — a benchmark config copied into your project from the Studio examples, selecting the Benchmark class and its settings.
  • --output_dir — defaults to ./results/benchmark.

Override benchmark settings on the CLI, e.g. --benchmark.num_episodes 10 --benchmark.num_envs 8.

Output

  • Prints results.summary() to stdout.
  • Writes results.json and results.csv into --output_dir.
  • Optional video via config video_dir + record_mode (all | failures | successes | none).

Workflow

  1. Choose API or CLI deliberately. Use the Python API for code-level tasks; use CLI for config/docs/entry-point tasks.
    • Done when: the selected path matches the user's requested surface area.
  2. Confirm the policy loads from the checkpoint/export before a full sweep:
    physicalai benchmark --config <benchmark-config.yaml> --policy <ClassPath> --ckpt_path <path> --benchmark.num_episodes 1
    • Done when: one episode runs end-to-end and a summary prints.
  3. Run the full benchmark with the intended episode/env counts.
    • Done when: results.json and results.csv are written and the success metric is populated.
  4. Interpret results via BenchmarkResults/TaskResult fields; compare against a baseline checkpoint on the same config.
  5. Record videos for qualitative review when a task regresses (record_mode: failures).

Required checks

  • The policy runs from both a .ckpt and an export dir if both are supported paths.
  • The Python API path (Benchmark(...).evaluate(...)) and CLI wrapper agree on supported inputs for the run.
  • Success/episode metrics are populated (not zero/NaN by accident) and reproducible across runs.
  • Env/episode counts match hardware; large num_envs fits memory.
  • Install the optional gym extra needed by the chosen environment before running it.

Related skills

  • pai-train-policy — to produce the checkpoint being benchmarked.
  • pai-export-policy — when benchmarking an exported artifact for deployment parity.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.