Skill Bench

Benchmark/eval lab for Agent Skills in the OpenClaw Android module.

Indexed Reference Map

• @[SKILL.md#L1-L105] Frontmatter, scope, SOP, packaging and relationship to skill-creator.
• @[agents/grader.md#L1-L1] Grading expectations resource.
• @[agents/comparator.md#L1-L1] Blind comparison resource.
• @[agents/analyzer.md#L1-L1] Analysis resource.

Scope

• In-scope:
  • create or modify eval sets for a skill
  • run benchmark with-skill vs baseline / previous version
  • grade outputs against expectations
  • aggregate benchmark metrics
  • blind compare outputs A/B
  • optimize trigger descriptions when the environment supports it
  • package skill .skill for distribute/test
• Out-of-scope:
  • define the canonical internal skill schema
  • decide kb_mode, indexed refs, or compliance rulebook
  • refactor the skill to the module's original governance standard

Bundled Resources

• agents/grader.md — grading expectations + eval critique
• agents/comparator.md — blind A/B comparison
• agents/analyzer.md — benchmark/comparison analysis
• references/schemas.md — JSON contracts for eval/benchmark artifacts
• scripts/aggregate_benchmark.py — aggregate run results
• scripts/quick_validate.py — lightweight validation for packaging compatibility
• scripts/package_skill.py — package .skill artifact
• scripts/run_eval.py — trigger eval runner (requires Claude CLI environment)
• scripts/improve_description.py — description improver (requires Claude CLI environment)
• scripts/run_loop.py — iterative description optimization loop

Trigger Rules

• Positive: use when the user wants to benchmark/test Agent Skills, compare iterations, grade outputs, generate benchmark artifacts, tune descriptions, or package artifacts for test/distribution.
• Negative: do not use when the main task is to create/refactor canonical skill structure, fix kb_mode, build indexed refs, or run compliance/self-audit according to the core spec.
• Escalation: if the task has external/network write or needs a tool/runtime unavailable in the current Android module, clearly report the capability gap and request approval if needed.

SOP

1. Determine whether the target skill is stable enough to benchmark; if not, switch to skill-creator first.
2. Choose mode:
   • benchmark outputs
   • blind compare
   • description optimization
   • packaging
3. For benchmark:
   • standardize eval prompts + expectations in evals/evals.json
   • run with-skill and baseline/old-skill under the same workspace layout
   • save eval_metadata.json, outputs, transcript, timing if any
4. Grade each run with agents/grader.md; write grading.json with correct schema field names (text, passed, evidence).
5. Aggregate with scripts/aggregate_benchmark.py to create benchmark.json + benchmark.md.
6. If A/B rigor is needed, use agents/comparator.md + agents/analyzer.md.
7. Only package when the artifact has passed minimum review.

Android Module Constraints

• All default paths must stay within the workspace.
• Benchmark/package artifacts should map into canonical .ops/* when operational in nature:
  • disposable run temp -> workspace/.ops/tmp/
  • candidate packaging/review outputs awaiting promote/use -> workspace/.ops/staging/
  • rollback-safe preserved outputs -> workspace/.ops/backups/
  • manifests/benchmark metadata/checkpoints -> workspace/.ops/state/
• Prefer the absolute wrapper /data/adb/openclaw/node/bin/python instead of hardcoding python3 or depending on PATH.
• Do not treat the lightweight validator in this skill as the canonical schema authority.
• If description optimization requires claude -p but the runtime lacks suitable CLI/auth, stop and report the capability gap instead of falling back vaguely.

Packaging Rules

• scripts/package_skill.py is only used to package artifacts, not define schema authority.
• Reject symlink and exclude build artifacts/runtime cache.
• The resulting .skill must be reproducible from any cwd in the workspace.
• Do not use ad hoc artifact roots like .tmp if .ops/tmp or .ops/staging is more appropriate.

Output Contract

• Deliverables may include:
  • evals/evals.json
  • per-run grading.json
  • benchmark.json / benchmark.md
  • comparison.json / analysis.json
  • .skill package path
• Every conclusion must clearly state benchmark scope, what baseline was used, and limitations if any.

Error Handling

• If runtime is missing (claude, model auth, input files): report what is missing and which modes can still run.
• If artifact schema has wrong field names: fail early and point out which field deviates from the contract.
• If benchmark cannot distinguish the value of the skill: flag weak assertion/eval instead of claiming false success.
• If packaging fails due to import path/cwd/symlink: report the root cause + specific path fix.

Relationship to skill-creator

• skill-creator = build the skill right.
• skill-bench = prove the skill works.
• When the task moves from governance to measurement, hand off to skill-bench.
• When benchmark detects structural/spec problems, hand off back to skill-creator.