agentleFS
UKGovernmentBEISUKGovernmentBEIS/inspect_evals/.claude/skills/generate-asset-actions/SKILL.md

generate-asset-actions

Skill747 starsChanged 36 days agoMITView on GitHub

Generate asset-actions.yaml from ASSETS.yaml by classifying assets into priority tiers. Use when the user asks to regenerate, update, or refresh the asset actions.

The file

We split this skill into the rules it gives an agent

6 rules. About 0 are common, 1 is distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If ASSETS.yaml may be stale, run uv run python tools/generate_asset_manifest.py first.

    line 10
  2. Very specific:

    Run uv run python tools/summarise_asset_manifest.py to get aggregate counts (by type, by state, totals).

    line 12
  3. Very specific:

    Read ASSETS.yaml. For each asset, determine target stage first, then priority.

    line 16
  4. Very specific:

    Every entry needs: eval, source, type, state, target, action, reason.

    line 37
  5. Distinctive:

    Update this list when new instability is discovered.

    line 41
  6. Very specific:

    Every floating asset in ASSETS.yaml appears in urgent, high, or medium

    line 59

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Generate Asset Policy

Regenerate internal/audits/asset-actions.yaml and internal/audits/audit-summary.md from ASSETS.yaml.

If ASSETS.yaml may be stale, run uv run python tools/generate_asset_manifest.py first.

Run uv run python tools/summarise_asset_manifest.py to get aggregate counts (by type, by state, totals). Use these numbers when populating audit-summary.md.

Classification

Read ASSETS.yaml. For each asset, determine target stage first, then priority. Process both state: floating assets AND state: pinned assets that match known-unstable sources (since their target is controlled, they are not yet at their target stage).

Target stages (per ADR-0007)

The target stage depends on host reliability, not asset type:

  • controlled (Stage 2) — any asset where upstream has broken before, maintainer is unresponsive/deprecated, OR host is unreliable (personal repos, Google Drive, .edu domains, university servers, any host without version control). This applies to git_clone, direct_url, and huggingface alike.
  • pinned (Stage 1) — assets on reliable, version-controlled hosts (GitHub, HuggingFace, well-known CDNs) with no history of breakage.

Per ADR-0007: "Anything hosted on a less reliable domain (personal websites, Google Drive, university servers, or any host without version control) should skip straight to Stage 2."

Priority tiers

  1. Urgent — all other floating refs on reliable hosts. Target is pinned.
  2. High — matches a known-unstable source (see registry below). Target is controlled.
  3. Medium — unreliable host (drive.google.com, .edu domains, personal repos/websites) not already in the known-unstable registry. Target is controlled.

For assets with state: pinned and a {SHA} placeholder but no checksum, classify as Low (target: pinned with checksum).

Omit assets already at their target stage.

Every entry needs: eval, source, type, state, target, action, reason.

Known-Unstable Sources

Update this list when new instability is discovered.

SourceEvalIncident
xlang-ai/OSWorldosworldFiles removed (PR #958)
openai/evalsmakemesayDeprecated upstream
corebench.cs.princeton.educore_benchUniversity server, no versioning
epatey/fontsosworldPersonal repo
ShishirPatil/gorillabfclData format issues (PR #954)
yunx-z/MLRC-Benchmlrc_benchBroken task
LRudL/sadsadUpstream bugs (issues #7, #8)
meg-tong/sycophancy-evalsycophancyInvalid JSON/NaN, workaround in code
josancamon/paperbenchpaperbenchPaper ID mismatch (HF discussion #2)
sentientfutures/moru-benchmarkmoruExact duplicate rows

Verification

  1. asset-actions.yaml parses as valid YAML
  2. Every floating asset in ASSETS.yaml appears in urgent, high, or medium
  3. floating_assets + needing_checksums + no_action_needed == total_external_assets
  4. Numbers in audit-summary.md match output of summarise_asset_manifest.py

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.