agentleFS
Terum-IncTerum-Inc/terum-skills/.claude/skills/eval/SKILL.md

eval

Skill28 starsChanged 58 days agoApache-2.0View on GitHub

Evaluate a local skill with the terum-skills CLI — hygiene, trigger evals and execution cases against a baseline, billed to the user's own Claude Code — after confirming the cost, and show the result as a Markdown board. Use when the user asks to eval, test, score or benchmark a skill, or to see whether an edit made it better.

43 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use docs-ci, docs-interfaces and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • jh941213Skill

    docs-ci

    문서 CI(링크·OpenAPI lint·breaking 게이트·mermaid 검증)나 문서 드리프트 감지(docs.yaml 신선도 검사)를 설치하거나 돌릴 때 사용. 문서 내용 작성은 docs-architecture·docs-interfaces·docs-manuals.

    Also in 3 projects that use this

  • jh941213Skill

    docs-interfaces

    API·이벤트 인터페이스 문서(OpenAPI·swagger·AsyncAPI 명세, 엔드포인트 레퍼런스, API 구성도·시퀀스, API CHANGELOG)를 만들거나 코드와 맞출 때 사용. 시스템 구성도는 docs-architecture, API 설계 원칙은 api-design-principles.

    Also in 3 projects that use this

  • open-coder-aiSkill

    policy-init

    Create conformant Chock policy from request. args(request, target_path,

    Also in 3 projects that use this

  • TheKREATORSSkill

    instinct-export

    Export instincts from project/global scope to a file

    Also in 3 projects that use this

  • TheKREATORSSkill

    instinct-import

    Import instincts from file or URL into project/global scope

    Also in 3 projects that use this

  • diego-tobalinaSkill

    build-fix

    Fix build and compilation errors systematically. Run when build fails.

    Also in 6 projects that use this

  • +4 more from jh941213

The file

We split this skill into the rules it gives an agent

4 rules. 0 are common, 1 is distinctive, and 3 are very specific to this project.

  • 0

    Common

    Most of the 34 edited versions we read keep these.

  • 1

    Distinctive

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run it in the background when your shell tool can, or with a long timeout: a full matrix takes minutes.

    line 17
  2. Very specific:

    No TTY: never pipe y, never drive the CLI with expect, never add --frames to dodge a question.

    line 36
  3. Very specific:

    Never use the skill-file !command injection.

    line 37
  4. Very specific:

    Do not cd; run from the current working directory and pass absolute paths.

    line 38

“Common” is measured: kept by at least 6 in 10 of the 34 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Run one evaluation of a Library skill and show its board. Eval never mints a version and never moves skill bytes. When the evaluated bytes are already a published version it publishes that run's receipt to the team (--no-commit keeps it on this machine); otherwise it prints a share hint.

Arguments: everything after the command (in Claude Code this arrives as "$ARGUMENTS"): one skill name (or nothing when the working directory is inside a Library skill folder) or several, then any of --k <n>, --triggers-only, --execution-only, --case <stem>, --model <m>, --judge-model <m>, --no-gen, --no-commit (keep the receipt on this machine instead of publishing it to the team), --parallel <n>, --pending (every shared skill with no receipt for its current version), --window overnight|later (queue instead of running; nothing is paid), passed through unchanged. Leave --batch <n> out: it asks "Continue?" before each further batch, which needs a terminal. The queue flags (--queue-list, --dequeue <skill>, --drain …) belong to the terum-skills skill; --dequeue and --drain need a confirmation first.

Command

npx -y terum-skills@latest eval <skill> [flags] --format md

Run it in the background when your shell tool can, or with a long timeout: a full matrix takes minutes. Show the board when it finishes.

Before

Confirm with the user first: trials and generation run their logged-in Claude Code and bill their account. State what the run buys — cases × k × arms (k defaults to 1; two arms, three when a published incumbent exists) plus generation calls for any missing cases or triggers, times the number of skills for a batch (several skills run as one batch after one preflight, --parallel <n> at a time, default four; confirm once for the whole batch) — and, when the skill-info board shows a previous run, quote its cost and duration as the estimate. --window overnight|later queues instead of running and costs nothing now. Suggest --triggers-only or --case <stem> for a first look. What eval does, so the user knows:

  1. Finds the local folder by name; it need not belong to a team. When a team is configured, the clone is fetched for policy and incumbent history, never for the candidate bytes.
  2. Runs hygiene on the local bytes, allowing absent managed fields; errors stop before any agent call.
  3. Preflight checks claude --version and a one-turn smoke task before paid trials.
  4. Uses existing cases and triggers, generating only the missing assets for the requested tiers; it prints the target path and the content-change consequence before writing them into the folder. Generating assets changes content identity and can mint a version on the next publish; --no-gen uses only existing assets, and regenerating cases means deleting evals/cases/ and re-running. The generated bytes are re-checked against HYG2/HYG3 (the same predicates and author exemption the next command uses) before the write-back; a failure there refuses the run and writes nothing.
  5. Runs trigger evals and execution cases, each k times, comparing the candidate with the baseline and, when available, a digest-different published incumbent chosen by receipt recency.
  6. Stores receipts and run artifacts under ~/.terum/skills/evals/local/<digest>/<run-id>/ and prints the report; a later publish attaches receipts whose digest matches the published bytes; when these exact bytes are already a published version the receipt goes to the team now (--no-commit keeps it local).

After

Show the board verbatim: the verdict band, the comparison table and any ⚠ marks stay as printed; never average holes into a clean number. Already evaluated these exact bytes is a result — the earlier receipt stands. A batch shows one board with Evaluated X of N and the queued remainder; offer the eval-report skill for each skill afterwards. Model flags pass through unchanged (--model, --judge-model; default sonnet). Editing the skill changes its digest, so the Library's score belongs to the bytes that were evaluated; installed receipts keep their original runner and are not proof this user ran anything. The one exit-1 that does ask for a re-run is the generated-assets hygiene refusal, which says so in its own text.

Rules

  • No TTY: never pipe y, never drive the CLI with expect, never add --frames to dodge a question. A question the CLI asks means the verb belongs in a terminal: say so and hand over the command.
  • Never use the skill-file !`command` injection; run every command with your shell tool and read its output.
  • Do not cd; run from the current working directory and pass absolute paths.
  • Exit 1 is a result, not a retry: show the failure block, do not re-run, and do not claim earlier steps were rolled back.

Sandbox

When CODEX_SANDBOX_NETWORK_DISABLED=1 is set and the command starts with npx, add --prefer-offline after npx so a cached package resolves without the registry; if npx still reports a network error, ask the user to run the command in a terminal. In that sandbox the verbs that need the network — sync, install, publish, invite, eval, and update's release probe — are handed to a terminal with the reason.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.