eval
Skill28 starsChanged 58 days agoApache-2.0View on GitHub
Evaluate a local skill with the terum-skills CLI — hygiene, trigger evals and execution cases against a baseline, billed to the user's own Claude Code — after confirming the cost, and show the result as a Markdown board. Use when the user asks to eval, test, score or benchmark a skill, or to see whether an edit made it better.
43 other projects use a version of this.
Adopted by
Most recently taken up by
- link7373/agentic-bi-teamtheir own version, first seen Nov 15, 2026
- relai-ai/agent-eval-looptheir own version, first seen Nov 15, 2026
- Integralist/agent-skillstheir own version, first seen Nov 14, 2026
- jisuzzzz/haechitheir own version, first seen Nov 8, 2026
- DataChuLee/Claude-Code-Settingtheir own version, first seen Nov 1, 2026
- jh941213/my-cc-harnesstheir own version, first seen Nov 1, 2026
Used with
Projects with it also use docs-ci, docs-interfaces and 10 more.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
jh941213Skilldocs-ci
문서 CI(링크·OpenAPI lint·breaking 게이트·mermaid 검증)나 문서 드리프트 감지(docs.yaml 신선도 검사)를 설치하거나 돌릴 때 사용. 문서 내용 작성은 docs-architecture·docs-interfaces·docs-manuals.
Also in 3 projects that use this
jh941213Skilldocs-interfaces
API·이벤트 인터페이스 문서(OpenAPI·swagger·AsyncAPI 명세, 엔드포인트 레퍼런스, API 구성도·시퀀스, API CHANGELOG)를 만들거나 코드와 맞출 때 사용. 시스템 구성도는 docs-architecture, API 설계 원칙은 api-design-principles.
Also in 3 projects that use this
open-coder-aiSkillpolicy-init
Create conformant Chock policy from request. args(request, target_path,
Also in 3 projects that use this
TheKREATORSSkillinstinct-export
Export instincts from project/global scope to a file
Also in 3 projects that use this
TheKREATORSSkillinstinct-import
Import instincts from file or URL into project/global scope
Also in 3 projects that use this
diego-tobalinaSkillbuild-fix
Fix build and compilation errors systematically. Run when build fails.
Also in 6 projects that use this
- +4 more from jh941213
Other versions43 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 43
- PrimeIntellect-ai/prime-rl★ 2.3ksame name
- rohanpsingh/LearningHumanoidWalking★ 1.2ksame name
- mikeyobrien/rho★ 372same name
- ai-analyst-lab/ai-analyst★ 304same name
- jh941213/ADLC-harness★ 125same name
- jh941213/my-cc-harness★ 125same name
- jh941213/my-claude-code-asset★ 125same name
- qtzx06/yolodex★ 96same name
- Kastalien-Research/thoughtbox★ 64same name
- Kanevry/session-orchestrator★ 53same name
33 more not listed
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository9 other files it gives its agents.
- AGENTS.mdAGENTS.md
- desktop/AGENTS.mdAGENTS.md
- eval-reportSkill
- list-skillsSkill
- search-skillsSkill
- skill-infoSkill
- skill-statusSkill
- sync-skillsSkill
- terum-skillsSkill
The file
We split this skill into the rules it gives an agent
4 rules. 0 are common, 1 is distinctive, and 3 are very specific to this project.
0
Common
Most of the 34 edited versions we read keep these.
1
Distinctive
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 17
Run it in the background when your shell tool can, or with a long timeout: a full matrix takes minutes.
- Very specific: line 36
No TTY: never pipe y, never drive the CLI with expect, never add --frames to dodge a question.
- Very specific: line 37
Never use the skill-file
!commandinjection. - Very specific: line 38
Do not cd; run from the current working directory and pass absolute paths.
“Common” is measured: kept by at least 6 in 10 of the 34 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Run one evaluation of a Library skill and show its board. Eval never mints a version and never moves skill bytes. When the evaluated bytes are already a published version it publishes that run's receipt to the team (--no-commit keeps it on this machine); otherwise it prints a share hint.
Arguments: everything after the command (in Claude Code this arrives as "$ARGUMENTS"): one skill name (or nothing when the working directory is inside a Library skill folder) or several, then any of --k <n>, --triggers-only, --execution-only, --case <stem>, --model <m>, --judge-model <m>, --no-gen, --no-commit (keep the receipt on this machine instead of publishing it to the team), --parallel <n>, --pending (every shared skill with no receipt for its current version), --window overnight|later (queue instead of running; nothing is paid), passed through unchanged. Leave --batch <n> out: it asks "Continue?" before each further batch, which needs a terminal. The queue flags (--queue-list, --dequeue <skill>, --drain …) belong to the terum-skills skill; --dequeue and --drain need a confirmation first.
Command
npx -y terum-skills@latest eval <skill> [flags] --format mdRun it in the background when your shell tool can, or with a long timeout: a full matrix takes minutes. Show the board when it finishes.
Before
Confirm with the user first: trials and generation run their logged-in Claude Code and bill their account. State what the run buys — cases × k × arms (k defaults to 1; two arms, three when a published incumbent exists) plus generation calls for any missing cases or triggers, times the number of skills for a batch (several skills run as one batch after one preflight, --parallel <n> at a time, default four; confirm once for the whole batch) — and, when the skill-info board shows a previous run, quote its cost and duration as the estimate. --window overnight|later queues instead of running and costs nothing now. Suggest --triggers-only or --case <stem> for a first look. What eval does, so the user knows:
- Finds the local folder by name; it need not belong to a team. When a team is configured, the clone is fetched for policy and incumbent history, never for the candidate bytes.
- Runs hygiene on the local bytes, allowing absent managed fields; errors stop before any agent call.
- Preflight checks
claude --versionand a one-turn smoke task before paid trials. - Uses existing cases and triggers, generating only the missing assets for the requested tiers; it prints the target path and the content-change consequence before writing them into the folder. Generating assets changes content identity and can mint a version on the next publish;
--no-genuses only existing assets, and regenerating cases means deletingevals/cases/and re-running. The generated bytes are re-checked against HYG2/HYG3 (the same predicates and author exemption the next command uses) before the write-back; a failure there refuses the run and writes nothing. - Runs trigger evals and execution cases, each k times, comparing the candidate with the baseline and, when available, a digest-different published incumbent chosen by receipt recency.
- Stores receipts and run artifacts under
~/.terum/skills/evals/local/<digest>/<run-id>/and prints the report; a later publish attaches receipts whose digest matches the published bytes; when these exact bytes are already a published version the receipt goes to the team now (--no-commitkeeps it local).
After
Show the board verbatim: the verdict band, the comparison table and any ⚠ marks stay as printed; never average holes into a clean number. Already evaluated these exact bytes is a result — the earlier receipt stands. A batch shows one board with Evaluated X of N and the queued remainder; offer the eval-report skill for each skill afterwards. Model flags pass through unchanged (--model, --judge-model; default sonnet). Editing the skill changes its digest, so the Library's score belongs to the bytes that were evaluated; installed receipts keep their original runner and are not proof this user ran anything. The one exit-1 that does ask for a re-run is the generated-assets hygiene refusal, which says so in its own text.
Rules
- No TTY: never pipe
y, never drive the CLI withexpect, never add--framesto dodge a question. A question the CLI asks means the verb belongs in a terminal: say so and hand over the command. - Never use the skill-file
!`command`injection; run every command with your shell tool and read its output. - Do not
cd; run from the current working directory and pass absolute paths. - Exit 1 is a result, not a retry: show the failure block, do not re-run, and do not claim earlier steps were rolled back.
Sandbox
When CODEX_SANDBOX_NETWORK_DISABLED=1 is set and the command starts with npx, add --prefer-offline after npx so a cached package resolves without the registry; if npx still reports a network error, ask the user to run the command in a terminal. In that sandbox the verbs that need the network — sync, install, publish, invite, eval, and update's release probe — are handed to a terminal with the reason.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

