eval
Skill4 starsChanged 6 days agoMITView on GitHub
Measure whether the team answers data questions correctly — run the eval suite of questions with pinned answers, grade the results, record the run, and turn failures into fixes. Use after changing an agent, skill, or knowledge file, on a schedule, when a stakeholder corrects an answer, and before announcing the team to a new audience.
43 other projects use a version of this.
Adopted by
Most recently taken up by
- relai-ai/agent-eval-looptheir own version, first seen Nov 15, 2026
- Integralist/agent-skillstheir own version, first seen Nov 14, 2026
- jisuzzzz/haechitheir own version, first seen Nov 8, 2026
- DataChuLee/Claude-Code-Settingtheir own version, first seen Nov 1, 2026
- jh941213/my-cc-harnesstheir own version, first seen Nov 1, 2026
- jh941213/ADLC-harnesstheir own version, first seen Nov 1, 2026
Used with
Projects with it also use docs-ci, docs-interfaces and 10 more.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
jh941213Skilldocs-ci
문서 CI(링크·OpenAPI lint·breaking 게이트·mermaid 검증)나 문서 드리프트 감지(docs.yaml 신선도 검사)를 설치하거나 돌릴 때 사용. 문서 내용 작성은 docs-architecture·docs-interfaces·docs-manuals.
Also in 3 projects that use this
jh941213Skilldocs-interfaces
API·이벤트 인터페이스 문서(OpenAPI·swagger·AsyncAPI 명세, 엔드포인트 레퍼런스, API 구성도·시퀀스, API CHANGELOG)를 만들거나 코드와 맞출 때 사용. 시스템 구성도는 docs-architecture, API 설계 원칙은 api-design-principles.
Also in 3 projects that use this
open-coder-aiSkillpolicy-init
Create conformant Chock policy from request. args(request, target_path,
Also in 3 projects that use this
TheKREATORSSkillinstinct-export
Export instincts from project/global scope to a file
Also in 3 projects that use this
TheKREATORSSkillinstinct-import
Import instincts from file or URL into project/global scope
Also in 3 projects that use this
diego-tobalinaSkillbuild-fix
Fix build and compilation errors systematically. Run when build fails.
Also in 6 projects that use this
- +4 more from jh941213
Other versions43 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 43
- PrimeIntellect-ai/prime-rl★ 2.2ksame name
- rohanpsingh/LearningHumanoidWalking★ 1.2ksame name
- mikeyobrien/rho★ 372same name
- ai-analyst-lab/ai-analyst★ 304same name
- jh941213/ADLC-harness★ 125same name
- jh941213/my-cc-harness★ 125same name
- jh941213/my-claude-code-asset★ 125same name
- qtzx06/yolodex★ 96same name
- Kastalien-Research/thoughtbox★ 64same name
- Kanevry/session-orchestrator★ 53same name
33 more not listed
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository21 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- analyzeSkill
- build-dashboardSkill
- build-modelSkill
- build-pipelineSkill
- causal-analysisSkill
- connect-dataSkill
- define-kpisSkill
- experimentSkill
- forecastSkill
- health-checkSkill
- investigate-metricSkill
- make-deliverableSkill
- powerbiSkill
- privacy-reviewSkill
- research-domainSkill
- scorecardSkill
- setup-teamSkill
- tableauSkill
- triageSkill
- upgradeSkill
The file
We split this skill into the rules it gives an agent
6 rules. 0 are common, 1 is distinctive, and 5 are very specific to this project.
0
Common
Most of the 34 edited versions we read keep these.
1
Distinctive
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 28
The answer key is off limits: No answering agent opens anything under evals/ — say so in each brief.
- Distinctive: line 63
Every time a stakeholder says "that's the wrong table" or "you forgot to exclude test accounts", the team has been handed a free eval.
- Very specific: line 65
Write the question as the stakeholder asked it, the reference SQL that produces the right answer, and the pinned answer, into …
- Very specific: line 66
Add must_mention for the thing that was missed ("excluding test accounts"), and a one-line why.
- Very specific: line 78
Gate new audiences: Before the team is announced to a new group of stakeholders, their domain's slice of evals/local/ clears the threshold …
- Very specific: line 80
Keep a short "tried it, did not help" list in knowledge/decision-log.md.
“Common” is measured: kept by at least 6 in 10 of the 34 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Eval — Is The Team Actually Right?
Owner: the orchestrator, with analysis-reviewer on failures. Args pick the mode and suite, e.g. /eval, /eval local, /eval add (after a correction), /eval check.
An analytics answer can be wrong, look plausible, and be used without objection. Nothing else in the kit measures that. An eval is a question with a known right answer; a suite of them tells you whether a change to a skill or a knowledge file helped, and catches the slow drift that happens as the data model changes under documentation that did not.
Two suites: evals/demo/ ships with the kit and runs against the pinned demo warehouse. evals/local/ is yours — questions about your business, on your data. The demo suite proves the kit works; the local suite proves your team does. Format and grading rules: evals/README.md.
Mode A — Run (default)
Check the answer key first.
python scripts/run_evals.py --check-referencesThis needs no model. If a reference drifted, the key is wrong, not the team — stop and fix that before grading anything against it.
- Get the paper.
python scripts/run_evals.py --paper --suite <suite> --jsonprints ids and questions only. For the demo suite, also runpython scripts/run_evals.py --build-dband note the path it prints: answers must come from that pinned database, not fromdemo/demo.db, whose dates move with today. Answer each question the way the team would for a stakeholder. Delegate each to
bi-analyst(in parallel, in batches) with only the question text and the database to use. The agent works exactly as it normally would: knowledge base, catalog first, data checks, privacy rules.The answer key is off limits. No answering agent opens anything under
evals/— say so in each brief. An eval the team has read the answers to measures nothing, and this is enforced by instruction, not by a lock, so it depends on you.Collect the answers into
evals/runs/<YYYY-MM-DD-HHMM>/answers.json:{ "meta": {"suite": "demo", "model": "<model id>", "notes": "after editing /analyze step 3"}, "answers": { "active-customers": {"behaviour": "answered", "value": 549, "text": "549 active customers as of 30 June 2026."}, "trap-ambiguous-churn": {"behaviour": "clarified", "text": "Do you mean logo churn or revenue churn, and for which period?"} } }behaviouris one ofanswered,clarified(asked a question instead of answering),refused(declined on policy grounds),flagged_data_issue(reported a data problem as the finding).valueis the headline number or label;textis the answer as the stakeholder would read it. Record what the agent actually did — do not tidy an answer on its way into the file.Grade and record.
python scripts/run_evals.py --grade evals/runs/<stamp>/answers.jsonThe run is appended to
evals/results/history.jsonlwith the git SHA, kit version, and model.- Report. Pass rate, per-category results, and each failure with its reason. Then
python scripts/run_evals.py --history— say whether this run is better or worse than the last, and which questions newly fail. Diagnose every failure — the eval is the symptom. For each one, name which of three things went wrong:
- Ambiguity: more than one plausible table, column, or definition, and the team picked the wrong one. → Fix in
knowledge/metrics-catalog.md(a definition, a named segment) or deprecate the near-duplicate. - Staleness: the documentation describes the data as it used to be. → Fix the knowledge file; check what else that file says.
- Retrieval: the right information exists and the team did not find or use it. → Fix the domain reference doc in
knowledge/domains/— usually a missing routing trigger or gotcha.
Fix the knowledge, not the answer. Then re-run the affected questions. Do not edit an eval to make it pass unless the eval was wrong, and if it was, say so in the commit.
- Ambiguity: more than one plausible table, column, or definition, and the team picked the wrong one. → Fix in
Mode B — Add an eval from a correction (/eval add)
Every time a stakeholder says "that's the wrong table" or "you forgot to exclude test accounts", the team has been handed a free eval.
- Write the question as the stakeholder asked it, the reference SQL that produces the right answer, and the pinned answer, into
evals/local/questions.md. Pin it: a fixed date range, an as-of date, a closed period — a question whose answer moves tomorrow is not an eval. - Add
must_mentionfor the thing that was missed ("excluding test accounts"), and a one-linewhy. - Fix the cause in the knowledge base (Mode A step 7), in the same change.
- Log the correction in
knowledge/decision-log.md.
Mode C — Check only (/eval check)
python scripts/run_evals.py --check-references and --history. No model calls. This is what CI runs, and what /health-check runs weekly.
How many, and when
- A few dozen questions per business domain is enough; returns diminish after that. Cover the top stakeholder questions, each domain's main KPIs, and every trap that has actually caught the team.
- Run the full suite after any change to an agent, a skill, the catalog, or a domain doc — and put the before/after pass rate in the commit message. "I improved the docs" is a claim; the delta is evidence. A well-meant addition that lowers the score is common.
- Gate new audiences. Before the team is announced to a new group of stakeholders, their domain's slice of
evals/local/clears the threshold (default 90%). - A pass rate near 100% means there are no known gaps, not that the team cannot be wrong. It does not replace
analysis-revieweror the provenance footer. - Keep a short "tried it, did not help" list in
knowledge/decision-log.md. Negative results are cheap to record and stop the next person re-running the experiment.
Created by Colin Beck — https://www.linkedin.com/in/beckcolin/
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

