agentleFS
Sign inStart free
kuker24kuker24/OpenCodeBestFriend/skills/eval-harness/SKILL.md

eval-harness

Skill0 starsChanged 2 months agoMITView on GitHub

Evaluation harness and benchmark design for prompts, agents, and LLM workflows. Implements eval-driven development (EDD), test suite generation, pass@k metrics, and regression testing. Use when validating agent behavior, comparing prompt iterations, or establishing objective quality gates for LLM outputs. Not for standard application unit tests (tdd), exploratory browser testing (playwright-qa), or security auditing (full-audit-keamanan).

77 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use brand-discovery, agent-sort and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • ysyecustSkill

    brand-discovery

    Use when a brand needs to discover or articulate its identity through structured multi-session interviews. Covers purpose, positioning, audience, personality, voice, narrative, and founder-brand tension across 8 modules using laddering, 5 Whys, and projective techniques. Produces a resumable session with disk-persisted state and a master brandbook (90_SYNTHESIS.md).

    Also in 6 projects that use this

  • niloykumarbarmanSkill

    agent-sort

    Build an evidence-backed ECC install plan for a specific repo by sorting skills, commands, rules, hooks, and extras into DAILY vs LIBRARY buckets using parallel repo-aware review passes. Use when ECC should be trimmed to what a project actually needs instead of loading the full bundle.

    Also in 14 projects that use this

  • royaluniondesign-sysSkill

    configure-ecc

    Interactive installer for Everything Claude Code — guides users through selecting and installing skills and rules to user-level or project-level directories, verifies paths, and optionally optimizes installed files.

    Also in 12 projects that use this

  • junimnjwSkill

    ralphinho-rfc-pipeline

    RFC-driven multi-agent DAG execution pattern with quality gates, merge queues, and work unit orchestration.

    Also in 12 projects that use this

  • loulanyueSkill

    nuxt4-patterns

    Nuxt 4 app patterns for hydration safety, performance, route rules, lazy loading, and SSR-safe data fetching with useFetch and useAsyncData.

    Also in 10 projects that use this

  • royaluniondesign-sysSkill

    content-hash-cache-pattern

    Cache expensive file processing results using SHA-256 content hashes — path-independent, auto-invalidating, with service layer separation.

    Also in 16 projects that use this

The file

We split this skill into the rules it gives an agent

2 rules. 0 are common, 2 are distinctive, and 0 are very specific to this project.

  • 0

    Common

    Most of the 71 edited versions we read keep these.

  • 2

    Distinctive

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run the eval suite on proposed changes and calculate the score delta.

    line 44
  2. Distinctive:

    Run the same dataset without the skill (Run A) and with it (Run B).

    line 49

“Common” is measured: kept by at least 6 in 10 of the 71 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Eval Harness

Evaluation and benchmark design specialist for LLM workflows, prompts, and agent configurations.

This skill establishes objective quality gates using Eval-Driven Development (EDD). It constructs ground-truth test datasets, defines deterministic and model-graded evaluation rubrics, computes reliability metrics (pass@1, pass@k), and guards against prompt regression.

Boundaries & Handoffs

NeedRoute
Project code test-driven development (red-green-refactor)tdd
Web application exploratory browser testing and UI flowsplaywright-qa
Application build verification profiles (FAST, STANDARD, RELEASE)rules/01-verification.md
Deep architectural failure diagnosis across agent layersagent-architecture-audit
Auditing cost tradeoffs across model tierscost-aware-llm-pipeline
Catalog verdicts and skill retirement decisionsskill-stocktake
Designing benchmarks, evaluation datasets, rubrics, and regression suiteseval-harness

Core Methodology

Consult references/methodology.md for detailed scoring criteria and evaluation templates:

  1. Dataset Construction:
    • Assemble representative input samples covering canonical happy-paths, subtle boundary conditions, and known failure modes.
    • Separate test sets into Development/Calibration and Held-Out Validation to prevent prompt over-fitting.
  2. Grading Assertions:
    • Deterministic Assertions (Fast & Objective): Exact substring matching, regular expressions, JSON schema validation, AST parsing, and status code verification.
    • Model-Graded Assertions (Semantic Evaluation): Structured evaluators using clear, blind rubrics with binary or Likert criteria. Avoid generic "rate 1-5" prompts; use explicit pass/fail definitions per grade.
  3. Metrics & Statistical Rigor:
    • Measure pass@1 (single run accuracy) and pass@k (probability of at least 1 success in k trials for stochastic workflows).
    • Track evaluation latency and token costs alongside accuracy scores.
  4. Regression Gate Workflow:
    • Establish a baseline score on the current stable prompt or agent setup.
    • Run the eval suite on proposed changes and calculate the score delta.
    • Reject any prompt change that introduces regressions on previously passing test cases.
  5. Skill Utility Gate (A/B):
    • Consult references/skill-utility.md when the question is whether a skill beats its own absence.
    • Run the same dataset without the skill (Run A) and with it (Run B); quality parity at lower token or latency cost still passes.
    • A ≈ B on the skill's claimed task is evidence for RETIRE or COMPRESS; route that verdict to skill-stocktake.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free