Eval Architect — Agent Reference

Quick-reference for the accelint-eval-architect skill. Load specific references only when the workflow step in SKILL.md names them.

Target types

• skill — a skill in this repo (SKILL.md at root). Assessment reads files.
• tool-repo — a standalone tool repo, esp. a RAG/retrieval pipeline. Assessment reads what it can, then interviews the developer for stages + invocation.

Modes (one-line each)

• ASSESS — read/interview the target → eval profile → verifiability gate (per stage for pipelines) → framework recommendation. Read-only.
• SCAFFOLD — copy a framework starter, generate the walking skeleton, wire the build, hand off calibration.
• AUDIT — run the decay checklist against an existing evals/ dir.
• EXTEND — add metrics/fixtures to an existing eval; scaffold_eval.py --layer pulls in another template's non-conflicting files.

Reference table of contents

Rules (one-line; full reasoning in SKILL.md "NEVER" section)

• Deterministic check before judge — always.
• Generate goldens from the live schema; never hand-write them.
• Thresholds start record-only; calibrate from a baseline before committing numbers.
• One fixture / one metric / one pass test / one regression test — skeleton, not suite.
• Every metric needs a regression test that proves it can fail.
• Derive persona×scenario from the skill; flat fixtures when single-mode.
• Commit eval source before the first run.
• Match harness toolchain to the target (vitest for Node, pytest for Python).
• Taste-based output → human-review checklist, not an LLM judge.
• Detectors ship recall + false-positive-resistance as a pair — either alone rewards degenerate behavior.
• Pipeline targets: run the verifiability gate per stage (ingest/retrieve are usually deterministic; only generate needs a judge).
• RAG gold sets are human-curated; never trust a bootstrap draft as-is.
• Tool-repo comprehension is interactive; confirm stages + invocation before recommending.

Scripts

• scripts/scaffold_eval.py — copy a template; --layer add-only into an existing eval (reports conflicts); --dest alternate dirname; fails fast on missing required placeholders.
• scripts/bootstrap_goldset.py — draft a RAG gold set (content-based corpus hash; PDFs via optional pypdf; mandatory human curation; refuses to overwrite an existing draft without --force).
• scripts/suggest_thresholds.py — suggest calibrated thresholds from results/run-*.json with evidence.
• scripts/audit_checks.py — mechanical audit checks (#2/#3/#4-partial/#7/#8/#11/#12/#13/#14); pairs with references/audit.md.

Scope

Scaffolds: DeepEval, deterministic-vitest, deterministic-pytest, human-review, rag (deterministic ingest+retrieve layer + gated Ragas answer layer in one template; no separate "deterministic-retrieval" key). Tool-repo support: RAG / retrieval pipelines. Recommends-but-hands-off (no scaffold yet): Promptfoo, Inspect AI, non-RAG tool types.