Calibre-LabsCalibre-Labs/reforge-ai-evals/llms.txt

reforge-ai-evals

llms.txt95 starsChanged 16 days agoMITView on GitHub

Open-source toolkit for evaluating AI agents and LLM features, by Sandhya Hegde and Justin Bauer of Calibre (calibrelabs.ai), built for the Reforge AI Evals course (https://www.reforge.com/courses/ai-evals). Covers LLM-as-a-judge prompt design, code-based evaluators, the User Input Grid (UIG) method for eval dataset coverage, judge alignment with human labels (TPR/TNR), AI PRD and eval rubric templates, and turning production failures into regression evals. Packaged as skills for AI coding agents (Claude Code, Codex, Cursor and others). Worked examples (Corner, Ledger, Mapper) show…

Create a free account to mount

Files

llms.txt

# AI Evals Toolkit (Reforge AI Evals course)

> Open-source toolkit for evaluating AI agents and LLM features, by Sandhya Hegde and Justin Bauer of Calibre (calibrelabs.ai), built for the Reforge AI Evals course (https://www.reforge.com/courses/ai-evals). Covers LLM-as-a-judge prompt design, code-based evaluators, the User Input Grid (UIG) method for eval dataset coverage, judge alignment with human labels (TPR/TNR), AI PRD and eval rubric templates, and turning production failures into regression evals. Packaged as skills for AI coding agents (Claude Code, Codex, Cursor and others). Worked examples (Corner, Ledger, Mapper) show each method; the methods apply to any AI product.

## Methods (reusable for any AI product)

Install in any coding agent with `npx skills add https://github.com/Calibre-Labs/reforge-ai-evals`, or as the `ai-evals` Claude Code plugin (`claude plugin marketplace add Calibre-Labs/reforge-ai-evals`, then `claude plugin install ai-evals@calibre-labs`).

- [Eval guide skill](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/skills/eval-guide/SKILL.md): entry point that routes to the right eval skill
- [Eval rubric skill](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/skills/eval-rubric/SKILL.md): write an AI agent's eval rubric (outcome, trajectory, governance, experience) and release thresholds, v1 from the PRD and v2 from trace codes
- [Trace review app skill](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/skills/trace-review-app/SKILL.md): build a one-file trace review app for one AI product, with a review queue, notes, pass/fail labels, trace-code tags and exports
- [Trace codes skill](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/skills/trace-codes/SKILL.md): cluster trace-analysis notes into named yes/no failure codes with counts
- [User Input Grid skill](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/skills/uig/SKILL.md): design eval datasets that cover the dimensions along which real user inputs vary, audit coverage, fill gaps
- [LLM-as-a-judge skill](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/skills/eval-llm-judge/SKILL.md): one criterion per judge, binary PASS/FAIL, 3 few-shot examples, structured JSON output, TPR/TNR validation
- [Code-based evaluator skill](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/skills/eval-code/SKILL.md): when and how to write deterministic evaluators
- [Judge alignment skill](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/skills/llm-align/SKILL.md): measure LLM judge agreement with human labels, investigate disagreements, fix judge prompts
- [Ticket-to-eval skill](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/skills/ticket-to-eval/SKILL.md): convert support tickets and production traces into PII-stripped regression eval rows

## Templates and worked examples

- [AI PRD template](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/templates/ai-prd-template.md): PRD for agents, with autonomy, eval rubric and release thresholds
- [Corner (consumer shopping agent) PRD](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/templates/corner/corner-prd-detailed.md) with v1 and v2 eval rubrics and an annotated trace
- [Ledger (enterprise finance agent) PRD](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/templates/ledger/ledger-prd-detailed.md) with v1 and v2 eval rubrics and an annotated trace
- [Mapper (market research agent) PRD](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/templates/mapper/mapper-prd-detailed.md) with v1 eval rubric, a recorded session and ten annotatable traces
- [Support triage agent PRD](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/templates/support-triage/support-triage-prd.md)
- [Market Map User Input Grid](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/docs/uig-market-map.md): complete worked UIG

## Optional

- [Full README](https://github.com/Calibre-Labs/reforge-ai-evals/blob/main/README.md)
- [Calibre](https://calibrelabs.ai) and [Calibre blog](https://blog.calibrelabs.ai)

Download now (ZIP)

In this repository9 other files it gives its agents.
- eval-codeSkill
- eval-guideSkill
- eval-llm-judgeSkill
- eval-rubricSkill
- llm-alignSkill
- ticket-to-evalSkill
- trace-codesSkill
- trace-review-appSkill
- uigSkill

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free