AI Evaluation Builder

Purpose

Design an offline and online evaluation harness: datasets, metrics, baselines, regression gates, and production monitoring for AI system quality.

Input: AI architecture or workflow spec, quality requirements, sample inputs/outputs (optional), risk level
Output: Evaluation Plan with dataset spec, metric definitions, harness design, and rollout gates Examples: See examples.md for worked input/output.

Workflow

Step 1: Define evaluation objectives

Map business requirements to measurable quality dimensions:

Step 2: Design offline evaluation

Dataset specification

Metrics

Document human eval protocol when automated metrics are insufficient.

Step 3: Design online evaluation

Plan A/B or shadow deployment for model changes.

Step 4: Build harness architecture

Test cases → Runner → Model/Workflow under test → Scorers → Report → Gate

Components:

• Runner — batch and single-case execution with config versioning
• Scorers — deterministic + model-based + human queue
• Reporter — diff vs baseline, trend charts, failure clusters
• Gate — pass/fail criteria blocking deployment

Step 5: Define regression gates

Step 6: Validate

Run Validation checklist.

Decision Rules

Validation

• Quality dimensions mapped to metrics with thresholds
• Golden, regression, and adversarial datasets specified
• Offline harness components documented
• Online monitoring signals and alert thresholds defined
• Regression gates with block/warn actions
• Baseline version identified for comparison
• Human eval protocol defined where automation insufficient
• Per-agent or per-stage eval for multi-agent workflows

Anti-patterns

• Vibe-based eval — "looks good" without numeric thresholds.
• Train-test leakage — eval cases drawn from fine-tuning data.
• Single metric obsession — optimizing accuracy while ignoring safety or cost.
• Stale golden set — dataset never updated after product changes.
• No baseline — comparing only to previous broken version.

Best Practices

• Start with 50 high-quality golden cases before scaling dataset size.
• Store eval runs with model version, prompt version, and config hash.
• Cluster failures by error type for targeted fixes.
• Run eval on every prompt or model change in CI.
• Pair with ai-cost-optimizer and ai-latency-optimizer for full NFR coverage.

Output Structure

# Evaluation Plan: [System Name]

## Objectives
| Dimension | Metric | Threshold |
|-----------|--------|-----------|

## Offline Datasets
| Split | Size | Source | Refresh |
|-------|------|--------|---------|

## Scoring Methods
| Metric | Method | Calibrated |
|--------|--------|------------|

## Harness Architecture
[Component diagram and config]

## Online Monitoring
| Signal | Source | Alert |
|--------|--------|-------|

## Regression Gates
| Gate | Criteria | Action |
|------|----------|--------|

## Baseline
[Version, scores, date established]

Next Skills