Agent Evaluation Skill

Evaluate AI agent task execution using world-class LLM-as-judge patterns from DeepEval, RAGAS, and G-Eval frameworks.

Output Format

Evaluation results are saved to evals/results/eval-${yyyy-mm-dd-hh-mm}-${commit_id}.md

Results Table

Run Evaluation

# Run E2E evaluation
npx tsx eval.ts

# Or via npm
npm run eval:e2e

# Output saved to: evals/results/eval-2026-01-28-12-30-abc1234.md

Evaluation Rubric (0-5)

Pass threshold: >= 3 (development), >= 4 (production)

Evaluation Prompt Template

Use this prompt for LLM-as-judge evaluation:

You are an expert evaluator assessing AI agent task completion.

## Original Task
{{task}}

## Execution Trace
{{trace}}

## Final Output
{{output}}

## Evaluation Criteria
1. Was the core objective achieved?
2. Were appropriate tools selected?
3. Were tool arguments correct?
4. Was execution efficient (minimal steps)?
5. Is the final output accurate and complete?

## Scoring Rubric
- 5: COMPLETE - All requirements met perfectly
- 4: MOSTLY_COMPLETE - Minor issues only
- 3: PARTIAL - Core done but significant gaps
- 2: ATTEMPTED - Progress made but failed
- 1: FAILED - Wrong approach or result
- 0: NO_ATTEMPT - No meaningful progress

## Instructions
1. Analyze the execution step-by-step
2. Identify specific issues or strengths
3. Score using the rubric
4. Provide actionable recommendations

## Response Format (JSON only)
{
  "reasoning": "<step-by-step analysis>",
  "score": <0-5>,
  "verdict": "<COMPLETE|MOSTLY_COMPLETE|PARTIAL|ATTEMPTED|FAILED|NO_ATTEMPT>",
  "feedback": "<1-2 sentence summary>",
  "recommendations": ["<improvement 1>", "<improvement 2>"]
}

Quick Evaluation Playbook

Step 1: Extract Data

# Get task from session
TASK=$(cat .reflection/session_*.json | jq -r '.task' | head -1)

# Get execution trace (last 20 messages)
TRACE=$(opencode session messages --limit 20 --format json)

# Get final output
OUTPUT=$(opencode session messages --last --format text)

Step 2: Run Evaluation

# Using promptfoo (recommended)
cd evals && npx promptfoo eval \
  -c agent-eval.yaml \
  --var task="$TASK" \
  --var trace="$TRACE" \
  --var output="$OUTPUT" \
  -o results/eval-$(date +%s).json

# Or using direct API call
curl -X POST "https://api.openai.com/v1/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "<EVAL_PROMPT>"}],
    "response_format": {"type": "json_object"}
  }' | jq '.choices[0].message.content | fromjson'

Step 3: Parse Results

# Extract score and feedback
cat results/eval-*.json | jq '{
  score: .score,
  verdict: .verdict,
  feedback: .feedback,
  recommendations: .recommendations
}'

Metrics Reference

Core Agent Metrics

Composite Scores

overall_score = (
  task_completion * 0.5 +
  tool_correctness * 0.2 +
  argument_accuracy * 0.2 +
  step_efficiency * 0.1
)

promptfoo Config Example

Create evals/agent-eval.yaml:

description: Agent task completion evaluation

prompts:
  - file://prompts/agent-evaluation.txt

providers:
  - id: azure:gpt-4.1-mini
    config:
      apiHost: eastus.api.cognitive.microsoft.com
      deployment_id: gpt-4.1-mini

defaultTest:
  assert:
    - type: is-json
    - type: javascript
      value: output.score >= 0 && output.score <= 5

tests:
  - vars:
      task: "Create a hello.js file that prints Hello World"
      trace: |
        1. Agent reads current directory
        2. Agent creates hello.js with console.log("Hello World")
        3. Agent confirms file created
      output: "Created hello.js with console.log('Hello World')"
    assert:
      - type: javascript
        value: JSON.parse(output).score >= 4

Integration with Reflection Plugin

The reflection plugin uses this evaluation pattern internally:

// reflection-3.ts - simplified evaluation flow
async function evaluateTask(sessionId: string): Promise<Evaluation> {
  const task = extractInitialTask(messages)
  const trace = formatExecutionTrace(messages)
  const output = extractFinalOutput(messages)
  
  const response = await llm.chat({
    messages: [{ role: "user", content: buildEvalPrompt(task, trace, output) }],
    response_format: { type: "json_object" }
  })
  
  return JSON.parse(response.content)
}

Benchmarks

Standard Test Cases

Running Benchmarks

# Run full benchmark suite
npm run eval

# Run specific benchmark
npm run eval:judge

# View results
npm run eval:view

Best Practices

1. Always include reasoning - Makes debugging possible
2. Use structured JSON output - Parse reliably
3. Score consistently - Same rubric across all evals
4. Track over time - Catch regressions
5. Calibrate with humans - Validate judge accuracy periodically
6. Separate outcome vs process - Score both what and how

Troubleshooting