Run Evaluations

When to Use

• After modifying any .prompty file to validate quality hasn't regressed
• Debugging failing or partial eval results
• Iterating on prompts with cached recordings to avoid LLM costs

Running Evals

Always activate the virtualenv first: .venv\Scripts\activate (Windows) or source .venv/bin/activate (Linux/macOS).

Via CLI

# Run all workflows
avc test eval

# Run a specific workflow
avc test eval --test-paths evals/tests/mention_action

# Run a single test file
avc test eval --test-paths evals/tests/filter_existing_comment/discard_azure_sdk_repeat_comment.yaml

# Multiple runs (median result kept)
avc test eval --num-runs 5 --test-paths evals/tests/filter_comment_metadata

# Use recordings (cached LLM responses) — first run saves, subsequent runs reuse
avc test eval --use-recording --test-paths evals/tests/mention_action

# Verbose output (show passing tests too)
avc test eval --style verbose

Via run.py directly

cd evals
python run.py --test-paths tests/mention_action

Existing Workflows

Evaluator kinds

• prompt — Action-based. Compares expected vs actual action, then similarity-scores the rationale. Wrong action = 0%.
• summarize_prompt — Summary-based. Uses SimilarityEvaluator on full output. Success threshold: score > 70%.

Recordings

• Stored in evals/recordings/<workflow_name>/<testcase_id>.json
• Gitignored — each dev builds their own cache
• If you change a test file, delete its recording or run without --use-recording
• --use-recording on first run makes LLM calls and saves; subsequent runs reuse cached responses

Gotchas

• Use python cli.py not .\avc: The avc.bat script calls bare python which may resolve to the system Python instead of the venv. Use .venv\Scripts\activate; python cli.py test eval ... to ensure the venv Python is used.
• Field name mismatch: Test YAML fields must exactly match target function parameter names (excluding testcase and response)
• Stale recordings: After changing a prompt, delete recordings or run without --use-recording to get fresh results
• Testcase uniqueness: The testcase field must be unique across all test files in a workflow — it's the cache key
• Kind validation: The kind in test-config.yaml must be registered in _config_loader.py (prompt or summarize_prompt)