Run QA Bot Evaluation

Run the Azure SDK QA bot evaluation locally with evals_run.py in the package tools/sdk-ai-bots/azure-sdk-qa-bot-evaluation. It calls the bot /completion endpoint concurrently to collect answers, then grades them with the Foundry builtin LLM evaluators.

Run all commands from tools/sdk-ai-bots/azure-sdk-qa-bot-evaluation with the package .venv active and az login done. See running evaluations for flags, the bot endpoint options, single-case and all-scenario recipes, and how to read results.

Triggers

USE FOR: run an evaluation on a curated dataset (basic or perf); run a single scenario; run all scenarios; run a single test case; grade against the local or deployed bot; inspect results and the pass/fail gate WHEN: "run evaluation", "run eval", "evaluate the bot", "run perf evaluation", "run basic evaluation", "run a single test case", "evaluate one question", "run all scenarios", "score the bot", "run evals locally" DO NOT USE FOR: preparing or uploading datasets, pipeline troubleshooting, knowledge-graph indexing

Rules

• --dataset accepts a local evaluation_datasets/<target>/<scenario>.jsonl path or a Foundry asset name qa-bot-<target>-<scenario>[:version] (the asset name resolves to the local file; the version is informational).
• There is no single-testcase flag. To run one case, make a one-row JSONL file and pass it as --dataset (see the reference).
• Use --is_ci False locally (uses az login); CI uses pipeline identity.
• The bot endpoint comes from BOT_SERVICE_ENDPOINT; if unset it defaults to the local http://localhost:8089 — start the agent server.py first for local runs.
• Default evaluators are all seven; pass --evaluators to subset.

Environment

Confirm and remind the user to set these before running (loads .env; copy and fill tools/sdk-ai-bots/azure-sdk-qa-bot-evaluation/env-variables):

Common commands

# One scenario (all evaluators), against the local server, no baseline gate:
python evals_run.py --dataset evaluation_datasets/perf/typespec.jsonl \
  --is_ci False --baseline_check False --cache_result full

# Same via the asset name:
python evals_run.py --dataset "qa-bot-perf-typespec:latest" --is_ci False --baseline_check False

# Subset of evaluators:
python evals_run.py --dataset "qa-bot-basic-python:latest" \
  --evaluators "bot_evals,groundedness" --is_ci False --baseline_check False

For a single test case, all scenarios, the deployed bot, and reading results / the gate, see running evaluations.

Steps

1. Ensure the required env vars are set; for a local run, start the agent server.py.
2. Choose the dataset: a scenario file, an asset name, or a one-row file for a single case.
3. Run python evals_run.py --dataset <...> --is_ci False (add --baseline_check False for ad-hoc runs, --evaluators to subset).
4. Open the printed Foundry Report URL and review the per-case score table.
5. With --cache_result full, inspect cache/<scenario>-*.json for per-case and failed-case detail.