agentleFS
Sign inStart free
MicrosoftKnown publishermicrosoft/skills-for-copilot-studio/skills/run-eval/SKILL.md

run-eval

Skill486 starsChanged 8 months agoMITView on GitHub

Run evaluations against a Copilot Studio agent via the Power Platform Evaluation API. Works on DRAFT agents — no publish step required. Lists test sets, starts a run, polls until complete, fetches results, and proposes YAML fixes for failures. Use when the user wants to test agent changes without publishing.

7 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

6 rules. 0 are common, 2 are distinctive, and 4 are very specific to this project.

  • 0

    Common

    Most of the 7 edited versions we read keep these.

  • 2

    Distinctive

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run evaluations against a Copilot Studio agent's draft — no publish needed.

    line 15
  2. Very specific:

    All eval-api commands run in the foreground.

    line 19
  3. Very specific:

    Add --connection-id <id> if the user provided a connection ID in Step 2.

    line 56
  4. Very specific:

    Add --published only if the user explicitly asked for published-bot testing.

    line 58
  5. Very specific:

    Stop when state is Completed, Failed, Abandoned, or Cancelled.

    line 68
  6. Distinctive:

    After applying: offer to push and re-run (go back to Step 3).

    line 89

“Common” is measured: kept by at least 6 in 10 of the 7 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Run Evaluation (PPAPI)

Run evaluations against a Copilot Studio agent's draft — no publish needed.

The caller (test agent) must provide --client-id and --workspace. If you don't have the client ID, return immediately and tell the caller to run test-auth first.

All eval-api commands run in the foreground. NEVER use run_in_background.

Step 1: List test sets and let the user choose

node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js list-testsets --workspace <path> --client-id <id>
  • No test sets found: Tell the user to create one in Copilot Studio (Evaluate tab > New evaluation). Stop.
  • One test set: Tell the user which one you're using and proceed.
  • Multiple test sets: Show them all and ask the user to pick. Do not proceed until they answer.

Step 2: Ask about authenticated execution — MANDATORY, do not skip

You MUST ask this question and wait for the user's answer before starting the run.

Ask the user:

Does your agent use authenticated knowledge sources or connector actions (tools) that require user identity? If so, you'll need to provide a connection ID — without it, the eval runs anonymously and tools and knowledge sources will not be used.

How to obtain the connection ID:

  1. Go to https://make.powerautomate.com
  2. Open Connections from the side menu
  3. Select the relevant Microsoft Copilot Studio connection
  4. Copy the connection ID from the URL (the GUID segment after /connections/)

If your agent doesn't use authenticated knowledge or tools, you can skip this.

Do not proceed to Step 3 until the user responds.

Step 3: Start the run

node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js start-run --workspace <path> --client-id <id> --testset-id <id> --run-name "Draft eval <date>"

Add --connection-id <id> if the user provided a connection ID in Step 2.

Add --published only if the user explicitly asked for published-bot testing.

Step 4: Poll until complete

node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js get-run --workspace <path> --client-id <id> --run-id <runId>

Poll every 15-30 seconds. Report progress: "Processing: 3/10 test cases..."

Stop when state is Completed, Failed, Abandoned, or Cancelled.

Step 5: Fetch and analyze results

node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js get-results --workspace <path> --client-id <id> --run-id <runId>

Present a summary table (total, passed, failed, errors). For failures:

MetricWhat to check
GeneralQuality FailWhich of relevance/completeness/groundedness/abstention failed
ExactMatch FailScore 0.0–1.0
CapabilityUse FailmissingInvocationSteps
Error statuserrorReason — often a test set config issue, not a YAML issue

Step 6: Propose fixes (if failures found)

For YAML authoring failures: find the relevant topic, read it, propose specific edits. Wait for user approval before applying.

After applying: offer to push and re-run (go back to Step 3).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free