agentleFS
Sign inStart free
MicrosoftKnown publishermicrosoft/Building-AI-Agents-From-Zero-To-Production/.github/skills/agent-evaluation/SKILL.md

agent-evaluation

Skill194 starsChanged 4 months agoMITView on GitHub

Help a learner evaluate and observe an AI agent on Microsoft Foundry — quality/safety evaluators, tracing, and reading results. Use for Lesson 3 tasks: measuring agent performance, adding observability, or interpreting evaluation output.

  • Reads credentials

We split this skill into the rules it gives an agent

3 rules. 0 are common, 2 are distinctive, and 1 is very specific to this project.

  • 0

    Common

    Most of the 16 edited versions we read keep these.

  • 2

    Distinctive

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Use the evaluation/observability APIs from the current agent-framework surface (Foundry evals + tracing).

    line 15
  2. Distinctive:

    When an eval fails, look at the trace first (bad tool call?

    line 29
  3. Distinctive:

    Keep evaluation runs cheap: small datasets, and delete resources afterward (see cost & cleanup).

    line 31

“Common” is measured: kept by at least 6 in 10 of the 16 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Agent evaluation & observability

Guidance for measuring and improving agents (Lesson 3).

Guardrails (always apply)

  • Platform: Microsoft Foundry; framework: Microsoft Agent Framework.
  • Use the evaluation/observability APIs from the current agent-framework surface (Foundry evals + tracing). Model for any judge/eval step: gpt-5.1 (never gpt-4o / gpt-4.1).
  • Endpoints/models from .env; az login for auth.

Concepts a learner should grasp

  1. Why evaluate: agents are non-deterministic — you need repeatable metrics (groundedness, relevance, coherence, safety) to know whether a change helped or hurt.
  2. Observability vs evaluation: tracing captures what happened on each run (spans, tool calls, tokens); evaluation scores those runs against criteria.
  3. Close the loop: trace → evaluate → adjust instructions/tools → re-evaluate.

How to help

  • Point the learner at lesson-3-agent-evals/ and its README for the exact evaluator setup.
  • Encourage a small, fixed test set of prompts so results are comparable across runs.
  • When an eval fails, look at the trace first (bad tool call? missing context? weak instructions?) before changing the model.
  • Keep evaluation runs cheap: small datasets, and delete resources afterward (see cost & cleanup).

Validate

  • python -m py_compile <file>.py.
  • An evaluation run should produce per-criterion scores you can compare between versions.

References

  • lesson-3-agent-evals/README.md
  • MIGRATION-GUIDE.md (current SDK patterns)

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free