agentleFS
Sign inStart free
Goodeye-LabsGoodeye-Labs/truesight-mcp-skills/skills/evaluate-trace/SKILL.md

evaluate-trace

Skill7 starsChanged 8 months agoMITView on GitHub

Evaluate one or more traces against an existing Truesight live evaluation. Use when a deployed live evaluation already exists and the user wants run outputs with optional handoff to review and promotion.

The file

We split this skill into the rules it gives an agent

12 rules. About 0 are common, 6 are distinctive, and 6 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈6

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 6

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use this skill when the user wants to evaluate traces with an existing live evaluation endpoint.

    line 8
  2. Very specific:

    BEFORE the first scoping question, search for a structured question tool (e.g., AskUserQuestion or similar interactive widget) and load it.

    line 13
  3. Distinctive:

    If context does not make scope clear, ask one question at a time using the structured question tool (loaded per the HARD-GATE above).

    line 16
  4. Distinctive:

    Ask exactly one clarifying question per message.

    line 28
  5. Distinctive:

    Use the structured question tool for every question.

    line 29
  6. Distinctive:

    Ask a single follow-up if needed, then proceed.

    line 30
  7. Very specific:

    If live evaluation id is unknown, call list_live_evaluations.

    line 35
  8. Very specific:

    Ensure inputs keys exactly match input_columns.

    line 38
  9. Very specific:

    Use the run_eval tool with live_evaluation_id and inputs for each trace.

    line 41
  10. Very specific:

    If human judgment is needed, route to review-and-promote-traces.

    line 47
  11. Very specific:

    Use deterministic trace ordering and log run_id for each input.

    line 51
  12. Distinctive:

    If a scope error occurs, ask the user to create an API key with the missing scope in Truesight Settings.

    line 60

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Evaluate Trace

Use this skill when the user wants to evaluate traces with an existing live evaluation endpoint.

Interactive Q&A protocol (mandatory)

<HARD-GATE> BEFORE the first scoping question, search for a structured question tool (e.g., `AskUserQuestion` or similar interactive widget) and load it. Use that tool for EVERY scoping question. Fall back to plain-text lettered options ONLY if no such tool exists in the environment. </HARD-GATE>

If context does not make scope clear, ask one question at a time using the structured question tool (loaded per the HARD-GATE above).

Example question structure:

Do you want to evaluate one trace or a batch?
A) One trace now
B) Small batch (up to 25)
C) Full batch loop

Rules:

  • Ask exactly one clarifying question per message.
  • Use the structured question tool for every question. Structure each with a short header, 2-4 options with labels and descriptions, and place the recommended option first. Do not add "(Recommended)" or similar annotations to option labels.
  • Ask a single follow-up if needed, then proceed.

Workflow

  1. Identify target live evaluation:
    • If live evaluation id is unknown, call list_live_evaluations.
    • Select public_id and verify required input_columns.
  2. Prepare inputs:
    • Ensure inputs keys exactly match input_columns.
    • Include media_url for multimodal evaluations when needed.
  3. Execute evaluation:
    • Use the run_eval tool with live_evaluation_id and inputs for each trace.
  4. Return useful outputs:
    • run_id
    • per-judgment scores/outcomes
    • brief interpretation for next action
  5. Optional handoff:
    • If human judgment is needed, route to review-and-promote-traces.

Batch mode guidance

  • Use deterministic trace ordering and log run_id for each input.
  • Apply retries with stable idempotency context in caller logic if needed.
  • Summarize failures by category or threshold, then propose review handoff.

Scopes reference

  • list_live_evaluations requires live-evaluations:read
  • run_eval requires live-evaluations:execute

If a scope error occurs, ask the user to create an API key with the missing scope in Truesight Settings.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free