agentleFS
Goodeye-LabsGoodeye-Labs/truesight-mcp-skills/skills/review-and-promote-traces/SKILL.md

review-and-promote-traces

Skill7 starsChanged 8 months agoMITView on GitHub

Judge flagged trace outputs and promote judged items back to datasets. Use when an evaluation run requires human judgment or when review queue items need to be judged for promotion into the dataset.

The file

Show its rules and text

We split this skill into the rules it gives an agent

16 rules. About 0 are common, 11 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈11

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use this skill for the judgment and promotion loop after trace evaluation.

    line 8
  2. Very specific:

    BEFORE the first scoping question, search for a structured question tool (e.g., AskUserQuestion or similar interactive widget) and load it.

    line 13
  3. Distinctive:

    If context is unclear, ask one question at a time using the structured question tool (loaded per the HARD-GATE above).

    line 16
  4. Distinctive:

    Ask one question per message.

    line 27
  5. Distinctive:

    Use the structured question tool for every question.

    line 28
  6. Distinctive:

    Ask one follow-up when needed, then continue.

    line 29
  7. Distinctive:

    If needed, flag evaluated run:

    line 33
  8. Very specific:

    Use flag_review_item with run_id.

    line 34
  9. Very specific:

    Use list_review_items with pagination.

    line 36
  10. Distinctive:

    Use the structured question tool (loaded per the HARD-GATE above) to present options that match the live evaluation setup:

    line 40
  11. Distinctive:

    If valid options are unclear, look them up before asking:

    line 51
  12. Very specific:

    Use get_result(run_id) for run-level context and chain outputs.

    line 53
  13. Very specific:

    If evaluation id is available in run metadata, call get_evaluation(evaluation_id) and read config/judgment criteria to derive valid …

    line 54
  14. Distinctive:

    When options remain ambiguous after lookup, ask one clarification question using the structured question tool before proceeding.

    line 55
  15. Distinctive:

    Keep an audit trail of judgment rationale in notes.

    line 71
  16. Distinctive:

    If a scope error occurs, ask the user to create a key with the missing scope in Truesight Settings.

    line 78

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Review and Promote Traces

Use this skill for the judgment and promotion loop after trace evaluation.

Interactive Q&A protocol (mandatory)

<HARD-GATE> BEFORE the first scoping question, search for a structured question tool (e.g., `AskUserQuestion` or similar interactive widget) and load it. Use that tool for EVERY scoping question. Fall back to plain-text lettered options ONLY if no such tool exists in the environment. </HARD-GATE>

If context is unclear, ask one question at a time using the structured question tool (loaded per the HARD-GATE above).

Example question structure:

What do you want to review first?
A) One specific run
B) Pending queue triage across runs

Rules:

  • Ask one question per message.
  • Use the structured question tool for every question. Structure each with a short header, 2-4 options with labels and descriptions, and place the recommended option first. Do not add "(Recommended)" or similar annotations to option labels.
  • Ask one follow-up when needed, then continue.

Workflow

  1. If needed, flag evaluated run:
    • Use flag_review_item with run_id.
  2. Retrieve review items:
    • Use list_review_items with pagination.
  3. Collect human judgments:
    • Present review items to the user and ask for judgment one item at a time when needed.
    • Capture judgment_value and optional notes for each item.
    • Use the structured question tool (loaded per the HARD-GATE above) to present options that match the live evaluation setup:
      • Binary example:
        • A) Pass
        • B) Fail
      • Categorical example:
        • A) <option 1>
        • B) <option 2>
        • C) <option 3>
      • Continuous example:
        • A) Enter numeric value (within configured range)
        • B) Skip this item for now
    • If valid options are unclear, look them up before asking:
      1. Inspect list_review_items payload for item result details and expected value hints.
      2. Use get_result(run_id) for run-level context and chain outputs.
      3. If evaluation id is available in run metadata, call get_evaluation(evaluation_id) and read config/judgment criteria to derive valid judgment options.
    • When options remain ambiguous after lookup, ask one clarification question using the structured question tool before proceeding.
  4. Submit judgments:
    • For each target item, call judge_review_item.
    • Pass the user-provided judgment_value and optional notes into judge_review_item.
    • Include notes when judgment context matters.
  5. Promote judged outputs:
    • Use add_reviewed_items_to_dataset with run_id.
  6. Report result:
    • number of items judged
    • promotion status and row counts
    • any skipped or blocked items

Queue-wide triage guidance

  • Group by run and status first.
  • Prioritize high-impact runs or oldest pending runs.
  • Keep an audit trail of judgment rationale in notes.

Scopes reference

  • list_review_items requires review:read
  • flag_review_item, judge_review_item, and add_reviewed_items_to_dataset require review:write

If a scope error occurs, ask the user to create a key with the missing scope in Truesight Settings.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.