review-and-promote-traces
Skill7 starsChanged 8 months agoMITView on GitHub
Judge flagged trace outputs and promote judged items back to datasets. Use when an evaluation run requires human judgment or when review queue items need to be judged for promotion into the dataset.
In this repository8 other files it gives its agents.
- bootstrap-template-evaluationSkill
- build-review-interfaceSkill
- create-evaluationSkill
- error-analysisSkill
- eval-auditSkill
- evaluate-traceSkill
- generate-synthetic-dataSkill
- truesight-workflowsSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
16 rules. About 0 are common, 11 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈11
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 8
Use this skill for the judgment and promotion loop after trace evaluation.
- Very specific: line 13
BEFORE the first scoping question, search for a structured question tool (e.g., AskUserQuestion or similar interactive widget) and load it.
- Distinctive: line 16
If context is unclear, ask one question at a time using the structured question tool (loaded per the HARD-GATE above).
- Distinctive: line 27
Ask one question per message.
- Distinctive: line 28
Use the structured question tool for every question.
- Distinctive: line 29
Ask one follow-up when needed, then continue.
- Distinctive: line 33
If needed, flag evaluated run:
- Very specific: line 34
Use flag_review_item with run_id.
- Very specific: line 36
Use list_review_items with pagination.
- Distinctive: line 40
Use the structured question tool (loaded per the HARD-GATE above) to present options that match the live evaluation setup:
- Distinctive: line 51
If valid options are unclear, look them up before asking:
- Very specific: line 53
Use get_result(run_id) for run-level context and chain outputs.
- Very specific: line 54
If evaluation id is available in run metadata, call get_evaluation(evaluation_id) and read config/judgment criteria to derive valid …
- Distinctive: line 55
When options remain ambiguous after lookup, ask one clarification question using the structured question tool before proceeding.
- Distinctive: line 71
Keep an audit trail of judgment rationale in notes.
- Distinctive: line 78
If a scope error occurs, ask the user to create a key with the missing scope in Truesight Settings.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Review and Promote Traces
Use this skill for the judgment and promotion loop after trace evaluation.
Interactive Q&A protocol (mandatory)
<HARD-GATE>
BEFORE the first scoping question, search for a structured question tool (e.g., `AskUserQuestion` or similar interactive widget) and load it. Use that tool for EVERY scoping question. Fall back to plain-text lettered options ONLY if no such tool exists in the environment.
</HARD-GATE>If context is unclear, ask one question at a time using the structured question tool (loaded per the HARD-GATE above).
Example question structure:
What do you want to review first?
A) One specific run
B) Pending queue triage across runsRules:
- Ask one question per message.
- Use the structured question tool for every question. Structure each with a short header, 2-4 options with labels and descriptions, and place the recommended option first. Do not add "(Recommended)" or similar annotations to option labels.
- Ask one follow-up when needed, then continue.
Workflow
- If needed, flag evaluated run:
- Use
flag_review_itemwithrun_id.
- Use
- Retrieve review items:
- Use
list_review_itemswith pagination.
- Use
- Collect human judgments:
- Present review items to the user and ask for judgment one item at a time when needed.
- Capture
judgment_valueand optionalnotesfor each item. - Use the structured question tool (loaded per the HARD-GATE above) to present options that match the live evaluation setup:
- Binary example:
- A) Pass
- B) Fail
- Categorical example:
- A) <option 1>
- B) <option 2>
- C) <option 3>
- Continuous example:
- A) Enter numeric value (within configured range)
- B) Skip this item for now
- Binary example:
- If valid options are unclear, look them up before asking:
- Inspect
list_review_itemspayload for item result details and expected value hints. - Use
get_result(run_id)for run-level context and chain outputs. - If evaluation id is available in run metadata, call
get_evaluation(evaluation_id)and read config/judgment criteria to derive valid judgment options.
- Inspect
- When options remain ambiguous after lookup, ask one clarification question using the structured question tool before proceeding.
- Submit judgments:
- For each target item, call
judge_review_item. - Pass the user-provided
judgment_valueand optionalnotesintojudge_review_item. - Include notes when judgment context matters.
- For each target item, call
- Promote judged outputs:
- Use
add_reviewed_items_to_datasetwithrun_id.
- Use
- Report result:
- number of items judged
- promotion status and row counts
- any skipped or blocked items
Queue-wide triage guidance
- Group by run and status first.
- Prioritize high-impact runs or oldest pending runs.
- Keep an audit trail of judgment rationale in notes.
Scopes reference
list_review_itemsrequiresreview:readflag_review_item,judge_review_item, andadd_reviewed_items_to_datasetrequirereview:write
If a scope error occurs, ask the user to create a key with the missing scope in Truesight Settings.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

