build-review-interface
Skill5 starsChanged 34 days agoApache-2.0View on GitHub
Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels.
2 other projects use a version of this.
Adopted by
Most recently taken up by
- ai-evals-course/evals-skillsword for word, first seen Aug 31, 2026
- Goodeye-Labs/truesight-mcp-skillstheir own version, first seen Mar 10, 2026
Other versions2 projects use a version of this.
Word for word · 1
- ai-evals-course/evals-skills★ 1.4k100% same
Edited · 0
None found.
Wrote their own version · 1
- Goodeye-Labs/truesight-mcp-skills★ 7same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository20 other files it gives its agents.
- AGENTS.mdAGENTS.md
- create-taskSkill
- error-discoverySkill
- eval-auditSkill
- evals-startSkill
- generate-synthetic-dataSkill
- harbor-execSkill
- rewardkitSkill
- validate-evaluatorSkill
- write-judge-promptSkill
- writing-for-agentsSkill
- writing-great-skillsSkill
- cerbos-audit-insightsSkill
- cerbos-authz-migrationSkill
- cerbos-embedded-pdpSkill
- cerbos-hub-setupSkill
- cerbos-pep-integrationSkill
- cerbos-policySkill
- cerbosSkill
- cerbos-synapse-extensionSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 7 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 19
Collapse repetitive elements: If every trace shares the same system prompt, put it in a <details> toggle.
- Distinctive: line 20
Extract and surface key metadata: If traces contain a property name, client type, or session ID buried in the data, extract it and display …
- Distinctive: line 21
Color-code by role or status: Use left-border colors to distinguish user messages, assistant messages, tool calls, and system prompts at a …
- Distinctive: line 79
After building the interface, verify it with Playwright.
- Distinctive: line 87
Functional test: Write a Playwright script that performs a full annotation workflow:
- Distinctive: line 93
Verify the trace counter updates correctly
- Distinctive: line 94
Verify auto-save by reloading the page and checking labels persist
- Distinctive: line 96
Test that all keyboard shortcuts trigger the correct actions
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Build a Custom Annotation Interface
Overview
Build an HTML page that loads traces from a data source (JSON/CSV file), displays one trace at a time with Pass/Fail buttons, a free-text notes field, and Next/Previous navigation. Save labels to a local file (CSV/SQLite/JSON). Then customize to the domain using the guidelines below.
Data Display
Format all data in the most human-readable representation for the domain. Emails should look like emails. Code should have syntax highlighting. Markdown should be rendered. Tables should be tables. JSON should be pretty-printed and collapsible.
- Collapse repetitive elements. If every trace shares the same system prompt, put it in a
<details>toggle. - Extract and surface key metadata. If traces contain a property name, client type, or session ID buried in the data, extract it and display it prominently as a header or badge.
- Color-code by role or status. Use left-border colors to distinguish user messages, assistant messages, tool calls, and system prompts at a glance.
- Group related elements visually. Tool calls and their responses should be visually linked (indentation, shared border).
- Collapse what doesn't help judgment. Verbose tool response JSON, intermediate reasoning steps, and debugging context go behind toggles.
- Highlight what matters most. Make the primary content reviewers judge visually dominant. Bold key entities (prices, dates, names). Use font size and spacing to create hierarchy.
- Show the full trace. Include all intermediate steps (tool calls, retrieved context, reasoning), not just the final output. Collapse them by default but keep them accessible.
- Sanitize rendered content. Strip raw HTML from LLM outputs before rendering. Disable images in rendered markdown if they could be tracking pixels.
Feedback Collection
Annotate at the trace level. The reviewer judges the whole trace, not individual spans.
- Binary Pass/Fail buttons as the primary action.
- Free-text notes field for the reviewer to describe what went wrong (or right).
- Defer button for uncertain cases.
- Auto-save on every action.
Once you have established failure categories from error analysis, you can later add predefined failure mode tags as clickable checkboxes, dropdowns or picklists so reviewers can select from known categories in addition to writing notes. But don't add these in the initial build.
Navigation and Status
- Next/Previous buttons and keyboard arrow keys.
- Trace counter showing position and progress ("12 of 87 remaining").
- Jump to specific trace by ID.
- Counts of labeled vs unlabeled traces.
Keyboard Shortcuts
Arrow keys = Navigate traces
1 = Pass 2 = Fail
D = Defer U = Undo last action
Cmd+S = Save Cmd+Enter = Save and nextSelecting Traces to Load
Build the app to accept traces from any source (JSON/CSV file). Keep sampling logic outside the app in a separate script. Start with random sampling.
Additional Features
Reference panel: Toggle-able panel showing ground truth, expected answers, or rubric definitions alongside the trace.
Filtering: Filter traces by metadata dimensions relevant to the product (channel, user type, pipeline version).
Clustering: Group traces by metadata or semantic similarity. Show representative traces per cluster with drill-down.
Design Checklist
- Same layout, controls, and terminology on every trace
- Pass and Fail buttons are visually distinct (color, size)
- Keyboard shortcuts work for all primary actions
- Full trace accessible even when sections are collapsed
- Labels persist automatically without explicit save
- Trace-level annotation (not span-level) as the default
- All data rendered in its native format (markdown as HTML, code with highlighting, JSON pretty-printed, tables as HTML tables, URLs as clickable links)
Testing
After building the interface, verify it with Playwright.
Visual review: Take screenshots of the interface with representative trace data loaded. Review each screenshot for:
- Layout and spacing: is the visual hierarchy clear? Can you immediately see what matters?
- Readability: is all data rendered in its native format? Are there any raw JSON blobs, unrendered markdown, or unstyled content?
- Aesthetics: does the interface look professional and clean? Would a domain expert use this?
- Responsiveness: does the layout hold at different window sizes?
Functional test: Write a Playwright script that performs a full annotation workflow:
- Load the app and verify traces are displayed
- Click Pass on a trace, verify the label is saved
- Click Fail on a trace, add a note, verify both are saved
- Click Defer, verify it is recorded
- Navigate forward and backward with buttons and keyboard shortcuts
- Verify the trace counter updates correctly
- Verify auto-save by reloading the page and checking labels persist
- Expand collapsed sections (system prompts, tool calls) and verify content is accessible
- Test that all keyboard shortcuts trigger the correct actions
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

