sota-finder
Skill81 starsChanged 7 days agoMITView on GitHub
Find recent or historical state-of-the-art results and build practical, source-aware leaderboards for a task or benchmark. Use for benchmark surveys, cross-paper result comparison, open-world SOTA discovery, or structured leaderboard artifacts from natural language, papers, URLs, repositories, tables, and existing result files. Keep comparable results in the main leaderboard and useful non-comparable candidates in a clearly labeled reference section. Do NOT use this skill for a general literature review with no result comparison, or for extracting one supplied paper's tables without broader SOTA discovery.
In this repository21 other files it gives its agents.
- aminer-academic-searchSkill
- aminer-daily-paperSkill
- aminer-deep-searchSkill
- aminer-experiment-to-paperSkill
- aminer-exp-extractionSkill
- aminer-free-academicSkill
- aminer-grant-applicationSkill
- aminer-paper-deep-readingSkill
- aminer-paper-writingSkill
- aminer-patent-tech-intelligenceSkill
- aminer-pdf-ocrSkill
- aminer-thesis-writingSkill
- aminer-topic-advisor-selectionSkill
- baseline-finderSkill
- citation-faithfulnessSkill
- deep-researchSkill
- paper-source-traceSkill
- pdf-citation-verifierSkill
- related-work-auditSkill
- scientific-taxonomySkill
- structured-reference-auditSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 5 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈5
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 41
Search the task name, full name, abbreviation, aliases, datasets, metrics, and major subtasks.
- Distinctive: line 47
Prefer official result pages and primary artifacts for ranking claims.
- Distinctive: line 48
Stop expanding when further checks are unlikely to change an important leader, scope, conflict, or practical recommendation.
- Distinctive: line 56
Do not turn this helper into a crawler or delay delivery to write one-off scraping infrastructure.
- Distinctive: line 94
Use plain language. Keep evidence IDs, search logs, routine validation, and defensive boilerplate out of chat.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
SOTA Finder
Find the useful answer first. Evidence supports the leaderboard; it must not crowd out the result.
Route the request
Choose the lightest effort that can answer the user:
direct: no real leaderboard research is needed, or supplied material already answers the request. Reply in chat; create files only when requested.quick: one named benchmark or narrow scope with an accessible official or primary results surface.standard: the default for several sources, multiple obvious scopes, a supplied seed, or a practical model-selection question.deep: no unified benchmark, broad historical coverage, many competing scopes, or consequential deployment and reproduction constraints.
Treat inputs and tools as independent branches: use supplied files and URLs first; an offline-only request disables web research; AMiner is optional. Unless the user restricts sources, search externally by default. Reserve enough time to synthesize and deliver instead of spending the whole run on discovery.
Research and build the leaderboard
- Search the task name, full name, abbreviation, aliases, datasets, metrics, and major subtasks. Cover official leaderboards or benchmark pages, papers and preprints, author repositories or model cards, and aggregators. Use aggregators to discover candidates, then verify important rows from stronger sources when practical.
- Cover the material scopes before deepening one of them. Split suites, subtasks, test sets, metric variants, and materially different settings when a single ranking would be misleading. There may be no universal winner.
- Classify each useful result:
- Main Leaderboard: same benchmark or dataset, split, metric, direction, and material evaluation setting, with a numeric score and a credible row-level source.
- Reference Results: useful recent or historical candidates that are not directly comparable, are only partially verified, or use a different protocol.
- Conflicts / Unavailable: contradictory, corrected, withdrawn, or inaccessible claims that affect interpretation.
- Prefer official result pages and primary artifacts for ranking claims. Record enough source, date, method, score, setting, code or weight availability, and practical constraints to explain the conclusion. Leave noncritical fields unknown rather than withholding useful results.
- Stop expanding when further checks are unlikely to change an important leader, scope, conflict, or practical recommendation. State only limitations that materially change interpretation.
For a known public HTML, JSON, CSV, XML, or text result endpoint whose rows are hard to read with native tools, the optional bounded helper can save one local copy:
python3 "<skill-root>/scripts/fetch_source.py" "<URL>"Do not turn this helper into a crawler or delay delivery to write one-off scraping infrastructure.
AMiner
AMiner may help with paper discovery, metadata, and related work, but it never replaces official pages, papers, repositories, or general web research. The Skill must start and finish without AMiner. Before paid or uncertain calls are expected to total about CNY 5 or more, ask once for confirmation.
Build the artifacts
Skip artifacts for direct unless the user requests them. Otherwise prepare one working JSON file outside the final output directory using references/data-contract.md, then validate and build:
python3 "<skill-root>/scripts/build_outputs.py" \
--input "/path/to/research-input.json" \
--output-dir "/path/to/output/bundle" \
--validate-only
python3 "<skill-root>/scripts/build_outputs.py" \
--input "/path/to/research-input.json" \
--output-dir "/path/to/output/bundle"Install requirements.txt if openpyxl is unavailable. The final directory contains:
brief-report.md: concise expanded report;full-report.html: complete self-contained report;leaderboards.xlsx: index plus leaderboard sheets;evidence.json: structured results and evidence.
Fix concrete validation errors in the working JSON and rebuild. If an artifact fails, still return the useful chat result and identify the missing file.
Chat response
Lead with the conclusion the user wants: the best-supported leader by scope, whether no single comparable winner exists, and the most practical reproducible option when relevant. Then:
- show the main leaderboard for every material scope; show the full table when it is still easy to scan;
- mention important non-comparable paper, preprint, repository, or aggregator candidates compactly;
- include only metric, cutoff-date, conflict, or coverage notes that can change the conclusion;
- link the four artifacts when they were created.
Use plain language. Keep evidence IDs, search logs, routine validation, and defensive boilerplate out of chat. Preserve practical information that can change selection or deployment, including code or weights, license, hardware, cost, and material settings.
Boundaries
Use the user's language for visible prose while keeping JSON keys, IDs, enum values, metric names, and other stable machine fields in English. Do not mix materially incomparable results into one main ranking, invent missing scores, or build a universal benchmark rule database inside the Skill.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

