sota-check
Skill49 starsChanged 14 days agoApache-2.0View on GitHub
One repeatable pass that stops fak from re-inventing known kernel art - before writing or optimizing a compute kernel (a quantized GEMM, a fused attention, a KV-cache reuse, a MoE dispatch, a Metal/CUDA kernel), it checks the SOTA prior-art matrix for the production reference (llama.cpp / Marlin / CUTLASS / FlashInfer / vLLM / SGLang / a named paper), decides the route (borrow / bind / stay-minimal), holds the result to the named oracle, and records what was consulted in. Use when this named workflow matches the task.
In this repository151 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .github/copilot-instructions.mdCopilot instructions
- llms-full.txtllms.txt
- llms.txtllms.txt
- agent-readinessSkill
- appeal-scoreSkill
- bottleneck-mapSkill
- ci-repairSkill
- claim-repro-scoreSkill
- clear-out-wipSkill
- commit-cleanSkill
- conflation-scoreSkill
- cross-validateSkill
- curate-clusterSkill
- debt-cleanSkill
- debt-orchestratorSkill
- disambiguation-scoreSkill
- dojo-rsi-scoreSkill
- dos-dispatch-loopSkill
- dos-dispatchSkill
- dos-next-upSkill
- dos-replanSkill
- dos-witness-claimSkill
- field-borrowSkill
- fleet-waveSkill
- git-subagent-syncSkill
- goalSkill
- guard-rsi-scoreSkill
- harness-creatorSkill
- harness-gardenSkill
- industry-scoreSkill
- issue-orchestratorSkill
- issue-queueSkill
- issue-triageSkill
- lightgap-scoreSkill
- memory-compactSkill
- milestone-scoreSkill
- modularizeSkill
- negframe-scoreSkill
- opencode-waveSkill
- operator-heaviness-scoreSkill
- persona-scoreSkill
- phased-planSkill
- plan-auditSkill
- quality-scoreSkill
- question-loopSkill
- refresh-cachedoc-numbersSkill
- refresh-readmeSkill
- releaseSkill
- resume-watchdog-auditSkill
- run-it-all-nightSkill
- score-2xSkill
- scorecardSkill
- scout-loopSkill
- signal-firstSkill
- skill-lifecycleSkill
- skill-overlapSkill
- skill-scoreSkill
- slop-scoreSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 6 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 32
Look up and capture the reference - BEFORE writing code: Run the installed first-class binary and save its read-only machine payload:
- Very specific: line 56
If the matrix had a blind spot, add the row: When fak sota did not know the operation, or the sota-coverage scorecard flags an uncovered …
- Distinctive: line 67
Do not collapse the matrix to one universal winner.
- Very specific: line 87
If you added a matrix row, python tools/sota_coverage_scorecard.py reads clean (the new row has an existing fak-path file, a primary link, …
- Distinctive: line 112
pre-attempt: - before each specific optimization or kernel variant, not once per campaign.
- Distinctive: line 113
post-failure: - when an attempt fails or underperforms, query again with the measured failure signature (e.g.
- Distinctive: line 116
plateau: - after two consecutive attempts without gain, re-search upstream PRs and releases since the last record before calling the axis …
- Distinctive: line 118
pre-claim: - before any "first", "novel", "#1", or "beats SOTA" statement, with a record no older than 7 days.
- Very specific: line 123
When fak gains a new compute operation, add one Op row to internal/sotamatrix/sotamatrix.go (keep the slice sorted by Slug).
- Very specific: line 131
Keep the scorecard and Go source ASCII-friendly in comments and help strings (use - for a dash, straight quotes).
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
sota-check - the prior-art-before-scratch pass
Why this skill exists
This repo has a documented failure mode: an agent reaches for "implement the Mac Q6_K fused MLP from scratch" or "hand-roll the amd64 kquant SIMD" without first checking that llama.cpp's GGML kernels, Marlin, CUTLASS, FlashInfer, or a named paper already solved the same contraction - and re-derives, badly, what is known art. Prior art WAS being researched (idea_scout, the docs/notes/RESEARCH-* corpus, the SOTA matrix) but the research was inert: nothing on the kernel-commit path forced an agent to consult it.
The law, in one line: the default answer to "should I write this kernel from scratch?" is NO - find the production reference first, route deliberately (borrow / bind / stay-minimal), prove against an oracle, and record what you read.
The single source of truth is internal/sotamatrix (a flat in-binary literal, same discipline
as internal/benchcatalog). The human front door is
docs/sota/README.md. Three surfaces read the matrix: the
fak sota command (lookup), the PRIOR_ART advisory gate (commit-time nudge), and
tools/sota_coverage_scorecard.py (keeps the matrix complete against the tree).
The pass
- Look up and capture the reference - BEFORE writing code. Run the installed
first-class binary and save its read-only machine payload:
It reports the SOTA stack,fak sota <slug> --json > sota-<slug>.json fak sota <the-file-you-are-about-to-edit> --json > sota-path.jsonPrimaryLink, chosen route, oracle, and named papers.fak sota listenumerates every operation. If the operation is NOT in the matrix, that is itself the finding - go to step 5. Persist a generated study note only through the separate explicitfak sota <slug> --writegate; JSON capture alone must not mutate the repository. - Read the primary reference. Open the
PrimaryLink. Record the research date, canonical URL, pinned revision/version, and license beside the captured JSON. The point is not to copy bytes (the licenses and the language differ) - it is to learn the technique (the fused dequant tile, the online-softmax recurrence, the radix-tree reuse) so the fak version is an informed implementation, not a naive one. A web search for the named paper is fair game here, but it supplements the captured local catalog result rather than replacing it. - Route honestly. Pick
stay-minimal(the bit-exact contract is fak's value, not raw throughput - most rows),bind(use the production library/format directly: cuBLAS fp16, the GGUF format), orborrow(adapt the reference technique). The hard rule: borrow a kernel only after a witness for the current path exists, so the choice is evidence-based and not a premature bet. - Prove against the oracle. The matrix row names the witness (almost always
cpureff32 with a cosine floor, or bit-identity). A kernel with no recorded oracle floor is not done - it can read green on a CPU box while being wrong on the device. - If the matrix had a blind spot, add the row. When
fak sotadid not know the operation, or the sota-coverage scorecard flags an uncovered kernel file, add oneOptointernal/sotamatrix/sotamatrix.go: the FileGlobs that should trigger the advisory, the verified fak path, the SOTA stack, a primary http(s) link, the route, and the oracle. This is the additive-leaf discipline - the next person inherits the reference you found. - Stamp the commit. End the kernel commit with a
Prior-art:trailer naming what you consulted - e.g.Prior-art: Marlin fused dequant-MMA (IST-DASLab/marlin); cosine >= 0.995 vs HF AWQ. This silences the advisory gate AND leaves a durable, greppable record.
Portfolio output — best default plus bounded coverage
Do not collapse the matrix to one universal winner. Produce both (a) the best net-true default
for fak's common supported case and (b) a bounded capability superset view of useful
alternatives for named cohorts. Route each viable technique through /field-borrow as
DEFAULT, OPTIONAL-MODULE, RECIPE, WATCH, or EXCLUDE. A globally second-place approach
may still deserve an isolated module or recipe for a different accelerator, provider, privacy
constraint, deployment topology, compatibility requirement, cost envelope, or operator
preference. It must not displace the default merely to increase feature count.
Bound the exercise: require dated evidence, a supported cohort, a modular seam, incremental
support cost, and a disconfirming witness. Mark genuinely dominated, unsafe,
license-incompatible, out-of-scope, and support-uneconomic approaches EXCLUDE; mark volatile
or immature moment-in-time findings WATCH with a review trigger. This prevents both ego-driven
"not invented here" rejection and boil-the-ocean accumulation.
What "done" proves
fak sota <op>resolves the operation and you read itsPrimaryLinkbefore writing code.- The kernel is held to the row's oracle (a recorded cosine floor / bit-identity), not just a "looks right" eyeball.
- The commit carries a
Prior-art:trailer. - If you added a matrix row,
python tools/sota_coverage_scorecard.pyreads clean (the new row has an existing fak-path file, a primary link, and an oracle) and the tree coverage is full.
The anti-gaming rule (specific to this surface)
Silence the PRIOR_ART advisory by actually checking prior art, never by stamping
Prior-art: n/a reflexively or muting the gate. The trailer is a record of a reference you
read - if you genuinely could find no prior art for a truly novel contraction, say that in
the trailer as a dated dry-hole record so the claim is falsifiable and the next person can
disprove it by naming the art you missed:
Prior-art: DRY-HOLE axis=<axis> searched=<class>,<class>,... observed=YYYY-MM-DD q="<query as run>"; oracle cpuref cosine >= <f>searched names at least 3 distinct source classes, at least 2 of them live-field
(github, prs, releases, papers, web; the internal classes are code, notes,
registry, sota) - silence from the matrix and our own notes alone is unmined, not dry.
A dry-hole record expires after 14 days (7 before a "first"/"novel"/"#1" claim). An empty
n/a on a GEMM that llama.cpp ships is the exact failure this skill exists to catch.
Re-check points (the field moves mid-campaign)
The pass above is not one-shot. Re-run step 1-2 and refresh the dry-hole record:
- pre-attempt - before each specific optimization or kernel variant, not once per campaign;
- post-failure - when an attempt fails or underperforms, query again with the measured failure signature (e.g. "decode GEMV at 55% of roofline, bandwidth-bound") - it is a sharper question than the one you started with;
- plateau - after two consecutive attempts without gain, re-search upstream PRs and releases since the last record before calling the axis exhausted;
- pre-claim - before any "first", "novel", "#1", or "beats SOTA" statement, with a record no older than 7 days.
Adding an operation to the matrix
When fak gains a new compute operation, add one Op row to
internal/sotamatrix/sotamatrix.go (keep the slice sorted by Slug). The sota-coverage
scorecard then holds it to the same discipline - the fak-path file must exist, the primary
link must be a real URL, the oracle must be named - and the PRIOR_ART gate starts surfacing
its reference whenever the matching kernel files are touched.
ASCII gotcha
Keep the scorecard and Go source ASCII-friendly in comments and help strings (use - for a
dash, straight quotes). The matrix DATA legitimately carries UTF-8 (cosine symbols, deltas) in
its string fields; that is output content, not source scaffolding, and is fine.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

