agentleFS
anthony-chaudharyanthony-chaudhary/fak/.agents/skills/sota-check/SKILL.md

sota-check

Skill49 starsChanged 14 days agoApache-2.0View on GitHub

One repeatable pass that stops fak from re-inventing known kernel art - before writing or optimizing a compute kernel (a quantized GEMM, a fused attention, a KV-cache reuse, a MoE dispatch, a Metal/CUDA kernel), it checks the SOTA prior-art matrix for the production reference (llama.cpp / Marlin / CUTLASS / FlashInfer / vLLM / SGLang / a named paper), decides the route (borrow / bind / stay-minimal), holds the result to the named oracle, and records what was consulted in. Use when this named workflow matches the task.

The file

We split this skill into the rules it gives an agent

10 rules. About 0 are common, 6 are distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈6

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Look up and capture the reference - BEFORE writing code: Run the installed first-class binary and save its read-only machine payload:

    line 32
  2. Very specific:

    If the matrix had a blind spot, add the row: When fak sota did not know the operation, or the sota-coverage scorecard flags an uncovered …

    line 56
  3. Distinctive:

    Do not collapse the matrix to one universal winner.

    line 67
  4. Very specific:

    If you added a matrix row, python tools/sota_coverage_scorecard.py reads clean (the new row has an existing fak-path file, a primary link, …

    line 87
  5. Distinctive:

    pre-attempt: - before each specific optimization or kernel variant, not once per campaign.

    line 112
  6. Distinctive:

    post-failure: - when an attempt fails or underperforms, query again with the measured failure signature (e.g.

    line 113
  7. Distinctive:

    plateau: - after two consecutive attempts without gain, re-search upstream PRs and releases since the last record before calling the axis …

    line 116
  8. Distinctive:

    pre-claim: - before any "first", "novel", "#1", or "beats SOTA" statement, with a record no older than 7 days.

    line 118
  9. Very specific:

    When fak gains a new compute operation, add one Op row to internal/sotamatrix/sotamatrix.go (keep the slice sorted by Slug).

    line 123
  10. Very specific:

    Keep the scorecard and Go source ASCII-friendly in comments and help strings (use - for a dash, straight quotes).

    line 131

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

sota-check - the prior-art-before-scratch pass

Why this skill exists

This repo has a documented failure mode: an agent reaches for "implement the Mac Q6_K fused MLP from scratch" or "hand-roll the amd64 kquant SIMD" without first checking that llama.cpp's GGML kernels, Marlin, CUTLASS, FlashInfer, or a named paper already solved the same contraction - and re-derives, badly, what is known art. Prior art WAS being researched (idea_scout, the docs/notes/RESEARCH-* corpus, the SOTA matrix) but the research was inert: nothing on the kernel-commit path forced an agent to consult it.

The law, in one line: the default answer to "should I write this kernel from scratch?" is NO - find the production reference first, route deliberately (borrow / bind / stay-minimal), prove against an oracle, and record what you read.

The single source of truth is internal/sotamatrix (a flat in-binary literal, same discipline as internal/benchcatalog). The human front door is docs/sota/README.md. Three surfaces read the matrix: the fak sota command (lookup), the PRIOR_ART advisory gate (commit-time nudge), and tools/sota_coverage_scorecard.py (keeps the matrix complete against the tree).

The pass

  1. Look up and capture the reference - BEFORE writing code. Run the installed first-class binary and save its read-only machine payload:
    fak sota <slug> --json > sota-<slug>.json
    fak sota <the-file-you-are-about-to-edit> --json > sota-path.json
    It reports the SOTA stack, PrimaryLink, chosen route, oracle, and named papers. fak sota list enumerates every operation. If the operation is NOT in the matrix, that is itself the finding - go to step 5. Persist a generated study note only through the separate explicit fak sota <slug> --write gate; JSON capture alone must not mutate the repository.
  2. Read the primary reference. Open the PrimaryLink. Record the research date, canonical URL, pinned revision/version, and license beside the captured JSON. The point is not to copy bytes (the licenses and the language differ) - it is to learn the technique (the fused dequant tile, the online-softmax recurrence, the radix-tree reuse) so the fak version is an informed implementation, not a naive one. A web search for the named paper is fair game here, but it supplements the captured local catalog result rather than replacing it.
  3. Route honestly. Pick stay-minimal (the bit-exact contract is fak's value, not raw throughput - most rows), bind (use the production library/format directly: cuBLAS fp16, the GGUF format), or borrow (adapt the reference technique). The hard rule: borrow a kernel only after a witness for the current path exists, so the choice is evidence-based and not a premature bet.
  4. Prove against the oracle. The matrix row names the witness (almost always cpuref f32 with a cosine floor, or bit-identity). A kernel with no recorded oracle floor is not done - it can read green on a CPU box while being wrong on the device.
  5. If the matrix had a blind spot, add the row. When fak sota did not know the operation, or the sota-coverage scorecard flags an uncovered kernel file, add one Op to internal/sotamatrix/sotamatrix.go: the FileGlobs that should trigger the advisory, the verified fak path, the SOTA stack, a primary http(s) link, the route, and the oracle. This is the additive-leaf discipline - the next person inherits the reference you found.
  6. Stamp the commit. End the kernel commit with a Prior-art: trailer naming what you consulted - e.g. Prior-art: Marlin fused dequant-MMA (IST-DASLab/marlin); cosine >= 0.995 vs HF AWQ. This silences the advisory gate AND leaves a durable, greppable record.

Portfolio output — best default plus bounded coverage

Do not collapse the matrix to one universal winner. Produce both (a) the best net-true default for fak's common supported case and (b) a bounded capability superset view of useful alternatives for named cohorts. Route each viable technique through /field-borrow as DEFAULT, OPTIONAL-MODULE, RECIPE, WATCH, or EXCLUDE. A globally second-place approach may still deserve an isolated module or recipe for a different accelerator, provider, privacy constraint, deployment topology, compatibility requirement, cost envelope, or operator preference. It must not displace the default merely to increase feature count.

Bound the exercise: require dated evidence, a supported cohort, a modular seam, incremental support cost, and a disconfirming witness. Mark genuinely dominated, unsafe, license-incompatible, out-of-scope, and support-uneconomic approaches EXCLUDE; mark volatile or immature moment-in-time findings WATCH with a review trigger. This prevents both ego-driven "not invented here" rejection and boil-the-ocean accumulation.

What "done" proves

  • fak sota <op> resolves the operation and you read its PrimaryLink before writing code.
  • The kernel is held to the row's oracle (a recorded cosine floor / bit-identity), not just a "looks right" eyeball.
  • The commit carries a Prior-art: trailer.
  • If you added a matrix row, python tools/sota_coverage_scorecard.py reads clean (the new row has an existing fak-path file, a primary link, and an oracle) and the tree coverage is full.

The anti-gaming rule (specific to this surface)

Silence the PRIOR_ART advisory by actually checking prior art, never by stamping Prior-art: n/a reflexively or muting the gate. The trailer is a record of a reference you read - if you genuinely could find no prior art for a truly novel contraction, say that in the trailer as a dated dry-hole record so the claim is falsifiable and the next person can disprove it by naming the art you missed:

Prior-art: DRY-HOLE axis=<axis> searched=<class>,<class>,... observed=YYYY-MM-DD q="<query as run>"; oracle cpuref cosine >= <f>

searched names at least 3 distinct source classes, at least 2 of them live-field (github, prs, releases, papers, web; the internal classes are code, notes, registry, sota) - silence from the matrix and our own notes alone is unmined, not dry. A dry-hole record expires after 14 days (7 before a "first"/"novel"/"#1" claim). An empty n/a on a GEMM that llama.cpp ships is the exact failure this skill exists to catch.

Re-check points (the field moves mid-campaign)

The pass above is not one-shot. Re-run step 1-2 and refresh the dry-hole record:

  • pre-attempt - before each specific optimization or kernel variant, not once per campaign;
  • post-failure - when an attempt fails or underperforms, query again with the measured failure signature (e.g. "decode GEMV at 55% of roofline, bandwidth-bound") - it is a sharper question than the one you started with;
  • plateau - after two consecutive attempts without gain, re-search upstream PRs and releases since the last record before calling the axis exhausted;
  • pre-claim - before any "first", "novel", "#1", or "beats SOTA" statement, with a record no older than 7 days.

Adding an operation to the matrix

When fak gains a new compute operation, add one Op row to internal/sotamatrix/sotamatrix.go (keep the slice sorted by Slug). The sota-coverage scorecard then holds it to the same discipline - the fak-path file must exist, the primary link must be a real URL, the oracle must be named - and the PRIOR_ART gate starts surfacing its reference whenever the matching kernel files are touched.

ASCII gotcha

Keep the scorecard and Go source ASCII-friendly in comments and help strings (use - for a dash, straight quotes). The matrix DATA legitimately carries UTF-8 (cosine symbols, deltas) in its string fields; that is output content, not source scaffolding, and is fine.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.