scorecard
Skill49 starsChanged 14 days agoApache-2.0View on GitHub
The generic scoring doctrine the whole fak scorecard family instantiates — how to BUILD a new deterministic, tree-cross-checked, debt-driving scorecard and how to RUN any of them as a repeatable RSI pass. Every sibling (code-quality, docs, doc-appeal, seo, demo-quality, repo-hygiene, observability, learning, industry, agent-readiness, product, persona) is the same machine pointed at a different surface: pure KPIs over a data-dir or the git-tracked tree, cross-checked against reality so the score can't be gamed by editing data, folded into one *-debt. Use when this named workflow matches the task.
5 other projects use a version of this.
Adopted by
Most recently taken up by
- link7373/agentic-bi-teamtheir own version, first seen Nov 10, 2026
- mangollc/claude-SDR-Manager-plugintheir own version, first seen Nov 1, 2026
- lgtm-hq/ai-skillstheir own version, first seen Oct 13, 2026
- sparkline-ventures/product-evaltheir own version, first seen Jun 29, 2026
- tansuasici/claude-research-kittheir own version, first seen Jun 2, 2026
Other versions5 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 5
- sparkline-ventures/product-eval★ 15same name
- link7373/agentic-bi-team★ 4same name
- tansuasici/claude-research-kit★ 4same name
- lgtm-hq/ai-skills★ 1same name
- mangollc/claude-SDR-Manager-plugin★ 1same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository151 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .github/copilot-instructions.mdCopilot instructions
- llms-full.txtllms.txt
- llms.txtllms.txt
- agent-readinessSkill
- appeal-scoreSkill
- bottleneck-mapSkill
- ci-repairSkill
- claim-repro-scoreSkill
- clear-out-wipSkill
- commit-cleanSkill
- conflation-scoreSkill
- cross-validateSkill
- curate-clusterSkill
- debt-cleanSkill
- debt-orchestratorSkill
- disambiguation-scoreSkill
- dojo-rsi-scoreSkill
- dos-dispatch-loopSkill
- dos-dispatchSkill
- dos-next-upSkill
- dos-replanSkill
- dos-witness-claimSkill
- field-borrowSkill
- fleet-waveSkill
- git-subagent-syncSkill
- goalSkill
- guard-rsi-scoreSkill
- harness-creatorSkill
- harness-gardenSkill
- industry-scoreSkill
- issue-orchestratorSkill
- issue-queueSkill
- issue-triageSkill
- lightgap-scoreSkill
- memory-compactSkill
- milestone-scoreSkill
- modularizeSkill
- negframe-scoreSkill
- opencode-waveSkill
- operator-heaviness-scoreSkill
- persona-scoreSkill
- phased-planSkill
- plan-auditSkill
- quality-scoreSkill
- question-loopSkill
- refresh-cachedoc-numbersSkill
- refresh-readmeSkill
- releaseSkill
- resume-watchdog-auditSkill
- run-it-all-nightSkill
- score-2xSkill
- scout-loopSkill
- signal-firstSkill
- skill-lifecycleSkill
- skill-overlapSkill
- skill-scoreSkill
- slop-scoreSkill
- sota-checkSkill
- The first 60 shown.
The file
Show its rules and text
We split this skill into the rules it gives an agent
13 rules. 0 are common, 4 are distinctive, and 9 are very specific to this project.
0
Common
Most of the 5 edited versions we read keep these.
4
Distinctive
This author's own choices. Worth a look.
9
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- could be automatic: a setting could enforce it
- Very specific: line 27
Every <surface>_debt value is an unbounded non-negative integer: the exhaustive count of all currently detected HARD defects and coverage …
- Very specific: line 59
Do not preserve a fixed list in this skill: the registry grows.
- Very specific: line 88
A control-pane payload: Every scorecard's --json emits the same envelope so the fold and any loop runner can read it uniformly:
- Distinctive: line 104
When a task names scoreboard debt, scorecard debt, score improvement, or a 2×/50% target, the individual card is not the whole acceptance …
- Distinctive: line 114
Treat the receipts as comparable only when workspace, corpus, and detector versions match.
- Very specific: line 151
Never git add -A. Push via fak sync push.
- Very specific: line 203
Do not use git archive HEAD | tar -x for this: it strips .git, so repo-hygiene and the churn-based cards error on the missing repo.
- Distinctive: line 208
When a Go-backed card errors, the control pane now says so directly (it tags the errored Go cards and points here).
- Very specific: line 211
Run go build ./.... If it FAILS, the errored cards are a working-tree
- Very specific: line 214
If go build ./... PASSES but a card still errors, it is a real card bug — debug that card's --json directly.
- Very specific: line 225
When a surface isn't measured yet, add one.
- Very specific: line 257
Wire it into the control pane + re-pin: Add a row to SCORECARDS in tools/scorecard_control_pane.py binding {key, debt, script, label}.
- Distinctive: line 320
Keep this skill refined as the method evolves: when a new scorecard teaches a better KPI shape, a sharper cross-check, or a new failure …
“Common” is measured: kept by at least 6 in 10 of the 5 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
scorecard — the generic scoring doctrine (build one, run any)
What this is. fak measures itself with a family of scorecards, one per surface a reviewer cares about. They look different on the outside but they are the same machine: a deterministic measurement that turns "is this surface getting better or worse" into one integer you can move and prove you moved. This skill is the shared doctrine — the anatomy every scorecard follows, the RSI loop every scorecard skill runs, and the recipe for adding a new one. The per-surface skills (
quality-score,appeal-score,industry-score,agent-readiness,product-score,persona-score) are instances; this is the template they share.
The whole idea in one line: a scorecard reads reality, folds it into a *-debt
integer the cross-check won't let you fake, and a paired skill drives that integer
down by adding the real thing — never by gaming the detector.
Raw debt is an unbounded discovery ledger
Every <surface>_debt value is an unbounded non-negative integer: the exhaustive
count of all currently detected HARD defects and coverage gaps in the named corpus
under a named metric version. “Unbounded” means the schema and process impose no
maximum; a finite repository still produces a finite count.
The 0–100 score and A–F grade are separate, bounded presentation signals. They must never replace, clamp, normalize away, sample, or top-N-truncate raw debt. A renderer may preview a work-list only if the machine payload still emits every defect and the preview labels both its limit and the full count. Zero means “metric version V found no debt in corpus C now,” not “no debt exists” or “stop discovering.” When a new check finds more debt, bump the metric version and preserve the prior snapshot so same-version improvement and cross-version discovery stay distinguishable.
Discover the live family; do not hard-code a roster
The registry printed by the first-class Go front door is the truth source:
fak score --helpSelect the scorecard named by the task from that live registry and capture its machine-readable result before editing:
fak score <name> --json > <name>-before.jsonRetire binding FAIL debt before advisory polish.
Do not preserve a fixed list in this skill: the registry grows. If a registered scorecard lacks
--json, record that as a concrete gap rather than silently omitting it. The historical
python tools/scorecard_control_pane.py path is a generation/debug fallback only; before trusting
its generated pane, compare its discovered inputs with fak score --help.
The five laws every scorecard obeys
- Deterministic. Two clones at one commit score identically. No clock, no
network, no randomness in the score. Read-only over the data; the only writes are
a generated doc folder under
--markdown-dir/--markdown. - Cross-checked against reality — ungameable by editing data. A data-driven
scorecard (industry / product / persona) keeps its rows in a JSON data dir, but
every claim in a row is verified against the real tree: a path that must
exist, a CLAIMS tag the concept actually carries, a command whose
./cmd/<dir>resolves, a doc that mentions a token. So you cannot drop debt by editing the data — you fix the real thing. A tree-reading scorecard (code / agent-readiness) reads the git-tracked tree directly, which is the same property by construction. - One headline
*-debtinteger. The count of concrete, re-derivable HARD defects (plus coverage gaps, for the catalog scorecards). Driving it to zero is the goal; the*-debtis what folds into the control pane. SOFT signals lower the score but are never debt — they're judgment nudges, not work-list items. - A pure core, an impure shell. The KPIs are pure functions over facts
(
kpi_*(...) -> {kpi, group, score, detail, defects, soft}); a thin disk/git shell gathers the facts and calls them. This is what makes the core unit-testable with fixtures and the live tree a single smoke test. A control-pane payload. Every scorecard's
--jsonemits the same envelope so the fold and any loop runner can read it uniformly:{ "schema": "...", "ok": false, "verdict": "ACTION", "finding": "...", "reason": "...", "next_action": "...", "corpus": { "score": 0, "grade": "A", "<surface>_debt": 0, ... }, "kpis": [ ... ] }The control pane reads
corpus.<debt>andcorpus.grade; keep those keys.
Default portfolio contract for scoreboard-debt work
When a task names scoreboard debt, scorecard debt, score improvement, or a 2×/50% target, the individual card is not the whole acceptance surface. Run the canonical portfolio pane before edits and after the final candidate:
python tools/scorecard_control_pane.py --json > <allocated-scratch>/pane-before.json
# run the owning leaf scorecard with the same workspace/corpus scope
python tools/scorecard_control_pane.py --json > <allocated-scratch>/pane-after.jsonTreat the receipts as comparable only when workspace, corpus, and detector
versions match. The pane's index_coverage is a completion gate:
index_coverage.ok must be true, unindexed_count must be zero, and each
excluded implementation must carry a production-owned reason. A card absent
from default reporting is unfinished integration, not an improved card.
Report both the owning card's leaf-debt delta and the pane's total_debt delta.
For a 2× / 50% reduction, prove debt_after <= debt_before / 2 against the
declared authoritative baseline. A grade change, narrower corpus, or stale
baseline comparison does not prove 2×.
Preserve the measuring stick. Detector logic, discovery, exclusions, weights,
thresholds, and the pinned baseline cannot count as debt retirement unless the
issue explicitly requests a separately reviewed measurement migration. Never
--pin merely to make residual debt or a regression green. Store same-scope
before/after receipts for substantive cohorts and bind them to the shipped
commit or issue witness.
Running one as an RSI pass (the loop every scorecard skill shares)
Whichever surface you're on, the pass is the same five steps:
- Run it —
python tools/<x>_scorecard.pybuilds the work-list;--jsonis the machine payload;--critical/--gaps(catalog scorecards) rank the backlog. - Retire
*-debtworst-first — fix the heaviest KPI (or worst-served row) first, by adding the real affordance / writing the real test / correcting the real overclaim. Never weaken a claim, a guard, or a verdict to score. - Weigh the SOFT signals, then stop — fix the cheap real ones; don't chase them to zero. A token added only to move a metric is the gaming this refuses.
- Re-measure + prove —
--compare baseline.jsonprints the debt delta and the 2×/3× verdict; regenerate the committed snapshot so the doc matches the tree. - Commit only the scorecard lane, by explicit path — run
fak sync reconcile --apply, then commit viafak commit --path:
Neverfak sync reconcile --apply fak commit --path tools/<x>_scorecard.py --path tools/<x>_scorecard_test.py --path <data dir> --path <doc dir> -F msg [--push] fak sync pushgit add -A. Push viafak sync push. End the subject with the(fak <leaf>)trailer.
The clean read (measure the committed tree, not the working-tree dirt)
A scorecard number is only trustworthy if two people at the same commit get the
same number. On a shared, multi-session tree that is not automatic — a
working tree carries untracked scratch, nested mirror checkouts
(.fak/tmp/…, .dos/_dos_park/_iso_build/…), mangled root paths a prior session
dumped, and your own half-finished WIP. Two pollution classes can corrupt a read:
- File-enumeration pollution — scratch / mirror / mangled paths get counted, inflating the occurrence-counters (slop, disambiguation) many-fold.
- Build-break pollution — uncommitted WIP that doesn't compile makes every
Go-backed card (
go run ./cmd/fak …) error and drop out of the fold, so a regression there can't be caught.
The canonical clean read — run from the repo ROOT, no extra checkout needed:
python tools/scorecard_control_pane.py # the folded portfolio read
python tools/scorecard_control_pane.py --json # machine payloadClass (1) is handled in-place, by construction — the occurrence-counters report the committed floor on a dirty tree just as on a clean one, via one of two immunity mechanisms:
git ls-files(the strongest, used bycode-slop,repo-hygiene,agent-readiness,stability, …): enumerate tracked paths directly and read working-tree bytes. An untracked path is structurally unreachable.- Tracked-subtree scoping (used by
concept-disambiguation, …): walk only the source subtrees (internal/,cmd/,docs/) and skip scratch dirs (.git, caches,vendor,.dispatch-runs,.goal-runs), so a nested mirror checkout at.fak/…or.dos/…is never descended into.
(This is why the fix for a polluted read is to make the measurement immune,
never to delete the untracked junk by hand — it may be a peer's scratch on a
shared tree.) When you build a NEW tree-reading scorecard, prefer git
ls-files over a root rglob — it is immune by construction, not by remembering
to exclude every scratch dir the fleet invents.
Class (2) has one requirement: the tree must COMPILE. If your working tree
has WIP that doesn't build, either commit/stash it first, or measure a pristine
checkout of HEAD that keeps .git (every git-based card needs it):
git worktree add --detach /path/to/clean HEAD # or: git clone --local . /path/to/clean
python tools/scorecard_control_pane.py --workspace /path/to/cleanDo not use git archive HEAD | tar -x for this: it strips .git, so
repo-hygiene and the churn-based cards error on the missing repo.
Regression note — a build break is not a card bug
When a Go-backed card errors, the control pane now says so directly (it tags the errored Go cards and points here). To triage by hand:
- Run
go build ./.... If it FAILS, the errored cards are a working-tree build break — commit/stash your WIP or measure a clean HEAD checkout; the cards are fine. - If
go build ./...PASSES but a card still errors, it is a real card bug — debug that card's--jsondirectly.
The live-smoke test test_live_collect_and_fold asserts zero errored cards on
the real tree, so a genuine card breakage reds CI; this note keeps a transient WIP
build-break from being mistaken for one.
Building a NEW scorecard (the recipe)
When a surface isn't measured yet, add one. The fastest path is to copy the closest
existing instance and re-point it: product_scorecard.py / persona_readiness_scorecard.py
for a catalog (data-dir rows cross-checked against the tree), internal/agentreadinessscore
for a tree-reading scorecard (no data dir).
- Pick the shape. Catalog (a roster of rows that evolves — concepts, personas, competitors) → data dir + coverage. Tree-reading (a fixed set of affordances the tree must have) → no data dir.
- Define the KPIs as pure functions. Each returns
{kpi, group, score (0-100), detail, defects: [str], soft: [str]}. Prefix every defect string with the row/area id ("<id>: …") so per-row debt is recoverable. Group the KPIs (e.g. well-formed / reality / honesty) and weight them. - Cross-check against the real tree. This is the ungameable part — load tree facts once (paths that exist, CLAIMS tags, cmd dirs, documented verbs, doc text) and have each KPI verify the data against them. If a check can pass by editing only the data file, it's not a real check.
- Fold to a payload. Compute the weighted composite, the
A–Fgrade (grade_letter), the<surface>_debtunbounded integer (exhaustive sum of HARD defects + coverage gaps, never clamped or truncated), and emit the control-pane envelope withcorpus.<surface>_debtandcorpus.grade. - Add renderers + flags.
render(terminal work-list),--json,--compare baseline.json(prove the drop), and a doc generator (--markdownor--markdown-dir). For catalog scorecards add--chart/--critical/--gaps. - Write the test. Fixtures for each KPI's defect trigger AND its clean case, the
fold to
*-debt, plus a live smoke asserting a named metric-version/corpus result. If it is zero, call it the current detector floor, not proof of completion. - Wire it into the control pane + re-pin. Add a row to
SCORECARDSintools/scorecard_control_pane.pybinding{key, debt, script, label}. Adding a scorecard raises the portfolio total, so re-pin the baseline:python tools/scorecard_control_pane.py --pin(and committools/scorecard_baseline.jsonin the same lane). Without the re-pin the--checkratchet reads the new debt as a regression. - Write the paired RSI skill under
.claude/skills/<x>-score/SKILL.md— the five-step loop above, the one anti-gaming rule, and the commit-by-path discipline. It is an instance of this doctrine; link back here.
The anti-gaming law (the thing that makes a scorecard worth anything)
A scorecard is only as honest as its cross-check. The single rule that protects every instance: retire a defect by changing reality, never by changing the detector. A missing-affordance defect is fixed by adding the affordance, not the keyword; an overclaim is fixed by correcting the claim, not by relaxing the check; an untested package is fixed by a real test, not a stubbed one. If "fixing" a defect would mean gaming the substring match instead of improving the surface, stop — that's not a real gap, and weakening the check to make it green is the one move that turns the whole family into theater.
The growth-invariant KPI shape (when "stays constant as it grows" is the goal)
Most scorecards measure an absolute surface and report a count — and that count mechanically WORSENS as the repo grows (a 3×-bigger tree has ~3× the surfaces, so the raw defect count climbs even when discipline is unchanged). That is correct for "how clean is the tree right now," but WRONG when the goal is "does effort stay flat as we grow" (steerability, or any future "constant-as-it-scales" surface). For that goal:
- Every KPI is a ratio, density, or distribution percentile — never a raw count. A 2×-larger surface with the same discipline must score identically. A percentile is definitionally scale-free; a rate (offenders / total) is scale-free; a Gini is scale-free in value. A raw count is not — drop it.
- The headline is a 0–100 INDEX (a weighted mean of the invariant KPIs), not a debt
pile. A debt count can't answer "is effort flat" because it trends with size. The
index can. You still emit a
*-debtinteger for control-pane membership, but — - A defect is emitted only when an invariant rate crosses a FIXED threshold. The score is the rate; the debt is the count of threshold crossings. So the debt stays orthogonal to size: a clean small tree and a clean 10×-bigger tree both emit 0.
- DROP any signal that is secretly size-coupled. Some ratios still trend with growth: a spread (files-per-package) widens as packages mature; a mean (LOC-per-package) drifts as the denominator grows non-linearly; a denominator-coupled share (one package's fan-in / total packages) falls "for free" as the repo grows. If it moves from growth alone with no change in discipline, it is SOFT (advisory) or dropped — never a gate.
- Stay orthogonal to count-based siblings in the shared portfolio. If another
scorecard already emits HARD debt for a defect class (code_quality owns god-files and
tests), do NOT re-emit it — score it on the invariant rate (SOFT) and let the sibling
own the count. Re-emitting double-counts the same defect in
total_debt.
steerability is the reference instance: its only HARD KPIs are the two whose cheapest
fix is genuinely real (split a dispatch monolith; commit the ratchet) — both 0 on a
disciplined tree — and its value lives in the index + the SOFT drift signals.
Keep this skill refined as the method evolves: when a new scorecard teaches a better KPI shape, a sharper cross-check, or a new failure mode (a SOFT signal that should be HARD, a debt that double-counts), fold the lesson back here so the next scorecard starts from it.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

