selection-regression
Skill1 starsChanged 5 months agoMITView on GitHub
Run the selection-logic regression check — replay the digest selection logic on a frozen input window (candidates + pinned base snapshot, no web, no memory writes) and verify that previously "hit" items do not drop in ranking. Also freezes a window into the regression set. Produces an A/B verdict on a fixed set; never changes selection logic itself. Invoked by the armarium agent.
In this repository6 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- global-context/CLAUDE.mdCLAUDE.md
- knowledge-base/CLAUDE.mdCLAUDE.md
- calibrationSkill
- digestSkill
- healthSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 3 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 31
Read via Bash (printf '%s' "${ARMARIUM_REGRESSION_K:-3}" etc.).
- Very specific: line 40
If the intent reads as a check but version A isn't named (no git ref/SHA of the logic before the edit) — this isn't a data edge but a …
- Distinctive: line 114
Idempotency: if the set already holds a record with the same run id / the same frozen input — do not create a duplicate.
- Distinctive: line 116
Write the record under the carrier-reliability lock (single writer).
- Very specific: line 141
Read (Read) .claude/armarium/eval/regression-set.md — the set records.
- Very specific: line 147
Slice unavailable: check that the slice's git-SHA resolves (git cat-file -e <sha> or git rev-parse --verify <sha>^{commit}).
- Distinctive: line 150
Stale set: if the frozen slice lags noticeably behind the current base — the input for checking the *logic* stays valid (the slice is …
- Very specific: line 153
Run both versions — A and B — through digest replay (contract above, via Agent) on the window's frozen input and base slice, each K times …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
selection-regression — A/B check of the selection logic
An active, targeted tool: when the selection logic changes (an edit to digest's prompts/principles
or to the taste profile), replay the same frozen window through both versions of the logic and
verify that previously "hit" items didn't drop. Not a recon mode and not a continuous metrics count — a
one-off check of a specific edit on a frozen input.
This is a headless procedure (invoked by the agent with no owner dialog): there are no status lines or checklist — there's no one to watch the run. The skill does not change the selection logic and doesn't apply/revert the edit — it hands back an objective verdict, the decision is the owner's.
One skill, two flows; the flow is chosen by the intent of the request.
Configuration (env, with defaults)
ARMARIUM_REGRESSION_K— the number of repeats of each version's run on one input (damping sampling noise), default 3, working range 3–5.ARMARIUM_REGRESSION_TOP— the passing top of the digest (the target volume), default 5.ARMARIUM_REGRESSION_MARGIN— the spread threshold: by how many positions a "hit" item's median rank in B must be worse than in A to count as a regression (a drop within the repeat spread doesn't count as a regression). Default 1. The exact form of the threshold is finalized on a dry run; the invariant that doesn't change is "hits don't drop from A to B".
Read via Bash (printf '%s' "${ARMARIUM_REGRESSION_K:-3}" etc.); on absence — the default.
When to use (flow choice by intent)
- Flow A — freeze a window into the set. "freeze the current/last window", "add the window to the regression set". A write action.
- Flow B — regression check. "check the selection regression", "did the edit break selection", "run A/B over the set". A read-only action (verdict to chat).
If the intent reads as a check but version A isn't named (no git ref/SHA of the logic before the
edit) — this isn't a data edge but a missing parameter of Flow B: request from the caller a reference to
version A (the previous digest commit); without it a true A/B is impossible.
Process indication
A headless run with no dialog gate: status lines (§indication) and the checklist are not opened — there's no one to present to. The verdict and edges are laid out in Flow B's final report.
Boundaries (what the skill does not do)
- Doesn't change the selection logic and doesn't apply/revert the edit — that's the owner's decision. This skill gives only an A/B verdict.
- No web. No
WebSearch/WebFetch: the logic is checked on a fixed input, not source drift. This is a load-bearing invariant — the absence of web guarantees we measure the edit, not the world. - Real logic, not a copy. The run goes through the
digestreplay mode (contract below); this skill doesn't duplicate the selection logic or read its text, it calls it. - Writes only to the service zone
.claude/armarium/eval/. Overknowledge-base/— read-only (the base slice is read at a commit). Doesn't write tostate/taste/sources/metrics. - The regression mode does NOT run
base-diff.sh— the taste diff isn't launched, the frontier doesn't move, the recon memory protocol isn't executed. The input is the set + a base slice by git-SHA. - Doesn't touch the
#from-curatormarker (only reads it by grep when producing the freeze verdict). - Only retention of "hits", not completeness — what was missed isn't measured (the unshown isn't visible).
- Replay with no side effects — writes no finish-memory, doesn't move the frontier, doesn't go to the web.
digest replay-mode contract
This regression check runs the real selection logic through digest, calling it in replay mode
via Agent. Replay mode is a delta to digest (owned there); here only the call contract that this
skill consumes is pinned. The Agent task text passes these parameters:
- frozen input — the window's candidate list as
scoutreturned it (title + link + raw material + domain); thescoutfan-out is not launched; - base slice by git-SHA — dedup and "base link" are computed against the base at the pinned commit (read-only), not the current one;
- logic version — a git ref/SHA of the
digestversion: A (before the edit) or B (after, current); - repeat index
kin 1..K — for reproducibility/repeat accounting; - no-side-effects mode — no web, no memory writes (
armarium-state/taste), no frontier move, nobase-diff.sh.
Replay returns the window's ranked candidate list with a rank and a pass-through mark for the passing top — by each version and each repeat. This skill reduces that to median ranks and compares.
Flow A — freeze a window into the set (write)
Freezes a real run's window as an immutable set record: input + base slice + shown items with their rank + a "hit/miss" verdict. The verdict is fixed as a historical fact at the moment of freezing and is not recomputed afterward.
- Determine the window to freeze: "current/last" → the last delivered run; or the one the owner
specified. Read (
Read).claude/armarium/memory/armarium-state.md, the "Shown" section — this run's shown items with their rank. - Take the run's frozen scout input from the capture that
digeststores on a run (a commitment it fulfills): the snapshot.claude/armarium/eval/scout-capture.md— header (run date +base-SHA) + the window's full candidate list (not just the shown ones), asscoutreturned it. This is a transient snapshot of the latest run: if the window being frozen isn't the last run and capture has already been overwritten — name the limitation, don't reconstruct the input by guesswork. Replay runs the logic on the same input. - Pin the git-SHA of the base slice at the window's moment (
Bash): for the current window —git rev-parse HEAD; for a past one — the SHA recorded for that run. The SHA refers to the state ofknowledge-base/. - Produce a "hit/miss" verdict for each shown item from the historical save signal (the feedback
loop): a shown item later saved by the owner with the
#from-curatormarker is a hit. Check by grep over the base (read-only, don't change the marker):
and match the found saves to the shown items by topic/link. A save without the marker is an ambiguous case: by the feedback-loop rule the marker = a direct hit; mark such an item "miss/ambiguous" and name the limitation, don't second-guess.grep -rIl -F -- "#from-curator" knowledge-base/ - Idempotency: if the set already holds a record with the same run id / the same frozen input — do not create a duplicate; name the existing record and exit.
- Write the record under the carrier-reliability lock (single writer). From the repo root:
Start: bash .claude/armarium/bin/run-guard.sh check (exit 3 → bash .claude/armarium/bin/run-guard.sh recover) (exit 1 at lock → another writer is active: skip the run, don't wait) bash .claude/armarium/bin/run-guard.sh lock regression Write: Write — append an immutable window record to .claude/armarium/eval/regression-set.md (in the format from that file's header; id = next in sequence) Finish: bash .claude/armarium/bin/run-guard.sh commit bash .claude/armarium/bin/run-guard.sh log --mode regression --outcome ok bash .claude/armarium/bin/run-guard.sh unlockcheckexit 4 (orphaned lock) — no need to clear it by hand,lockclears it.commitis idempotent (NOTHING-TO-COMMITis not an error).
Freezing only the output isn't enough: without a frozen input and base slice, the run isn't reproducible (scout will return something else, the base moves ahead) — it'd measure input drift, not the edit. So we freeze the input + base slice + verdict.
Flow B — regression check, a true A/B (read-only)
A read-only run (replay writes no memory, the set doesn't change) — the run-guard lock is not taken.
- Read (
Read).claude/armarium/eval/regression-set.md— the set records.- Empty set (not a single window) → honestly: "the set is empty, there's nothing to check the regression against". No fake "passed". Exit.
- Determine the logic versions: A — a git ref/SHA of
digestbefore the edit; B — the current one (after the edit). Version A arrives as a parameter (see "When to use"). - For each window in the set:
- Slice unavailable: check that the slice's git-SHA resolves (
git cat-file -e <sha>orgit rev-parse --verify <sha>^{commit}). If it doesn't (history rewritten/squashed) → mark the window "slice unavailable", exclude it from the run, don't reconstruct. - Stale set: if the frozen slice lags noticeably behind the current base — the input for checking the logic stays valid (the slice is pinned by SHA), but the verdict's relevance is lower: name the age of the window and suggest an update (Flow A), don't mask it.
- Run both versions — A and B — through
digestreplay (contract above, viaAgent) on the window's frozen input and base slice, eachKtimes (ARMARIUM_REGRESSION_K). Collect a ranked list with pass-through to the top for each run. - For each item take the median rank over the
Krepeats — separately in A and in B. Remember the within-version repeat spread (for the threshold). - Determine the window's "hits" by the frozen "hit/miss" verdict from the record. The frozen verdict is only a label (which items are "hits"), not a reference rank.
- Slice unavailable: check that the slice's git-SHA resolves (
- Regression criterion (comparison of median ranks A↔B, deterministic):
- a previously "hit" item in B doesn't fall out of the passing top (
ARMARIUM_REGRESSION_TOP, guide 3–5) and doesn't drop below its median rank in A (or below "miss" items in the same window); - a drop counts as a regression only if it exceeds the within-version repeat spread (beyond
ARMARIUM_REGRESSION_MARGIN) — a single run doesn't render the verdict; - at least one hit that dropped this way → regression, the window fails;
- not counted as a regression: new items rising and "miss" items reordering — the criterion is about retaining "hits", not about the whole ranking being unchanged. A different top composition without hits dropping → not a regression.
- a previously "hit" item in B doesn't fall out of the passing top (
- A/B verdict (report to chat, nothing is written):
- per window — passed / regression; on a regression — the concrete dropped "hits" with their median rank in A and in B;
- windows excluded as "slice unavailable" and stale ones — name them separately (not silently);
- aggregate over the set: the edit is safe to accept (neutral/improves) / the edit causes a regression.
- This skill does not decide to apply/revert the edit — the verdict is objective, the choice is the owner's.
Edges (roundup)
- Empty set → "nothing to check against", no fake "passed".
- Stale set → name the window's age, suggest an update; the logic verdict stays valid.
- Unresolvable slice git-SHA → mark "slice unavailable", exclude the window from the run.
- Different top composition without hits dropping → not a regression.
- Save without
#from-curator(at freezing) → by the feedback-loop rule treated as "miss/ambiguous"; name the limitation, don't second-guess. - Lock held by another writer (Flow A,
lockexit 1) → skip the run, don't wait.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

