tiered-rubric-loop
Skill28 starsChanged yesterdayMITView on GitHub
Drops a combined deterministic and semantic rubric verification loop into any project, acting as the loop component of an agentic harness. Automatically leverages existing Spec-Driven Development (SDD) and Test-Driven Development (TDD) artefacts—or creates lightweight ones inline—routing each check across Tier 1 deterministic gates ($0), Tier 1.5 calibrated scoring via the Gemini Decisions API (`gemini-decision-1`, ~200 ms, 0 output tokens), and Tier 2 stateless semantic critique only for grey-zone escalations or complex `[arch]` invariants. Use when running iterative implementation/refinement loops, gating PRs or complex slices against `B1`–`Bn` assertions (`SPEC.md` / `LOOPS.md`), or scoring diffs and multimodal diagrams.
- Reads credentials
In this repository10 other files it gives its agents.
The file
Show its rules and text
We split this skill into the rules it gives an agent
13 rules. About 0 are common, 4 are distinctive, and 9 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
9
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 27
Instead of waking an expensive reasoning subagent on every single pass—or trusting a zero-thinking classifier blindly on deep security and …
- Very specific: line 82
Keep assertions atomic: Test one falsifiable invariant per Bi rather than bundling three functions or eight files into one …
- Very specific: line 83
Name concrete paths in
Verification Target: Prefer src/auth.py, tests/unit/test_auth.py over generic Doc review / Code review so … - Very specific: line 84
Separate canonical registries from summary copy: Don't assert identical title formatting across both machine registries (navigation.yaml) …
- Very specific: line 106
Observe: Read the specification (docs/SPEC.md, SPEC.md, or LOOPS.md), git status --short, and current failing tests/checks.
- Distinctive: line 108
Act (TDD): Write or update the failing test first (Red), confirm it fails, then implement the minimal clean change (Green) and refactor …
- Very specific: line 110
Tier 1: Run deterministic checks (make check or repo lint + test commands).
- Very specific: line 111
Tier 1.5: Run eval_rubric.py --json over the spec, diff, and touched files (plus --image / --pdf for diagram/UI specs).
- Very specific: line 163
Default calibrated thresholds (overridable via --pass-threshold, --fail-threshold, --min-confidence):
- Distinctive: line 171
Always spawn Tier 2 as a fresh, stateless, read-only subagent that never sees the coding agent's prior chain-of-thought or failed attempts:
- Very specific: line 172
Antigravity: Call invoke_subagent with TypeName: "research" (Model: "flash" for ESCALATE_L2_FLASH.
- Very specific: line 177
When local-code-review is not installed, pass this exact prompt (filling in only the target Bi IDs, spec path, and changed files):
- Distinctive: line 197
Stop the loop whenever any of these three conditions triggers:
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Tiered Rubric Loop (tiered-rubric-loop)
A fast, loosely coupled way to drop a combined deterministic + semantic verification loop into any repository as the verification engine of an agentic harness.
It automatically reuses any Spec-Driven Development (SDD) (docs/SPEC.md or SPEC.md) and Test-Driven Development (TDD) artefacts already in the repo—or bootstraps a lightweight B1–Bn binary checklist inline if none exist.
Instead of waking an expensive reasoning subagent on every single pass—or trusting a zero-thinking classifier blindly on deep security and concurrency invariants—each assertion flows through three escalating stages:
- Tier 1 — Deterministic Checks (
$0, seconds): Runs local linters, typecheckers, and unit tests (make check,codespell,ruff,pyright,pytest) first. Any failure stops the pass immediately so you never waste API calls on broken code. - Tier 1.5 — Fast Semantic Scoring (
~200 ms,0output tokens): Evaluates open binary (yes/no) rubric criteria (B1–Bn) in a single prefill pass using the Gemini Decisions API (gemini-decision-1viascripts/eval_rubric.py). Straightforward editorial and doc-sync checks pass immediately, while obvious omissions fail fast without waking a subagent. - Tier 2 — Stateless Semantic Critique: Spawns a fresh, read-only reviewer subagent only when Tier 1.5 is uncertain ("grey zone"), or for complex architectural and security invariants (
[arch]) that need deeper multi-file reasoning.
Table of Contents
- How the 3-Stage Funnel Works
- Plain-Markdown
B1–BnContract & Tagging - Loose Coupling & Companion Skills Matrix
- Bounded Loop Workflow
- Running Tier 1.5 (
eval_rubric.py) - Threshold Routing & Stateless Tier 2 Critic
- Circuit Breaker Trio & Continuous Ratchet
- Loop Receipt Format (
LOOPS.md)
How the 3-Stage Funnel Works
| Tier | Mechanism | Latency & Cost | What It Owns |
| Tier 1 (Deterministic) | Local linters, typecheckers, unit tests (make check, codespell, ruff, pyright, pytest) | Seconds, $0 | Syntax, static types, formatting, and deterministic test execution. Any failure short-circuits immediately before Tier 1.5 or Tier 2 runs. |
| Tier 1.5 (Calibrated Probabilistic) | gemini-decision-1 via client.decisions.create (v1beta) in scripts/eval_rubric.py | ~150–250 ms, $0.10 / 1M input tokens, 0 output / thinking tokens | Evaluates up to 1,000 closed-ended noul (yes/no) assertions over a single shared diff/spec/media payload. Terminates clear mechanical passes (PASS), rejects obvious omissions immediately (FAIL_FAST), and flags uncertain items for L2. |
| Tier 2 (Stateless Semantic Critic) | Fresh read-only subagent with zero prior coding chain-of-thought | 10–40 s, standard model tokens | Diagnoses grey-zone escalations (flash for mechanical, inherit / frontier model for [arch]) and performs one final frontier sign-off on [arch] items after Tier 1.5 is green. |
Key safety rule: Because gemini-decision-1 runs in a single prefill forward pass with zero thinking tokens and zero file-reading tools, it can produce false positives on subtle logic, race conditions, or auth flaws. Therefore, Tier 1.5 is terminal for mechanical/editorial checks, fail-fast on clear omissions, and provisional for [arch] assertions.
Plain-Markdown B1–Bn Contract & Tagging
The sole shared contract is a Markdown table or bullet list of numbered binary assertions (B1–Bn) in docs/SPEC.md, SPEC.md, or LOOPS.md (with optional --spec <path> override):
| ID | Falsifiable Condition | Verification Target |
| :--- | :--- | :--- |
| `B1 [det]` | Static gates (`codespell`, `ruff`, `pyright`) and offline tests exit `0` | `make check` |
| `B2` | `.env.template` and `README.md` document every newly added variable and flag | Doc review |
| `B3 [arch]` | Auth middleware rejects expired tokens and never retries `4xx` responses | `tests/unit/test_auth.py` |Bullet assertions (- **B4**: CLI help panel renders in English (UK)), atomic sub-IDs (B4a, B11b), numbered items (1. \B5 [docs]`: ...), and Markdown task checkboxes (- [ ] B6 [concurrency]: ...or- [x] B6: ...) are all supported, alongside extended tags ([mech], [docs], [ui], [visual], [perf], [concurrency]). Unrecognised tags warn on stderr and default safely to [mech]` instead of silently dropping the assertion.
| Tag | Meaning | Tier 1.5 Handling |
[det] (or deterministic Verification Target such as make check, pytest tests/..., codespell -s, ruff check .) | Already proven by Tier 1 shell exit codes | Skipped by Tier 1.5 by default (--include-det overrides) so the model never guesses whether pytest passed from a text diff. |
(untagged, [mech], [docs], [ui], [visual], [multimodal]) | Mechanical, editorial, doc-sync, schema, or UI convention check | Terminal PASS when P(yes) >= 0.90, confidence >= 0.80, and no evidence was truncated; FAIL_FAST when <= 0.20; otherwise ESCALATE_L2_FLASH. |
[arch] (also [security], [logic], [concurrency], [perf]) | Concurrency, security, state invariant, performance, or test-realism check | Provisional pass (PROVISIONAL_ARCH_PASS) at Tier 1.5; gated for one final frontier L2 review once all items clear Tier 1.5. |
Repo-relative file paths and local images named in Verification Target or inline backticks are automatically resolved and attached (blocking .env, secret env files, *.enc, and paths outside --repo-root; pass --no-auto-target-files to disable).
Writing High-Signal B1–Bn Assertions
- Keep assertions atomic: Test one falsifiable invariant per
Birather than bundling three functions or eight files into one question—single-invariant assertions give sharpP(yes)separation and triggerFAIL_FASTaccurately on regressions. - Name concrete paths in
Verification Target: Prefersrc/auth.py, tests/unit/test_auth.pyover genericDoc review/Code reviewsoeval_rubric.pyauto-attaches untouched context files even in narrow working-tree diffs. - Separate canonical registries from summary copy: Don't assert identical title formatting across both machine registries (
navigation.yaml) and summarised overview pages (README.md, spotlight cards) unless exact parity is truly required.
Loose Coupling & Companion Skills Matrix
tiered-rubric-loop requires zero hard dependencies on other skills and degrades cleanly when optional companions or API keys are absent:
| Companion Skill / Dependency | When Present | Standalone Fallback (When Absent) |
maintaining-core-documentation / spec-driven-development | Reads B1–Bn directly from docs/SPEC.md or SPEC.md. | Reads any local SPEC.md / LOOPS.md, or asks you to agree a compact B1–Bn list inline before coding. |
test-driven-development | Delegates the Act step to full Red-Green-Refactor execution. | Enforces the core TDD rule inline: write/update a failing test first, change one semantic behaviour per pass, never weaken assertions. |
loopy | Supplies the outer Observe -> Choose -> Act -> Verify -> Record -> Stop governance and LOOPS.md receipts. | Runs the identical bounded loop directly with the receipt format below. |
local-code-review | Stateless Tier 2 read-only subagent runs local-code-review on escalated/[arch] items. | Uses the built-in Stateless Tier 2 Critic Prompt below. |
GEMINI_API_KEY / GOOGLE_API_KEY | Enables ~200 ms Tier 1.5 screening via eval_rubric.py. | When eval_rubric.py exits 3 (missing/invalid key), skips Tier 1.5 cleanly and executes Tier 1 -> Tier 2 directly while noting how to enable Tier 1.5. |
Bounded Loop Workflow
Track every pass around Observe -> Choose -> Act -> Verify -> Record -> Stop:
- Observe: Read the specification (
docs/SPEC.md,SPEC.md, orLOOPS.md),git status --short, and current failing tests/checks. ConfirmB1–Bnexist (draft and confirm them first if missing). - Choose: Pick one highest-value unverified semantic behaviour or failing assertion for this pass (mechanical lint/spelling fixes may be batched).
- Act (TDD): Write or update the failing test first (Red), confirm it fails, then implement the minimal clean change (Green) and refactor without weakening tests.
- Verify (3-Stage Funnel):
- Tier 1: Run deterministic checks (
make checkor repo lint + test commands). If Tier 1 fails, stop verification immediately and return to Choose/Act—do not spend tokens on Tier 1.5 or Tier 2. - Tier 1.5: Run
eval_rubric.py --jsonover the spec, diff, and touched files (plus--image/--pdffor diagram/UI specs). - Tier 2: Escalate only grey-zone items immediately, or run a single final
[arch]critic pass once Tier 1.5 reports zeroFAIL_FASTand zero grey-zone items.
- Tier 1: Run deterministic checks (
- Record: Append pass outcome,
ratchet_score(meanP(yes)), and remaining failures toLOOPS.md(or session progress notes). - Repeat or Stop: Enforce the Circuit Breaker Trio before starting another pass.
Running Tier 1.5 (eval_rubric.py)
Resolve scripts/eval_rubric.py relative to where tiered-rubric-loop is installed:
- Global Antigravity path:
~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py - Project-local path:
.agents/skills/tiered-rubric-loop/scripts/eval_rubric.py
In sandboxed agent runtimes that block outbound HTTPS by default, invoke uv run with unsandboxed network permission (BypassSandbox: true in Antigravity) for live API calls, or pass --offline for sandboxed --dry-run checks.
# 1. Working-tree loop: auto-discover SPEC.md / docs/SPEC.md / LOOPS.md
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --json
# 2. Branch or merged PR gating: evaluate committed changes against base ref / range
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --base main --json
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py \
--git-range origin/main...HEAD \
--only B3,B4 \
--json
# 3. Explicit spec, scoped files, and multimodal diagram/UI screenshot check
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py \
--spec docs/SPEC.md \
--file src/auth.py \
--image docs/images/architecture.png \
--json
# 4. Offline dry-run inside sandbox (verifies rubric, targets, and truncation zero-API)
uv run --offline ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --dry-run --jsonFor large PRs (> 8 files or > 100 KB), shard evaluations by assertion group via --only B3,B4 so files stay within --max-file-chars (24,000) and --max-evidence-chars (180,000). If any file or combined payload is truncated, eval_rubric.py populates truncated_sources and downgrades mechanical PASS to ESCALATE_L2_FLASH unless --allow-truncated-pass is explicitly set.
Exit Codes
| Code | Meaning | Loop Action |
0 | Tier 1.5 clear (all evaluated items are PASS or PROVISIONAL_ARCH_PASS) | If final_l2_arch_ids is non-empty, run one Tier 2 frontier critic pass on those IDs; otherwise exit loop as Success. |
1 | One or more FAIL_FAST or ESCALATE_L2_* items (or API call error) | Address FAIL_FAST items directly; escalate grey-zone items (ESCALATE_L2_*) only when root cause is non-obvious. |
2 | Usage error, missing spec, or no B1–Bn assertions parsed | Fix spec path or rubric formatting before continuing. |
3 | Missing/invalid GEMINI_API_KEY or GOOGLE_API_KEY | Degrade gracefully to Tier 1 -> Tier 2 and inform the user once. |
Threshold Routing & Stateless Tier 2 Critic
Default calibrated thresholds (overridable via --pass-threshold, --fail-threshold, --min-confidence):
PASS(P(yes) >= 0.90,confidence >= 0.80, mechanical): Terminal pass. No L2 model runs.PROVISIONAL_ARCH_PASS(P(yes) >= 0.90,confidence >= 0.80,[arch]): Held until all other assertions pass, then verified once by a frontier L2 critic.FAIL_FAST(P(yes) <= 0.20,confidence >= 0.80): Returned directly to the coding agent—do not burn an L2 turn repeating an obvious omission.ESCALATE_L2_FLASH/ESCALATE_L2_INHERIT(0.20 < P(yes) < 0.90orconfidence < 0.80): Escalate only those specificBiIDs to a fresh read-only subagent.
Subagent Routing Across Coding Agents
Always spawn Tier 2 as a fresh, stateless, read-only subagent that never sees the coding agent's prior chain-of-thought or failed attempts:
- Antigravity: Call
invoke_subagentwithTypeName: "research"(Model: "flash"forESCALATE_L2_FLASH;Model: "inherit"forESCALATE_L2_INHERITand finalPROVISIONAL_ARCH_PASSsign-off, optionally applyinglocal-code-review). - Claude Code / Gemini CLI / Other Agents: Spawn a read-only reviewer subagent (Haiku/Flash tier for mechanical escalations; Opus/Sonnet/Frontier tier for
[arch]escalations).
Built-In Stateless Tier 2 Critic Prompt
When local-code-review is not installed, pass this exact prompt (filling in only the target Bi IDs, spec path, and changed files):
You are a stateless, read-only Tier 2 semantic critic. You have NOT seen the
coding session and must judge solely from the specification and actual files on disk.
Evaluate ONLY these rubric assertions: [INSERT B_i IDs AND CONDITIONS].
Read: [SPEC.md / docs/SPEC.md] and inspect the changed/untracked files + git diff.
Rules:
1. Cite exact file paths, line numbers, and observed code/test behaviour.
2. Check whether tests verify real behaviour rather than mocking away logic.
3. For each assertion ID, return PASS or FAIL with a concise root-cause note.
4. Do not invent fixes or assume uninspected files pass.Circuit Breaker Trio & Continuous Ratchet
Stop the loop whenever any of these three conditions triggers:
- 100% Pass: Tier 1 passes, Tier 1.5 reports
exit_code == 0, and anyfinal_l2_arch_idsclear their final frontier Tier 2 review (Success). - Pass Budget Exhausted: Maximum 4–5 passes per slice reached (
Exhausted—escalate remaining blockers to the user). - 2 Consecutive Non-Improving Passes: Neither failing assertion count decreases nor
ratchet_score(meanP(yes)across evaluatedB1–Bnitems) improves across two consecutive passes (No progress—escalate to the user rather than spinning).
Tracking the continuous ratchet_score avoids premature stalls when a pass genuinely moves an assertion from e.g. P(yes) = 0.25 to 0.82 while still short of 0.90, and catches silent regressions immediately.
Loop Receipt Format (LOOPS.md)
Record completed runs (or append per-pass telemetry) concisely:
## Tiered Rubric Loop Receipt
- **Target**: [Slice / Feature / PR]
- **Spec**: [docs/SPEC.md] (`B1`–`Bn`)
- **Passes**: [N / 5] | **Outcome**: `Success` | `Exhausted` | `No progress` | `Blocked`
- **Tier 1**: `make check` -> PASS
- **Tier 1.5 Ratchet (`gemini-decision-1`)**: `0.58 -> 0.94 -> 0.98` (skipped `[det]`: `B1`)
- **Tier 2 (`Model: "inherit"`)**: Verified `[arch]` IDs `B3` -> PASS
- **Remaining / Escalation**: None (or exact failing `Bi` + file:line)Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

