agentleFS
derailed-dashderailed-dash/dazbo-agent-skills/skills/tiered-rubric-loop/SKILL.md

tiered-rubric-loop

Skill28 starsChanged yesterdayMITView on GitHub

Drops a combined deterministic and semantic rubric verification loop into any project, acting as the loop component of an agentic harness. Automatically leverages existing Spec-Driven Development (SDD) and Test-Driven Development (TDD) artefacts—or creates lightweight ones inline—routing each check across Tier 1 deterministic gates ($0), Tier 1.5 calibrated scoring via the Gemini Decisions API (`gemini-decision-1`, ~200 ms, 0 output tokens), and Tier 2 stateless semantic critique only for grey-zone escalations or complex `[arch]` invariants. Use when running iterative implementation/refinement loops, gating PRs or complex slices against `B1`–`Bn` assertions (`SPEC.md` / `LOOPS.md`), or scoring diffs and multimodal diagrams.

  • Reads credentials

The file

Show its rules and text

We split this skill into the rules it gives an agent

13 rules. About 0 are common, 4 are distinctive, and 9 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈4

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 9

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Instead of waking an expensive reasoning subagent on every single pass—or trusting a zero-thinking classifier blindly on deep security and …

    line 27
  2. Very specific:

    Keep assertions atomic: Test one falsifiable invariant per Bi rather than bundling three functions or eight files into one …

    line 82
  3. Very specific:

    Name concrete paths in Verification Target: Prefer src/auth.py, tests/unit/test_auth.py over generic Doc review / Code review so …

    line 83
  4. Very specific:

    Separate canonical registries from summary copy: Don't assert identical title formatting across both machine registries (navigation.yaml) …

    line 84
  5. Very specific:

    Observe: Read the specification (docs/SPEC.md, SPEC.md, or LOOPS.md), git status --short, and current failing tests/checks.

    line 106
  6. Distinctive:

    Act (TDD): Write or update the failing test first (Red), confirm it fails, then implement the minimal clean change (Green) and refactor …

    line 108
  7. Very specific:

    Tier 1: Run deterministic checks (make check or repo lint + test commands).

    line 110
  8. Very specific:

    Tier 1.5: Run eval_rubric.py --json over the spec, diff, and touched files (plus --image / --pdf for diagram/UI specs).

    line 111
  9. Very specific:

    Default calibrated thresholds (overridable via --pass-threshold, --fail-threshold, --min-confidence):

    line 163
  10. Distinctive:

    Always spawn Tier 2 as a fresh, stateless, read-only subagent that never sees the coding agent's prior chain-of-thought or failed attempts:

    line 171
  11. Very specific:

    Antigravity: Call invoke_subagent with TypeName: "research" (Model: "flash" for ESCALATE_L2_FLASH.

    line 172
  12. Very specific:

    When local-code-review is not installed, pass this exact prompt (filling in only the target Bi IDs, spec path, and changed files):

    line 177
  13. Distinctive:

    Stop the loop whenever any of these three conditions triggers:

    line 197

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Tiered Rubric Loop (tiered-rubric-loop)

A fast, loosely coupled way to drop a combined deterministic + semantic verification loop into any repository as the verification engine of an agentic harness.

It automatically reuses any Spec-Driven Development (SDD) (docs/SPEC.md or SPEC.md) and Test-Driven Development (TDD) artefacts already in the repo—or bootstraps a lightweight B1–Bn binary checklist inline if none exist.

Instead of waking an expensive reasoning subagent on every single pass—or trusting a zero-thinking classifier blindly on deep security and concurrency invariants—each assertion flows through three escalating stages:

  1. Tier 1 — Deterministic Checks ($0, seconds): Runs local linters, typecheckers, and unit tests (make check, codespell, ruff, pyright, pytest) first. Any failure stops the pass immediately so you never waste API calls on broken code.
  2. Tier 1.5 — Fast Semantic Scoring (~200 ms, 0 output tokens): Evaluates open binary (yes/no) rubric criteria (B1–Bn) in a single prefill pass using the Gemini Decisions API (gemini-decision-1 via scripts/eval_rubric.py). Straightforward editorial and doc-sync checks pass immediately, while obvious omissions fail fast without waking a subagent.
  3. Tier 2 — Stateless Semantic Critique: Spawns a fresh, read-only reviewer subagent only when Tier 1.5 is uncertain ("grey zone"), or for complex architectural and security invariants ([arch]) that need deeper multi-file reasoning.

Table of Contents

  • How the 3-Stage Funnel Works
  • Plain-Markdown B1–Bn Contract & Tagging
  • Loose Coupling & Companion Skills Matrix
  • Bounded Loop Workflow
  • Running Tier 1.5 (eval_rubric.py)
  • Threshold Routing & Stateless Tier 2 Critic
  • Circuit Breaker Trio & Continuous Ratchet
  • Loop Receipt Format (LOOPS.md)

How the 3-Stage Funnel Works

TierMechanismLatency & CostWhat It Owns
Tier 1 (Deterministic)Local linters, typecheckers, unit tests (make check, codespell, ruff, pyright, pytest)Seconds, $0Syntax, static types, formatting, and deterministic test execution. Any failure short-circuits immediately before Tier 1.5 or Tier 2 runs.
Tier 1.5 (Calibrated Probabilistic)gemini-decision-1 via client.decisions.create (v1beta) in scripts/eval_rubric.py~150–250 ms, $0.10 / 1M input tokens, 0 output / thinking tokensEvaluates up to 1,000 closed-ended noul (yes/no) assertions over a single shared diff/spec/media payload. Terminates clear mechanical passes (PASS), rejects obvious omissions immediately (FAIL_FAST), and flags uncertain items for L2.
Tier 2 (Stateless Semantic Critic)Fresh read-only subagent with zero prior coding chain-of-thought10–40 s, standard model tokensDiagnoses grey-zone escalations (flash for mechanical, inherit / frontier model for [arch]) and performs one final frontier sign-off on [arch] items after Tier 1.5 is green.

Key safety rule: Because gemini-decision-1 runs in a single prefill forward pass with zero thinking tokens and zero file-reading tools, it can produce false positives on subtle logic, race conditions, or auth flaws. Therefore, Tier 1.5 is terminal for mechanical/editorial checks, fail-fast on clear omissions, and provisional for [arch] assertions.


Plain-Markdown B1–Bn Contract & Tagging

The sole shared contract is a Markdown table or bullet list of numbered binary assertions (B1–Bn) in docs/SPEC.md, SPEC.md, or LOOPS.md (with optional --spec <path> override):

| ID | Falsifiable Condition | Verification Target |
| :--- | :--- | :--- |
| `B1 [det]` | Static gates (`codespell`, `ruff`, `pyright`) and offline tests exit `0` | `make check` |
| `B2` | `.env.template` and `README.md` document every newly added variable and flag | Doc review |
| `B3 [arch]` | Auth middleware rejects expired tokens and never retries `4xx` responses | `tests/unit/test_auth.py` |

Bullet assertions (- **B4**: CLI help panel renders in English (UK)), atomic sub-IDs (B4a, B11b), numbered items (1. \B5 [docs]`: ...), and Markdown task checkboxes (- [ ] B6 [concurrency]: ...or- [x] B6: ...) are all supported, alongside extended tags ([mech], [docs], [ui], [visual], [perf], [concurrency]). Unrecognised tags warn on stderr and default safely to [mech]` instead of silently dropping the assertion.

TagMeaningTier 1.5 Handling
[det] (or deterministic Verification Target such as make check, pytest tests/..., codespell -s, ruff check .)Already proven by Tier 1 shell exit codesSkipped by Tier 1.5 by default (--include-det overrides) so the model never guesses whether pytest passed from a text diff.
(untagged, [mech], [docs], [ui], [visual], [multimodal])Mechanical, editorial, doc-sync, schema, or UI convention checkTerminal PASS when P(yes) >= 0.90, confidence >= 0.80, and no evidence was truncated; FAIL_FAST when <= 0.20; otherwise ESCALATE_L2_FLASH.
[arch] (also [security], [logic], [concurrency], [perf])Concurrency, security, state invariant, performance, or test-realism checkProvisional pass (PROVISIONAL_ARCH_PASS) at Tier 1.5; gated for one final frontier L2 review once all items clear Tier 1.5.

Repo-relative file paths and local images named in Verification Target or inline backticks are automatically resolved and attached (blocking .env, secret env files, *.enc, and paths outside --repo-root; pass --no-auto-target-files to disable).

Writing High-Signal B1–Bn Assertions

  • Keep assertions atomic: Test one falsifiable invariant per Bi rather than bundling three functions or eight files into one question—single-invariant assertions give sharp P(yes) separation and trigger FAIL_FAST accurately on regressions.
  • Name concrete paths in Verification Target: Prefer src/auth.py, tests/unit/test_auth.py over generic Doc review / Code review so eval_rubric.py auto-attaches untouched context files even in narrow working-tree diffs.
  • Separate canonical registries from summary copy: Don't assert identical title formatting across both machine registries (navigation.yaml) and summarised overview pages (README.md, spotlight cards) unless exact parity is truly required.

Loose Coupling & Companion Skills Matrix

tiered-rubric-loop requires zero hard dependencies on other skills and degrades cleanly when optional companions or API keys are absent:

Companion Skill / DependencyWhen PresentStandalone Fallback (When Absent)
maintaining-core-documentation / spec-driven-developmentReads B1–Bn directly from docs/SPEC.md or SPEC.md.Reads any local SPEC.md / LOOPS.md, or asks you to agree a compact B1–Bn list inline before coding.
test-driven-developmentDelegates the Act step to full Red-Green-Refactor execution.Enforces the core TDD rule inline: write/update a failing test first, change one semantic behaviour per pass, never weaken assertions.
loopySupplies the outer Observe -> Choose -> Act -> Verify -> Record -> Stop governance and LOOPS.md receipts.Runs the identical bounded loop directly with the receipt format below.
local-code-reviewStateless Tier 2 read-only subagent runs local-code-review on escalated/[arch] items.Uses the built-in Stateless Tier 2 Critic Prompt below.
GEMINI_API_KEY / GOOGLE_API_KEYEnables ~200 ms Tier 1.5 screening via eval_rubric.py.When eval_rubric.py exits 3 (missing/invalid key), skips Tier 1.5 cleanly and executes Tier 1 -> Tier 2 directly while noting how to enable Tier 1.5.

Bounded Loop Workflow

Track every pass around Observe -> Choose -> Act -> Verify -> Record -> Stop:

  1. Observe: Read the specification (docs/SPEC.md, SPEC.md, or LOOPS.md), git status --short, and current failing tests/checks. Confirm B1–Bn exist (draft and confirm them first if missing).
  2. Choose: Pick one highest-value unverified semantic behaviour or failing assertion for this pass (mechanical lint/spelling fixes may be batched).
  3. Act (TDD): Write or update the failing test first (Red), confirm it fails, then implement the minimal clean change (Green) and refactor without weakening tests.
  4. Verify (3-Stage Funnel):
    • Tier 1: Run deterministic checks (make check or repo lint + test commands). If Tier 1 fails, stop verification immediately and return to Choose/Act—do not spend tokens on Tier 1.5 or Tier 2.
    • Tier 1.5: Run eval_rubric.py --json over the spec, diff, and touched files (plus --image / --pdf for diagram/UI specs).
    • Tier 2: Escalate only grey-zone items immediately, or run a single final [arch] critic pass once Tier 1.5 reports zero FAIL_FAST and zero grey-zone items.
  5. Record: Append pass outcome, ratchet_score (mean P(yes)), and remaining failures to LOOPS.md (or session progress notes).
  6. Repeat or Stop: Enforce the Circuit Breaker Trio before starting another pass.

Running Tier 1.5 (eval_rubric.py)

Resolve scripts/eval_rubric.py relative to where tiered-rubric-loop is installed:

  • Global Antigravity path: ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py
  • Project-local path: .agents/skills/tiered-rubric-loop/scripts/eval_rubric.py

In sandboxed agent runtimes that block outbound HTTPS by default, invoke uv run with unsandboxed network permission (BypassSandbox: true in Antigravity) for live API calls, or pass --offline for sandboxed --dry-run checks.

# 1. Working-tree loop: auto-discover SPEC.md / docs/SPEC.md / LOOPS.md
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --json

# 2. Branch or merged PR gating: evaluate committed changes against base ref / range
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --base main --json
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py \
  --git-range origin/main...HEAD \
  --only B3,B4 \
  --json

# 3. Explicit spec, scoped files, and multimodal diagram/UI screenshot check
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py \
  --spec docs/SPEC.md \
  --file src/auth.py \
  --image docs/images/architecture.png \
  --json

# 4. Offline dry-run inside sandbox (verifies rubric, targets, and truncation zero-API)
uv run --offline ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --dry-run --json

For large PRs (> 8 files or > 100 KB), shard evaluations by assertion group via --only B3,B4 so files stay within --max-file-chars (24,000) and --max-evidence-chars (180,000). If any file or combined payload is truncated, eval_rubric.py populates truncated_sources and downgrades mechanical PASS to ESCALATE_L2_FLASH unless --allow-truncated-pass is explicitly set.

Exit Codes

CodeMeaningLoop Action
0Tier 1.5 clear (all evaluated items are PASS or PROVISIONAL_ARCH_PASS)If final_l2_arch_ids is non-empty, run one Tier 2 frontier critic pass on those IDs; otherwise exit loop as Success.
1One or more FAIL_FAST or ESCALATE_L2_* items (or API call error)Address FAIL_FAST items directly; escalate grey-zone items (ESCALATE_L2_*) only when root cause is non-obvious.
2Usage error, missing spec, or no B1–Bn assertions parsedFix spec path or rubric formatting before continuing.
3Missing/invalid GEMINI_API_KEY or GOOGLE_API_KEYDegrade gracefully to Tier 1 -> Tier 2 and inform the user once.

Threshold Routing & Stateless Tier 2 Critic

Default calibrated thresholds (overridable via --pass-threshold, --fail-threshold, --min-confidence):

  • PASS (P(yes) >= 0.90, confidence >= 0.80, mechanical): Terminal pass. No L2 model runs.
  • PROVISIONAL_ARCH_PASS (P(yes) >= 0.90, confidence >= 0.80, [arch]): Held until all other assertions pass, then verified once by a frontier L2 critic.
  • FAIL_FAST (P(yes) <= 0.20, confidence >= 0.80): Returned directly to the coding agent—do not burn an L2 turn repeating an obvious omission.
  • ESCALATE_L2_FLASH / ESCALATE_L2_INHERIT (0.20 < P(yes) < 0.90 or confidence < 0.80): Escalate only those specific Bi IDs to a fresh read-only subagent.

Subagent Routing Across Coding Agents

Always spawn Tier 2 as a fresh, stateless, read-only subagent that never sees the coding agent's prior chain-of-thought or failed attempts:

  • Antigravity: Call invoke_subagent with TypeName: "research" (Model: "flash" for ESCALATE_L2_FLASH; Model: "inherit" for ESCALATE_L2_INHERIT and final PROVISIONAL_ARCH_PASS sign-off, optionally applying local-code-review).
  • Claude Code / Gemini CLI / Other Agents: Spawn a read-only reviewer subagent (Haiku/Flash tier for mechanical escalations; Opus/Sonnet/Frontier tier for [arch] escalations).

Built-In Stateless Tier 2 Critic Prompt

When local-code-review is not installed, pass this exact prompt (filling in only the target Bi IDs, spec path, and changed files):

You are a stateless, read-only Tier 2 semantic critic. You have NOT seen the
coding session and must judge solely from the specification and actual files on disk.

Evaluate ONLY these rubric assertions: [INSERT B_i IDs AND CONDITIONS].
Read: [SPEC.md / docs/SPEC.md] and inspect the changed/untracked files + git diff.

Rules:
1. Cite exact file paths, line numbers, and observed code/test behaviour.
2. Check whether tests verify real behaviour rather than mocking away logic.
3. For each assertion ID, return PASS or FAIL with a concise root-cause note.
4. Do not invent fixes or assume uninspected files pass.

Circuit Breaker Trio & Continuous Ratchet

Stop the loop whenever any of these three conditions triggers:

  1. 100% Pass: Tier 1 passes, Tier 1.5 reports exit_code == 0, and any final_l2_arch_ids clear their final frontier Tier 2 review (Success).
  2. Pass Budget Exhausted: Maximum 4–5 passes per slice reached (Exhausted—escalate remaining blockers to the user).
  3. 2 Consecutive Non-Improving Passes: Neither failing assertion count decreases nor ratchet_score (mean P(yes) across evaluated B1–Bn items) improves across two consecutive passes (No progress—escalate to the user rather than spinning).

Tracking the continuous ratchet_score avoids premature stalls when a pass genuinely moves an assertion from e.g. P(yes) = 0.25 to 0.82 while still short of 0.90, and catches silent regressions immediately.


Loop Receipt Format (LOOPS.md)

Record completed runs (or append per-pass telemetry) concisely:

## Tiered Rubric Loop Receipt

- **Target**: [Slice / Feature / PR]
- **Spec**: [docs/SPEC.md] (`B1`–`Bn`)
- **Passes**: [N / 5] | **Outcome**: `Success` | `Exhausted` | `No progress` | `Blocked`
- **Tier 1**: `make check` -> PASS
- **Tier 1.5 Ratchet (`gemini-decision-1`)**: `0.58 -> 0.94 -> 0.98` (skipped `[det]`: `B1`)
- **Tier 2 (`Model: "inherit"`)**: Verified `[arch]` IDs `B3` -> PASS
- **Remaining / Escalation**: None (or exact failing `Bi` + file:line)

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.