ai-council-review
Skill2 starsChanged 16 days agoMITView on GitHub
Convene a council of ~4 frontier models via the OpenRouter API for independent parallel reviews of a PR, plan doc, or files, then synthesize agreements and dissents in-session. Use for high-stakes or contested changes, before irreversible decisions, or when one extra model's opinion is not enough — costs real money per run. Triggers on /ai-council-review, "council review", "panel review", "multi-model review", "review with multiple models", "ask several models", "get a third opinion", "high-stakes review", "openrouter review". For a quick single-model second opinion, use ai-review instead.
- Reads credentials
In this repository17 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- skills/agent-workspace/templates/CLAUDE.mdCLAUDE.md
- agent-workspaceSkill
- ai-reviewSkill
- apple-mailSkill
- apple-notesSkill
- ballotSkill
- chrome-browserSkill
- dossierSkill
- lab-notesSkill
- pandocSkill
- paprika-recipesSkill
- pdf-zineSkill
- private-podcast-feedSkill
- send-to-kindleSkill
- text-to-speechSkill
- tmux-controlSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 3 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 42
Before the FIRST council run in a session, confirm with your human partner that sending this content to external providers is acceptable — …
- Distinctive: line 65
Never fabricate or substitute a council opinion: If members fail or quorum is not met, report that and ask your human partner how to …
- Very specific: line 137
Follow ${CLAUDE_SKILL_DIR}/references/synthesis.md step by step.
- Very specific: line 147
Write report.md into the run dir per ${CLAUDE_SKILL_DIR}/references/report-template.md.
- Very specific: line 155
Only on explicit user request, post findings with gh pr comment (or a formal review via gh api).
- Very specific: line 163
Run --preset budget (3 cheaper members) first, or --preset crowd (6 members) for broader coverage at still-modest cost.
- Distinctive: line 184
Stuck rule: if the same blocking findings (by fingerprint) survive two consecutive runs, do not propose a third run — the council has said …
- Very specific: line 217
Run artifacts live under ~/.local/state/ai-council-review/<repo>/<run-id>/ (override: --out).
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
AI Council Review
Several frontier models review the same material in parallel — blind to each other — via one API (OpenRouter). A dispatch script handles fan-out, cost gating, and clustering; you synthesize, because you can do the one thing the council cannot: verify findings against the actual repository.
This is the heavyweight sibling of ai-review (one model, streamed, ~$0.06 by default). A
council run costs real money — from well under a cent (budget) to over $1
(flagship, which trips the confirmation gate on anything beyond a small
diff). See the per-preset tables in README.md for exact pricing. Use it for
high-stakes decisions, not routine diffs.
Data leaves the machine — confirm before first dispatch
Everything submitted (diffs, plan docs, file contents) is sent to the OpenRouter API and forwarded to multiple third-party model providers (different companies, different data-retention policies). For private or proprietary code this means unreleased source leaves the machine.
- Before the FIRST council run in a session, confirm with your human partner that sending this content to external providers is acceptable — especially in private repos or anything under NDA. Consent given for this repo earlier in this session carries over; do not re-ask per run.
- Reduce exposure: review the diff, not the whole tree; never include
.envfiles, credentials, or customer data in the input. - Account-wide tightening: OpenRouter privacy settings (disable logging, restrict to ZDR-eligible providers).
Prerequisites
- Node.js >= 20 (
node --version) AI_COUNCIL_OPENROUTER_API_KEYexported — the council's own variable, so it can hold a dedicated, budget-capped key rather than whatever generic key a shell happens to export (get one: https://openrouter.ai/keys).OPENROUTER_API_KEYstill works as a fallback, and a run that falls back says so on stderr. The script reads the key from the environment only and never prints it. If neither is set, dispatch exits with code 4 — ask your human partner; do NOT hunt for keys in keychains, dotfiles, or session archives.
CRITICAL rules
- Never fabricate or substitute a council opinion. If members fail or quorum is not met, report that and ask your human partner how to proceed — do not write "what the model would probably say", and do not pass your own review off as a member's.
- Never pass
--yeson your own.--yesasserts that your human partner saw the printed cost estimate and approved it in this session. Budget gate refusals (exit 3) are a stop, not an obstacle to route around. - Never skip the synthesis protocol. Raw per-model output is not the deliverable; the verified, dissent-preserving report is.
Workflow
Script: ${CLAUDE_SKILL_DIR}/scripts/council.mjs (always via this absolute
path — never a relative ./scripts/...).
1. Classify
| Input | Flags | Rubric |
| GitHub PR | --pr <N> | --rubric code |
| Unstaged / staged / branch diff | (default) / --staged / --branch [base] | --rubric code |
| Plan or design doc | <file> or --plan <file> | --rubric plan |
| Any other document/files | <file...> or --input-file <f> | --rubric doc |
Preset: the default preset is already the code-review roster (--rubric
code needs no flag). For plan/doc rubrics, use --preset prose — the
prior default roster, better suited to prose than the codex-tuned default.
--models a,b,c overrides for custom councils.
--personas runs the council in coverage mode: each member gets a
distinct focus lens (correctness, security, design, testing, operations)
on top of the shared rubric. This trades consensus signal for breadth —
agreement counting is invalid across different assignments, and the
synthesis protocol switches to coverage mode. Prefer it for broad material
(a large plan, a many-concern diff); prefer the identical-rubric default
when the question is "is this correct?" and agreement should mean
something.
2. Estimate (spend-free)
node ${CLAUDE_SKILL_DIR}/scripts/council.mjs review <flags> --dry-runShow the user the per-model table and total. If the estimate exceeds the confirmation threshold (default $1), you MUST get an explicit go-ahead in this session before step 3.
3. Dispatch (1–10 minutes — run in background)
node ${CLAUDE_SKILL_DIR}/scripts/council.mjs review <same flags> [--yes]The script fans out in parallel, retries once on 429/5xx or an empty
completion (the latter only if the budget cap leaves room for it), aborts
hung members at the timeout, persists everything, and prints
RUN_DIR=<path> as its last line. A member that failed, needed a retry or
hit the output limit leaves its full responses in failures/ (named in its
manifest entry as diagnostics). That file names models: it is for diagnosing a
flaky seat, not for synthesis.
| Exit | Meaning | Your move |
| 0 | Quorum met (check degraded in manifest) | Synthesize (step 4) |
| 1 | Usage/input error (bad slug, empty diff, oversized payload) | Fix per the message; slug errors include suggestions — update roster, retry |
| 2 | Quorum failed | Report which members failed and why; ask your human partner (fewer members, another preset, retry later); do NOT synthesize or self-substitute. ai-review uses the same API and key, so it is not a fallback for an OpenRouter outage |
| 3 | Budget gate blocked (nothing sent) | Relay the estimate to your human partner; only proceed how they decide |
| 4 | API key missing/rejected | Ask your human partner to set AI_COUNCIL_OPENROUTER_API_KEY (or OPENROUTER_API_KEY) |
4. Synthesize
Follow ${CLAUDE_SKILL_DIR}/references/synthesis.md step by step. Core
moves: members are anonymized (member-A…) — do not open
roster-key.json until the report step; agreement is counted over
delivered members; parse-failed members' raw text is read and included; top
findings are verified against the actual repo before being reported;
dissent is preserved as contested items and minority reports; ranking is
verified > severity > agreement > confidence.
5. Report
Write report.md into the run dir per
${CLAUDE_SKILL_DIR}/references/report-template.md; give the user the
condensed version in chat with actual vs estimated cost. Then record the
verification outcomes (outcomes record, synthesis protocol step 8) — the
archive is what makes future rosters evidence-based.
6. Optional: post to the PR
Only on explicit user request, post findings with gh pr comment (or a
formal review via gh api). The script never posts anywhere.
Cost pattern: two-stage triage (optional)
A suggestion for material that may not need a frontier council — nothing enforces it:
- Run
--preset budget(3 cheaper members) first, or--preset crowd(6 members) for broader coverage at still-modest cost. - Escalate to
default/maxonly if the budget/crowd run produced majors/blockers or contested clusters — objective properties ofclusters.json, not a feeling. - A clean budget/crowd run (approvals + nits) is a result: report it and stop.
Each stage passes the same estimate and consent gates; escalation is a second run with its own estimate. Prior art: cascade cost controls (cheap-first with objective escalation triggers).
Re-reviewing after fixes
Each cluster in clusters.json has a fingerprint stable across runs
(built from files + title tokens, not member labels or line numbers). When
a council run follows an earlier run of the same material:
- Compare fingerprints against the previous run's
clusters.json; classify clusters as new / persisting / resolved. Fingerprints are a hint — confirm matches semantically before reporting them. - Stuck rule: if the same blocking findings (by fingerprint) survive two consecutive runs, do not propose a third run — the council has said what it has to say. Recommend human judgment on the persisting blockers instead. (Cost/termination gate from prior art; see README provenance.)
Configuration
Precedence: flags > AI_COUNCIL_* env > repo .ai-council.json >
~/.config/ai-council-review/config.json > bundled
references/presets.json (presets: default, code, prose, budget,
crowd, max, flagship, smoke — see README.md for per-preset rosters
and pricing; scalars: budgetUsd, confirmThresholdUsd, timeoutMs,
quorum, preset, reasoningEffort, providerSort, and the object
expectedOutputTokens, set in config files only, whose models map merges
across the layers).
The cost estimate prices each member from OpenRouter's live catalog on every run and assumes the average output length measured for that model (reasoning models write about 9,000 tokens); the hard cap uses the worst case.
reasoningEffort (--reasoning-effort / AI_COUNCIL_REASONING_EFFORT,
bundled default "medium") sets reasoning.effort on every request — but
only to models that advertise "reasoning" in the live catalog's
supported_parameters; sending it to a model that doesn't support it is a
400, not a graceful ignore, so dispatch gates it per-member. Set to "none"
to disable reasoning outright, or "low" for the cheapest/fastest council
presets. providerSort (--provider-sort / AI_COUNCIL_PROVIDER_SORT,
bundled default "throughput") sets provider.sort — a routing hint, not a
model capability, so it's sent to every member unconditionally.
Useful env: OPENROUTER_BASE_URL (testing), COUNCIL_TIMEOUT_MS,
REVIEW_BASE_BRANCH (for --branch).
Run artifacts live under ~/.local/state/ai-council-review/<repo>/<run-id>/
(override: --out). They contain the reviewed source — treat as sensitive;
never commit them.
Inspecting the roster
node ${CLAUDE_SKILL_DIR}/scripts/council.mjs models # current council + live pricing
node ${CLAUDE_SKILL_DIR}/scripts/council.mjs models --verify a,b # diagnose slugsModel slugs churn. When dispatch exits 1 with "Unknown model slug", the fix
is a roster update (config override or a patch to references/presets.json)
— pick the suggested replacement closest in capability.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

