agentleFS
eins78eins78/agent-skills/skills/ai-council-review/SKILL.md

ai-council-review

Skill2 starsChanged 16 days agoMITView on GitHub

Convene a council of ~4 frontier models via the OpenRouter API for independent parallel reviews of a PR, plan doc, or files, then synthesize agreements and dissents in-session. Use for high-stakes or contested changes, before irreversible decisions, or when one extra model's opinion is not enough — costs real money per run. Triggers on /ai-council-review, "council review", "panel review", "multi-model review", "review with multiple models", "ask several models", "get a third opinion", "high-stakes review", "openrouter review". For a quick single-model second opinion, use ai-review instead.

  • Reads credentials

The file

We split this skill into the rules it gives an agent

8 rules. About 0 are common, 3 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Before the FIRST council run in a session, confirm with your human partner that sending this content to external providers is acceptable — …

    line 42
  2. Distinctive:

    Never fabricate or substitute a council opinion: If members fail or quorum is not met, report that and ask your human partner how to …

    line 65
  3. Very specific:

    Follow ${CLAUDE_SKILL_DIR}/references/synthesis.md step by step.

    line 137
  4. Very specific:

    Write report.md into the run dir per ${CLAUDE_SKILL_DIR}/references/report-template.md.

    line 147
  5. Very specific:

    Only on explicit user request, post findings with gh pr comment (or a formal review via gh api).

    line 155
  6. Very specific:

    Run --preset budget (3 cheaper members) first, or --preset crowd (6 members) for broader coverage at still-modest cost.

    line 163
  7. Distinctive:

    Stuck rule: if the same blocking findings (by fingerprint) survive two consecutive runs, do not propose a third run — the council has said …

    line 184
  8. Very specific:

    Run artifacts live under ~/.local/state/ai-council-review/<repo>/<run-id>/ (override: --out).

    line 217

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

AI Council Review

Several frontier models review the same material in parallel — blind to each other — via one API (OpenRouter). A dispatch script handles fan-out, cost gating, and clustering; you synthesize, because you can do the one thing the council cannot: verify findings against the actual repository.

This is the heavyweight sibling of ai-review (one model, streamed, ~$0.06 by default). A council run costs real money — from well under a cent (budget) to over $1 (flagship, which trips the confirmation gate on anything beyond a small diff). See the per-preset tables in README.md for exact pricing. Use it for high-stakes decisions, not routine diffs.

Data leaves the machine — confirm before first dispatch

Everything submitted (diffs, plan docs, file contents) is sent to the OpenRouter API and forwarded to multiple third-party model providers (different companies, different data-retention policies). For private or proprietary code this means unreleased source leaves the machine.

  • Before the FIRST council run in a session, confirm with your human partner that sending this content to external providers is acceptable — especially in private repos or anything under NDA. Consent given for this repo earlier in this session carries over; do not re-ask per run.
  • Reduce exposure: review the diff, not the whole tree; never include .env files, credentials, or customer data in the input.
  • Account-wide tightening: OpenRouter privacy settings (disable logging, restrict to ZDR-eligible providers).

Prerequisites

  • Node.js >= 20 (node --version)
  • AI_COUNCIL_OPENROUTER_API_KEY exported — the council's own variable, so it can hold a dedicated, budget-capped key rather than whatever generic key a shell happens to export (get one: https://openrouter.ai/keys). OPENROUTER_API_KEY still works as a fallback, and a run that falls back says so on stderr. The script reads the key from the environment only and never prints it. If neither is set, dispatch exits with code 4 — ask your human partner; do NOT hunt for keys in keychains, dotfiles, or session archives.

CRITICAL rules

  • Never fabricate or substitute a council opinion. If members fail or quorum is not met, report that and ask your human partner how to proceed — do not write "what the model would probably say", and do not pass your own review off as a member's.
  • Never pass --yes on your own. --yes asserts that your human partner saw the printed cost estimate and approved it in this session. Budget gate refusals (exit 3) are a stop, not an obstacle to route around.
  • Never skip the synthesis protocol. Raw per-model output is not the deliverable; the verified, dissent-preserving report is.

Workflow

Script: ${CLAUDE_SKILL_DIR}/scripts/council.mjs (always via this absolute path — never a relative ./scripts/...).

1. Classify

InputFlagsRubric
GitHub PR--pr <N>--rubric code
Unstaged / staged / branch diff(default) / --staged / --branch [base]--rubric code
Plan or design doc<file> or --plan <file>--rubric plan
Any other document/files<file...> or --input-file <f>--rubric doc

Preset: the default preset is already the code-review roster (--rubric code needs no flag). For plan/doc rubrics, use --preset prose — the prior default roster, better suited to prose than the codex-tuned default. --models a,b,c overrides for custom councils.

--personas runs the council in coverage mode: each member gets a distinct focus lens (correctness, security, design, testing, operations) on top of the shared rubric. This trades consensus signal for breadth — agreement counting is invalid across different assignments, and the synthesis protocol switches to coverage mode. Prefer it for broad material (a large plan, a many-concern diff); prefer the identical-rubric default when the question is "is this correct?" and agreement should mean something.

2. Estimate (spend-free)

node ${CLAUDE_SKILL_DIR}/scripts/council.mjs review <flags> --dry-run

Show the user the per-model table and total. If the estimate exceeds the confirmation threshold (default $1), you MUST get an explicit go-ahead in this session before step 3.

3. Dispatch (1–10 minutes — run in background)

node ${CLAUDE_SKILL_DIR}/scripts/council.mjs review <same flags> [--yes]

The script fans out in parallel, retries once on 429/5xx or an empty completion (the latter only if the budget cap leaves room for it), aborts hung members at the timeout, persists everything, and prints RUN_DIR=<path> as its last line. A member that failed, needed a retry or hit the output limit leaves its full responses in failures/ (named in its manifest entry as diagnostics). That file names models: it is for diagnosing a flaky seat, not for synthesis.

ExitMeaningYour move
0Quorum met (check degraded in manifest)Synthesize (step 4)
1Usage/input error (bad slug, empty diff, oversized payload)Fix per the message; slug errors include suggestions — update roster, retry
2Quorum failedReport which members failed and why; ask your human partner (fewer members, another preset, retry later); do NOT synthesize or self-substitute. ai-review uses the same API and key, so it is not a fallback for an OpenRouter outage
3Budget gate blocked (nothing sent)Relay the estimate to your human partner; only proceed how they decide
4API key missing/rejectedAsk your human partner to set AI_COUNCIL_OPENROUTER_API_KEY (or OPENROUTER_API_KEY)

4. Synthesize

Follow ${CLAUDE_SKILL_DIR}/references/synthesis.md step by step. Core moves: members are anonymized (member-A…) — do not open roster-key.json until the report step; agreement is counted over delivered members; parse-failed members' raw text is read and included; top findings are verified against the actual repo before being reported; dissent is preserved as contested items and minority reports; ranking is verified > severity > agreement > confidence.

5. Report

Write report.md into the run dir per ${CLAUDE_SKILL_DIR}/references/report-template.md; give the user the condensed version in chat with actual vs estimated cost. Then record the verification outcomes (outcomes record, synthesis protocol step 8) — the archive is what makes future rosters evidence-based.

6. Optional: post to the PR

Only on explicit user request, post findings with gh pr comment (or a formal review via gh api). The script never posts anywhere.

Cost pattern: two-stage triage (optional)

A suggestion for material that may not need a frontier council — nothing enforces it:

  1. Run --preset budget (3 cheaper members) first, or --preset crowd (6 members) for broader coverage at still-modest cost.
  2. Escalate to default/max only if the budget/crowd run produced majors/blockers or contested clusters — objective properties of clusters.json, not a feeling.
  3. A clean budget/crowd run (approvals + nits) is a result: report it and stop.

Each stage passes the same estimate and consent gates; escalation is a second run with its own estimate. Prior art: cascade cost controls (cheap-first with objective escalation triggers).

Re-reviewing after fixes

Each cluster in clusters.json has a fingerprint stable across runs (built from files + title tokens, not member labels or line numbers). When a council run follows an earlier run of the same material:

  1. Compare fingerprints against the previous run's clusters.json; classify clusters as new / persisting / resolved. Fingerprints are a hint — confirm matches semantically before reporting them.
  2. Stuck rule: if the same blocking findings (by fingerprint) survive two consecutive runs, do not propose a third run — the council has said what it has to say. Recommend human judgment on the persisting blockers instead. (Cost/termination gate from prior art; see README provenance.)

Configuration

Precedence: flags > AI_COUNCIL_* env > repo .ai-council.json > ~/.config/ai-council-review/config.json > bundled references/presets.json (presets: default, code, prose, budget, crowd, max, flagship, smoke — see README.md for per-preset rosters and pricing; scalars: budgetUsd, confirmThresholdUsd, timeoutMs, quorum, preset, reasoningEffort, providerSort, and the object expectedOutputTokens, set in config files only, whose models map merges across the layers).

The cost estimate prices each member from OpenRouter's live catalog on every run and assumes the average output length measured for that model (reasoning models write about 9,000 tokens); the hard cap uses the worst case.

reasoningEffort (--reasoning-effort / AI_COUNCIL_REASONING_EFFORT, bundled default "medium") sets reasoning.effort on every request — but only to models that advertise "reasoning" in the live catalog's supported_parameters; sending it to a model that doesn't support it is a 400, not a graceful ignore, so dispatch gates it per-member. Set to "none" to disable reasoning outright, or "low" for the cheapest/fastest council presets. providerSort (--provider-sort / AI_COUNCIL_PROVIDER_SORT, bundled default "throughput") sets provider.sort — a routing hint, not a model capability, so it's sent to every member unconditionally.

Useful env: OPENROUTER_BASE_URL (testing), COUNCIL_TIMEOUT_MS, REVIEW_BASE_BRANCH (for --branch).

Run artifacts live under ~/.local/state/ai-council-review/<repo>/<run-id>/ (override: --out). They contain the reviewed source — treat as sensitive; never commit them.

Inspecting the roster

node ${CLAUDE_SKILL_DIR}/scripts/council.mjs models                 # current council + live pricing
node ${CLAUDE_SKILL_DIR}/scripts/council.mjs models --verify a,b    # diagnose slugs

Model slugs churn. When dispatch exits 1 with "Unknown model slug", the fix is a roster update (config override or a patch to references/presets.json) — pick the suggested replacement closest in capability.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.