agentleFS
peterbamuhigirepeterbamuhigire/chwezi-dev-engine/skills/sdlc-meta/santa-method/SKILL.md

santa-method

Skill31 starsChanged 18 days agoMITView on GitHub

Use when output must clear two independent adversarial reviewers before it ships — production code, published content, or any deliverable where a single self-reviewing agent shares the blind spots that produced the output. Not for internal drafts or deterministic checks (use build/test/lint pipelines for those).

12 other projects use a version of this, 12 of them edited.

Adopted by

Most recently taken up by

Used with

Projects with it also use agent-payment-x402, openclaw-persona-forge and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • TironFoxSkill

    agent-payment-x402

    Add x402 payment execution to AI agents — per-task budgets, spending controls, and non-custodial wallets via MCP tools. Use when agents need to pay for APIs, services, or other agents.

    Also in 6 projects that use this

  • TironFoxSkill

    openclaw-persona-forge

    为 OpenClaw AI Agent 锻造完整的龙虾灵魂方案。根据用户偏好或随机抽卡, 输出身份定位、灵魂描述(SOUL.md)、角色化底线规则、名字和头像生图提示词。 如当前环境提供已审核的生图 skill,可自动生成统一风格头像图片。 当用户需要创建、设计或定制 OpenClaw 龙虾灵魂时使用。 不适用于:微调已有 SOUL.md、非 OpenClaw 平台的角色设计、纯工具型无性格 Agent。 触发词:龙虾灵魂、虾魂、OpenClaw 灵魂、养虾灵魂、龙虾角色、龙虾定位、 龙虾剧本杀角色、龙虾游戏角色、龙虾 NPC、龙虾性格、龙虾背景故事、 lobster soul、lobster character、抽卡、随机龙虾、龙虾 SOUL、gacha。

    Also in 5 projects that use this

  • renepfSkill

    motion-advanced

    Advanced motion patterns for React / Next.js — drag & drop, gestures, text animations, SVG path drawing, custom hooks, imperative sequences (useAnimate), loaders, and the full API decision tree. Requires motion-foundations.

    Also in 4 projects that use this

  • RojiosSkill

    dart-flutter-patterns

    Production-ready Dart and Flutter patterns covering null safety, immutable state, async composition, widget architecture, popular state management frameworks (BLoC, Riverpod, Provider), GoRouter navigation, Dio networking, Freezed code generation, and clean architecture.

    Also in 5 projects that use this

  • RojiosSkill

    defi-amm-security

    Security checklist for Solidity AMM contracts, liquidity pools, and swap flows. Covers reentrancy, CEI ordering, donation or inflation attacks, oracle manipulation, slippage, admin controls, and integer math.

    Also in 5 projects that use this

  • renepfSkill

    prediction-market-oracle-research

    Research prediction markets as data sources or oracle signals for products, agents, dashboards, and corporate decision intelligence. Use for source-grounded analysis of market-implied probabilities, caveats, and integration patterns without investment advice.

    Also in 3 projects that use this

  • +1 more from TironFox
  • +1 more from renepf

The file

We split this skill into the rules it gives an agent

6 rules. 1 is common, 4 are distinctive, and 1 is very specific to this project.

  • 1

    Common

    Most of the 12 edited versions we read keep these.

  • 4

    Distinctive

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Do not use for internal drafts, exploratory research, or tasks with deterministic verification — run this engine's build/type/lint/test …

    line 41
  2. Distinctive:

    Run the primary task normally.

    line 65
  3. Distinctive:

    Every criterion must have an objective pass/fail condition — vague rubrics produce vague reviews.

    line 111
  4. Distinctive:

    After 3 rounds without a double-PASS, stop and escalate to a human instead of looping further.

    line 142
  5. Common:

    Run Santa Method on a random sample (10-15% of the batch, minimum 5 items).

    line 148
  6. Distinctive:

    If a systematic pattern emerges, apply a targeted fix to the whole batch, not just the sample.

    line 150

“Common” is measured: kept by at least 6 in 10 of the 12 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Santa Method

Multi-agent adversarial verification with a convergence loop. Two independent reviewers, no shared context, must both PASS before output ships.

The Problem This Solves

A single agent reviewing its own output shares the same biases, knowledge gaps, and systematic errors that produced the output. This engine already has sdlc-meta/skill-engine-audit (audits a whole skill engine) and digital-research-engine/skills/peer-review-loop (reviewer independence for research claims, with a red-team memo and dissent disposition). Santa Method is narrower and more mechanical than either: it is the specific dual-independent-PASS gate for a single deliverable — code, a generated document, a batch of generated content — not a whole-engine audit or a research claim review. Use it as the gate; use skill-engine-audit or peer-review-loop for their own broader jobs.

<!-- dual-compat-start -->

Use When

  • Output will be published, deployed, or consumed by end users
  • Compliance, regulatory, or brand constraints must be enforced
  • Code ships to production without human review
  • Content accuracy matters (technical docs, educational material, customer-facing copy)
  • Batch generation at scale where spot-checking misses systemic patterns
  • Hallucination risk is elevated (claims, statistics, API references, legal language)

Do Not Use When

Do not use for internal drafts, exploratory research, or tasks with deterministic verification — run this engine's build/type/lint/test pipeline for those instead (see rules/common/verification.md).

Required Inputs

  • The task specification the output was generated against.
  • The output under review, in full (or the sample, under Batch Sampling).
  • A rubric whose every criterion has an objective pass/fail condition (see Rubric Design).
  • Authority to launch two fresh, non-fork reviewer agents per round; without it, see Degraded Mode under Quality Standards.

Workflow

GENERATE  ->  DUAL INDEPENDENT REVIEW (B, C; no shared context, same rubric)
          ->  VERDICT GATE: B=PASS AND C=PASS -> SHIP
                             otherwise         -> FIX CYCLE (max 3 rounds, fresh reviewers each
                                                   round) -> re-review -> escalate to human if
                                                   still NAUGHTY after 3 rounds

Phase 1: Generate

Run the primary task normally. Santa Method is a post-generation verification layer, not a generation strategy.

Phase 2: Dual Independent Review

Spawn two review agents (use the Agent tool with subagent_type other than fork — a fork inherits your context, which defeats the independence requirement here). Critical invariants:

  1. Context isolation — neither reviewer sees the other's assessment, and neither is a fork of the generating session.
  2. Identical rubric — both receive the same evaluation criteria.
  3. Same inputs — both receive the original spec AND the generated output.
  4. Structured output — each returns a typed verdict, not free prose.

Reviewer prompt shape:

You are an independent quality reviewer. You have NOT seen any other review of this output.

## Task Specification
{task_spec}

## Output Under Review
{output}

## Evaluation Rubric
{rubric}

## Instructions
Evaluate the output against EACH rubric criterion:
- PASS: criterion fully met, no issues
- FAIL: specific issue found (cite the exact problem, file/line where applicable)

Return:
{
  "verdict": "PASS" | "FAIL",
  "checks": [{"criterion": "...", "result": "PASS|FAIL", "detail": "..."}],
  "critical_issues": ["..."],
  "suggestions": ["..."]
}

Be rigorous. Your job is to find problems, not to approve.

Rubric Design

Every criterion must have an objective pass/fail condition — vague rubrics produce vague reviews.

CriterionPass ConditionFailure Signal
Factual accuracyAll claims verifiable against source materialInvented statistics, wrong version numbers, nonexistent APIs
Hallucination-freeNo fabricated entities, quotes, URLs, referencesLinks or citations that don't resolve
CompletenessEvery requirement in the spec is addressedMissing sections, skipped edge cases
CompliancePasses project-specific constraintsBanned terms, tone violations, regulatory gaps
Internal consistencyNo contradictions within the outputSection A says X, section B says not-X
Technical correctnessCode compiles/runs, logic is soundSyntax errors, logic bugs

Domain extensions for this engine's typical deliverables:

  • Code (PHP/TypeScript): type safety, error handling coverage, no hardcoded secrets, input validation, test coverage for new paths — this overlaps with agents/code-reviewer.md, agents/php-reviewer.md, agents/typescript-reviewer.md; use one of those agents as a Santa reviewer directly rather than re-deriving the rubric.
  • Content/documentation: banned-slop terms (see rules/common/coding-style.md and the design engine's anti-slop doctrine where the output is presentation-layer), claims traceable to a real source per digital-research-engine.

Phase 3: Verdict Gate

Both reviewers must PASS. No partial credit. If only one reviewer catches an issue, that issue is real — the other reviewer's blind spot is exactly the failure mode this skill exists to eliminate.

Phase 4: Fix Until Nice (Convergence Loop)

  • Max 3 iterations. Fix ONLY the flagged critical issues — no unrequested refactors.
  • Each round spawns fresh reviewer agents with no memory of the previous round; prior context creates anchoring bias.
  • After 3 rounds without a double-PASS, stop and escalate to a human instead of looping further.

Pattern: Batch Sampling

For large batches (100+ items), full dual review on every item is cost-prohibitive:

  1. Run Santa Method on a random sample (10-15% of the batch, minimum 5 items).
  2. Classify failures by type (hallucination, compliance, completeness, etc.).
  3. If a systematic pattern emerges, apply a targeted fix to the whole batch, not just the sample.
  4. Re-sample and re-verify the fixed batch. Repeat until a clean sample passes.

Quality Standards

  • Both reviewers PASS in the same round; there is no partial credit and no majority vote.
  • Reviewers never see each other's assessment, and each round uses fresh reviewers.
  • Fixes address only the flagged critical issues; three rounds without a double PASS escalate to a human.
  • Degraded mode: when independent agents cannot be launched, the gate cannot be met. Record the verdict as NOT_ASSESSED for the dual-PASS gate and hand the deliverable to a human reviewer, rather than reporting a self-review as a Santa PASS.

Outputs

  • One structured verdict per reviewer per round.
  • A reconciliation record ending in SHIP or escalation to a human.

Anti-Patterns

Failure modeSymptomMitigation
Infinite loopReviewers keep finding new issues after fixesMax iteration cap (3), then escalate
Rubber stampingBoth reviewers pass everythingAdversarial prompt: "Your job is to find problems, not approve"
Subjective driftReviewers flag style, not errorsTight rubric, objective pass/fail only
Fix regressionFixing A introduces BFresh reviewers each round catch it
Agreement biasBoth miss the same thingMitigated by independence, not eliminated — add a third reviewer for critical output
Cost explosionToo many iterations on large outputsBatch sampling pattern; budget caps per cycle

Evidence Produced

CategoryArtifactFormatExample
CorrectnessTwo independent reviewer verdicts per round, each against the same rubricJSON in the reviewer-prompt shape (verdict, checks, critical_issues, suggestions)reviews/santa/round-1-reviewer-b.json, reviews/santa/round-1-reviewer-c.json
Release evidenceReconciliation record: rounds run, both verdicts per round, issues flagged by one or both reviewers, fixes applied, final outcome (SHIP or escalated)Markdownreviews/santa/reconciliation.md — "Round 1: B=FAIL (invented API), C=PASS; fixed; round 2: B=PASS, C=PASS; SHIP"
CorrectnessBatch sample record (Batch Sampling only): sample size and selection, failure taxonomy, batch-wide fix, re-sample resultMarkdown table"120 items, 15 sampled; 3 hallucinated citations; citation check applied to all 120; clean re-sample of 15"

References

Companion rules, agents and skills:

Skill/agentRelationship
rules/common/verification.mdRun deterministic checks (build, type, lint, test) first; Santa Method handles the semantic checks those pipelines cannot.
agents/code-reviewer.md, php-reviewer.md, typescript-reviewer.mdUse as the reviewer agents themselves for code deliverables instead of writing a fresh rubric each time.
sdlc-meta/skill-engine-auditUse for whole-engine audits; Santa Method is for a single deliverable.
digital-research-engine/skills/peer-review-loopUse for research-claim review; Santa Method is the mechanical dual-PASS gate, narrower in scope.

Metrics

  • First-pass rate: % of outputs that pass on round 1 (target >70%)
  • Mean iterations to convergence: average rounds to a double PASS (target <1.5)
  • Issue taxonomy: distribution of failure types
  • Reviewer agreement: % of issues flagged by both reviewers vs. only one — low agreement means the rubric needs tightening
  • Escape rate: issues found post-ship that this gate should have caught (target 0)
<!-- dual-compat-end -->

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.