Forge playbook — AI chatbot / RAG

Provenance

Forge-native playbook. references/evals.md adds eval method adapted in our own words (no file copied verbatim) from ai-evals-course/evals-skills (eval-audit, error-discovery, write-judge-prompt, validate-evaluator, generate-synthetic-data, evaluate-rag; Apache-2.0, main at 80d5f7b, seen 2026-11-02) and anthropics/skills claude-api build-eval (Apache-2.0); see LICENSE.txt. Forge changes: no review web app, no code samples, no observability-MCP step; paid eval runs owner-approved.

When to use

• Use for: AI chatbots and RAG: grounded answers with citations, honest fallback, human handoff, and measuring them (retrieval metrics, LLM judges).
• Not for: a tool-using agent (forge-agent), plain web search scraping (forge-scraping).

Do not duplicate ECC skills — defer to: agentdb-vector-search / agentdb-memory-patterns (vector store), docs-lookup (SDK docs), learn-codebase (grounding on a repo), claude-api (if Anthropic-based).

Hard rules

• Grounding: answers are source-aware and cite their sources.
• Clear fallback response when retrieval is empty/low-confidence — no guessing.
• No hallucinated business facts (names, prices, contacts, claims).
• Safe data handling; conversation logging on; a defined human-handoff path.
• Lead capture is consent-aware; secrets/API keys in env only.
• Measured, not felt: retrieval and generation are evaluated separately on a real Q/A set; an LLM judge counts only after it is validated against human labels (references/evals.md).

Retrieval design (defaults, change with evidence)

• Chunk on natural boundaries (sections, paragraphs, steps) and prefix each chunk with document title + heading; chunk size and overlap are tuned on the eval set, not guessed.
• Hybrid search (keyword/BM25 + vector, merged with reciprocal rank fusion) for names, codes and prices that embeddings miss; a reranker when precision matters.
• Metadata filters (tenant, language, product, date) applied before similarity search; tenant isolation is a filter, never a prompt instruction.
• A relevance threshold decides "nothing good found" → fallback, instead of always passing the top k.
• Ingestion is idempotent (stable chunk IDs, content hash) and re-embeds when the source or the embedding model changes.

Team (conditional)

Lead: architect. Specialists: mle-reviewer (pipeline/serving), data-scientist (eval set, judge validation, honest metrics), python-reviewer / typescript-reviewer, security-reviewer, database-reviewer (vector store).

Skills / commands / MCP

agentdb-vector-search, docs-lookup, learn-codebase; claude-api reference for model IDs/params if using Claude. security-reviewer on the serving layer. forge-graded-verify to grade answers at handoff.

Fan-out & flow

L3 typical. Parallel: ingestion/embeddings pipeline ∥ chat/retrieval layer ∥ frontend ∥ eval set (questions + gold chunks). Serial: system prompt → retrieval → fallback → error analysis on traces → evaluators → safety eval.

Domain gates

• System prompt reviewed; retrieval + embeddings correct; ingestion idempotent; citations present; fallback verified; no invented facts; logging + handoff defined.
• Retrieval is measured on an eval set on known Q/A with gold chunks: Recall@k (and MRR for single-fact lookups) reported; faithfulness and answer relevance checked separately from retrieval.
• Code checks first (citation present, cited ID in the retrieved set, refusal on out-of-scope); any LLM judge is binary, one failure mode per judge, with TPR/TNR measured on held-out labels.

Ship-readiness (unique)

Fallback exercised with real output on empty or low-confidence retrieval; answers cite sources, and every answer in the eval set cites a retrievable source; out-of-scope questions are refused; no invented business info; PII + keys in env; logging on; handoff tested; regression eval run with retrieval metrics and judge results reported (with their TPR/TNR). The ship-readiness AI/RAG checklist is advisory; optionally run codex-reviewer (Codex) on important code — not a blocker.

Safety, approval and honesty

• Approval / safety: lead capture is consent-aware; secrets/API keys in env only.
• Approval / safety: production transcripts used for evals only after PII/retention is checked; a paid full eval run waits for the owner's yes after an oracle/null check and a measured cost estimate.
• Honest failure: no hallucinated business facts; empty or low-confidence retrieval gets the fallback, not a guess.
• Honest failure: an unvalidated judge's pass rate is not a quality claim; report raw and corrected rates with an interval, or say the judge is unvalidated.
• Untrusted content: see .claude/skills/forge-integration/references/untrusted-content.md.
• Shared gates for every playbook (honesty, owner approval, secrets, reviews): .claude/docs/forge-reference/playbook-template.md.

References

• .claude/skills/forge-integration/references/untrusted-content.md — untrusted-content injection defense (shared, word-for-word the same for integration, rag and scraping).
• references/evals.md — error analysis, retrieval vs generation metrics, judge prompts, judge validation (TPR/TNR, bias correction), synthetic data, eval audit.