agentleFS
ats4321ats4321/claude-engineering-skills/skills/llm-system-design/SKILL.md

LLM System Design

Skill0 starsChanged 4 months agoMITView on GitHub

Decide WHETHER and WHERE an LLM belongs in a system, and shape the pipeline around it. Auto-load when designing, architecting, or proposing any LLM-backed feature or system — "should we use an LLM for this", choosing between local (Ollama) and hosted (Anthropic/OpenAI) models, selecting model size, choosing pipeline shape (single call, chain, fan-out, agent loop), budgeting cost/latency per call, or designing the failure story for an AI feature. NOT for debugging malformed outputs or parsing (that is llm-integration-reliability) and NOT for building the agent loop itself (that is agent-engineering).

  • Reads credentials

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 2 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Run the existence gate: Before any LLM design, ask in order (this is engineering-minimalism's ladder applied to models):

    line 44
  2. Distinctive:

    Choose local vs hosted with an explicit tradeoff table: Write the table into the design.

    line 55
  3. Very specific:

    Design idempotency for side effects: If the pipeline posts, writes, or publishes anywhere, re-running it must not duplicate output …

    line 93
  4. Distinctive:

    Every quantity in the feature is traceably computed by code or explicitly justified as model-produced.

    line 175

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

LLM System Design

Purpose

Most LLM-system failures are design failures: the model was given a job deterministic code should own, the pipeline shape amplifies cost, or nobody designed what the user sees when the model is wrong. This skill is the design-time discipline — decide whether an LLM belongs at all, give it the narrowest possible role, and design the failure story before the happy path.

Metadata

  • Prerequisites: engineering-minimalism (the existence gate below is its ladder applied to LLMs); architecture-analysis when adding to an existing system.
  • Related Skills: llm-integration-reliability (implementation-level reliability), agent-engineering (if the design lands on an agent loop), prompt-and-context-engineering (what goes into each call), evaluation-frameworks (proving the design works), system-design (general system design).
  • Owns: LLM-role selection (generator / interpreter / judge / router); local-vs-hosted model tradeoffs; pipeline shapes; cost/latency budgeting for LLM calls; design-time measurement/interpretation split.

When to Use / When NOT to Use

Use when:

  • Proposing or reviewing any feature that would call an LLM.
  • Choosing between a local model and a hosted API, or between model sizes.
  • Deciding pipeline shape: one call, a chain, parallel fan-out, or an agent loop.
  • An existing LLM feature is too expensive, too slow, or too wrong — redesign starts here, not at the prompt.

Do NOT use (load the sibling instead):

  • The design is settled and outputs are misbehaving → llm-integration-reliability.
  • You are building the agent loop, tools, or memory → agent-engineering.
  • You are writing or tuning the prompt text → prompt-and-context-engineering.
  • You need to measure whether the system works → evaluation-frameworks.
  • The system has no LLM component → system-design.

Definitions & Mental Model

  • Role: the single job the model performs. Four roles cover nearly everything: generator (produce novel text/code), interpreter (turn supplied structured facts into language), judge (score supplied evidence against criteria), router (choose one of N branches).
  • Pipeline shape: how calls compose — single call, sequential chain, parallel fan-out over chunks, or agent loop (model chooses next action).
  • Measurement/interpretation split: deterministic code computes every number; the model only speaks about numbers it is handed.
  • Failure story: the designed behavior when the model is down, slow, or wrong — as much a requirement as the happy path.

A principal engineer treats an LLM as an expensive, non-deterministic, occasionally-wrong subroutine. You would not put such a subroutine anywhere you could avoid it, you would give it the smallest job it can do well, you would bound its blast radius, and you would meter its cost. Every design decision below follows from that framing. The most reliable, cheapest, fastest LLM call is the one your design eliminated.

Core Methodology

  1. Run the existence gate. Before any LLM design, ask in order (this is engineering-minimalism's ladder applied to models):
    • Can deterministic code do it? Parsing, arithmetic, lookup, matching, validation, and formatting are NOT LLM jobs — regex, SQL, and a library are cheaper, faster, and always right.
    • Can deterministic code do most of it? Then the LLM's role shrinks to interpreting the deterministic result (see step 3).
    • Is the task actually language-shaped (summarize, explain, judge free text, converse)? Only then does an LLM earn a place.
  2. State the job in one sentence with a testable output. "Summarize the diff into ≤3 bullet points" is a design. "Use AI to improve review" is not. If you cannot state the output contract, return to step 1.
  3. Assign the narrowest role. Pick exactly one per call:
    • Interpreter is the safest role — the model receives measured facts and produces language about them; it structurally cannot invent the facts.
    • Judge requires ground-truth injection (evidence in the prompt — mechanics owned by llm-integration-reliability step 6).
    • Router requires an enum output contract and a default branch for low-confidence.
    • Generator is the highest-risk role; demand it only when novelty is the product. Apply the measurement/interpretation split: enumerate every quantity in the feature and move each computable one into code.
  4. Choose local vs hosted with an explicit tradeoff table. Write the table into the design; do not decide by default:

    AxisLocal (e.g. Ollama)Hosted API (e.g. Anthropic/OpenAI)
    Data privacyData never leaves the machineData crosses a trust boundary — check policy
    Marginal cost~zero per call; hardware fixed costPer-token; scales with usage
    LatencyHardware-bound; no networkNetwork + queue; usually faster per token on large models
    Capability ceilingLimited by local hardwareFrontier models available
    AvailabilityYou operate it; fails when host machine doesProvider SLA; fails when network/provider does
    ConcurrencyCollapses fast under parallel load — cap itRate limits — cap it differently

    Rule: pick the least capable model that passes your evaluation (see evaluation-frameworks); step up only on measured failure, never on vibes.

  5. Choose the pipeline shape (decision tree):
What does one unit of work require?
├─ One question, one answer, all context fits comfortably
│    → SINGLE CALL. Default. Cheapest to build, test, and debug.
├─ Output of step A is required input to step B (different roles)
│    → CHAIN. Validate A's output with a deterministic gate
│      before B consumes it (contracts: llm-integration-reliability).
├─ Same operation over many independent pieces (chunks, files, items)
│    → PARALLEL FAN-OUT. Bound chunk size explicitly; cap concurrency
│      with a semaphore sized to the backend; one bad item must not
│      kill the batch (recoverable-per-item taxonomy).
└─ The sequence of steps cannot be known in advance —
   the model must decide what to do next based on observations
     → AGENT LOOP. Last resort: hardest to test, bound, and price.
       Most "agent" ideas are actually chains — re-check.
       If truly needed → agent-engineering.
  1. Budget cost and latency at design time. For each call in the shape: estimate tokens in and out, multiply by calls per user action, multiply by expected volume. Write the number down. If a per-action budget does not exist, the design is not finished. Two levers dominate: cache anything generated once and reused (separate generation from evaluation — split owned by llm-integration-reliability step 7), and shrink context (see prompt-and-context-engineering).
  2. Design the failure story before the happy path. Answer in the design doc:
    • Backend unreachable → what does the user see? (Fail fast with an actionable message; never a hang.)
    • Output malformed after bounded retries → degraded output, or explicit "unavailable"? (Never silent wrongness.)
    • Output plausible but wrong → what limits the damage? (Human review? Enum constraints? The interpreter role?)
    • Define which failures are fatal vs recoverable per the taxonomy owned by architecture-analysis.
  3. Design idempotency for side effects. If the pipeline posts, writes, or publishes anywhere, re-running it must not duplicate output (idempotency owned by architecture-analysis; a worked instance appears in Repository Examples).
  4. Record the design as a lightweight ADR (format owned by system-design): role, model choice with tradeoff table, pipeline shape, budgets, failure story, and what you rejected.

Design review checklist

  • Existence gate run: every deterministic sub-task moved out of the model
  • The model's job stated in one sentence with a testable output contract
  • Exactly one role per call (generator / interpreter / judge / router)
  • Every computable quantity computed by code, not the model
  • Local-vs-hosted decided by written tradeoff table, not default
  • Least-capable-model-that-passes-evals selected
  • Pipeline shape justified by the decision tree; agent loop only if steps are truly unknowable
  • Per-action cost and latency budget written down
  • Failure story designed: down / malformed / plausibly-wrong all answered
  • Side effects idempotent on re-run
  • Evaluation plan exists before build (evaluation-frameworks)

Discovery & Audit Commands

Audit an existing codebase's LLM design before changing it:

# Where are the LLM call sites and which providers?
grep -rn -E "import (anthropic|openai|ollama)|from (anthropic|openai|ollama)|@anthropic-ai|api\.anthropic\.com|api\.openai\.com|localhost:11434" --include="*.py" --include="*.js" --include="*.ts" . | grep -v node_modules

# What roles do the prompts imply? (read each hit)
grep -rln -E "system_prompt|SYSTEM_PROMPT|prompt" --include="*.py" --include="*.js" --include="*.ts" . | grep -v node_modules

# Is there a pipeline shape already? (loops, fan-out, chains)
grep -rn -E "Semaphore|gather|Promise\.all|while|for .* in chunks" --include="*.py" --include="*.ts" . | grep -v node_modules | head -20

# Are budgets/caps designed in? (absence is a finding)
grep -rn -E "max_tokens|MAX_.*CHUNK|budget|timeout" --include="*.py" --include="*.ts" . | grep -v node_modules | head -20

# Which model(s), and is the choice configurable?
grep -rniE "model.*=.*['\"]|OLLAMA_MODEL|claude-|gpt-" --include="*.py" --include="*.ts" --include="*.env.example" . | grep -v node_modules | head -20

Failure Modes & Anti-patterns

SymptomMistakeCorrection
LLM asked to add numbers, parse dates, or match stringsSkipped the existence gateDeterministic code for computable tasks; model only for language-shaped work
Model reports metrics that were never measuredGenerator role where interpreter belongsCode measures; model interprets supplied numbers (step 3)
Bill or latency 10× estimateNo per-action budget at design timeToken math × volume, written down, before building (step 6)
Feature hangs when the model host is downFailure story never designedFail fast with actionable message; fatal-vs-recoverable decided up front (step 7)
"Agent" that always executes the same 3 stepsAgent loop chosen for a knowable sequenceIt's a chain; agent loops only when the step sequence is genuinely unknowable
Frontier-priced model doing enum classificationModel size chosen by prestigeLeast capable model that passes evals; step up on measured failure only
Same content regenerated on every requestGeneration fused with per-request workCache the generation; split generation from evaluation
Re-runs post duplicate comments/messagesSide effects not idempotentDelete-or-replace own prior output; design idempotency in (step 8)
Sensitive data sent to a hosted API by defaultLocal-vs-hosted never actually decidedWrite the tradeoff table; privacy axis first
One bad chunk kills a 40-chunk batchFan-out without per-item error taxonomyRecoverable-per-item; log and skip; fatal only for whole-backend failure

Worked Example

Task: "Add AI-powered categorization of customer support tickets into 6 queues."

  1. Existence gate: 40% of tickets contain an order-ID pattern that maps deterministically to a queue — regex handles those with zero model calls. Remainder is genuinely language-shaped.
  2. Job sentence: "Given ticket title+body, output one of 6 queue names." Output contract: single enum value.
  3. Role: router. Enum contract {"queue": "billing|shipping|returns|technical|account|other"}, with other as the low-confidence default routed to a human.
  4. Local vs hosted: tickets contain PII → privacy axis dominates → local model, or hosted with a signed data agreement; table written into the ADR.
  5. Shape: single call per ticket. Fan-out only exists at the queue-consumer level, which is ordinary code.
  6. Budget: ~800 tokens in, ~10 out; 3,000 tickets/day → the token math goes in the ADR; a small local model is plausible — test it first.
  7. Failure story: backend down → tickets flow to other (human triage), banner raised for operators; malformed output after 2 strikes → other. Wrongness bounded: worst case equals today's manual triage.
  8. Evals before build: 200 historically-labeled tickets as the golden set; the small model ships only if it beats the current keyword rules (evaluation-frameworks).

The resulting design uses the model for 60% of tickets, in the narrowest role, with a failure mode no worse than the status quo.

Repository Examples

Repo facts below are point-in-time illustrations (as of 2026-07-04) — examples, never assumptions about your system.

  • prism (~/prism) — a complete design instance: role = judge over injected PR diffs (never recalls code from memory); local model via Ollama (OLLAMA_MODEL, default llama3.2) — privacy + zero marginal cost for a self-hosted reviewer; shape = parallel fan-out over diff chunks bounded by MAX_LINES_PER_CHUNK=120 and asyncio.Semaphore(5); failure story designed: OllamaUnavailableError aborts fast, per-chunk timeout degrades gracefully; scope constrained in the prompt to genuine bugs/security only, with "post nothing" as first-class success; idempotent posting (deletes its own prior comments).
  • orphy (~/orphy) — the measurement/interpretation split as architecture: DSP code measures pitch/onsets/cents error; the LLM only interprets those numbers into coaching language — hallucinated measurements are structurally impossible.
  • NYTW (~/NYTW) — cost-structure design: question generation is one cacheable call; grading is per-attempt with fresh evidence — generation/evaluation split along cost/frequency lines.
  • ragit (~/ragit) — model-selection asymmetry: embedding model fixed (nomic-embed-text — the index's embedding space is a stable contract) while the chat model is a swappable ranked list — the interpreter can change; the index contract cannot.

Validation Criteria

You applied this skill correctly when:

  1. The design document states the model's one-sentence job, role, and output contract — and a reviewer can test the contract mechanically.
  2. Every quantity in the feature is traceably computed by code or explicitly justified as model-produced.
  3. The local-vs-hosted tradeoff table exists with the privacy axis answered.
  4. Per-action token/cost/latency budgets are written and arithmetic-checkable.
  5. The failure story answers down/malformed/plausibly-wrong, and each failure is classified fatal or recoverable.
  6. The pipeline shape matches the decision tree, and any agent loop is justified by genuinely unknowable step sequences.
  7. An evaluation plan predates the build.

Provenance & Maintenance

  • Sources: ~/prism, ~/orphy, ~/NYTW, ~/ragit — investigated 2026-07-04. Owner context: primary consumers are Sonnet-class agents; LLM output reliability is the owner's hardest recurring problem; minimalism doctrine owner-confirmed 2026-07-04. Methodology is repo-independent; skill authored 2026-07-06.
  • Assumptions: the local-vs-hosted table reflects the 2026 provider landscape (pricing models, local-model capability) — re-check axes as the market moves. Repo constants (Semaphore(5), MAX_LINES_PER_CHUNK=120, model names) are point-in-time.
  • Re-verification commands:
    grep -rn "Semaphore\|MAX_LINES_PER_CHUNK\|OLLAMA_MODEL" ~/prism/prism ~/prism/.env.example
    grep -rn "nomic-embed-text" ~/ragit/ragit
  • Likely to drift: provider names and pricing structures; local-model capability ceiling; the example repos' model choices and constants.
  • Maintenance checklist:
    • Re-run re-verification commands; re-stamp Repository Examples.
    • Re-validate the tradeoff table axes against the current provider landscape annually.
    • Confirm cross-referenced skills still exist under their directory names.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.