build-llm-pipeline
Skill4 starsChanged 6 days agoMITView on GitHub
Builds an LLM pipeline for extraction, classification or analysis with verification, evaluation and confidentiality controls.
In this repository210 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .cursor/rules/ai-ml-code.mdcCursor rule
- .cursor/rules/code-translation.mdcCursor rule
- .cursor/rules/data-pipelines.mdcCursor rule
- .cursor/rules/documentation.mdcCursor rule
- .cursor/rules/dsge-code.mdcCursor rule
- .cursor/rules/econometrics-code.mdcCursor rule
- .cursor/rules/eviews-code.mdcCursor rule
- .cursor/rules/forecasts.mdcCursor rule
- .cursor/rules/gempack-code.mdcCursor rule
- .cursor/rules/lego-manifests.mdcCursor rule
- .cursor/rules/llm-code.mdcCursor rule
- .cursor/rules/model-code.mdcCursor rule
- .cursor/rules/notebooks-and-scripts.mdcCursor rule
- .cursor/rules/r-code.mdcCursor rule
- .cursor/rules/restricted-data.mdcCursor rule
- .cursor/rules/rl-code.mdcCursor rule
- .cursor/rules/sectoral-trade-climate-code.mdcCursor rule
- .cursor/rules/semi-structural-code.mdcCursor rule
- .cursor/rules/stata-code.mdcCursor rule
- .cursor/rules/tests.mdcCursor rule
- .cursor/rules/validation.mdcCursor rule
- adoptSkill
- ai-auditSkill
- backtestSkill
- build-abmSkill
- build-causal-mlSkill
- build-cgeSkill
- build-climate-macroSkill
- build-deep-forecasterSkill
- build-drl-policySkill
- build-dsgeSkill
- build-financial-programmingSkill
- build-fpasSkill
- build-game-modelSkill
- build-hankSkill
- build-io-samSkill
- build-llm-agentsSkill
- build-multicountrySkill
- build-neural-solverSkill
- build-rag-kbSkill
- build-semi-structuralSkill
- build-sfcSkill
- build-trade-modelSkill
- calibrate-modelSkill
- calibration-planSkill
- carbon-pricing-analysisSkill
- chart-packSkill
- climate-scenarioSkill
- code-inventorySkill
- code-reviewSkill
- country-typologySkill
- create-exerciseSkill
- data-requirementsSkill
- debt-decompositionSkill
- debt-sustainabilitySkill
- design-trainingSkill
- distributional-analysisSkill
- equivalence-testSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 6 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 26
If a required input is ABSENT, that part is NOT ASSESSED — NO DATA: do not estimate or invent it.
- Very specific: line 28
Collaboration: Follow .claude/docs/collaboration-protocol.md: ask questions first, present options, get approval before writing files, and …
- Distinctive: line 49
Create a labelled evaluation set before building prompts
- Distinctive: line 54
Document failure modes, human review requirements and monitoring
- Distinctive: line 58
After the confidentiality check: confirm the endpoint
- Distinctive: line 59
After the evaluation set: confirm thresholds
- Distinctive: line 60
After evaluation: approve release and review level
- Distinctive: line 77
Every extracted number is traceable to its source
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
<!-- Generated from .claude/skills/build-llm-pipeline/SKILL.md by tools/sync_agents.py. Edit the source, then run the script. -->/build-llm-pipeline — Build LLM Pipeline
Category: Build (AI) · Primary agent: llm-economic-analyst
Builds an LLM pipeline for extraction, classification or analysis with verification, evaluation and confidentiality controls.
Arguments: whatever the user wrote after the command name.
Purpose
Build an LLM pipeline for extraction, classification or analysis of economic documents that is evaluated like a measurement instrument: confidentiality cleared first, a labelled evaluation set built before prompts, structured outputs verified against sources, and accuracy measured against agreed thresholds. The output is a pipeline with an evaluation card and defined human review.
Use it when large document collections must be read for numbers, measures, stances or facts, or text must be classified at scale.
Do not use it when questions are answered from a document collection with citations (/build-rag-kb), or the task is to simulate agents (/build-llm-agents).
Before you start
- Check required inputs. For each input below record
FOUNDorABSENT(never 'assumed'):- Task definition
- Labelled evaluation sample
If a required input is
ABSENT, that part isNOT ASSESSED — NO DATA: do not estimate or invent it; name what is missing and which skill produces it. If all required inputs are absent, stop and say so.
- Check confidentiality. Identify the data classification level of the inputs (
.claude/docs/confidentiality-and-ai-use.md). Do not proceed with restricted or confidential data in a way the policy does not allow. - Collaboration. Follow
.claude/docs/collaboration-protocol.md: ask questions first, present options, get approval before writing files, and updateproduction/session-state/active.mdafter each approved section.
Agents
llm-economic-analyst— primary agent; coordinates the work and integrates resultsnlp-text-as-data-specialist— Turns text into economic datallm-red-team-specialist— Red-teams LLM systemsdata-governance-metadata-specialist— Owns data governance and metadata
Each agent is defined in .claude/agents/<name>.md. If your tool runs subagents, run these agents as subagents, in parallel where the steps are independent; otherwise read each agent's file and apply its instructions in turn. Subagents only write to the paths named in your instructions.
Key design decisions (present options, let the user choose)
| Decision | Typical options | What it changes |
| Model and endpoint | internal, approved external, local | Confidentiality and quality |
| Output format | JSON schema, labels, short text | Verifiability |
| Verification | source quotation, cross-check with databases, consistency reruns | Error detection |
| Thresholds | accuracy and numeric faithfulness targets | Release decision |
| Human review | full, sampled, exceptions only | Cost and risk |
Workflow
- Classify input data by confidentiality; confirm the allowed model/endpoint before any prompt is written
- Create a labelled evaluation set before building prompts
- Cover document types, languages and hard cases; keep a held-out portion
- Design structured-output prompts and verification steps (source cross-check, schema validation)
- Evaluate accuracy, numeric faithfulness and stability across prompt/model versions
- Error analysis by field and document type
- Document failure modes, human review requirements and monitoring
- Freeze prompt versions; rerun evaluation after any model change
Checkpoints with the user
- After the confidentiality check: confirm the endpoint
- After the evaluation set: confirm thresholds
- After evaluation: approve release and review level
Common pitfalls
- Prompts tuned on the evaluation set
- Fluent outputs accepted without source checks
- Silent model-version changes
- Confidential documents sent to unapproved services
Output
models/llm/<name>/ and docs/models/<name>/llm-eval-card.md
The pipeline folder holds prompts, schemas, code, the evaluation set and results; the evaluation card documents accuracy, failure modes, human review and monitoring. Every run saves its results to output/<run-id>/ in the model folder, with a run log (.claude/rules/model-code.md).
Gate / verdict
- Confidentiality approved
- Evaluation on a labelled set meets agreed thresholds
- Human review defined
- Every extracted number is traceable to its source
- Evaluation is rerun after any model or prompt change
Suggested next steps
/ai-audit/validate-model
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

