build-data-pipeline
Skill178 starsChanged 4 months agoMITView on GitHub
Guide a developer through building a Solana data pipeline or indexer. Use when a user says "build an indexer", "data pipeline", "analytics", "track transactions", "monitor wallets", "webhook", "index accounts", or "real-time data". Reads build-context.md from a prior scaffold phase if available.
- Sends data out
In this repository34 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- animation-reverse-engineeringSkill
- brand-designSkill
- build-defi-protocolSkill
- build-mobileSkill
- build-with-claudeSkill
- csoSkill
- debug-programSkill
- design-tasteSkill
- frontend-design-guidelinesSkill
- launch-tokenSkill
- navigate-skillsSkill
- number-formattingSkill
- page-load-animationsSkill
- product-reviewSkill
- review-and-iterateSkill
- roast-my-productSkill
- scaffold-projectSkill
- verify-humanity-pohSkill
- virtual-solana-incubatorSkill
- colosseum-copilotSkill
- competitive-landscapeSkill
- defillama-researchSkill
- find-next-crypto-ideaSkill
- learnSkill
- solana-beginnerSkill
- validate-ideaSkill
- apply-grantSkill
- create-pitch-deckSkill
- deploy-to-mainnetSkill
- marketing-videoSkill
- submit-to-hackathonSkill
- video-craftSkill
The file
We split this skill into the rules it gives an agent
12 rules. About 0 are common, 9 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈9
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 26
If TEL_PROMPTED is no: Before starting the skill workflow, ask the user about telemetry.
- Distinctive: line 36
If A: run this bash:
- Distinctive: line 43
If B: run this bash:
- Very specific: line 62
Check for .superstack/build-context.md.
- Distinctive: line 63
Read references/indexing-patterns.md to select the right ingestion method.
- Distinctive: line 64
Read references/data-storage.md to design the storage schema.
- Distinctive: line 71
Test with live devnet data, then switch to mainnet when ready.
- Distinctive: line 76
Always implement idempotent writes — webhooks and WebSockets can deliver duplicates.
- Distinctive: line 78
Never store raw transaction blobs without parsing — they are expensive to query later.
- Distinctive: line 79
Use Helius enhanced transactions for parsed data instead of raw RPC when possible.
- Distinctive: line 94
When updating, deep-merge — don't overwrite existing fields.
- Very specific: line 133
After the skill workflow completes (success, error, or abort), log the telemetry event.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Preamble (run first)
_TEL_TIER=$(cat ~/.superstack/config.json 2>/dev/null | grep -o '"telemetryTier": *"[^"]*"' | head -1 | sed 's/.*"telemetryTier": *"//;s/"$//' || echo "anonymous")
_TEL_TIER="${_TEL_TIER:-anonymous}"
_TEL_PROMPTED=$([ -f ~/.superstack/.telemetry-prompted ] && echo "yes" || echo "no")
_TEL_START=$(date +%s)
_SESSION_ID="$$-$(date +%s)"
mkdir -p ~/.superstack
echo "TELEMETRY: $_TEL_TIER"
echo "TEL_PROMPTED: $_TEL_PROMPTED"
if [ "$_TEL_TIER" != "off" ]; then
_TEL_EVENT='{"skill":"build-data-pipeline","phase":"build","event":"started","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}'
echo "$_TEL_EVENT" >> ~/.superstack/telemetry.jsonl 2>/dev/null || true
_CONVEX_URL=$(cat ~/.superstack/config.json 2>/dev/null | grep -o '"convexUrl":"[^"]*"' | head -1 | cut -d'"' -f4 || echo "")
[ -n "$_CONVEX_URL" ] && curl -s -X POST "$_CONVEX_URL/api/mutation" -H "Content-Type: application/json" -d '{"path":"telemetry:track","args":{"skill":"build-data-pipeline","phase":"build","status":"success","version":"0.2.0","platform":"'$(uname -s)-$(uname -m)'","timestamp":'$(date +%s)000'}}' >/dev/null 2>&1 &
true
fiIf TEL_PROMPTED is no: Before starting the skill workflow, ask the user about telemetry.
Use AskUserQuestion:
Help superstack get better! We track which skills get used and how long they take — no code, no file paths, no PII. Change anytime in
~/.superstack/config.json.
Options:
- A) Sure, help superstack improve (anonymous)
- B) No thanks
If A: run this bash:
echo '{"telemetryTier":"anonymous"}' > ~/.superstack/config.json
_TEL_TIER="anonymous"
touch ~/.superstack/.telemetry-promptedIf B: run this bash:
echo '{"telemetryTier":"off"}' > ~/.superstack/config.json
_TEL_TIER="off"
touch ~/.superstack/.telemetry-promptedThis only happens once. If TEL_PROMPTED is yes, skip this entirely and proceed to the skill workflow.
Wrong skill? See SKILL_ROUTER.md for all available skills.
Build Data Pipeline
Overview
Guide the user through building a data pipeline that ingests, transforms, and stores Solana on-chain data. Covers real-time event streaming via webhooks and WebSockets, historical backfilling, account state indexing, and building query-friendly storage. Uses Helius where it is the best fit for production-grade ingestion, while keeping the storage and indexing design provider-agnostic.
Workflow
- Check for
.superstack/build-context.md. If found, use stack decisions. If not, ask: what data do you need (transactions, account state, token transfers, program events)? Real-time, historical, or both? Write.superstack/build-context.mdwith the context gathered so future skills can use it. - Read references/indexing-patterns.md to select the right ingestion method.
- Read references/data-storage.md to design the storage schema.
- Implement in milestones: a. Set up data ingestion (Helius webhook, WebSocket subscription, or geyser plugin) b. Build the parser/transformer for your specific data format c. Design and create the database schema d. Implement write path (ingestion → parse → store) e. Build query API or dashboard on top of stored data
- Test with live devnet data, then switch to mainnet when ready.
- Monitor ingestion lag and handle missed events with backfill logic.
Non-Negotiables
- Always implement idempotent writes — webhooks and WebSockets can deliver duplicates.
- Include a backfill mechanism — you will miss events during deploys, restarts, and outages.
- Never store raw transaction blobs without parsing — they are expensive to query later.
- Use Helius enhanced transactions for parsed data instead of raw RPC when possible.
- Monitor ingestion lag — if your pipeline falls behind, you need alerts, not silent data loss.
- Store the slot number with every record for ordering and deduplication.
Phase Handoff
This skill is Phase 2 (Build) in the Idea → Build → Launch journey.
Reads: .superstack/build-context.md
Writes/Updates: .superstack/build-context.md (creates if missing) with:
pipeline.ingestion_method: "webhook" | "websocket" | "geyser" | "rpc-polling"pipeline.data_types: string[] (e.g., ["transactions", "account-state", "token-transfers"])pipeline.storage: "postgresql" | "redis" | "custom"pipeline.backfill_implemented: boolean
When updating, deep-merge — don't overwrite existing fields.
See ../../data/specs/phase-handoff.md for the full JSON contract.
Quick Start
# Fastest: Helius webhooks (no infrastructure needed)
# 1. Get Helius API key from dashboard.helius.dev
# 2. Create webhook:
curl -X POST https://api-mainnet.helius-rpc.com/v0/webhooks?api-key=YOUR_KEY \
-H 'Content-Type: application/json' \
-d '{
"webhookURL": "https://your-app.com/webhook",
"transactionTypes": ["TRANSFER"],
"accountAddresses": ["YOUR_PROGRAM_ID"]
}'
# For WebSocket (real-time, more control):
# Use: solana logs <PROGRAM_ID> --url mainnet-beta
# Or: @solana/web3.js Connection.onLogs()Decision Points
- Which ingestion method? Webhooks (simplest, Helius) → WebSocket (real-time) → Geyser (highest throughput) → Polling (last resort).
- Which RPC? Helius is required if you want Helius webhooks or Helius-only enhanced APIs. DAS is available from multiple providers (for example Helius, MetaMask/Infura, and QuickNode), so choose based on pricing, limits, and where the rest of your stack already lives.
- Database? PostgreSQL for relational data + transactions. Redis for caching + real-time state. SQLite for small/local indexers.
- Hosting? Railway or Fly.io for webhook receivers. AWS/GCP for Geyser plugins.
Resources
references/
- references/indexing-patterns.md
- references/data-storage.md
Telemetry (run last)
After the skill workflow completes (success, error, or abort), log the telemetry event.
Determine the outcome from the workflow result: success if completed normally, error
if it failed, abort if the user interrupted.
Run this bash:
_TEL_END=$(date +%s)
_TEL_DUR=$(( _TEL_END - ${_TEL_START:-$_TEL_END} ))
_TEL_TIER=$(cat ~/.superstack/config.json 2>/dev/null | grep -o '"telemetryTier": *"[^"]*"' | head -1 | sed 's/.*"telemetryTier": *"//;s/"$//' || echo "anonymous")
if [ "$_TEL_TIER" != "off" ]; then
echo '{"skill":"build-data-pipeline","phase":"build","event":"completed","outcome":"OUTCOME","duration_s":"'"$_TEL_DUR"'","session":"'"$_SESSION_ID"'","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","platform":"'$(uname -s)-$(uname -m)'"}' >> ~/.superstack/telemetry.jsonl 2>/dev/null || true
true
fiReplace OUTCOME with success/error/abort based on the workflow result.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

