agentleFS
masteranimemasteranime/n8n-claude-skills/skills/enrichment-waterfall/SKILL.md

enrichment-waterfall

Skill32 starsChanged 7 months agoMITView on GitHub

Build multi-vendor data enrichment waterfalls in n8n — cascading API calls across SerpAPI, Hunter.io, Apollo, Clearbit, LLM extractors, and scrapers with cost-aware fallbacks. Use this skill whenever the user wants to enrich leads, contacts, companies, or any entity with external data in n8n — phrases like "lead enrichment", "email finder", "data waterfall", "Clay alternative", "find LinkedIn profile", "get company info", "enrich this list of leads". Also use when designing any flow where multiple vendors are tried in sequence until one succeeds. Use this skill before designing such workflows because naive sequential API calls produce $10/lead costs — the waterfall pattern drops that to $0.10 by ordering vendors correctly.

1 other project uses a version of this.

Adopted by

Most recently taken up by

  • Other versions1 project uses a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 1

    1. guerrilla2799/ops-and-scale-os★ 1same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • In this repository4 other files it gives its agents.

The file

We split this skill into the rules it gives an agent

9 rules. About 0 are common, 4 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈4

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    After each vendor, check BOTH response status AND payload content:

    line 75
  2. Very specific:

    Don't just check .error — vendors often return 200 with empty results on a miss.

    line 86
  3. Very specific:

    Before calling ANY vendor, SELECT on input_key WHERE enriched_at > NOW() - INTERVAL 30 DAY.

    line 102
  4. Distinctive:

    When paid vendors miss, SerpAPI + LLM extract works 80%+ of the time:

    line 106
  5. Very specific:

    IF confidence < 0.7 → treat as miss

    line 118
  6. Very specific:

    Use Groq llama-3.3-70b-versatile for extract — it's fast enough that even at 90% hit rate, per-lead cost stays under 2 cents.

    line 120
  7. Distinctive:

    Every vendor has limits. Hitting them burns waterfalls silently.

    line 124
  8. Distinctive:

    No vendor health monitoring: Add a dashboard query: hit rate per vendor per week.

    line 139
  9. Very specific:

    Add a final Set node to normalize each vendor's response into this schema before returning.

    line 158

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Enrichment Waterfall for n8n

A waterfall = try cheapest/fastest vendor first, fall through to more expensive/accurate vendors only when the previous fails. This is how Clay, Clearbit, and every production enrichment pipeline actually works.

The core pattern

Input (name, email, or domain)
  ↓
Vendor 1 (cheap, fast, ~60% hit rate) — e.g., Hunter.io
  ↓ IF no match
Vendor 2 (medium cost, ~80% cumulative) — e.g., Apollo
  ↓ IF no match
Vendor 3 (expensive / LLM extract, ~95% cumulative) — e.g., SerpAPI + LLM
  ↓ IF no match
Dead letter: log as "unenrichable"

At each step, a hit short-circuits the rest. You pay only for what the cheap vendors miss.

Ordering: cost × accuracy × rate limit

Order vendors by expected cost per successful enrichment, not sticker price. Calculate:

effective_cost = price_per_call / hit_rate

Example for email-from-name+company:

VendorPrice/callHit rateEffective cost
Hunter.io$0.00455%$0.007
Apollo bulk$0.0175%$0.013
SerpAPI + LLM extract$0.0290%$0.022
Manual LinkedIn scrape$0.0560%$0.083

Order: Hunter → Apollo → SerpAPI+LLM → dead letter. Effective cost per enriched lead ≈ $0.012 vs $0.083 if you'd started with the scraper.

n8n implementation

Structure

1. Trigger (Webhook / Schedule / Manual)
2. Set — normalize input (lowercase email, strip whitespace, extract domain)
3. MySQL / Google Sheets — check cache (was this already enriched in last 30 days?)
4. IF cache hit → return cached → END
5. HTTP Request: Hunter.io
6. IF match found → Set enriched data → merge back → END
7. HTTP Request: Apollo (on Hunter miss)
8. IF match → merge → END
9. HTTP Request: SerpAPI
10. Information Extractor (LangChain) — extract contact from SERP results
11. IF match → merge → END
12. MySQL insert — dead letter table

Critical configuration

Each HTTP Request node needs:

  • continueOnFail: true — so one vendor's 500 doesn't kill the pipeline
  • retry.maxTries: 2 with retry.waitBetweenTries: 3000
  • Timeout: 10s. Waterfalls with 6 vendors at 30s timeouts = 3-minute-per-lead worst case
  • Auth via n8n credentials, never inline

The IF check pattern

After each vendor, check BOTH response status AND payload content:

// In an IF node expression:
={{ 
  $('Hunter Request').item.json.error 
    ? false 
    : $('Hunter Request').item.json.data?.email != null 
}}

Don't just check .error — vendors often return 200 with empty results on a miss.

Cache layer (mandatory)

Enrichment data goes stale in ~30 days but doesn't change daily. Cache aggressively:

CREATE TABLE enrichment_cache (
  input_key VARCHAR(255) PRIMARY KEY,  -- normalized email/domain
  enriched_data JSON,
  source VARCHAR(50),                   -- which vendor hit
  enriched_at TIMESTAMP,
  INDEX idx_enriched_at (enriched_at)
);

Before calling ANY vendor, SELECT on input_key WHERE enriched_at > NOW() - INTERVAL 30 DAY. Cache hit rate of 40% is normal after a few weeks — that's 40% cost reduction for free.

LLM extract stage (the Stage 3 secret weapon)

When paid vendors miss, SerpAPI + LLM extract works 80%+ of the time:

  1. HTTP Request → SerpAPI search: "{{ $json.first_name }} {{ $json.last_name }}" "{{ $json.company }}" site:linkedin.com
  2. Information Extractor with schema:
    {
      "linkedin_url": "string",
      "title": "string",
      "location": "string",
      "confidence": "number (0-1)"
    }
  3. IF confidence < 0.7 → treat as miss

Use Groq llama-3.3-70b-versatile for extract — it's fast enough that even at 90% hit rate, per-lead cost stays under 2 cents.

Rate limits (the silent killer)

Every vendor has limits. Hitting them burns waterfalls silently.

VendorTypical limitn8n handling
Hunter.io50/min (free tier 25/day)Split In Batches size=1, Wait 1200ms between
Apollo600/min enterpriseUsually fine at batch size 10
SerpAPIPlan-dependentCheck headers, backoff if X-RateLimit-Remaining < 5

For high-volume pipelines, run the waterfall as a sub-workflow called from a Split In Batches parent with batchSize: 10, waitBetweenBatches: 60000.

Anti-patterns

  • Parallelizing vendors. Running all 3 in parallel and picking the best defeats the point — you pay for all 3 on every lead. Waterfall = sequential with early exit.
  • No dead letter. Unenrichable leads must go somewhere queryable so you can spot-check WHY they're missing. Otherwise vendor regressions go unnoticed.
  • Same timeout across vendors. Cheap vendors are usually fast (set 5s). Scrapers need 30s. Tune per vendor.
  • No vendor health monitoring. Add a dashboard query: hit rate per vendor per week. When Hunter's hit rate drops from 55% to 20%, you want to know immediately.

Output contract

Whatever the source, the waterfall should output a unified schema downstream consumers can rely on:

{
  "email": "string",
  "full_name": "string",
  "company": "string",
  "linkedin_url": "string|null",
  "title": "string|null",
  "enrichment_source": "hunter|apollo|serpapi_llm",
  "confidence": "number (0-1)",
  "enriched_at": "ISO timestamp"
}

Add a final Set node to normalize each vendor's response into this schema before returning.

Reference

  • references/waterfall-template.json — importable 18-node starter waterfall

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.