Positive Reply Scoring

Pipeline contract

Stage 17 of 20. Runs after the human has sent the campaign. Reads replies from Smartlead (scripts below) OR 17-replies-raw.csv (lead_id, email, reply_text, replied_at) exported from any sender. Writes 17-replies.csv (adds label, is_positive, next_action) and 17-reply-score.json.

• Save to the run folder, not the ~/cold-email-ai-skills/ path below.
• unsubscribe and negative_hostile rows go to the do-not-contact list in the same session — that list feeds stage 10's hard disqualifiers.
• Positive rows hand off to /lead-qualification.

Files live in runs/<run>/. Column definitions: /prospecting-system. Standalone use: ignore this block.

Reply rate tells you if people are paying attention. Positive reply rate tells you if they want what you're selling. This skill computes the second.

Why this exists

A campaign can get 5% reply rate and still be a disaster. If 90% of those replies are "unsubscribe" and "not a fit," you're burning your domains for nothing.

The metric that matters is:

positive_reply_rate = positive_replies / total_sent

Compared side-by-side:

• Campaign A: 1% reply rate, 70% positive → 0.7% positive reply rate
• Campaign B: 5% reply rate, 10% positive → 0.5% positive reply rate
• Campaign A wins.

Classification schema

Every reply is classified into exactly one bucket:

Positive reply rate = (positive_interested + positive_soft + positive_referral) / total_sent

Inputs

• Smartlead API key (env: SMARTLEAD_API_KEY)
• Campaign ID to score
• Optional: client_id (if using a sub-client setup)
• Optional: date range (defaults to full campaign)

Steps

1. Fetch all leads + replies from the campaign

Run the fetch script:

npx tsx scripts/fetch-campaign-replies.ts --campaign-id=12345 --out=/tmp/replies.json

This walks /campaigns/{id}/leads paginated, identifies leads with replies (has_reply = true), then fetches /campaigns/{id}/leads/{lead_id}/message-history for each, and writes them to a JSON file with one object per reply.

Output schema per reply:

{
  "lead_id": "...",
  "email": "...",
  "lead_first_name": "...",
  "company": "...",
  "reply_time": "ISO timestamp",
  "reply_subject": "...",
  "reply_body": "... full text ...",
  "sequence_step": 1
}

2. Classify replies in the Claude Code conversation

Once the JSON is written, Claude (the one running this skill) reads the file and classifies each reply. For speed, fan out in batches of 20-30 via the Task tool (see personalization-subagent-pattern skill for fan-out mechanics).

Classification prompt (per batch):

Classify each reply as one of:
- positive_interested, positive_soft, positive_referral
- neutral_question
- negative_notnow, negative_notfit, negative_hostile
- unsubscribe, ooo, bounce, other

For each reply, output: { lead_id, label, confidence: 0.0-1.0, one_line_reason }

Rules:
- OOO auto-replies ("I'm out of office") → ooo
- Bounces (delivery failure messages) → bounce
- "Take me off your list", "unsubscribe", "STOP" → unsubscribe
- "Not interested", "not a fit" → negative_notfit
- "Not right now, circle back in Q3" → negative_notnow
- "Try [other person]" → positive_referral
- "Send more info" or "Tell me more" → positive_soft
- "Yes, let's book a call", "what times work" → positive_interested
- Insults, reports, legal threats → negative_hostile

If confidence < 0.7, label as `other`.

3. Aggregate + compute rates

Run the aggregator:

npx tsx scripts/aggregate-scores.ts --replies=/tmp/classified-replies.json --campaign-id=12345

Output (to stdout + optional --out):

Campaign 12345 — Positive Reply Scoring

Total sent:              5,284
Total replies:              212 (4.01%)
  ooo/bounce (excluded):    34
  Net replies:             178

Breakdown:
  positive_interested:    22
  positive_soft:          31
  positive_referral:       8
  neutral_question:       14
  negative_notnow:        28
  negative_notfit:        52
  negative_hostile:        3
  unsubscribe:            20
  other:                   0

Positive reply rate:     1.15% (61 / 5,284)
Positive % of replies:   34.3% (61 / 178)
Negative hostile risk:    0.06% (3 / 5,284)
Unsub rate:              0.38% (20 / 5,284)

Benchmarks (B2B cold email):
  Good positive reply rate: ≥1%
  Great: ≥2%
  Hostile >0.3% or unsub >2% → deliverability risk, pause campaign

4. Save to disk

Write aggregate results to:

~/cold-email-ai-skills/profiles/<business-slug>/scores/<campaign-id>-<YYYY-MM-DD>.json

This builds a history so you can trend positive reply rate over campaigns.

5. Flag action items

At the end, surface:

• Positive replies that need a human response — list the top 10 positive_interested leads and their reply bodies. The user should reply to these within 30 seconds of seeing this report.
• Referrals that need follow-up — positive_referral labels. Add the referred contacts to a new outreach list.
• Hostile flags — any negative_hostile replies. Read them manually; consider pausing the inbox if someone is genuinely angry.
• Unsubscribes — confirm they're globally suppressed (Smartlead does this automatically, but double-check).

When to use this skill

• After a campaign has run for at least 14 days (otherwise sample is too small)
• When comparing two campaigns in an experiment (use the same cutoff date for both)
• Weekly as a quality check on running campaigns
• Before deciding to kill or scale a campaign

Common gotchas

• Don't trust reply rate alone. A 5% reply rate from spam-trap replies and unsubscribes is worse than a 2% reply rate from real buyers.
• Exclude OOO + bounce from denominators. They're not real replies. The script does this automatically.
• Smartlead's built-in AI categorization exists but is less controllable. This skill uses Claude directly for transparency and prompt-tunable classification.
• Small samples lie. Below ~500 sent, the positive reply rate has too much noise. Wait for more volume before declaring winners/losers.
• Classify only FIRST reply per lead. If a lead replied, you replied, they replied again — only the first reply is the signal. Later messages are the conversation, not the scoring.

What to do next

Respond to every positive_interested reply within 30 seconds of seeing it. Then /experiment-design to plan the next iteration based on what worked.

If positive reply rate is <1% after 200+ sends: the 1% rule failed. Run /email-deliverability-audit (are you reaching the inbox?) (check for vague CTAs, generic first lines, em dashes).

Or wait: this skill is the Wednesday task in /pipeline-review. Run it weekly going forward.

Related skills

• /experiment-design — uses positive reply rate as the success metric
• /email-deliverability-audit — if hostile + unsub are elevated, run this next
• /cold-email-starter-kit → 10-reply-handling.md for what to do with the positive replies once flagged

Scripts

• scripts/fetch-campaign-replies.ts — pulls replies via Smartlead API
• scripts/aggregate-scores.ts — computes rates from classified JSON