agentleFS
reatlatreatlat/fullstory-claude-plugin/skills/changelog-detective/SKILL.md

changelog-detective

Skill36 starsChanged 17 days agoMITView on GitHub

Detect what changed in the product by comparing before and after a deploy or date — find new behaviors, new errors, changed user flows, and unexpected side effects that weren't in the release notes. Use after a deploy or when the user suspects something changed but doesn't know what.

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Use free fullstory:discover_groups and fullstory:get_opportunity_stats for live signals.

    line 22
  2. Distinctive:

    Ask: "I'll compare 48 hours before and after the deploy (excluding the deploy hour).

    line 32
  3. Distinctive:

    If you find undocumented changes that look like bugs, flag them immediately.

    line 134

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Changelog Detective

Find what actually changed after a deploy — not what the release notes say, but what the session data shows. Catch undocumented changes, side effects, and surprises.

When to Use

  • "What actually changed after the deploy?"
  • "Something feels different since Tuesday — what changed?"
  • "Check for undocumented changes in the last release"
  • "Did the deploy introduce any new user behaviors?"
  • "What new errors appeared this week that weren't here last week?"

Mental Model

Release notes tell you what was intended to change. Session data tells you what actually changed. These often differ. A "minor CSS fix" might accidentally break form validation. A "performance improvement" might introduce a new race condition. This skill hunts for the gaps between intent and reality.

Use free fullstory:discover_groups and fullstory:get_opportunity_stats for live signals. Discovery accepts one scope at a time (page_id, domain, app_name, segment_id, or funnel_id); use a segment for device/user filters and the live schema for time ranges. Use computed metrics for exact custom windows.

Workflow

Step 1: Define the comparison window

Two time periods:

  • Before: 24-72 hours before the deploy/change date
  • After: 24-72 hours after (but not including the deploy window itself — things are chaotic during rollout)

Ask: "I'll compare 48 hours before and after the deploy (excluding the deploy hour). That OK?"

Step 2: Before/After diff — Errors

What errors existed before vs after?

fullstory:build_metric(query="errors by type", output_type="top_n")
fullstory:compute_metric(metric_id, time_range=before)
fullstory:compute_metric(metric_id, time_range=after)

Look for:

  • New errors: Error types that appear in After but not Before → these are deploy regressions
  • Resolved errors: Error types in Before but not After → these are fixed (celebrate these)
  • Changed errors: Same error type, different frequency → possible impact change

Step 3: Before/After diff — Frustrations

fullstory:discover_groups(compare_to_previous=true)
→ identify changed frustration signals
→ build metrics for those signals and compute for the exact before/after windows

Look for:

  • New frustration signals: Rage clicks or dead clicks on elements that weren't problematic before
  • Resolved frustrations: Frustrations that disappeared (fix worked!)
  • Changed patterns: Same frustration, different page or device distribution

Step 4: Before/After diff — User behavior

Did user flows change?

fullstory:build_metric(query="page views by page", output_type="top_n")
→ compare before/after: any pages suddenly getting more/less traffic?

fullstory:build_metric(query="conversion rate", output_type="single_number")
→ compare before/after: did conversion change?

Look for:

  • Traffic shifts: Pages with significantly different traffic share
  • Flow changes: New common page sequences (users navigating differently)
  • Time changes: Sessions getting longer or shorter
  • Conversion changes: Key funnels performing differently

Step 5: Before/After diff — Device/Browser

Did the change affect specific platforms?

fullstory:build_metric(query="sessions by device type", output_type="top_n")
→ compare before/after: did the device mix change?

fullstory:build_metric(query="errors on Safari", output_type="single_number")
→ compare before/after: any browser-specific regressions?

Step 6: Assemble the changelog

## Actual Changelog — v2.3.1 Deploy (Aug 4 15:00 UTC)

### What the release notes said:
- "Fixed promo code validation"
- "Updated checkout button styling"  
- "Performance improvements"

### What actually changed (from session data):

✅ **Promo code fixed**: Rage clicks on "Apply Promo" dropped from 412/week to 24/week (94% reduction)
✅ **Button styling**: No new dead clicks on checkout elements — visual change was clean
🟡 **Performance**: Page load time unchanged (was 2.1s, now 2.1s). "Improvements" may not be user-visible.

### Undocumented changes:

🔴 **New error introduced**: `TypeError: Cannot read property 'validate' of undefined` on /checkout — 34 users in 48 hours. Did not exist before deploy. Stack trace points to the new promo validation code.
🔴 **Mobile regression**: Checkout form fields shifted 20px down on iOS Safari. Not visible on desktop. Affects ~15% of mobile checkout users.
🟡 **New user behavior**: 8% of users now click "Apply Promo" before entering a code (dead click). The new button styling may make it look like it does something before the field is filled.

### Net assessment:
The deploy fixed the #1 frustration (promo code rage clicks) but introduced a new error and a mobile layout regression. Worth fixing both, but the fix was net positive — 412 fewer frustrated users per week vs 34 users seeing a new error.

Changelog Diff Checklist

DimensionBeforeAfterChange
Top 5 errors[list][list]New? Resolved? Changed frequency?
Top frustrations[list][list]New signals? Resolved? Shifted?
Conversion rateX%Y%Δ = Y-X
Bounce rateX%Y%Δ
Session durationX minY minΔ
Device mixD/M/T%D/M/T%Shift?
Browser errorsPer browserPer browserNew browser-specific issues?
Page traffic shareTop pagesTop pagesAny pages gaining/losing >20%?

Guidelines

  • The deploy window (first hour after deploy) is noisy — caches warming, users getting new bundles, partial rollouts. Exclude it from before/after comparison. Start "after" at +1 hour.
  • New errors are the strongest signal. An error that didn't exist before the deploy is almost certainly deploy-related. Prioritize these.
  • Behavioral changes can be positive. If users are suddenly using a feature more, that might be intentional (better UX) rather than a bug. Investigate before flagging.
  • Some changes are coincidental, not causal. A traffic spike might be a marketing campaign, not the deploy. Cross-reference with campaign-tracker before attributing to the deploy.
  • If you find undocumented changes that look like bugs, flag them immediately. The team might not know. "Heads up: the deploy introduced a TypeError on /checkout that wasn't in the release notes. 34 users affected in 48 hours."
  • For major deploys (50+ files changed), run this skill. For minor deploys (hotfix, config change), deploy-radar is usually sufficient.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.