agentleFS
dkpapadopoulosdkpapadopoulos/auto-claude-skills/skills/outcome-review/SKILL.md

outcome-review

Skill6 starsChanged 54 days agoMITView on GitHub

Use when reviewing a shipped feature's real-world outcome in the LEARN phase — checking adoption, error, or experiment metrics after release, validating ship-time hypotheses, or deciding follow-up work — querying PostHog and creating gated follow-up Jira work

1 other project uses a version of this.

Adopted by

Most recently taken up by

The file

Show its rules and text

We split this skill into the rules it gives an agent

20 rules. About 0 are common, 10 are distinctive, and 10 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈10

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 10

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Check which MCP tools are available:

    line 12
  2. Very specific:

    If you have access to query-run, get-experiment, list-experiments, get-feature-flag, or create-annotation as MCP tools, use Tier 1.

    line 15
  3. Distinctive:

    If no PostHog MCP tools are available, ask the user to provide metrics directly:

    line 18
  4. Very specific:

    Check for a learn baseline file in ~/.claude/.skill-learn-baselines/:

    line 27
  5. Very specific:

    If found, use the baseline's shipped_at, ship_method, hypotheses, and jira_ticket fields

    line 29
  6. Very specific:

    If ship_method is "pull_request", verify the PR was actually merged before proceeding (check pr_url via gh pr view)

    line 30
  7. Distinctive:

    If no baseline found, ask the user:

    line 31
  8. Very specific:

    If the baseline has non-null hypotheses, use each hypothesis's metric field to target specific events/properties instead of generic …

    line 41
  9. Distinctive:

    Check experiment results if applicable:

    line 42
  10. Distinctive:

    Check feature flag status:

    line 45
  11. Distinctive:

    Ask the user to share metrics from their dashboards

    line 52
  12. Very specific:

    If the baseline has hypotheses, present each hypothesis and its metric to the user: "For H1 ([description]), I need the current value of …

    line 53
  13. Distinctive:

    Ask about any observed regressions or improvements

    line 54
  14. Very specific:

    Add the chosen cause to the row (or list it under the table) for each non-Confirmed hypothesis (this includes Partially confirmed and …

    line 95
  15. Very specific:

    When hypotheses is null in the baseline (or no baseline found): skip this section entirely.

    line 97
  16. Distinctive:

    Wait for the user's choice.

    line 109
  17. Very specific:

    If the baseline lacks jira_ticket or the feature's parent ticket is unknown, find it first: call search(cloudId, "<feature name>") — the …

    line 115
  18. Very specific:

    Only after explicit approval: createJiraIssue to create the ticket.

    line 118
  19. Distinctive:

    If follow-up work was identified:

    line 127
  20. Distinctive:

    If the loop is closed:

    line 130

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Outcome Review

Query analytics for a shipped feature, synthesize an outcome report, and optionally create follow-up Jira work. Entered independently after shipping — days or weeks later.

Step 1: Detect Available Tools

Check which MCP tools are available:

Tier 1 — PostHog MCP: If you have access to query-run, get-experiment, list-experiments, get-feature-flag, or create-annotation as MCP tools, use Tier 1.

Tier 2 — Manual Metrics: If no PostHog MCP tools are available, ask the user to provide metrics directly:

"I don't have PostHog MCP access. Please share any of the following:

  • Dashboard screenshots or metric summaries
  • Adoption numbers, funnel data, or error rates
  • Experiment results if applicable
  • Any specific concerns about the shipped feature"

Step 2: Identify the Feature

  1. Check for a learn baseline file in ~/.claude/.skill-learn-baselines/:
    • List files, match by feature name or branch name from the user's prompt
    • If found, use the baseline's shipped_at, ship_method, hypotheses, and jira_ticket fields
    • If ship_method is "pull_request", verify the PR was actually merged before proceeding (check pr_url via gh pr view)
  2. If no baseline found, ask the user:

    "Which feature should I review? Please provide the feature name, branch name, or Jira ticket ID."

Step 3: Gather Metrics

Tier 1 (PostHog MCP available):

  1. Query adoption metrics via query-run with HogQL:
    • Event counts for the feature's key events since shipped_at
    • Compare to the period before shipping (same duration)
    • If the baseline has non-null hypotheses, use each hypothesis's metric field to target specific events/properties instead of generic adoption queries
  2. Check experiment results if applicable:
    • list-experiments to find experiments linked to the feature
    • get-experiment for results, significance, and variant performance
  3. Check feature flag status:
    • get-feature-flag for rollout percentage and targeting rules
  4. Check error rates:
    • query-run for error events associated with the feature

Tier 2 (Manual):

  1. Ask the user to share metrics from their dashboards
    • If the baseline has hypotheses, present each hypothesis and its metric to the user: "For H1 ([description]), I need the current value of [metric]. What is it?"
  2. Ask about any observed regressions or improvements
  3. Synthesize from what the user provides

Step 4: Synthesize Outcome Report

Present a structured report:

Outcome Report

Feature: [name] | Shipped: [date] | Branch: [name]

Adoption: [metrics summary — event counts, trend direction, comparison to pre-ship baseline]

Quality: [error rates, regression indicators]

Experiments: [results if applicable — significance, winning variant, effect size]

Assessment: One of:

  • Positive — Metrics improved, no regressions. Close the loop.
  • Regression detected — [specific metric] degraded by [amount]. Investigate.
  • Inconclusive — Insufficient data. Revisit in [N] days.
  • Mixed — [positive metrics] improved but [negative metrics] regressed. Judgment call.

Hypothesis Validation (when baseline has non-null hypotheses):

IDHypothesisMetricBaselineTargetActualStatusCause
H1[description][metric][baseline][target][measured value][status][cause if status ≠ Confirmed, else —]

Status values:

  • Confirmed — Actual meets or exceeds target
  • Not confirmed — Actual does not meet target
  • Inconclusive — Insufficient data, or validation window has not elapsed
  • Partially confirmed — Directionally correct but below target threshold

Cause (required for any non-Confirmed hypothesis) — diagnose WHY before recommending. A low metric does not by itself mean the feature failed; mistaking the cause is how a working feature gets reverted. Classify the cause and let it drive the recommendation:

  • instrumentation-broken — the metric pipeline is wrong, not the feature (e.g. event renamed/never fired, 0 events while manual QA confirms the flow works). → Fix tracking and re-measure; do NOT change the feature.
  • adoption-gap — the feature works but is under-exposed or undiscovered (e.g. flag at 5% rollout; healthy within the exposed cohort). → Increase rollout/discoverability, then re-measure before judging the feature.
  • product-miss — fully exposed and correctly measured, users still don't convert. → Iterate on or roll back the feature.
  • inconclusive-data — not enough signal yet. → Extend the window; do not act.

Add the chosen cause to the row (or list it under the table) for each non-Confirmed hypothesis (this includes Partially confirmed and Inconclusive, not only Not confirmed), and make the recommendation follow from it.

When hypotheses is null in the baseline (or no baseline found): skip this section entirely. Fall back to the existing generic metrics flow with no behavioral change.

Recommendations: Specific next actions based on the assessment — and, for each non-Confirmed hypothesis, consistent with its diagnosed Cause above.

Step 5: User Decision Gate

Present the report and ask:

"Based on this outcome review, would you like me to:

  1. Close the loop — no follow-up needed
  2. Create follow-up Jira tickets — I'll draft tickets for the recommended actions (requires your approval before creation)
  3. Investigate further — dig deeper into a specific metric or regression"

Wait for the user's choice.

Step 6: Follow-Up Actions

If "Create follow-up tickets" (and Atlassian Rovo MCP available):

  1. If the baseline lacks jira_ticket or the feature's parent ticket is unknown, find it first: call search(cloudId, "<feature name>") — the Rovo cross-system search returns the original ticket alongside any linked Confluence docs in one call. Fall back to searchJiraIssuesUsingJql only if search returns no matches.
  2. Draft the ticket(s) — title, description, acceptance criteria, priority.
  3. Present each draft to the user for approval.
  4. Only after explicit approval: createJiraIssue to create the ticket.
  5. addCommentToJiraIssue on the original ticket with the outcome summary.

If Atlassian Rovo MCP unavailable:

"I don't have Atlassian Rovo MCP access. Here are the recommended follow-up tickets — please create them manually: [formatted ticket descriptions]"

Step 7: Transition

If follow-up work was identified:

"If follow-up work is needed, invoke Skill(auto-claude-skills:product-discovery) or Skill(superpowers:brainstorming) to begin the next cycle."

If the loop is closed:

"Outcome review complete. The feature loop is closed."

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.