ai-feature-experiment-designer
Skill1 starsChanged 23 days agoMITView on GitHub
Designs A/B tests for non-deterministic Artificial Intelligence (AI) features: an offline-evaluation gate, variance-aware sample sizing, a primary metric with guardrails for cost per task, latency, and override rate, novelty-effect handling, and pre-registered ship / iterate / kill decision rules. Use when testing a new model, prompt, or agent behavior in production; comparing two Large Language Model (LLM) providers or versions; deciding whether an AI feature actually moves a business metric; or an online test result contradicts offline evaluations.
In this repository41 other files it gives its agents.
- ai-activation-optimizerSkill
- ai-content-quality-gateSkill
- ai-launch-playbookSkill
- ai-pricing-margin-modelerSkill
- ai-product-positioningSkill
- ai-search-visibility-optimizerSkill
- agent-autonomy-level-mapperSkill
- ai-failure-state-designerSkill
- ai-usability-test-plannerSkill
- ai-ux-auditSkill
- human-in-the-loop-designerSkill
- prompt-surface-designerSkill
- uncertainty-and-explainability-patternsSkill
- ai-agent-financial-analystSkill
- burn-rate-runway-plannerSkill
- capital-allocation-frameworkSkill
- dcf-valuation-builderSkill
- financial-statement-analyzerSkill
- saas-cohort-analyzerSkill
- competitive-response-modelerSkill
- mechanism-design-plannerSkill
- nash-equilibrium-solverSkill
- negotiation-strategistSkill
- repeated-game-strategistSkill
- ai-decision-engineSkill
- ai-product-strategySkill
- ai-stakeholder-translatorSkill
- customer-interview-synthesizerSkill
- jobs-to-be-done-extractorSkill
- opportunity-solution-treeSkill
- product-discovery-coachSkill
- product-roadmap-prioritizerSkill
- blue-ocean-strategy-canvasSkill
- go-to-market-strategySkill
- gtm-engineeringSkill
- market-sizingSkill
- okr-cascade-plannerSkill
- porters-five-forces-analyzerSkill
- pricing-strategy-analyzerSkill
- strategic-landscape-mapperSkill
- swot-tows-analyzerSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 7 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 45
Run before any traffic is exposed.
- Distinctive: line 67
Prefer behavioral signals (accepted, edited, copied, regenerated) over voluntary ratings, which suffer from selection bias.
- Distinctive: line 81
When randomizing by user but measuring per task, inflate for clustering:
- Distinctive: line 91
Run at least 2 full weekly cycles, and longer if the feature is used weekly rather than daily
- Distinctive: line 128
Write decision rules before the test starts
- Distinctive: line 132
Use voluntary thumbs-up rate as the primary metric
- Distinctive: line 133
Stop the test early at the first significant reading without a sequential method
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
AI Feature Experiment Designer
The same input can produce different outputs, costs, and latencies - an AI experiment has to measure all three before it declares a winner.
What this skill is
An experiment-design workflow for features whose behavior is stochastic and whose quality, cost, and speed move together. It pairs offline evaluations (cheap, fast, controlled) with online A/B tests (real users, real outcomes), sizes samples for the extra variance that model outputs add, and pre-registers guardrails so a variant that lifts engagement while doubling inference cost or eroding trust does not ship by accident.
What it solves
- Shipping a new prompt or model because offline evaluation scores rose, with no evidence users or revenue benefit
- Underpowered tests that read noise from output variance as a win
- Variants that win the primary metric while silently raising cost per task or latency
- Novelty spikes in the first week mistaken for durable lift
- Peeking at results daily and stopping the test at the first significant reading
- Metrics such as thumbs-up rate that are biased by who chooses to rate
When to invoke
- Swapping or upgrading the model behind a production feature
- Testing prompt, retrieval, or agent-planning changes with real users
- Launching an AI feature and needing proof of incremental business impact
- Choosing between providers on quality, cost, and latency together
- Reconciling an online result that disagrees with offline evaluations
Phase 1: Frame the hypothesis and unit
| Field | Example |
| Change | New retrieval step plus updated system prompt |
| Hypothesis | Raises task success because answers cite the customer's own documents |
| Unit of randomization | User or account (not request - avoids one user seeing both variants) |
| Population | Active users who trigger the feature at least once in the window |
| Exposure point | First time the feature is invoked after assignment |
Randomize by user or account whenever the experience accumulates (memory, history, learned preferences). Request-level randomization is acceptable only for stateless, single-turn features.
Phase 2: Offline evaluation gate
Run before any traffic is exposed.
| Check | Pass condition |
| Task-quality evaluation | Variant is not worse than control on a held-out set of 200+ realistic tasks, scored by rubric or graded model with human spot checks |
| Regression suite | No new failures on known critical cases |
| Safety and policy | No increase in policy violations on adversarial prompts |
| Cost and latency | Projected cost per task and 95th percentile (p95) latency within guardrail budgets |
Offline evaluation decides whether a variant is safe to test. It does not decide whether to ship.
Phase 3: Metric stack
| Role | Metric | Notes |
| Primary | Task success rate, accepted-output rate, or downstream conversion | One metric, tied to user or business value |
| Secondary | Retention of feature users at day 7 and 28, tasks per user | Directional support |
| Guardrail: cost | Inference cost per successful task | Includes retries and tool calls |
| Guardrail: speed | p95 time to first token and to complete answer | |
| Guardrail: trust | Override rate (user edits, regenerates, or discards output), complaint rate | |
| Guardrail: safety | Flagged outputs per 1,000 tasks |
Prefer behavioral signals (accepted, edited, copied, regenerated) over voluntary ratings, which suffer from selection bias.
Phase 4: Variance-aware sample sizing
Model outputs add within-user variance, so use the observed variance of the primary metric from the current production feature, not textbook values.
n per arm = 2 x (z_alpha/2 + z_beta)^2 x sigma^2 / MDE^2- z_alpha/2 = 1.96 for 95% confidence; z_beta = 0.84 for 80% power
- sigma^2 = per-user variance of the primary metric (for a rate p, use p x (1 - p) adjusted by the design effect below)
- MDE = Minimum Detectable Effect, stated in absolute terms
When randomizing by user but measuring per task, inflate for clustering:
Design effect = 1 + (m - 1) x ICCwhere m is the average tasks per user and ICC is the Intra-Class Correlation of the metric within users. Multiply n by the design effect. Use Controlled-experiment Using Pre-Experiment Data (CUPED) variance reduction with a pre-period covariate when available.
Phase 5: Duration and novelty handling
- Run at least 2 full weekly cycles, and longer if the feature is used weekly rather than daily
- Plot the treatment effect by day since first exposure; if it decays, report the stabilized effect from weeks 2 onward
- Hold out a small long-term control (for example 5%) for major AI changes to measure durable impact over 4-8 weeks
- Freeze the model version and prompt for both arms during the test; a silent provider update invalidates the comparison
Phase 6: Pre-registered decision rules
| Outcome | Decision |
| Primary significant and positive, all guardrails within limits | Ship |
| Primary positive, one guardrail breached (for example cost per task +25%) | Iterate on cost or latency, re-test |
| Primary flat, guardrails improved (cheaper or faster at same quality) | Ship as an efficiency win |
| Primary flat, guardrails flat or worse | Kill or return to offline work |
| Primary negative or safety guardrail breached | Kill and run a post-mortem |
Set guardrail limits numerically before launch (for example: cost per successful task no more than +10%, p95 latency no more than +500 milliseconds, override rate no more than +2 points). Use a sequential testing method if the team will look before the planned end.
Phase 7: Readout
Report the effect with confidence intervals, the guardrail table, the novelty curve, segment cuts (new versus existing users, heavy versus light users, task type), and the offline-to-online agreement: did offline evaluation predict the online result? Track that agreement over time to learn how much to trust offline evaluations.
Output
- Experiment brief: hypothesis, unit, population, exposure point
- Offline evaluation gate results
- Metric stack with numeric guardrail limits
- Sample size calculation including design effect and planned duration
- Pre-registered decision table
- Readout template with novelty curve and offline-to-online agreement
Operating rules
Always
- Gate every online test behind an offline evaluation pass
- Include cost per successful task and p95 latency as guardrails
- Randomize at the user or account level for stateful features
- Pin model and prompt versions for the whole test window
- Write decision rules before the test starts
Never
- Ship on offline evaluation scores alone
- Use voluntary thumbs-up rate as the primary metric
- Stop the test early at the first significant reading without a sequential method
- Compare variants measured in different weeks or on different model versions
- Report a first-week lift as durable without checking the novelty curve
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

