A/B Testing: From Hypothesis to Impact

7 min read · Posted Aug 2025

A crisp A/B test prevents analysis theater. My checklist:

1. Clear hypothesis with expected direction and mechanism.
2. Primary metric + guardrails (e.g., activation ↑, bounce rate not worse than −1%).
3. Pre-computed sample size & test duration.
4. Segment sanity checks (new vs. returning users).

Common pitfalls

• Peeking early → inflate false positives.
• Multiple comparisons without correction.
• Shipping local maxima without follow-ups.