📘 How do experiments prove a business change worked?

What you’ll learn

1. Why Experiment? Causation, Correlation, and the CounterfactualExplain why controlled experimentation is the most reliable way to establish that a business change causes an outcome rather than merely correlating with it.Business decisions hinge on causal claims, but observational data alone cannot separate the effect of a change from the confounders that travel with it. A controlled experiment manufactures a credible counterfactual by randomly assigning units to treatment and control, so the only systematic difference between groups is the change being tested. This is why randomized experiments are treated as the gold standard for causal inference, a framing made rigorous by R. A. Fisher's principles of randomization, replication, and blocking.
2. Designing a Trustworthy Experiment: Hypotheses, Metrics, and PowerTranslate a business question into a testable hypothesis with a defined success metric, guardrails, and an adequately powered sample size before any data is collected.A trustworthy experiment is designed before it runs. You state a falsifiable hypothesis, choose an Overall Evaluation Criterion that captures success while guardrail metrics protect against hidden harm, and then use the baseline rate, a minimum detectable effect, the significance level, and the desired power to compute the required sample size and run time. Locking these decisions in advance is what prevents the analysis stage from drifting into wishful interpretation.
3. Running Experiments and Reading Results HonestlyInterpret experiment results correctly by distinguishing signal from noise and recognizing the pitfalls that produce false or misleading conclusions.Even a well-designed experiment can be misread. Confidence intervals and p-values quantify uncertainty but are routinely misinterpreted, and practices like peeking, testing many metrics, and aggregating across imbalanced segments inflate false positives or reverse true effects. Sanity checks such as sample-ratio mismatch and a clear-eyed view of statistical versus practical significance separate trustworthy readings from convenient ones.
4. When You Cannot Randomize: Quasi-Experiments and Their LimitsRecognize situations where randomized experiments are infeasible and apply quasi-experimental designs while understanding the stronger assumptions they require.Randomization is sometimes impossible for practical, ethical, or structural reasons, such as evaluating a price change, a regulation, or a network feature. Quasi-experimental designs, including natural experiments and difference-in-differences, recover causal estimates by exploiting comparison groups and timing rather than random assignment. They are powerful but rest on assumptions like parallel trends that, unlike randomization, cannot be guaranteed and must be argued and probed.
5. Build It: A Mini Experiment Design from Question to DecisionProduce a complete, self-consistent experiment plan for a real business question, integrating hypothesis, metrics, power, analysis, and a decision rule into one artifact.This capstone walks you through building a one-page experiment design document, the artifact that experimentation teams use to make a test trustworthy before it launches. You will state the hypothesis, define the OEC and guardrails, set the unit of randomization, compute the required sample size from your MDE and power, pre-commit an analysis and decision rule, and choose between a randomized and a quasi-experimental approach when randomization is constrained. The result is a concrete, reusable template you can apply to any future business decision.

Questions this course answers

Grounded in trusted sources

• Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (Cambridge University Press, 2020), ch. 1
• R. A. Fisher, The Design of Experiments (Oliver & Boyd, 1935)
• Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (Cambridge University Press, 2020), chs. 2 and 17
• Jacob Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. (Lawrence Erlbaum, 1988)
• Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (Cambridge University Press, 2020), chs. 3 and 17-19
• American Statistical Association, 'ASA Statement on Statistical Significance and P-Values,' The American Statistician 70, no. 2 (2016): 129-133

Every Wunder lesson is built from real, reputable sources — never invented.

Related courses