🔬 Experimental Design and Statistical Methods

What you’ll learn

1. The Real Enemy Is Being FooledFrame the course: all methods exist to stop us fooling ourselves, defending against chance, confounds, and bias.Feynman's principle — you are the easiest person to fool — organizes everything. Chance fakes patterns, confounds masquerade as effects, and bias tips our measurements. Statistics fights chance, design fights confounds, and blinding fights bias.
2. The Question and Its VariablesTurn vague questions into testable hypotheses and identify independent, dependent, and confounding variables.A testable hypothesis is specific and falsifiable and names its variables: the independent (changed), the dependent (measured), and confounds (uncontrolled third factors). The confound is the central villain because you can't anticipate them all.
3. Comparison and the Control GroupExplain why a control group is essential and tell the story of Lind's 1747 scurvy trial.An effect is a difference, so you need a comparison group identical except for the tested variable. James Lind's 1747 controlled comparison aboard HMS Salisbury showed citrus cured scurvy long before vitamin C was known.
4. Randomization: Beating the Confounds You Never ImaginedExplain randomization and why it balances even unknown confounds, per Fisher.Random assignment spreads known and unknown confounds evenly across groups, which careful matching cannot do. Fisher's The Design of Experiments (1935) and the 'lady tasting tea' established randomization as the basis for valid inference.
5. Blinding: Taking the Thumb Off the ScaleExplain placebo control and single vs. double blinding as defenses against bias.Belief itself heals (placebo effect), so controls get an identical inert pill; single-blinding hides assignment from subjects, double-blinding from the measurer too. The randomized, double-blind, placebo-controlled trial stacks all three defenses.
6. Samples, Variability, and Why Size MattersExplain representative sampling, sampling variability, the square-root law, and statistical power.Samples must be representative, not just large; size can't fix bias. Fair samples still wobble (sampling variability), and uncertainty shrinks only with the square root of n, so halving it needs 4x the data. Underpowered studies miss real effects and inflate flukes.
7. Describing Data: Center, Spread, and ShapeTeach describing data honestly with center (mean/median/mode), spread, and distribution shape.The mean is distorted by outliers; the median better describes skewed data. Spread (standard deviation) is often the more important half. Distribution shape matters, and the normal curve arises via the central limit theorem when many small effects add up.
8. The p-value: What It Does and Doesn't MeanDefine the p-value precisely and correct its four most common misinterpretations.The p-value is the probability of data at least as extreme as observed if the null were true. It is NOT the probability the hypothesis is true, 'no effect' when non-significant, meaningful at exactly 0.05, or a measure of effect size.
9. Effect Size, Confidence, and the Replication CrisisDistinguish significance from effect size, introduce confidence intervals, and use the replication crisis as the payoff.Significance asks 'real?'; effect size asks 'big enough?'; a confidence interval gives a plausible range for the true effect. The 2015 Open Science Collaboration replicated only ~36% of 100 psychology studies (by significance), with effects about half as large — validating rigor over p-hunting.

Questions this course answers

Grounded in trusted sources

• Design of experiments — Wikipedia: https://en.wikipedia.org/wiki/Design_of_experiments
• James Lind & the first controlled trial — James Lind Library: https://www.jameslindlibrary.org/articles/james-lind-and-scurvy/
• The Design of Experiments (Fisher) — Wikipedia: https://en.wikipedia.org/wiki/The_Design_of_Experiments
• Blinded experiment / placebo — Wikipedia: https://en.wikipedia.org/wiki/Blinded_experiment
• The ASA statement on p-values — Wasserstein & Lazar (2016): https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1154108
• Estimating the reproducibility of psychological science — Open Science Collaboration, Science 2015: https://www.science.org/doi/10.1126/science.aac4716

Every Wunder lesson is built from real, reputable sources — never invented.

Related Science courses