📊 Statistics II

What you’ll learn

1. The Question Worth AskingShift from hypothesis testing to estimation — reporting effect sizes with intervals — and recognise that no statistical machinery can repair a broken study design.'Is there an effect?' is nearly the least useful question to ask of data: with a large enough sample, a trivial effect is highly significant, while an important effect in a small study may not reach significance at all. Reporting an estimate with a confidence interval preserves the effect's size, its precision, and the reader's ability to judge relevance — everything a significance verdict discards. The course's central claim is stated here: every technique ahead is a machine turning data into a claim about an unobserved world, and no machine can repair a design that was broken before the data arrived.
2. The Sampling Distribution: The Engine Under EverythingExplain the sampling distribution as the distribution of a statistic across hypothetical repeated studies, and derive the consequences of SE = σ/√n and the Central Limit Theorem.The sampling distribution — the distribution of a statistic across infinitely many hypothetical repetitions of the same study — is the object every inferential technique reasons about, even though a researcher only ever draws from it once. It is far narrower than the data itself, with standard error σ/√n, because averaging cancels extremes; and by the Central Limit Theorem it is approximately normal for large enough n regardless of the shape of the underlying data, which is why a small set of formulas serves so many fields. The square root in σ/√n governs the economics of research: halving uncertainty requires quadrupling the sample, and no analysis escapes it.
3. What a Confidence Interval Actually SaysConstruct and correctly interpret confidence intervals, and diagnose the four standard misreadings.A confidence interval extends roughly 2 standard errors either side of an estimate, and its 95% is a property of the procedure — across hypothetical repetitions, 95% of such intervals would contain the truth — not of the single interval computed, in which the randomness has already been spent. Four misreadings recur in published work: treating the interval as a probability statement about the truth, confusing it with the spread of the data, confusing it with a prediction interval for future studies (roughly 83% of replications fall inside a 95% CI), and reading an interval containing zero as proof of no effect. The width matters more than whether zero is inside: a narrow interval near zero is informative, while a wide one is a confession of ignorance, and the significance verdict makes the two look identical.
4. Comparing Two Groups, HonestlyEstimate the difference between two groups with an interval, and rank the t-test's assumptions by their real-world consequences.Comparing two groups means estimating their difference with an interval, not issuing a verdict; the estimate is a subtraction, and the work lies in combining two sources of noise. Because the true σ is never known, Gosset's t-distribution widens intervals to pay for using the sample's s instead — a substantial penalty at n = 5 and a negligible one by n = 100. Of the t-test's three assumptions, normality is forgiving thanks to the CLT and unequal variances is solved for free by Welch's test, but independence is fatal and routinely unchecked: clustered data has less information than its n claims, making every reported number wrong in the direction that flatters the researcher.
5. Power, and Why Small Significant Studies MisleadExplain statistical power as a design decision and account for Type M and Type S errors as the mechanism behind the replication crisis.Power is the probability of detecting a real effect, and its importance lies less in the misses than in what an underpowered study does when it succeeds: because only estimates far from zero clear the significance threshold, a noisy study's significant results are precisely the ones that overshot, making the published estimate systematically inflated (a Type M error) and, at very low power, sometimes wrong in sign (Type S). This inverts a common instinct — significance from a small sample is a warning, not a triumph. The 2015 Open Science Collaboration replication of 100 psychology studies found 36% significant against 97% in the originals, with replication effects around half the size, exactly the signature this mechanism predicts, and power analysis must therefore precede the study rather than follow it.
6. The Garden of Forking PathsDistinguish the multiple-comparisons problem from the garden of forking paths, and explain why preregistration, not correction, is the remedy.Twenty tests on pure noise yield at least one significant result about 64% of the time, and corrections like Bonferroni address that directly — but the deeper problem needs no multiple tests at all. Gelman and Loken's garden of forking paths describes an honest researcher who runs one analysis yet chose it, after seeing the data, from dozens of equally defensible alternatives she would have taken had the data differed; no correction applies because the alternative tests were never run and cannot be counted. Simmons, Nelson and Simonsohn demonstrated the stakes in 2011 by using then-standard practices to obtain a significant, published finding that the Beatles' 'When I'm Sixty-Four' made listeners a year and a half younger (p = .040), with simulations showing such flexibility can drive false-positive rates to 61%. The remedy is temporal — preregistering the analysis plan before data exists — which does not forbid exploration but requires labelling it as such.
7. Regression Is a Conditional MeanInterpret regression as a model of the conditional mean of Y given X, state the slope in comparative rather than causal language, and identify regression to the mean.Regression estimates the average value of Y among cases sharing a given X, and its one structural assumption — that those conditional means fall on a line — buys enormous efficiency by replacing a separate mean at every X with a slope and an intercept. The slope licenses a comparison, not a consequence: cases differing by one unit in X differ on average by 2.3 units in Y, which is a description of how the world varies rather than a prediction of what happens when it is pushed. Regression to the mean, the phenomenon that gave the method its name, follows from measurement being signal plus re-rolling noise, and it means anything selected for being extreme will drift toward average without any intervention at all — a trap that manufactures false confidence across entire industries.
8. Reading Regression Output CriticallyRead a regression table column by column, interpret R² and its limits, and diagnose model failures from residual plots.A regression table's useful information lies in the columns readers skip: the coefficient must be read in units before significance, and the standard error — from which coefficient ± 2 SE gives the interval — distinguishes a precise finding from a vague one that carries the same asterisk. R² measures how much of Y's variance the model tracks, which is largely a fact about Y's variability rather than the model's correctness; what counts as high is entirely field-dependent, and because R² never decreases when a predictor is added, it cannot compare models of different sizes. Residual plots are the essential diagnostic — curves reveal nonlinearity, fans reveal heteroskedasticity that invalidates every standard error, and single distant points can silently drag the whole line because least squares squares the distances — a lesson Anscombe made unforgettable in 1973 with four visibly different datasets sharing every summary statistic.
9. What 'Controlling For' Really DoesInterpret multiple regression's 'controlling for' precisely, and identify unmeasured confounding, residual confounding, lack of overlap, and mediator/collider bias as its failure modes.Multiple regression estimates how Y differs with X among cases sharing the values of the other predictors — a genuine like-with-like comparison, but one bounded by what was measured. 'Controlling for income and education' rules out only the confounders that happened to be columns in the dataset, leaves residual confounding behind whenever a control is measured crudely, and will silently extrapolate from its linear assumption when no comparable cases exist. Worse, adding controls is not monotonically good: controlling for a mediator subtracts out the very effect being measured, and controlling for a collider manufactures association from nothing. Which variables to control for is therefore not a statistical question at all — the fit improves either way and the table looks identical — but a question only a causal model brought from outside the data can answer.
10. Confounders, Colliders, and Simpson's ParadoxExplain Simpson's paradox and collider bias, and recognise that choosing what to condition on is a causal question that data cannot answer.In 1973 UC Berkeley admitted about 44% of male and 35% of female graduate applicants, yet Bickel, Hammel and O'Connell reported in Science (1975) that few individual departments departed significantly from expected female admission rates and that properly pooled data showed a small bias favouring women — because women disproportionately applied to departments that were more competitive for everyone. Both figures are arithmetically correct, and the mathematics cannot say which answers your question; that depends on an imported claim about what causes what. Collider bias runs the opposite way from confounding: conditioning on a variable caused by both X and Y — restaurant survival, hospital admission, celebrity status — manufactures association between genuinely independent causes, which is why controlling for every measurable variable is a bad instinct rather than a safe default.
11. Randomization: The One Thing That Actually WorksExplain why randomization is uniquely able to address unmeasured confounding, and delimit what it does and does not guarantee.Confounding, selection, collider bias, and Simpson's paradox are one problem in four costumes — units with different X differ in other ways too — and every statistical patch shares one limitation: it handles only the confounders that were measured. Randomization is the sole exception, because a coin's assignment is independent of every characteristic of the subject, including ones nobody has named, so no confounder can correlate with treatment. It does not make groups identical; it converts imbalance from unmeasurable bias into quantified variance already reflected in the confidence interval. And it buys internal validity only: differential dropout can break a randomized trial after the fact, and a clean trial on undergraduates establishes a clean causal claim about undergraduates, since external validity is beyond the reach of any statistical procedure.
12. When You Can't RandomizeApply quasi-experimental designs when randomization is unavailable, identify the untestable assumption each one buys, and explain how triangulation establishes causation from observational evidence.Most consequential questions cannot be randomized, so the practical task is finding variation in X that is defensibly unrelated to everything else — a coin nature already flipped. Natural experiments rely on arbitrary real-world assignment; difference-in-differences compares changes rather than levels, netting out common shocks at the cost of an unprovable parallel-trends assumption; instrumental variables isolate the variation an outside nudge explains, resting on the untestable claim that the instrument reaches Y only through X; regression discontinuity compares people either side of an arbitrary cutoff, buying credibility at the price of speaking only about people at the cutoff. Smoking shows observational evidence can still be conclusive: Fisher's genetic-confounding objection was answered not by one study but by triangulation — dose-response, cessation effects, cross-country consistency, animal experiments, and an identified mechanism — many imperfect designs whose flaws do not overlap.
13. Reading a Study Like a StatisticianEvaluate a published study by interrogating design, sampling, effect size, interval width, analytic flexibility, and causal warrant, in that order.Reading a study well means asking the design questions first, because nothing downstream repairs them: what was the design, who is missing from the sample, how big is the effect in real units, how wide is the interval, was the analysis preregistered, what makes any causal claim causal, and has it replicated. The recurring red flags each map to a chapter — a large effect from a small sample is a Type M error advertising itself, an unplanned subgroup finding is multiple comparisons, 'after controlling for' is a claim bounded by the dataset's columns, and a success story about treating the worst performers is regression to the mean. The course's argument closes here: every technique hit the same ceiling, because statistics can say what data is consistent with but not what it means — meaning depends on how the data came to exist, which is a fact about the design and not about the arithmetic.

Questions this course answers

Grounded in trusted sources

• David Freedman, Robert Pisani & Roger Purves, Statistics (4th ed., W. W. Norton)
• Andrew Gelman, Jennifer Hill & Aki Vehtari, Regression and Other Stories (Cambridge University Press, 2020)
• Judea Pearl & Dana Mackenzie, The Book of Why (Basic Books, 2018)
• Miguel Hernán & James Robins, Causal Inference: What If (Chapman & Hall/CRC, 2020)
• Open Science Collaboration, 'Estimating the reproducibility of psychological science', Science 349:6251 (2015) — https://www.science.org/doi/10.1126/science.aac4716
• Joseph Simmons, Leif Nelson & Uri Simonsohn, 'False-Positive Psychology', Psychological Science 22:11 (2011) — https://journals.sagepub.com/doi/10.1177/0956797611417632
• Peter Bickel, Eugene Hammel & J. William O'Connell, 'Sex Bias in Graduate Admissions: Data from Berkeley', Science 187:4175 (1975) — https://www.science.org/doi/10.1126/science.187.4175.398
• Andrew Gelman & John Carlin, 'Beyond Power Calculations: Assessing Type S and Type M Errors', Perspectives on Psychological Science 9:6 (2014)

Every Wunder lesson is built from real, reputable sources — never invented.

Related Math courses