The big ideas of statistics
Statistics is the science of learning from data when the world won't sit still. Here are twelve ideas that made that possible. Each one comes with a small experiment you can run in your browser.
Variation
Every measurement is one draw from a distribution.
The law of large numbers
Averages settle down, even though individual events don't.
The central limit theorem
Why the bell curve shows up nearly everywhere.
Random sampling
How you choose the data matters more than how much you collect.
Confidence intervals
An honest estimate always comes with a margin of error.
Hypothesis testing
Could pure chance have produced what you saw?
Bayesian reasoning
Start from what you believed, then update on the evidence.
Regression to the mean
Extreme results tend to be followed by more ordinary ones.
Confounding & Simpson's paradox
A trend in the whole can reverse in every part.
Randomized experiments
A coin flip can rule out every confounder at once.
Chance has structure
A single coin flip can't be predicted, but ten thousand flips are very predictable. The first three ideas explain how order emerges from randomness, and they are the foundation for everything after.
Variation is the signal, not the noise
Every measurement is one draw from a distribution. To understand data, look at the whole shape.
Measure the height of a thousand people and you get a thousand different answers. Statistics starts by accepting this. The variation isn't a nuisance to average away: it is the thing to be studied. The pattern of values, called the distribution, says what is typical, what is rare and what is impossible.
Summary numbers compress a distribution. The mean is its balance point. The median is the middle value. The standard deviation is the typical distance from the mean. For a symmetric shape like heights, mean and median agree. For a lopsided one like incomes, a few very large values pull the mean far above the median, and "average income" tells a different story than "typical income."
History
In 1835 Adolphe Quetelet measured soldiers' chest sizes and found the same bell shape that astronomers saw in their measurement errors. He imagined an "average man". A generation later Francis Galton turned this around: he argued that the spread around the average was the interesting part, because it is what heredity and selection act on.
Watch out
Don't summarize a skewed or two-humped distribution with only its mean. Plot it first.
The law of large numbers
Single events are unpredictable, but averages over many of them are not.
Flip a fair coin and you have no idea what will happen. Flip it ten thousand times and you can bet the share of heads will be very close to 50%. The law of large numbers says that the average of many independent trials converges to the true expected value.
Casinos, insurers and opinion polls depend on this. A casino doesn't know who will win tonight, but it knows almost exactly what fraction of all bets it will keep over a year.
The convergence is slow. The typical error shrinks like 1/√n, so you need four times the data to halve it. The shaded funnel in the chart shows this rate.
History
Jacob Bernoulli spent about twenty years proving the first version, which he called his "golden theorem." It was published in Ars Conjectandi in 1713, eight years after he died.
Watch out — the gambler's fallacy
The law does not say a run of heads will be "corrected" by extra tails. The coin has no memory. The proportion settles because a streak gets diluted by many later flips, not because it gets cancelled. In the readout, the raw gap between heads and tails often keeps growing even as the proportion converges.
The central limit theorem
Add up enough random things and you get a bell curve, almost whatever you started with.
This is probably the most surprising result in the subject. Take any population, whether it is flat, skewed or U-shaped. Draw a sample of size n and compute its mean. Do that again and again. The sample means form a distribution of their own, and as n grows that distribution becomes a normal (bell) curve, centered on the true mean with spread σ/√n.
This explains why the bell curve is everywhere. Heights, measurement errors and test scores are each the sum of many small, independent influences. It also explains why statistical methods built on the normal curve work for data that isn't normal at all, because they are usually applied to averages.
History
Abraham de Moivre found the bell curve in 1733 as an approximation to coin-flip counts. Pierre-Simon Laplace generalized the result in 1810. Galton built the board at the top of this page to demonstrate it, and called the law "the supreme law of Unreason."
Watch out
"Large enough" depends on the population. A very skewed or spiky parent needs a much bigger n, and some heavy-tailed distributions (where the variance is infinite) never converge to a bell curve.
Learning from a sample
We almost never see the whole population. We see a sample and try to reason back to everything else. These four ideas cover how to do that honestly: how to collect the sample, how uncertain the result is, and how to tell a real effect from luck.
Random sampling beats big data
How a sample was chosen matters more than how big it is.
It seems odd that interviewing 1,000 people can tell you how 300 million will vote. It works because of random sampling. If every person has a known chance of being picked, the sample is a small but fair copy of the population, and ideas 2 and 3 tell you how precise it is.
If the selection is not random, for example people who answer online polls or customers who leave reviews, the sample can be biased in ways that no amount of data will fix. Adding more data shrinks random error but leaves systematic error untouched. You end up very precise about the wrong answer.
History
In 1936 the Literary Digest mailed ballots to millions of people, drawn largely from telephone directories and car registrations, and got about 2.4 million back. It predicted a landslide for Alf Landon. George Gallup used a far smaller sample, chosen more carefully, and correctly called it for Franklin Roosevelt, who won 46 of 48 states. Jerzy Neyman had laid out the theory of random sampling two years earlier, in 1934.
Watch out
A huge dataset collected for a different purpose, such as app users, social media posts or patients who chose a treatment, is still a convenience sample. Ask who is missing and why.
2,500 voters. The 30% in the wealthy district (left, shaded) lean against the measure, and most of the town supports it. True support: .
Confidence intervals
An estimate without a margin of error is incomplete.
Your sample mean is almost never exactly the true mean. So report a range: the estimate plus or minus a margin of error. A 95% confidence interval comes from a procedure that, if repeated on many fresh samples, would capture the true value 95% of the time.
The "95%" describes the method, not any single interval. The chart makes this concrete. Each line is one experiment's interval. Most catch the true mean (the vertical line) and some miss. Over many runs, the share that catch it settles near the confidence level.
Two levers set the width. Higher confidence gives wider intervals, and more data gives narrower ones (again by 1/√n).
History
Jerzy Neyman formalized confidence intervals in 1937. They became the standard way to report uncertainty in science, polling ("±3 points") and medicine.
Watch out
"There's a 95% chance the true value is in this interval" is a common misreading in the frequentist framework, where the true value is fixed and the interval is what varies. (The Bayesian credible interval in idea 7 does mean that.) Also, an interval only covers random error. It says nothing about bias from idea 4.
Hypothesis testing & the p-value
Before calling an effect real, ask how often pure chance would produce it.
Say plants given a new fertilizer grew 2 cm more than plants without it. Is the fertilizer working, or did the better plants end up in that group by luck? A significance test makes the skeptic's case precise. Assume the null hypothesis, that the fertilizer does nothing, and ask how surprising your data would be in that world.
A permutation test answers by brute force. If the treatment does nothing, the labels "fertilizer" and "control" are arbitrary, so shuffle them. Each shuffle shows a difference chance alone could produce. The p-value is the fraction of shuffles at least as extreme as what you actually observed.
History
Ronald Fisher described the "lady tasting tea" experiment and the permutation logic behind it in The Design of Experiments (1935). Jerzy Neyman and Egon Pearson added the framework of error rates and power in 1933. In 2016 the American Statistical Association issued an unusual formal statement warning about widespread misuse of p-values.
Watch out
A p-value is not the probability that the null is true, and "significant" doesn't mean "important." Run enough tests and some will come up significant by chance, as the jelly-bean demo below shows.
Scientists test whether jelly beans cause acne, one color at a time. In this simulation, no color has any effect.
Bayesian reasoning
Evidence doesn't speak for itself. It updates what you already believed.
A disease affects 1% of people. A test catches 90% of cases and wrongly flags 9% of healthy people. You test positive. What are the chances you have the disease? Most people, including many doctors, guess about 90%. The real answer is about 9%.
The healthy group is so much larger that its 9% false alarms outnumber the true cases. Bayes' theorem combines the prior (how common the disease is) with the likelihood (how well the test discriminates) to get the posterior (what you should believe now). It's easiest to see with counts of people, as in the grid.
The same logic drives Bayesian statistics more broadly. Treat an unknown quantity, such as a coin's bias, as uncertain. Describe that uncertainty with a distribution, then update it as data arrives. Each posterior becomes the prior for the next observation.
History
The Reverend Thomas Bayes's essay was published after his death, in 1763, by his friend Richard Price. Laplace independently developed and applied the idea. After falling out of favor for much of the 20th century, Bayesian methods came back once computers made them practical (with Markov chain Monte Carlo, around 1990).
Watch out — base-rate neglect
Ignoring how rare something is leads to wildly overconfident conclusions. Screening tests, fraud detectors and facial recognition all have this problem.
Relationships and causes
The most useful questions involve relationships: does X go with Y, and does X cause Y? These are also where data most easily misleads. The next three ideas cover the traps and the main way out of them.
Regression to the mean
Extreme results are partly luck, and luck doesn't usually repeat.
Very tall parents tend to have children who are tall, but less tall than they are. Very short parents have children who are short, but less short. This isn't a biological force pulling people toward average. It happens whenever two measurements are imperfectly correlated. An extreme value is usually part real and part good luck, and the luck part doesn't carry over.
This is where the word regression comes from, and it is the basis of the regression line. The best prediction of Y from X is pulled toward the average by a factor equal to the correlation r.
History
Francis Galton noticed the pattern in sweet-pea seeds and then in human heights, and published "Regression towards mediocrity in hereditary stature" in 1886. Karl Pearson later turned Galton's ideas into the correlation coefficient.
Watch out
Regression to the mean makes useless treatments look effective. Patients seek help when symptoms are at their worst, and they would improve somewhat anyway. Daniel Kahneman described flight instructors who believed praise made pilots worse and criticism made them better. After an exceptionally good or bad landing, the next one was simply more ordinary.
Confounding & Simpson's paradox
Correlation isn't causation, and sometimes the whole points one way while every part points the other.
A confounder is a hidden third factor that affects both things you're comparing. Ice cream sales and drownings rise together because both go up in summer, not because ice cream causes drowning. Confounders can create correlations out of nothing, hide real ones, or even reverse them.
That reversal is Simpson's paradox. In the kidney-stone data below, treatment A has the better success rate for small stones and for large stones, yet B looks better overall. Doctors tended to give A to the harder, large-stone cases. Stone size is the confounder.
Which answer is correct, combined or split, isn't decided by the numbers alone. It depends on the causal story. Here stone size affects both the treatment choice and the outcome, so you should compare within sizes.
History
Edward Simpson described the paradox in 1951. A famous case is UC Berkeley's 1973 graduate admissions. Men were admitted at a higher overall rate, but most departments admitted women at equal or higher rates. Women had applied more often to the most competitive departments.
Watch out
"Controlling for" variables is not a cure-all. Adjusting for the wrong variable, such as one caused by the treatment itself, can create bias. Thinking about causes has to come before the arithmetic.
| Stone size | Treatment A | Treatment B |
|---|---|---|
| Small stones | 81/87 = 93% | 234/270 = 87% |
| Large stones | 192/263 = 73% | 55/80 = 69% |
| All patients | 273/350 = 78% | 289/350 = 83% |
Randomized experiments
Let a coin decide who gets the treatment, and every confounder, including ones you never thought of, is balanced at once.
Idea 9 left a hard problem. You can only adjust for confounders you know about and have measured. Health-conscious people both take vitamins and exercise, eat well and see doctors. An observational study of vitamins can't fully separate these.
The fix is random assignment. If a coin flip decides who gets the treatment, then on average the two groups match on everything: age, habits, genes and factors nobody has named. Any difference that remains is caused by the treatment or by chance, and idea 6 can measure the chance part.
This is why the randomized controlled trial is the gold standard in medicine, and why technology companies run A/B tests.
History
Fisher developed randomization while running crop experiments at Rothamsted in the 1920s. The 1948 Medical Research Council trial of streptomycin for tuberculosis, designed by Austin Bradford Hill, is often called the first modern randomized clinical trial. Donald Rubin's "potential outcomes" framework (1974) gave causal inference a precise mathematical language.
Watch out
Randomization can't always be used, because it can be unethical or impossible, as with smoking. Fields like epidemiology and economics developed careful designs (natural experiments, instrumental variables, difference-in-differences) to approximate it. These rely on assumptions that randomization doesn't need.
Each study has 200 people. A hidden trait, health-consciousness, raises the outcome score. In the observational study, health-conscious people are more likely to choose the vitamin.
The computer age
Cheap computing changed statistics. Instead of deriving every answer by hand, we can simulate, resample and fit flexible models. That made new mistakes possible too. These last two ideas are the core of modern data science and machine learning.
Overfitting & the bias–variance tradeoff
A model that explains your data perfectly has probably memorized its noise.
A flexible enough model can pass through every data point. That sounds ideal, but the data contain noise, and a model that fits the noise will predict new data badly. This is overfitting.
There are errors in both directions. A model that is too simple (high bias) misses real structure. A model that is too flexible (high variance) changes wildly with each new sample. The best model sits in between, and you find it by checking predictions on data the model hasn't seen. Error on the training data always falls as complexity grows. Error on test data falls and then rises.
This is why cross-validation (Stone, 1974), information criteria like AIC (Akaike, 1973) and regularization such as the lasso (Tibshirani, 1996) are so important. They all measure or penalize flexibility so that a model is judged on what it predicts, not on how well it fits.
Watch out
Any data that influenced your choices, including which features to use, which model or which settings, is no longer a fair test set. Leaking test data into model choices quietly brings overfitting back.
The bootstrap
To learn how much your answer could have varied, resample your own data.
Ideas 3 and 5 give formulas for the uncertainty of a mean. What about a median, a correlation, or the output of a complicated algorithm? Often there's no formula.
Bradley Efron's answer was the bootstrap, published in 1979. Treat your sample as a stand-in for the population. Draw a new sample of the same size from it, with replacement, so some points appear twice and some not at all. Recompute your statistic, and repeat thousands of times. The spread of those results estimates how much the statistic would vary if you could repeat the whole study.
The name comes from "pulling yourself up by your bootstraps": the method gets the uncertainty from the same data as the estimate. It works remarkably well in practice and has a solid theoretical basis.
History
Efron's 1979 paper in the Annals of Statistics was titled "Bootstrap methods: another look at the jackknife." It became one of the most widely used ideas in modern statistics, and Efron received the International Prize in Statistics for it in 2019.
Watch out
The bootstrap can't fix a bad sample, because it only resamples what you have. It also struggles with very small samples and with statistics such as the maximum.
You asked 15 coworkers how long their commute takes (minutes).
Three threads run through all twelve ideas
Aggregation
Individual cases are noisy, but averages are stable (ideas 1–3). Combining information is the basic move in statistics, and it is why an error that shrinks like 1/√n shows up everywhere.
Design beats analysis
How data is collected, by random sampling or random assignment, decides which conclusions are possible at all (ideas 4, 9, 10). Clever analysis can't rescue a biased design.
Quantified humility
Intervals, p-values, posteriors, test-set error and the bootstrap all describe how wrong you might be (ideas 5–7, 11–12). Statistics is mostly a discipline of honest uncertainty.
Further reading
- Stephen Stigler, The Seven Pillars of Statistical Wisdom (2016). A short, elegant history of the field's core ideas.
- Andrew Gelman & Aki Vehtari, "What are the most important statistical ideas of the past 50 years?" JASA (2021). An expert's list of the modern ideas.
- David Salsburg, The Lady Tasting Tea (2001). The people behind the ideas, written as stories.
- David Spiegelhalter, The Art of Statistics (2019). The best general-audience introduction, built around real questions.
- Richard McElreath, Statistical Rethinking. A popular, intuition-first course in Bayesian modeling and causal thinking.
- Bradley Efron & Trevor Hastie, Computer Age Statistical Inference (2016). How computing changed the field, from the bootstrap to deep learning.