The big ideas of data analysis and modeling

In 1973 the statistician Francis Anscombe published four small data sets of eleven points each. All four have the same mean of x (9), the same mean of y (7.5), the same correlation (0.82) and the same fitted regression line, y = 3 + 0.5x. A regression table for them is identical, line by line.

Plotted, the first is a plain noisy line. The second is a smooth curve, so a straight line is the wrong model. The third is a perfect line with one outlier that tilts the fit. The fourth has a single point at x = 19 that alone creates the whole slope. Each needs a different response: leave it, fit a curve, check the outlier, or admit you have one data point.

The breakdown point of a statistic is the fraction of the data you'd have to corrupt to push it to any value you like. For the mean it is 1 in n: change one number and the average follows. For the median it is nearly one half: you must corrupt half the sample before it can be dragged away.

Robust doesn't mean the outlier is wrong. It may be a typo, a sensor glitch or a data-entry unit error, or it may be the most important observation you have. A robust summary tells you what the bulk looks like, and the gap between robust and ordinary summaries tells you how much one or two values are doing.

Donald Rubin's taxonomy has three cases. Missing completely at random (MCAR): a coin decided who skipped the question. Throwing away incomplete rows loses information but not accuracy. Missing at random (MAR): missingness depends on something you observed, such as age. You can correct for it by modeling on the observed variable. Missing not at random (MNAR): missingness depends on the unseen value itself, as when high earners decline to state their income. The data alone can't tell you this has happened.

The lab below has a survey where income rises with age. Because no mechanism is visible in the data you keep, you have to argue from how the data were collected.

Ordinary regression assumes the x values are exact and puts all the noise in y. When x is measured with error, the fitted slope is multiplied by the reliability ratio: the share of the variance in the observed x that is signal rather than noise. Half the variance is noise, and you recover half the slope. This is called regression dilution or attenuation.

Noise in y only widens the scatter. Noise in x biases the result, and a bigger sample makes you more sure of the wrong slope. The same thing happens to controls: if you adjust for a confounder you measured badly, you only partly remove its effect.

Legendre published the method of least squares in 1805 and Gauss claimed to have used it since 1795, while predicting the orbit of the asteroid Ceres. The recipe is to choose the intercept a and slope b that make the sum of squared residuals as small as possible. Squaring makes big misses count more, and it makes the answer a closed-form formula.

A good fit leaves residuals that look like shapeless noise around zero. If the residuals bend, you've used a straight line on a curve. If they fan out, the noise grows with the size of x. If one point sits far from the rest, it may be steering the line. The residual plot is the most valuable diagnostic in regression because it shows what the model didn't explain.

A line says "each step adds the same amount." Prices, populations, infection counts and incomes more often say "each step multiplies by the same factor." Taking logarithms turns multiplication into addition, so such data fit a straight line on the log scale and the slope becomes a growth rate.

Logs also fix a second problem. When errors are proportional to the level, the raw scale shows a cone of scatter that violates the assumptions of ordinary least squares, while on the log scale the scatter is even. Fit in the scale where the noise is steady, then convert predictions back.

Training error rewards memorization. The fix is to hold some data back: fit on one part and score on the other. Cross-validation repeats this so every row serves once as the test set, and it gives an error estimate with less luck in it.

The estimate is only honest if the held-out rows are really unseen. Leakage is any way that information from the test rows reaches the model: selecting features on the full data before splitting, normalizing with statistics from all rows, or having the same patient in both folds. Leakage is subtle because the code runs and the score looks excellent.

The lab uses data where the labels are coin flips and the features are noise, so the right accuracy is 50%. Pick the "best" features using every row first and then cross-validate, and you'll see a brilliant score for a model that knows nothing.

Least squares is unbiased: on average it recovers the true coefficients. But with 15 correlated predictors and 30 rows, any single fit swings wildly, and the unbiased answer is far from the truth in nearly every sample. Ridge regression adds a penalty on the size of the coefficients, which shrinks them all toward zero and stabilizes them.

Too little penalty and the fit is noisy. Too much and every coefficient collapses toward zero and the model underfits. The best λ sits between, and test error against λ is a U shape. The lasso replaces the squared penalty with absolute values, which sets some coefficients exactly to zero and so selects variables.

Most classifiers give a score, and a threshold turns it into a yes or no. Move the threshold up and you catch fewer false alarms but also miss more cases. Sensitivity is the share of real cases flagged. Specificity is the share of non-cases cleared. Neither tells you what a flag means.

That is the positive predictive value, and it depends on prevalence. If one person in a thousand has the condition, even a test that is 95% sensitive and 95% specific produces about 50 false alarms for every true one. This is Bayes' rule, and it is the usual reason that screening and fraud flags disappoint.

The k-means algorithm alternates two steps: assign each point to its nearest center, then move each center to the mean of its points. It always converges, quickly, to something. Where it ends up depends on the starting centers, and the result is only a local optimum.

The total within-cluster squared distance always falls as k grows, down to zero when every point is its own cluster, so it can't choose k for you. The usual heuristic is the "elbow": the value of k where adding another cluster stops paying for itself. On data with clear groups the elbow is sharp. On a single smooth blob, there is none, and any partition is just an arbitrary slicing.

Imagine a cloud of points shaped like a tilted ellipse. Height and weight of adults, or two highly correlated test scores, are like that. The long axis of the ellipse is a better single summary than either original variable: projecting each point onto it keeps as much of the spread as any one direction can.

That axis is the first principal component. The second is perpendicular to it and carries the most remaining variance, and so on. Mathematically they are the eigenvectors of the covariance matrix, and the eigenvalues give the variance each carries. With 100 correlated measurements, the first three components might hold 90% of the information.

A random walk adds a random step each period and never returns to a mean. Any two of them will drift, and over a stretch of time they'll move together or apart purely by chance. The usual correlation test assumes independent observations, but consecutive points of a random walk are nearly the same, so there are far fewer independent pieces of evidence than rows.

Granger and Newbold showed in 1974 that regressing one independent random walk on another produces "significant" results far more often than the 5% a test promises. The cure is to difference: look at the period-to-period changes. If the changes in two series move together, there's a relationship. If only the levels do, there probably isn't.