The big ideas of statistics, with the fine print

Statement

In a regular parametric family, the score $U_n(\theta)=\partial_\theta \ell_n(\theta)$ has mean zero and variance $nI(\theta)$. For any unbiased $T$, the Cramér–Rao bound gives $\operatorname{Var}_\theta T \ge \{nI(\theta)\}^{-1}$, and the MLE attains it asymptotically:

The modern form of "efficiency" doesn't need unbiasedness. Le Cam's local asymptotic normality (LAN) says that

Hájek's convolution theorem then shows that every regular estimator's limit is $N(0,I^{-1})$ convolved with independent noise. Hodges' superefficient estimator shows why the regularity requirement is needed: it beats $I^{-1}$ at a single point and pays for it nearby.

Why it's true

Expand $0=U_n(\hat\theta)\approx U_n(\theta)+(\hat\theta-\theta)\,\partial_\theta U_n(\theta)$. The CLT gives $U_n/\sqrt n\rightsquigarrow N(0,I)$, the LLN gives $-\partial_\theta U_n/n\to I$, and Slutsky finishes the argument. The Wald, score and likelihood-ratio statistics are three readings of the same quadratic. They agree to first order and differ at $O(n^{-1/2})$.

For a binomial proportion, the Wald test gives $\hat p\pm z\sqrt{\hat p(1-\hat p)/n}$. Inverting the score test gives the Wilson interval, which evaluates the variance at the null value rather than at $\hat p$:

Statement

Let $\Theta_0\subset\Theta\subseteq\mathbb R^k$ be smooth with $\dim\Theta_0=k_0$, and let $\theta_0$ be an interior point. Under regularity, $\Lambda_n=2\{\sup_\Theta\ell_n-\sup_{\Theta_0}\ell_n\}\rightsquigarrow\chi^2_{k-k_0}$. In general (Chernoff, 1954), the limit is

where $C_\Theta$ and $C_{\Theta_0}$ are the tangent cones at $\theta_0$ and $\|v\|_I^2=v^\top I(\theta_0)v$. At an interior point the cones are linear subspaces, and the difference of squared projection distances is $\chi^2_{k-k_0}$.

Why it's true

By LAN, the log-likelihood ratio is locally $h^\top\Delta_n-\frac12 h^\top I h$. Maximizing a concave quadratic over a cone means projecting the Gaussian $I^{-1}\Delta_n$ onto that cone. For $X_i\sim N(\mu,1)$ with $\mu\ge0$ and $H_0:\mu=0$, the cone is a half-line, so $\Lambda_n = n\max(\bar X,0)^2$ and

This is exactly the situation when testing a variance component $\sigma_b^2=0$ in a mixed model. Comparing against $\chi^2_1$ halves the true size: the test is conservative, and power is lost. In finite samples the Bartlett correction $\Lambda_n/\{1+b/n\}$ reduces the $\chi^2$ approximation error from $O(n^{-1})$ to $O(n^{-2})$.

Statement

Glivenko–Cantelli: $\|F_n-F\|_\infty\to0$ almost surely. Donsker: $\sqrt n(F_n-F)\rightsquigarrow \mathbb B\circ F$ in $\ell^\infty(\mathbb R)$, where $\mathbb B$ is a Brownian bridge. For continuous $F$ the continuous mapping theorem then gives Kolmogorov's distribution-free limit. DKW, with Massart's sharp constant, gives a finite-sample band:

Why it matters

Replace half-lines $\{(-\infty,x]\}$ by a general class $\mathcal F$ and study $\|\mathbb P_n-P\|_{\mathcal F}=\sup_{f\in\mathcal F}|(\mathbb P_n-P)f|$. Uniform laws hold when $\mathcal F$ is small enough, as measured by VC dimension, covering numbers or bracketing entropy. This one idea gives consistency of M-estimators (argmax continuous mapping), the functional delta method (quantiles, bootstrap of smooth functionals), generalization bounds in learning theory, and the Donsker conditions behind semiparametric efficiency (idea 12). Chaining (Dudley's entropy integral) is the main proof technique.

Statement

Let $X\sim N_p(\theta,I_p)$ with loss $\|\hat\theta-\theta\|^2$. The MLE $\hat\theta=X$ is minimax with constant risk $p$, yet for $p\ge3$ it is inadmissible (Stein, 1956). The James–Stein estimator $\hat\theta^{JS}=\big(1-\tfrac{p-2}{\|X\|^2}\big)X$ satisfies

and its positive part $\hat\theta^{JS+}=(1-\tfrac{p-2}{\|X\|^2})_+X$ dominates it in turn (and is itself inadmissible). At $\theta=0$ the risk falls from $p$ to $2$.

Why it's true

Stein's lemma: $\mathbb E[(X_i-\theta_i)g(X)]=\mathbb E[\partial_i g(X)]$. Write $\hat\theta=X+g(X)$. Then $R=p+\mathbb E[\|g\|^2+2\nabla\!\cdot g]$, and SURE is an unbiased risk estimate. With $g=-(p-2)X/\|X\|^2$ we get $\nabla\!\cdot g=-(p-2)^2/\|X\|^2$. Empirical Bayes reading: if $\theta_i\stackrel{iid}{\sim}N(0,\tau^2)$, the Bayes rule is $(1-B)X$ with $B=1/(1+\tau^2)$. Since $\|X\|^2\sim(1+\tau^2)\chi^2_p$ and $\mathbb E[1/\chi^2_p]=1/(p-2)$, the factor $(p-2)/\|X\|^2$ is unbiased for $B$. James–Stein is what you get by estimating the prior from the data (Efron & Morris, 1973).

Why $p\ge3$? Brown (1971) showed that admissibility of generalized Bayes rules is equivalent to recurrence of an associated diffusion. Brownian motion is recurrent in dimensions 1 and 2 and transient in 3 or more.

Statement

Metropolis–Hastings proposes $y\sim q(x,\cdot)$ and accepts with probability

which enforces detailed balance $\pi(x)P(x,y)=\pi(y)P(y,x)$, so $\pi$ is invariant. A $\pi$-irreducible, aperiodic chain satisfies the ergodic theorem. Under geometric ergodicity a Markov chain CLT also holds, with asymptotic variance $\sigma_f^2\,(1+2\sum_{k\ge1}\rho_k)$. The effective sample size is $\mathrm{ESS}=N/(1+2\sum_k\rho_k)$.

Optimal scaling

For random-walk Metropolis on product targets in dimension $d$ with proposal $N(0,\ell^2 I_d/d)$, each coordinate converges to a Langevin diffusion as $d\to\infty$. Its speed $h(\ell)=2\ell^2\Phi(-\ell\sqrt{J}/2)$ is maximized when the acceptance rate is about 0.234 (Roberts, Gelman & Gilks, 1997). Steps that are too small give high acceptance but a slow random walk. Steps that are too large almost always get rejected. The number of steps to mix scales as $O(d)$ for RWM, $O(d^{1/3})$ for MALA, and $O(d^{1/4})$ for HMC.

Statement

Given $m$ p-values, let $V$ be the number of false rejections and $R$ the total number of rejections. Then $\mathrm{FDR}=\mathbb E[V/(R\vee1)]$. The Benjamini–Hochberg step-up procedure rejects the hypotheses with the $\hat k$ smallest p-values, where

Why it's true

For each null $i$, write $\mathbf 1\{i\text{ rejected}\}/R=\sum_k \mathbf 1\{p_i\le qk/m,\ R=k\}/k$. Conditional on the other p-values, $R$ is a function of $p_i$ that is constant on the event that $i$ is rejected (the leave-one-out trick). Uniformity of $p_i$ then gives $\mathbb E[V_i/R]=q/m$, and summing over the $m_0$ nulls gives $\pi_0 q$. The same proof works under PRDS dependence (Benjamini & Yekutieli, 2001). Under arbitrary dependence, replace $q$ by $q/\sum_{i\le m}i^{-1}$.

Two-groups model. If $p_i\sim\pi_0\,\mathrm{U}(0,1)+(1-\pi_0)F_1$, BH is approximately thresholding the tail-area $\mathrm{Fdr}(t)=\pi_0t/G(t)$, where $G$ is the marginal CDF of the p-values. The local fdr, $\pi_0f_0/f$, is the posterior null probability (Efron). In the histogram lab you can see both: a flat floor at height $\pi_0$ and a spike at zero.

Statement

Let $X_i=\theta_i+\varepsilon_i$ with $\varepsilon_i\sim N(0,1)$ for $i=1,\dots,m$, and report $i^*=\arg\max_i X_i$. Then $\mathbb E[X_{i^*}-\theta_{i^*}]\gt0$. When the $\theta_i$ are equal, the bias grows like $\sqrt{2\log m}$, and $X_{i^*}\pm1.96$ undercovers badly. There are four principled responses:

(i) Sample splitting. Select on one half of the data and infer on the other. It's valid, and you pay in power. (ii) Conditional (selective) inference. Condition on the selection event. For polyhedral events (lasso, marginal screening), $\eta^\top y$ given selection is a truncated Gaussian (Lee et al., 2016; Fithian, Sun & Taylor, 2014). (iii) Simultaneous inference. Cover every model you might have selected (PoSI; Berk et al., 2013). (iv) Bayes. If the prior is correct, the posterior is unaffected by data-dependent selection (Dawid, 1994). Tweedie's formula estimates the needed shrinkage from the marginal density alone:

The lab uses the Gaussian special case, with $\hat\tau^2=\max(0,\overline{X^2}-1)$ estimated from all $m$ observations. That is honest empirical Bayes, not an oracle.

Statement

The bootstrap approximates the law of $r_n\{T(F_n)-T(F)\}$ by the conditional law of $r_n\{T(F_n^*)-T(F_n)\}$. If $T$ is Hadamard differentiable at $F$ tangentially to the support of the Donsker limit, then the functional delta method and the bootstrap Donsker theorem (Giné & Zinn, 1990) imply consistency. For asymptotically pivotal statistics (studentized), the bootstrap is second-order correct: the coverage error is $O(n^{-1})$, compared with $O(n^{-1/2})$ for the normal approximation (Hall, 1992). This is the theoretical basis for the bootstrap-$t$ and BCa intervals.

A canonical failure

Take $X_i\sim\mathrm U(0,\theta)$ and $\hat\theta=X_{(n)}$. Then $n(\theta-\hat\theta)/\theta\rightsquigarrow\mathrm{Exp}(1)$, a continuous law. But the resampled maximum equals $\hat\theta$ whenever the largest observation is drawn:

The bootstrap law has an atom at 0 that the true law doesn't, so it is inconsistent. The fix is the $m$-out-of-$n$ bootstrap ($m\to\infty$, $m/n\to0$) or subsampling, which needs only that a limit law exists (Politis & Romano, 1994).

Statement

For the kernel density estimator $\hat f_h(x)=\frac1{nh}\sum_iK\big(\frac{x-X_i}h\big)$ with a second-order kernel, the bias is $\tfrac12h^2\mu_2(K)f''(x)+o(h^2)$ and the variance is $R(K)f(x)/(nh)+o((nh)^{-1})$. Integrating,

Stone (1980, 1982) showed this is optimal. Over Hölder classes of smoothness $\beta$ in $d$ dimensions, the minimax rate for squared error is $n^{-2\beta/(2\beta+d)}$. Holding $\beta$ fixed, the exponent goes to zero as $d$ grows, and that is the precise content of the "curse of dimensionality." Nonparametric rates are bought with structure: additivity, sparsity, single-index models, or low intrinsic dimension.

Choosing $h$

Least-squares cross-validation minimizes $\int\hat f_h^2-\frac2n\sum_i\hat f_{h,-i}(X_i)$, which is unbiased for $\mathrm{MISE}(h)-\int f^2$. In a single sample it differs from $\mathrm{ISE}(h)-\int f^2$ by a term of order $n^{-1/2}$ that hardly depends on $h$, so its level drifts while its minimizer tracks $h^*$. Even that minimizer is notoriously noisy: $\hat h/h^*-1=O_p(n^{-1/10})$ (Hall & Marron, 1987). Plug-in selectors (Sheather & Jones, 1991) trade some of that variance for bias. Silverman's rule of thumb assumes $f$ is Gaussian and oversmooths multimodal densities, which the lab shows.

Statement

$\hat\beta_\lambda=\arg\min_\beta\ \frac1{2n}\|y-X\beta\|_2^2+\lambda\|\beta\|_1$. Suppose $\beta$ is $s$-sparse, the noise is Gaussian with variance $\sigma^2$, and $X$ satisfies a restricted eigenvalue condition with constant $\kappa$. Taking $\lambda\asymp\sigma\sqrt{\log p/n}$ gives, with high probability (Bickel, Ritov & Tsybakov, 2009),

The $\log p$ factor is the whole price of not knowing the support. Exact support recovery needs more. You need a beta-min condition, and the irrepresentable condition (Zhao & Yu, 2006; Wainwright, 2009):

Why it's true

The choice $\lambda\ge2\|X^\top\varepsilon/n\|_\infty$, which holds with high probability at the $\sqrt{\log p/n}$ scale by a union bound over $p$ Gaussians, forces the error into a cone $\|\Delta_{S^c}\|_1\le3\|\Delta_S\|_1$. On that cone the restricted eigenvalue condition makes the quadratic loss strongly convex. Coordinate descent with soft-thresholding $S_\lambda(z)=\operatorname{sign}(z)(|z|-\lambda)_+$ and warm starts computes the whole path cheaply (Friedman, Hastie & Tibshirani, 2010). The lab uses it.

Statement

Let $X\in\mathbb R^{n\times p}$ have iid $N(0,1)$ entries, so $\Sigma=I_p$, and let $S=X^\top X/n$. If $p/n\to\gamma\in(0,1]$, the empirical spectral distribution of $S$ converges almost surely to the Marchenko–Pastur law:

Every population eigenvalue equals 1, yet the sample eigenvalues fill $[a,b]$. The largest one converges to $b$ with Tracy–Widom fluctuations of order $n^{-2/3}$ (Johnstone, 2001). Spiked model: a population spike $\ell$ produces an outlier sample eigenvalue only if $\ell\gt1+\sqrt\gamma$, the BBP phase transition. In that case the outlier sits at $\ell(1+\gamma/(\ell-1))$, and the squared cosine between the sample and population eigenvectors converges to $\frac{1-\gamma/(\ell-1)^2}{1+\gamma/(\ell-1)}$.

Why it matters

PCA is inconsistent when $p\asymp n$. Plug-in Markowitz portfolios underestimate risk. Hotelling's $T^2$ loses power and then can't be computed. Covariance shrinkage, either linear (Ledoit & Wolf, 2004) or nonlinear, which inverts the MP map, is the principled response. The proofs use the Stieltjes transform $m(z)=\int\frac{dF(x)}{x-z}$, which satisfies a quadratic self-consistency equation.

Statement

Observe $O=(X,A,Y)$ and target $\tau=\mathbb E[Y(1)-Y(0)]$. Under consistency, unconfoundedness $Y(a)\perp\!\!\!\perp A\mid X$ and overlap $0\lt e(X)\lt1$, the effect is identified by both $\mathbb E[\mu_1(X)-\mu_0(X)]$ and $\mathbb E\big[\frac{AY}{e(X)}-\frac{(1-A)Y}{1-e(X)}\big]$. The efficient influence function (Hahn, 1998) is

and AIPW is the plug-in sample mean of $\varphi+\tau$. Its bias is a product of nuisance errors:

Why it matters

If either $\hat e$ or $\hat\mu$ is consistent, the product goes to zero, which is model double robustness. If both converge faster than $n^{-1/4}$, the product is $o_p(n^{-1/2})$, and AIPW attains the semiparametric efficiency bound $\operatorname{Var}\varphi$ even when the nuisances come from flexible machine-learning fits. That is rate double robustness, and it is the foundation of double/debiased ML. Cross-fitting removes the Donsker condition on the nuisance estimators (Chernozhukov et al., 2018). The same "Neyman orthogonality" principle drives TMLE.