Large-sample reasoning for methods research
From repeated sampling to a first-order approximation
Before reading: Means, variance, elementary calculus, and independent sampling. Use A bridge between mathematics and statistical inference as a notation reference when needed.
1 Start with a random sample and a fixed target
Suppose independent people have complete measurements Y_1,\ldots,Y_n from the same population. We assume these observations are independent and identically distributed (i.i.d.), with population mean \mu=E[Y] and finite, positive variance \sigma^2=\operatorname{Var}(Y). The population SD is \sigma>0; this notation does not assume normally distributed measurements. The target \mu stays fixed as we repeat sampling. The sample mean \bar Y=n^{-1}\sum_iY_i is an estimator: its value changes from one random sample to another.
The first question is whether this procedure learns the intended population quantity as the sample grows. An estimator is consistent for a target if its probability of missing that target by more than any fixed positive tolerance tends to zero. For the sample mean, the law of large numbers (LLN) supplies precisely this result. It connects an average of observations to the expectation under the population that produced them.
To see the connection, write the estimation error as an average of centered measurements:
\bar Y-\mu=\frac1n\sum_{i=1}^n(Y_i-\mu).
Each centered measurement has expectation zero. The LLN says their average approaches zero in probability, establishing consistency of \bar Y for \mu. Thus, for a sufficiently large sample, a fixed-size estimation error becomes unlikely. This does not require each new observation to move the estimate closer to \mu, and it does not make the estimate equal to \mu in a finite sample.
Consistency is a statement about the procedure and the population actually sampled. If we average only observed outcomes from a selected subset, the LLN may establish consistency for E[Y\mid R=1]. It does not turn that quantity into the eligible-population mean E[Y]. The target distinction from the previous guide still matters.
After consistency comes a second question: how much does the estimator fluctuate around its limit, and how can we describe that uncertainty? The central limit theorem (CLT) supplies a scale and a distributional approximation under additional conditions. Consistency alone supplies neither.
2 The probability tools and what they do
The notation below separates three tasks: establishing a limit, describing a sampling distribution, and comparing the sizes of random errors. Throughout, probabilities refer to repeated sampling under the stated population law.
2.1 Convergence in probability
We write
X_n\xrightarrow{p}c.
This reads “X_n converges in probability to c.” For a fixed number c, the definition requires, for every fixed \varepsilon>0,
\Pr(|X_n-c|>\varepsilon)\longrightarrow0.
Convergence in probability controls departures from a fixed quantity. For example, \bar Y\xrightarrow{p}\mu means that the chance of a sample mean more than 0.1 units from \mu tends to zero; the same must hold for every other fixed positive tolerance. The distribution of \bar Y concentrates increasingly near \mu. The definition does not prescribe how quickly this happens.
2.2 Almost sure convergence: a stronger, pathwise property
You will also encounter
X_n\xrightarrow{\mathrm{a.s.}}c.
This reads “X_n converges almost surely to c.” With the whole sequence defined on a common probability space, it means
\Pr\!\left(\lim_{n\to\infty}X_n=c\right)=1.
Think of drawing one infinite sequence of observations and computing an estimate from its first n entries as n grows. Each realization produces a numerical sequence of estimates: a sample path. Almost sure convergence says that, except on a set of paths with probability zero, this numerical sequence converges in the ordinary sense. For each fixed positive tolerance, all estimates after some path-dependent sample size stay within that tolerance. The estimates can still fluctuate along the way.
This is stronger than convergence in probability. The latter controls the chance of a departure at sample size n; it does not by itself guarantee that a single path makes only finitely many fixed-size departures. Almost sure convergence implies convergence in probability, but the converse need not hold.
The stronger property is useful when the claim concerns an entire sequence:
- Strong consistency. The strong LLN shows that the sample mean converges almost surely to \mu for i.i.d. observations with E[|Y|]<\infty. This describes what happens along almost every growing dataset, rather than only giving a probability bound at each sample size.
- Eventual control of errors. The first Borel–Cantelli lemma says that if \sum_n\Pr(E_n)<\infty, then the failure events E_n occur only finitely often with probability one. Applying this to errors exceeding each tolerance 1,1/2,1/3,\ldots is one way to prove almost sure convergence. This summable bound is stronger than merely having \Pr(E_n)\to0.
- Long-run behavior of random algorithms. Some stochastic-approximation and martingale-convergence proofs establish that an iterative procedure converges along almost every random trajectory. Those results require their own stability and integrability conditions.
For the inference arguments in these guides, we will use consistency in probability, negligible remainders expressed through o_p, and CLTs in distribution. We will not need almost sure convergence as a proof step. It is a useful stronger conclusion when available, but even almost sure consistency alone supplies neither a root-n rate nor a normal sampling approximation.
2.3 Convergence in distribution, or law
We write
X_n\xrightarrow{d}X.
This reads “X_n converges in distribution (or in law) to X.” Let F_X(t)=\Pr(X\le t) be the cumulative distribution function (CDF) of X. The definition requires, at every continuity point t of F_X,
\Pr(X_n\le t)\longrightarrow F_X(t).
In words, at each such threshold t, the probability of a repeated-sample outcome below that threshold approaches the probability specified by the limiting law.
Convergence in distribution describes a limiting law. It need not concentrate at one number. For example, \sqrt n(\bar Y-\mu)\xrightarrow{d}N(0,\sigma^2) says that the scaled estimation errors have a distribution approaching a normal law with variance \sigma^2. Their unscaled errors shrink, but multiplying by \sqrt n reveals fluctuations that remain visible.
Convergence in law also does not mean that two random variables take nearly equal values. If X_n and X are independent standard normals, their distributions already agree for every n, yet X_n-X has the same N(0,2) distribution for every n. Its probability of exceeding a fixed positive tolerance in absolute value stays positive, so there is no convergence in probability of X_n to that particular X. Convergence in probability implies convergence in distribution; the reverse implication holds when the limiting law is a point mass at a constant, but not in general.
2.4 Big-O and little-o for random errors
Let a_n>0 be a deterministic scale, such as n^{-1/2}. The statement X_n=O_p(a_n) means that X_n/a_n is bounded in probability. For every tail-probability tolerance \varepsilon>0, there are a finite M>0 and an n_0 such that
\sup_{n\ge n_0}\Pr\!\left(\frac{|X_n|}{a_n}>M\right) <\varepsilon.
After rescaling, sufficiently large values remain uniformly unlikely for large n. The bound M can depend on \varepsilon, but not on n. This is a probability bound, not a claim that every possible realization is bounded by M.
The stronger statement X_n=o_p(a_n) means X_n/a_n\xrightarrow{p}0. Equivalently, for every fixed \delta>0,
\Pr\!\left(\frac{|X_n|}{a_n}>\delta\right)\longrightarrow0.
For example, n^{-1/2} is O_p(n^{-1/2}) but not o_p(n^{-1/2}): dividing by that scale gives 1. In contrast, n^{-1} is o_p(n^{-1/2}), because the ratio is n^{-1/2}\to0. Random errors obey the same scale comparison, with convergence understood in probability.
Useful product rules are
\begin{aligned} O_p(a_n)O_p(b_n)&=O_p(a_nb_n),\\ O_p(a_n)o_p(b_n)&=o_p(a_nb_n). \end{aligned}
These rules do not require independence. In particular, a remainder that is merely O_p(n^{-1/2}) need not disappear after multiplication by \sqrt n. To remove it from a first-order limit, we generally need o_p(n^{-1/2}).
2.5 Population error for a fitted function
Later we will measure a regression fit by \|\hat m-m\|_2^2=E[(\hat m(Z)-m(Z))^2]. Here the expectation averages over a fresh population covariate Z, holding the fitted function fixed. The norm is still random across training samples. Thus \|\hat m-m\|_2\xrightarrow{p}0 describes consistency in population prediction error; it is not a statement that training error is small. This distinction matters when a proof contains functions learned from data.
3 Two statements about the same sample mean
Return to the i.i.d. measurements with mean \mu and variance \sigma^2. The LLN and CLT answer different questions about the same estimator:
\begin{gathered} \bar Y\xrightarrow{p}\mu,\\ \sqrt n(\bar Y-\mu)\xrightarrow{d}N(0,\sigma^2). \end{gathered}
The first establishes consistency: where the estimator settles as information grows. The second describes its fluctuations on the root-n scale. For sufficiently large n, the sampling distribution of \bar Y is approximately N(\mu,\sigma^2/n). This approximation concerns the sample mean; the individual measurements can have a nonnormal distribution.
3.1 Measurement variability and estimation uncertainty
The population standard deviation \sigma describes differences between individual measurements. The standard error of \bar Y describes differences between sample means obtained in repeated studies of size n. These are different random objects: one measurement Y versus an average \bar Y.
Independence explains why averaging reduces the second kind of variation:
\begin{aligned} E[\bar Y]&=\mu,\\ \operatorname{Var}(\bar Y)&=\frac{1}{n^2}\sum_{i=1}^n\operatorname{Var}(Y_i)\\ &=\frac{\sigma^2}{n},\\ \operatorname{SE}(\bar Y)&=\sqrt{\operatorname{Var}(\bar Y)}\\ &=\frac{\sigma}{\sqrt n}. \end{aligned}
The covariance terms vanish because the observations are independent. These mean and variance identities are exact under our assumptions; the CLT supplies the approximate normal shape of the sampling distribution.
For example, suppose a feature has population mean 50 and population SD 10, in the feature’s measurement units:
| Number of independent people | SD of an individual measurement | SE of the sample mean |
|---|---|---|
| 25 | 10 | 2 |
| 100 | 10 | 1 |
| 400 | 10 | 0.5 |
In a study of 100 people, individual measurements still vary with SD 10. Across repeated studies of 100 people, the means vary around 50 with SD 1. Collecting more people makes the population mean easier to estimate; it does not make the people themselves more alike. Quadrupling the sample size halves the SE, while the population SD stays the same.
3.2 Replace the unknown population variance
In an actual study, \sigma^2 is usually unknown. For n\ge2, estimate it using the sample variance:
\begin{aligned} s_Y^2&=\frac{1}{n-1}\sum_{i=1}^n(Y_i-\bar Y)^2,\\ \widehat{\operatorname{SE}}(\bar Y)&=\frac{s_Y}{\sqrt n}. \end{aligned}
The sample SD s_Y estimates the spread of individual measurements. Dividing by \sqrt n turns it into an estimated SE for their mean. Reporting s_Y as an SE would confuse population heterogeneity with uncertainty about \mu.
There is a consistency argument behind this substitution. Write
s_Y^2=\frac{n}{n-1} \left\{\frac1n\sum_iY_i^2-\bar Y^2\right\}.
The LLN gives n^{-1}\sum_iY_i^2\xrightarrow{p}E[Y^2] and \bar Y\xrightarrow{p}\mu. Continuous mapping and n/(n-1)\to1 then give s_Y^2\xrightarrow{p}E[Y^2]-\mu^2=\sigma^2. Thus the estimated contribution variance approaches the population variance.
Finally, combine the CLT with this consistency result:
\begin{aligned} \frac{\bar Y-\mu}{s_Y/\sqrt n}&=\frac{\sqrt n(\bar Y-\mu)}{s_Y}\\ &\xrightarrow{d}N(0,1). \end{aligned}
This last step uses Slutsky’s theorem, stated below. It justifies using an estimated standard error in a large-sample interval or test. Merely knowing that the mean is consistent would not justify that step.
A finite population mean without finite variance can still support an LLN while defeating this particular CLT and variance calculation. Clustered measurements require the independent sampling unit and its contribution to be reconsidered. The theorem must match the problem, not just the formula used by software.
4 Core working theorems
These are the versions used throughout the guides. Each result has a particular job; a proof usually combines several rather than invoking “large samples” as a single justification. For the sample-average results, continue with the i.i.d. observations Y_1,\ldots,Y_n from a population distribution P.
Let f be a fixed function applied to each observation. We will compare the sample average n^{-1}\sum_{i=1}^n f(Y_i) with its population expectation E_P(f(Y)). The subscript P makes explicit the population distribution underlying the expectation written E earlier. For the mean example, take f(Y_i)=Y_i, so the average is \bar Y and the expectation is \mu. In the LLN and CLT below, f is fixed, rather than selected using those same observations. We will first write these results in full, then introduce shorthand for the two kinds of average.
Scope of notation and assumptions. The sample-mean notation Y_i, \mu, and \sigma^2 carries through its applications. Each theorem states the conditions it needs; these can be weaker than the finite-variance assumptions used for that example. Auxiliary variables and sequences introduced within a theorem are local to that statement, and their assumptions do not carry into the next theorem.
4.1 Law of large numbers: establish consistency
If E_P(|f(Y)|)<\infty, the weak LLN gives
\frac1n\sum_{i=1}^nf(Y_i)\xrightarrow{p}E_P(f(Y)).
The sample average consistently estimates its population expectation. Apply this separately to Y and Y^2 to begin the variance-consistency argument above. The second application requires E[Y^2]<\infty; the first only requires E[|Y|]<\infty. A pointwise LLN for a fixed f does not automatically apply to a fitted function \hat f chosen from the same data.
Under the same i.i.d. integrability condition, the strong LLN gives
\frac1n\sum_{i=1}^nf(Y_i)\xrightarrow{\mathrm{a.s.}}E_P(f(Y)).
Our consistency arguments use the weaker convergence-in-probability conclusion stated above.
4.2 Central limit theorem: obtain the fluctuation law
Applying the same fixed function f to each observation gives i.i.d. values f(Y_1),\ldots,f(Y_n). Their population mean is E_P(f(Y)) and their population variance is
\sigma_f^2=\operatorname{Var}_P(f(Y)).
If 0<\sigma_f^2<\infty, the central limit theorem gives
\sqrt n\left\{ \frac1n\sum_{i=1}^nf(Y_i)-E_P(f(Y)) \right\} \xrightarrow{d}N(0,\sigma_f^2).
Inside the braces is the difference between a sample average and its population expectation. The sample average changes across repeated studies; its population expectation stays fixed. The theorem describes the distribution of that estimation error after multiplying it by \sqrt n.
The LLN tells us that the difference approaches zero in probability. The CLT goes further: it describes the fluctuations that remain visible after rescaling. Under the stated assumptions, those fluctuations approach a normal distribution. The individual values f(Y_i) do not themselves need to be normally distributed. Their average fluctuates around E_P(f(Y)) on the n^{-1/2} scale.
For the sample mean, choose f(y)=y. Then E_P(f(Y))=\mu, \sigma_f^2=\sigma^2, and the statement becomes
\sqrt n(\bar Y-\mu)\xrightarrow{d}N(0,\sigma^2).
When combining several sample averages, we need their joint limiting distribution. The multivariate CLT applies to a fixed vector of functions with finite second moments and supplies a normal limit with their joint covariance matrix. Those covariances matter when we subsequently transform or combine the averages.
From explicit averages to expectation shorthand. Both quantities in the CLT can be written as expectations, but under different distributions. The population distribution P describes how observations arise in the population. The empirical distribution P_n places probability 1/n on each of the n observed values, counting repeated values separately. Given the data, drawing from P_n means selecting one of those observations uniformly. An expectation under P_n therefore averages over the observed sample:
\begin{aligned} P_nf&=E_{P_n}(f(Y))\\ &=\frac1n\sum_{i=1}^nf(Y_i),\\ Pf&=E_P(f(Y)). \end{aligned}
These are exact definitions of the shorthand. In particular, P_nf includes the factor 1/n: it is an average, not an unnormalized sum. The empirical distribution P_n, and hence P_nf, changes across samples; the population distribution P and expectation Pf stay fixed. Replacing P by P_n changes the distribution being averaged over, while keeping the function f the same. Neither expression means multiplication of P or P_n by f.
The difference between the two averages can now be abbreviated as
\begin{aligned} (P_n-P)f&=P_nf-Pf\\ &=\frac1n\sum_{i=1}^nf(Y_i)-E_P(f(Y)). \end{aligned}
Thus the full LLN and CLT statements above become
P_nf\xrightarrow{p}Pf,
\sqrt n(P_n-P)f\xrightarrow{d}N(0,\sigma_f^2).
The notation is shorter; the averages and the theorem assumptions are unchanged. For f(y)=y, P_nf=\bar Y and Pf=\mu. Use the expanded identities above whenever the compact notation hides what is being averaged or which distribution supplies the expectation.
4.3 Markov and Chebyshev: turn moments into probability bounds
For a nonnegative random variable W with finite expectation, Markov’s inequality states
\begin{aligned} \Pr(W\ge t)&\le\frac{E[W]}t,\\ t&>0. \end{aligned}
Applying it to a squared centered variable gives Chebyshev’s inequality. For X with finite variance and \varepsilon>0,
\Pr(|X-E[X]|\ge\varepsilon) \le\frac{\operatorname{Var}(X)}{\varepsilon^2}.
For the sample mean, the bound is \sigma^2/(n\varepsilon^2), which tends to zero and directly proves consistency under finite variance. Taking \varepsilon=M/\sqrt n also gives \Pr(\sqrt n|\bar Y-\mu|\ge M)\le \sigma^2/M^2, establishing \bar Y-\mu=O_p(n^{-1/2}). Neither bound identifies a normal limiting law. Later, conditional versions of this argument will control held-out empirical averages given their training data.
4.4 Continuous mapping: carry a limit through a function
If X_n\xrightarrow{p}c and g is continuous at c, then
g(X_n)\xrightarrow{p}g(c).
For example, variance consistency and \sigma^2>0 imply s_Y\xrightarrow{p}\sigma and 1/s_Y\xrightarrow{p}1/\sigma (with the reciprocal defined arbitrarily when s_Y=0). The zero-denominator event has probability tending to zero. Continuity at the limit matters: reciprocal substitution is not justified by convergence to zero.
There is also a distributional version: X_n\xrightarrow{d}X implies g(X_n)\xrightarrow{d}g(X) if g is continuous with probability one under the limiting law. Both statements extend to vectors. For two estimates converging in probability to constants, their vector converges to the vector of constants, so continuous functions of both have the corresponding limit.
4.5 Slutsky’s theorem: combine a limiting law with consistent quantities
Let Z_n and B_n be sequences of random variables satisfying Z_n\xrightarrow{d}Z and B_n\xrightarrow{p}c, where c is a constant. Here B_n is a new sequence whose required condition is convergence in probability to that constant; no sign or moment assumptions are added. Then
\begin{aligned} Z_n+B_n&\xrightarrow{d}Z+c,\\ Z_nB_n&\xrightarrow{d}cZ,\\ Z_n/B_n&\xrightarrow{d}Z/c\qquad(c\ne0). \end{aligned}
The theorem works because (Z_n,B_n) converges jointly in distribution to (Z,c), after which continuous mapping applies. Independence of Z_n and B_n is not required. This is why we can use the sample mean and sample variance calculated from the same people.
For the mean, set Z_n=\sqrt n(\bar Y-\mu) and B_n=s_Y. The CLT gives the first limit, and variance consistency gives the second with c=\sigma>0. Division yields a standard normal limit. Replacing a population variance by an estimate therefore needs a consistency proof, not an assertion that the estimate is “close” in one dataset.
Slutsky also removes a negligible remainder. If Z_n\xrightarrow{d}N(0,V) and A_n=o_p(1), then Z_n+A_n\xrightarrow{d}N(0,V). This is the step used after proving \sqrt nR_n=o_p(1) in an estimator expansion. If the added term is only O_p(1), it can change the limit. Likewise, two separate nondegenerate distributional limits do not by themselves justify multiplication or division; their joint behavior matters.
4.6 The delta method is the same first-order idea
The function f above acts on each observation before averaging. Here a function g acts on an estimator after it has been calculated. For a transformed population mean, the connection is
\begin{aligned} \hat\eta&=P_nf\\ &=\bar Y,\\ \eta_0&=Pf\\ &=\mu,\\ f(y)&=y. \end{aligned}
For example, g(u)=\sqrt u gives g(\hat\eta)=\sqrt{\bar Y} as an estimator of g(\eta_0)=\sqrt\mu when \mu>0. Transforming an average generally differs from averaging transformed observations. We use a separate letter g to distinguish these two roles. The general result below also applies to estimators that are not simple averages, and allows \hat\eta and \eta_0 to be vectors.
If \sqrt n(\hat\eta-\eta_0)\to_d N(0,\Sigma) and g is differentiable at \eta_0, then
\sqrt n\{g(\hat\eta)-g(\eta_0)\} \to_d N\!(0,\nabla g^\top\Sigma\nabla g).
The gradient is evaluated at \eta_0. A first-order Taylor expansion gives the leading term \nabla g(\eta_0)^\top\sqrt n(\hat\eta-\eta_0); differentiability and the root-n rate make the remaining scaled error negligible. Continuous mapping and Slutsky then give the displayed limit.
For a positive population mean \mu and g(u)=\sqrt u, the derivative is 1/(2\sqrt\mu), so
\sqrt n(\sqrt{\bar Y}-\sqrt\mu) \xrightarrow{d}N\!\left(0,\frac{\sigma^2}{4\mu}\right).
If measurements can be negative, consistency and \mu>0 ensure that \bar Y>0 with probability tending to one; the transformed estimator can be defined arbitrarily on the remaining event for this asymptotic argument. A feasible delta-method SE is s_Y/(2\sqrt{\bar Y}\sqrt n) on positive sample means.
For a ratio or correlation, the delta method propagates the joint covariance, not just separate standard errors. Dropping covariance between components can give the wrong uncertainty. If the gradient vanishes, the displayed first-order limit may be degenerate and a higher-order analysis may be needed.
5 The first-order pattern for a more complicated estimator
Now move from the particular mean \mu to a general scalar target \psi_0. The central goal is an expansion
\begin{aligned} \widehat\psi-\psi_0&=(P_n-P)D_0+R_n,\\ PD_0&=0,\\ R_n&=o_p(n^{-1/2}). \end{aligned}
The sample mean fits this pattern with D_0(Y)=Y-\mu and remainder exactly zero. For a more complicated estimator, the remainder needs its own argument. When 0<PD_0^2<\infty, a fixed-function CLT handles the average. The condition R_n=o_p(n^{-1/2}) makes \sqrt nR_n=o_p(1), so Slutsky leaves the same normal limit after adding the remainder.
A feasible standard error additionally needs consistent estimation of the contribution variance. This follows the sample-mean walkthrough: establish the population limit, isolate the leading random average, control what remains, and justify the estimated variance. Calling a function an influence function or fitting a flexible regression does not establish these steps.
Estimating equations and asymptotic linearity derives this pattern by linearizing a sample equation. Uncertainty, confidence intervals, and tests explains what the resulting approximation lets us claim.
6 Check your understanding
Checkpoints are completion targets for this page. Additional practice below is optional. These four checkpoints check the fundamental interpretations and calculations used throughout the later theory. Try each before opening its hint or solution outline. The longer smooth-root derivation is SP-01 on the next page.
Checkpoint · LS-01 — Interpret consistency and a limiting law
Task type: Interpretation and probability notation.
Let Y_1,\ldots,Y_n be i.i.d., with mean \mu and finite, positive variance \sigma^2.
- Write the probability statement that defines \bar Y\xrightarrow{p}\mu. Explain what a fixed tolerance \varepsilon>0 means in this statement.
- Explain how this differs from \sqrt n(\bar Y-\mu)\xrightarrow{d}N(0,\sigma^2). Why can the unscaled error approach zero while the scaled error has a nondegenerate limiting distribution? Does either statement say that individual measurements are normally distributed?
- What would almost sure convergence add to the consistency statement? Is that stronger conclusion required for the CLT and feasible standard-error argument on this page?
Ready to continue when: you can distinguish concentration near a target, a limiting law for scaled errors, and convergence along almost every sample path.
Transfer: Suppose an estimator converges in probability to \mu+b for a fixed b\ne0. Is it consistent for \mu? Would collecting more observations remove that target mismatch?
Hint
For consistency, start with the event |\bar Y-\mu|>\varepsilon. For the CLT, identify which random quantity has a limiting normal law. For almost sure convergence, think about one growing dataset rather than a separate probability at each sample size.
Solution outline — open after trying
Consistency means that, for every fixed \varepsilon>0,
\Pr(|\bar Y-\mu|>\varepsilon)\longrightarrow0.
The tolerance stays fixed as n grows. The probability of exceeding it tends to zero; this does not say that each additional observation brings the estimate closer to \mu.
The CLT instead describes the repeated-sample distribution of \sqrt n(\bar Y-\mu). The error shrinks on the n^{-1/2} scale, so multiplying by \sqrt n reveals fluctuations that remain visible. Neither statement requires normally distributed individual measurements, and the CLT is a limiting statement rather than an exact finite-sample normality claim.
Almost sure convergence says that along almost every infinite sample path, the estimates eventually stay within each fixed positive tolerance. We do not need that stronger conclusion for the inference argument here. It would not, by itself, supply a root-n rate or a normal limit.
In the transfer question, the estimator is consistent for \mu+b, not \mu. For example, its probability of lying within |b|/2 of \mu tends to zero. More observations concentrate it near the wrong target.
Checkpoint · LS-02 — Expand empirical and population expectations
Task type: Calculation and notation.
The observed values are Y_1=1, Y_2=1, and Y_3=4. Take f(y)=y^2. No population distribution P is supplied.
- What probabilities does the empirical distribution P_n assign to the distinct values 1 and 4?
- Write P_nf as both E_{P_n}(f(Y)) and an explicit average, and calculate its value. Write Pf as a population expectation. Can you calculate that population quantity from the three observations alone?
- Expand (P_n-P)f in this example. Which part is known from the sample?
- Take g(u)=u^2. Calculate g(\bar Y) and compare it with P_nf. Explain the difference between these operations.
Ready to continue when: you can expand the expectation shorthand, identify its distribution, and distinguish averaging transformed observations from transforming an average.
Transfer: Now take f(y)=\mathbf{1}\{y>2\}, the indicator that equals 1 when y>2 and 0 otherwise. What are P_nf and the population quantity Pf?
Hint
Drawing from P_n means selecting one of the three observed records with equal probability. The value 1 occurs twice. Keep the factor 1/3 when averaging squares; for g(\bar Y), average first and then square.
Solution outline — open after trying
The empirical distribution gives probability 2/3 to 1 and 1/3 to 4. Thus
\begin{aligned} P_nf&=E_{P_n}(Y^2)\\ &=\frac{1^2+1^2+4^2}{3}\\ &=6. \end{aligned}
The population quantity is Pf=E_P(Y^2). The sample provides an estimate of that expectation, not its exact value. Without specifying P, its value is unknown. Therefore (P_n-P)f=6-E_P(Y^2); only the first term is known.
Here \bar Y=2, so g(\bar Y)=\bar Y^2=4, whereas P_nf=6. Squaring each observation and then averaging differs from squaring their average.
For the indicator in the transfer question, P_nf=1/3 is the observed fraction exceeding 2. Its population counterpart is Pf=E_P(\mathbf{1}\{Y>2\})=\Pr_P(Y>2), which is not known exactly from these observations.
Checkpoint · LS-03 — Separate measurement spread from mean uncertainty
Task type: Calculation and interpretation.
Suppose independent measurements from one population have SD \sigma=12.
- Calculate the exact variance and SE of the sample mean for n=36 and n=144. What happens to the SD of individual measurements?
- In practice, \sigma is unknown. If a study of 36 people gives sample SD s_Y=15, what estimated SE should be reported for \bar Y? Explain why 15 is not that estimated SE.
- A colleague says, “With enough people, the measurements themselves become less variable and normally distributed.” Explain what is wrong with both parts of this statement.
Ready to continue when: you can calculate the variance and SE of a sample mean and explain why averaging reduces estimation uncertainty without reducing population heterogeneity.
Transfer: Keeping the population and independent sampling design fixed, how must n change to halve the SE?
Hint
Use \operatorname{Var}(\bar Y)=\sigma^2/n and \operatorname{SE}(\bar Y)=\sigma/\sqrt n. Label the random object whose spread each number describes.
Solution outline — open after trying
For n=36, the variance is 144/36=4 and the SE is 2. For n=144, the variance is 144/144=1 and the SE is 1. The population SD of individual measurements remains 12 in both cases.
The estimated SE from the observed sample is 15/\sqrt{36}=2.5. The sample SD 15 estimates the spread of individual measurements; dividing by \sqrt n estimates the spread of sample means across repeated studies. That estimated SE need not equal the true SE in one sample.
Increasing n changes the uncertainty of the average, not the population distribution of individual measurements. The CLT concerns the limiting distribution of scaled errors in the average. It does not make individual measurements normal. The variance and SE identities here are exact under independent, identically distributed sampling with finite variance; a normal approximation is a separate step.
Since SE is proportional to 1/\sqrt n, halving it requires quadrupling n under the same assumptions.
Checkpoint · LS-04 — Combine theorems and check a remainder
Task type: Theorem selection and rate reasoning.
Continue with i.i.d. measurements of mean \mu and finite, positive variance \sigma^2, and the sample SD s_Y.
Explain what the LLN, CLT, continuous mapping, and Slutsky’s theorem each contribute to
\frac{\sqrt n(\bar Y-\mu)}{s_Y}\xrightarrow{d}N(0,1).
Include why s_Y^2 consistently estimates \sigma^2. Must \bar Y and s_Y be independent?
Consider \tilde\mu_n=\bar Y+R_n. For each deterministic remainder R_n=n^{-1} and R_n=n^{-1/2}, calculate \sqrt nR_n. Which remainder is o_p(n^{-1/2})? Which is O_p(n^{-1/2})?
For which remainder can you retain the standard normal limit after replacing \bar Y with \tilde\mu_n in the display? Does merely proving R_n\to0 settle this question?
Ready to continue when: you can justify studentization with the core theorems and distinguish a negligible root-n remainder from an error that changes the limiting law.
Transfer: If \mu>0, what derivative does the delta method use for \sqrt{\bar Y}, and what large-sample SE estimate follows on the event \bar Y>0?
Hint
Write s_Y^2 in terms of the average of Y_i^2 and \bar Y^2. For the remainders, divide by n^{-1/2} before deciding whether the ratio tends to zero or stays bounded.
Solution outline — open after trying
The LLN gives \bar Y\xrightarrow{p}\mu and n^{-1}\sum_iY_i^2\xrightarrow{p}E_P(Y^2). The second use needs a finite second moment, which our finite-variance assumption supplies. Continuous mapping and n/(n-1)\to1 give
\begin{aligned} s_Y^2&=\frac{n}{n-1} \left\{\frac1n\sum_iY_i^2-\bar Y^2\right\}\\ &\xrightarrow{p}\sigma^2. \end{aligned}
Continuity of the square root gives s_Y\xrightarrow{p}\sigma>0. The CLT gives \sqrt n(\bar Y-\mu)\xrightarrow{d}N(0,\sigma^2), and Slutsky justifies division by s_Y to obtain N(0,1). Independence of \bar Y and s_Y is not required. As in the theorem discussion, define the ratio arbitrarily if s_Y=0; that event has probability tending to zero.
For R_n=n^{-1}, \sqrt nR_n=n^{-1/2}\to0, so R_n=o_p(n^{-1/2}) and also O_p(n^{-1/2}). For R_n=n^{-1/2}, \sqrt nR_n=1, so it is O_p(n^{-1/2}) but not o_p(n^{-1/2}).
The first remainder vanishes in the scaled limit. The second adds 1/s_Y\to_p1/\sigma to the studentized statistic, shifting its limit to N(1/\sigma,1). Both remainders tend to zero and preserve consistency for \mu, but only the first preserves the centered standard normal limit.
For the transfer, g'(\mu)=1/(2\sqrt\mu). The delta-method SE is estimated by s_Y/(2\sqrt{\bar Y}\sqrt n) when \bar Y>0. Consistency and \mu>0 make that event have probability tending to one.
Additional practice (optional)
- Apply the delta method to \log\bar Y for a positive population mean. Derive the limiting variance and an estimated SE, and state the domain condition.
- Explain why a consistent estimator need not have a root-n rate. Construct an example by adding a deterministic shrinking error to \bar Y.
- For the numerical example in section 3, calculate how many independent people would be needed to reduce the SE from 1 to 0.25. What happens to the population SD?
- State what changes in the argument if the contribution is fitted using the same observations.
7 Later transfer: quantiles
A discontinuous sample equation needs a different local argument. Quantiles and local expansions develops that transfer after valid paths and influence functions have been introduced; it retains checkpoint SP-06 and states the extra empirical-CDF lemma explicitly.