Continue with a completely observed feature mean \mu=E[Y], estimated by \hat\mu=\bar Y from n independent people. Let \sigma^2=\operatorname{Var}(Y). The population SD \sigma describes variation between people. The standard error \sigma/\sqrt n describes the spread of sample means across repeated studies of this size. Estimate it by s_Y/\sqrt n, where s_Y is the sample SD.
For example, a study of 100 people might have mean 2.5 and SD 2. Its estimated SE is 2/\sqrt{100}=0.2. The measurements vary on the scale of 2 units; the estimated mean’s sampling uncertainty is on the scale of 0.2 units. A confidence interval for the population mean will use the latter. Predicting a new person’s measurement requires a different calculation that includes individual variability.
The inferential starting point is the standardized error
The numerator measures estimation error in outcome units; dividing by the estimated SE expresses it in units of sampling uncertainty. This is called studentization. The CLT, variance consistency, and Slutsky’s theorem justify it for the i.i.d. finite-variance mean.
The same construction applies to a more general scalar target \theta_0 when \hat\theta-\theta_0=P_nD_0+o_p(n^{-1/2}), PD_0=0, and 0<PD_0^2<\infty, with a consistently estimated contribution variance. The sampling unit, target, centering, and remainder conditions remain part of the argument.
2 A confidence interval is a random procedure
For a standard normal variable Z, \Pr(-1.96\le Z\le1.96)\approx0.95. Apply this reference to the standardized estimation error:
For the example above, its margin is 1.96(0.2)=0.392, giving [2.108,2.892]. The point estimate and both endpoints would change if we collected another sample. The population mean would remain fixed.
Coverage is the probability that this random interval procedure includes the fixed population mean. Under the approximation’s conditions, about 95% of intervals from repeated studies cover it. In 1,000 simulated studies, the number covering need not be exactly 950. Once a particular dataset is observed, its endpoints are fixed; the frequentist coverage statement does not assign a 95% posterior probability to that realized interval containing \mu.
For independent normal measurements, replacing 1.96 by the appropriate t_{n-1} quantile gives an exact t interval. For more general asymptotically linear estimators, normal calibration is a large-sample approximation whose accuracy must be assessed for the setting.
A narrow interval can be centered at the wrong population quantity. If the observed-outcome mean targets 0.8 while the eligible-population mean is 2, a correct SE around 0.8 does not turn it into inference about 2. As the SE shrinks around the wrong limit, coverage for the intended target can deteriorate. Identification and target alignment precede interval calibration.
3 A test asks about a stated null
Specify the null before calculating a p-value. For the mean, take the point null H_0:\mu=\mu_\star. Under that null, the standardized statistic has an approximately standard normal distribution:
Let z_{\mathrm{obs}} be the statistic’s observed value. A two-sided p-value is the probability, under the null reference law, of a statistic at least this far from zero in either direction:
Here \Phi(t)=\Pr(Z\le t) is the standard normal CDF. In the numerical example, testing \mu_\star=2 gives z_{\mathrm{obs}}=(2.5-2)/0.2=2.5 and p\approx0.0124. The observed mean is 2.5 estimated SEs above the null value. About 1.24% of a standard normal reference distribution lies at least that far from zero in absolute value.
With valid calibration, rejecting when p<0.05 gives a rejection probability of approximately 0.05 when the null is true. This is a repeated-sampling Type I error statement. It is not the probability that this particular null is true or that this particular finding is a false positive. The estimated difference, 0.5 outcome units here, describes magnitude; its practical importance requires scientific context.
For the same two-sided Wald procedure, a 95% interval excludes a null value exactly when its p-value is below 0.05, apart from rounding. Both decisions compare the distance from the estimate to the null with 1.96\widehat{\operatorname{SE}}. The example’s interval excludes 2, agreeing with its p-value. This agreement requires matched procedures; an arbitrary interval and test need not agree.
A non-rejection can reflect limited precision even when a scientifically relevant difference exists. To assess which differences the data support, inspect the estimate and interval. Establishing equivalence requires a specified equivalence margin and a corresponding inferential procedure.
4 See coverage across repeated samples
This synthetic experiment draws complete normal measurements with mean 2 and SD 1. It uses the approximate Wald rule with an estimated SE. It evaluates one prespecified mean, rather than a family of selected features.
Each replication draws a fresh group of 40 people, calculates its own mean and SE, and builds its own interval. The indicator covered records whether that interval contains the known \mu=2. In this matched example, noncoverage is also rejection of the true point null; it is not a power calculation under an alternative.
Figure 1: Eighty intervals from a synthetic repeated-sampling experiment. The vertical line is the known population mean. Intervals missing that target are highlighted. The simulation also reports coverage over all 500 samples.
Coverage in a finite simulation fluctuates. With 500 replications and coverage near 0.95, the Monte Carlo SE is about \sqrt{0.95(0.05)/500}\approx0.01, or one percentage point. More simulated datasets improve the precision of this evaluation; increasing the number of people in each dataset changes the inferential problem itself. The plot shows 80 intervals for readability, while the numerical summary uses all 500.
One interval containing the truth is not evidence that a procedure has 95% coverage, and one miss is not evidence that it fails. Repeated-sampling evaluation studies the procedure. Its assumptions and asymptotic argument explain when we expect that evaluation to transfer beyond this generator.
5 Individual uncertainty and a family of claims
A pointwise interval concerns one prespecified target. Searching many genes, selecting the smallest p-values, and reporting those intervals is a different procedure. A collection of pointwise 95% intervals is not automatically a simultaneous confidence statement or a false-discovery guarantee. Multiple testing when the tests are not independent handles the family-level question after the individual inferential argument has been established.
6 Exercises
The checkpoint is a completion target. Additional practice is optional.
Checkpoint · INF-01 — Interpret a standard error, interval, and test
Task type: Calculation and interpretation.
For 100 independent people, a prespecified feature has sample mean 2.4 and sample SD 2. Calculate the mean’s SE, an approximate 95% Wald interval, and the two-sided Wald p-value for a population mean of 2. Explain the coverage claim, what the p-value does not mean, and why these calculations would not repair a mismatch between the scientific target and the estimator’s limit.
Ready to continue when: you can connect a sampling approximation to an interval and test without confusing measurement variability, coverage, null probability, and target alignment.
Transfer: Would the same SE apply if the 100 rows were five visits from each of 20 people?
Hint
The SD of the observations is 2. Divide by the square root of the independent sample size to estimate the mean’s SE.
Solution outline — open after trying
The SE is 0.2; the Wald interval is approximately [2.008, 2.792]. The null statistic is 2, giving p approximately 0.0455. The matched interval excludes 2. Under the procedure’s conditions, its coverage concerns repeated samples of intervals around a fixed target. The p-value is a null-tail probability, not a posterior null probability. Neither calculation establishes identification or centers an estimator at a different scientific quantity. Repeated rows need a variance calculation based on the independent people and their within-person dependence.
Additional practice (optional)
Compare the Wald and exact t procedures in the normal simulation.
Simulate a biased estimator whose SE shrinks with n and inspect coverage for the intended target.
Explain which additional question arises when 15,000 features are examined.
---title: "Uncertainty, confidence intervals, and tests"subtitle: "Turn a justified sampling approximation into a defensible claim"---**Before reading:** [Large-sample reasoning for methods research](large-sample-reasoning.qmd) and [Estimating equations and asymptotic linearity](estimating-equations.qmd).## What a standard error describesContinue with a completely observed feature mean $\mu=E[Y]$, estimated by $\hat\mu=\bar Y$ from $n$ independent people. Let $\sigma^2=\operatorname{Var}(Y)$. The population SD $\sigma$ describes variation between people. The standard error $\sigma/\sqrt n$ describes the spread of sample means across repeated studies of this size. Estimate it by $s_Y/\sqrt n$, where $s_Y$ is the sample SD.For example, a study of 100 people might have mean 2.5 and SD 2. Its estimated SE is $2/\sqrt{100}=0.2$. The measurements vary on the scale of 2 units; the estimated mean's sampling uncertainty is on the scale of 0.2 units. A confidence interval for the population mean will use the latter. Predicting a new person's measurement requires a different calculation that includes individual variability.The inferential starting point is the standardized error$$\frac{\hat\mu-\mu}{\widehat{\operatorname{SE}}(\hat\mu)}\xrightarrow{d}N(0,1).$$The numerator measures estimation error in outcome units; dividing by the estimated SE expresses it in units of sampling uncertainty. This is called **studentization**. The CLT, variance consistency, and Slutsky's theorem justify it for the i.i.d. finite-variance mean.The same construction applies to a more general scalar target $\theta_0$ when $\hat\theta-\theta_0=P_nD_0+o_p(n^{-1/2})$, $PD_0=0$, and $0<PD_0^2<\infty$, with a consistently estimated contribution variance. The sampling unit, target, centering, and remainder conditions remain part of the argument.## A confidence interval is a random procedureFor a standard normal variable $Z$, $\Pr(-1.96\le Z\le1.96)\approx0.95$. Apply this reference to the standardized estimation error:$$\Pr\!\left(\frac{|\hat\mu-\mu|}{\widehat{\operatorname{SE}}(\hat\mu)}\le1.96\right)\approx0.95.$$Rearranging the inequality for the unknown $\mu$ produces the approximate 95% **Wald interval**:$$\left[\hat\mu-1.96\widehat{\operatorname{SE}},\;\hat\mu+1.96\widehat{\operatorname{SE}}\right].$$For the example above, its margin is $1.96(0.2)=0.392$, giving $[2.108,2.892]$. The point estimate and both endpoints would change if we collected another sample. The population mean would remain fixed.**Coverage** is the probability that this random interval procedure includes the fixed population mean. Under the approximation's conditions, about 95% of intervals from repeated studies cover it. In 1,000 simulated studies, the number covering need not be exactly 950. Once a particular dataset is observed, its endpoints are fixed; the frequentist coverage statement does not assign a 95% posterior probability to that realized interval containing $\mu$.For independent normal measurements, replacing 1.96 by the appropriate $t_{n-1}$ quantile gives an exact t interval. For more general asymptotically linear estimators, normal calibration is a large-sample approximation whose accuracy must be assessed for the setting.A narrow interval can be centered at the wrong population quantity. If the observed-outcome mean targets 0.8 while the eligible-population mean is 2, a correct SE around 0.8 does not turn it into inference about 2. As the SE shrinks around the wrong limit, coverage for the intended target can deteriorate. Identification and target alignment precede interval calibration.## A test asks about a stated nullSpecify the null before calculating a p-value. For the mean, take the point null $H_0:\mu=\mu_\star$. Under that null, the standardized statistic has an approximately standard normal distribution:$$Z_n=\frac{\hat\mu-\mu_\star}{\widehat{\operatorname{SE}}}.$$Let $z_{\mathrm{obs}}$ be the statistic's observed value. A two-sided p-value is the probability, under the null reference law, of a statistic at least this far from zero in either direction:$$\begin{aligned}p&\approx\Pr(|Z|\ge|z_{\mathrm{obs}}|)\\&=2\{1-\Phi(|z_{\mathrm{obs}}|)\},\\Z&\sim N(0,1).\end{aligned}$$Here $\Phi(t)=\Pr(Z\le t)$ is the standard normal CDF. In the numerical example, testing $\mu_\star=2$ gives $z_{\mathrm{obs}}=(2.5-2)/0.2=2.5$ and $p\approx0.0124$. The observed mean is 2.5 estimated SEs above the null value. About 1.24% of a standard normal reference distribution lies at least that far from zero in absolute value.With valid calibration, rejecting when $p<0.05$ gives a rejection probability of approximately 0.05 **when the null is true**. This is a repeated-sampling Type I error statement. It is not the probability that this particular null is true or that this particular finding is a false positive. The estimated difference, 0.5 outcome units here, describes magnitude; its practical importance requires scientific context.For the same two-sided Wald procedure, a 95% interval excludes a null value exactly when its p-value is below 0.05, apart from rounding. Both decisions compare the distance from the estimate to the null with $1.96\widehat{\operatorname{SE}}$. The example's interval excludes 2, agreeing with its p-value. This agreement requires matched procedures; an arbitrary interval and test need not agree.A non-rejection can reflect limited precision even when a scientifically relevant difference exists. To assess which differences the data support, inspect the estimate and interval. Establishing equivalence requires a specified equivalence margin and a corresponding inferential procedure.## See coverage across repeated samplesThis synthetic experiment draws complete normal measurements with mean 2 and SD 1. It uses the approximate Wald rule with an estimated SE. It evaluates one prespecified mean, rather than a family of selected features.Each replication draws a fresh group of 40 people, calculates its own mean and SE, and builds its own interval. The indicator `covered` records whether that interval contains the known $\mu=2$. In this matched example, noncoverage is also rejection of the true point null; it is not a power calculation under an alternative.```{r}#| label: fig-mean-coverage#| fig-cap: "Eighty intervals from a synthetic repeated-sampling experiment. The vertical line is the known population mean. Intervals missing that target are highlighted. The simulation also reports coverage over all 500 samples."#| fig-alt: "Horizontal sample-mean intervals mostly cross the vertical population mean, with a few misses highlighted."set.seed(812)n <-40replicates <-500means <- ses <-numeric(replicates)for (b inseq_len(replicates)) { y <-rnorm(n, mean =2, sd =1) means[b] <-mean(y) ses[b] <-sd(y) /sqrt(n)}covered <-abs(means -2) <=1.96* sesc(coverage =mean(covered),rejection_under_null =mean(!covered),coverage_MCSE =sqrt(mean(covered) * (1-mean(covered)) / replicates))shown <-seq_len(80)plot(means[shown], shown, xlim =range(means[shown] +1.96* ses[shown], means[shown] -1.96* ses[shown]), xlab ="Feature mean", ylab ="Independent sample",pch =16, col =ifelse(covered[shown], "#2F80C8", "#D55E00"))segments(means[shown] -1.96* ses[shown], shown, means[shown] +1.96* ses[shown], shown,col =ifelse(covered[shown], "#2F80C8", "#D55E00"))abline(v =2, lty =2)```Coverage in a finite simulation fluctuates. With 500 replications and coverage near 0.95, the Monte Carlo SE is about $\sqrt{0.95(0.05)/500}\approx0.01$, or one percentage point. More simulated datasets improve the precision of this evaluation; increasing the number of people in each dataset changes the inferential problem itself. The plot shows 80 intervals for readability, while the numerical summary uses all 500.One interval containing the truth is not evidence that a procedure has 95% coverage, and one miss is not evidence that it fails. Repeated-sampling evaluation studies the procedure. Its assumptions and asymptotic argument explain when we expect that evaluation to transfer beyond this generator.## Individual uncertainty and a family of claimsA pointwise interval concerns one prespecified target. Searching many genes, selecting the smallest p-values, and reporting those intervals is a different procedure. A collection of pointwise 95% intervals is not automatically a simultaneous confidence statement or a false-discovery guarantee. [Multiple testing when the tests are not independent](../analysis-problems/multiple-testing.qmd) handles the family-level question after the individual inferential argument has been established.## ExercisesThe checkpoint is a completion target. Additional practice is optional.::: {#checkpoint-inf-01 .checkpoint}### Checkpoint · INF-01 — Interpret a standard error, interval, and test {.unnumbered}**Task type:** Calculation and interpretation.For 100 independent people, a prespecified feature has sample mean 2.4 and sample SD 2. Calculate the mean’s SE, an approximate 95% Wald interval, and the two-sided Wald p-value for a population mean of 2. Explain the coverage claim, what the p-value does not mean, and why these calculations would not repair a mismatch between the scientific target and the estimator’s limit.**Ready to continue when:** you can connect a sampling approximation to an interval and test without confusing measurement variability, coverage, null probability, and target alignment.**Transfer:** Would the same SE apply if the 100 rows were five visits from each of 20 people?<details class="checkpoint-help"><summary>Hint</summary>The SD of the observations is 2. Divide by the square root of the independent sample size to estimate the mean’s SE.</details><details class="checkpoint-help"><summary>Solution outline — open after trying</summary>The SE is 0.2; the Wald interval is approximately [2.008, 2.792]. The null statistic is 2, giving p approximately 0.0455. The matched interval excludes 2. Under the procedure’s conditions, its coverage concerns repeated samples of intervals around a fixed target. The p-value is a null-tail probability, not a posterior null probability. Neither calculation establishes identification or centers an estimator at a different scientific quantity. Repeated rows need a variance calculation based on the independent people and their within-person dependence.</details>:::### Additional practice (optional) {.unnumbered}1. Compare the Wald and exact t procedures in the normal simulation.2. Simulate a biased estimator whose SE shrinks with n and inspect coverage for the intended target.3. Explain which additional question arises when 15,000 features are examined.
