---
title: "Estimating nuisance functions for inference"
subtitle: "Prediction error, smoothness, and the product-rate requirement."
guide-status: "Available"
---
**Before reading:** [Conditional expectation and L2 geometry](../statistical-models/conditional-expectation.qmd), [Large-sample reasoning for methods research](../estimation-and-inference/large-sample-reasoning.qmd), and [Orthogonality and cross-fitting](orthogonality-and-cross-fitting.qmd).
## A nuisance fit has a specific job
A nuisance function is a population object needed to estimate the target: a conditional mean, an observation probability, a conditional variance, or another component. Naming a learning algorithm does not define that object. Start by writing the nuisance target and the norm or error property required by the estimator's theorem.
For the running missing-outcome mean $\mu=E[Y]$, the true nuisances are $m_0(z)=E[Y\mid Z=z,R=1]$ and $\pi_0(z)=P(R=1\mid Z=z)$. The outcome regression uses observed outcomes, while the observation model uses the recorded indicator for all eligible units. Under MAR, $m_0$ is also the full-population conditional mean. The target remains a single mean even when these nuisance functions are complicated.
For a fitted function, define its population L2 error by
$$
\|\widehat m-m_0\|_2=\sqrt{E_Z[(\widehat m(Z)-m_0(Z))^2]}.
$$
Hold the training sample fixed in this expectation. The norm averages prediction-function error over new eligible people's covariates; it does not include their irreducible outcome noise. Across training datasets, the fitted function and this norm are random. A rate such as $O_p(n^{-2/5})$ describes how that random error shrinks as training sample size grows.
## Prediction risk and nuisance error
First consider a regression problem in a specified evaluation population, with conditional mean $m(Z)=E[Y\mid Z]$. Hold the fitted predictor fixed. Decompose a new prediction error as $(Y-m)+(m-\widehat m)$. Its cross term has expectation zero because $E[Y-m\mid Z]=0$. Thus
$$
E[(Y-\hat m(Z))^2\mid\text{training}]=E[\operatorname{Var}(Y\mid Z)]+\|\hat m-m\|_2^2.
$$
The first term is irreducible outcome noise; the second is the squared norm needed for learning. If the noise variance is 1 and squared nuisance error is 0.04, the prediction risk is 1.04. A learner reducing the latter error to 0.01 changes total risk to 1.01. A small change in total prediction loss can therefore represent a substantial improvement in the function estimate.
Training error can be optimistic because the same outcomes helped select the predictor. Held-out prediction risk estimates performance on new observations from the evaluation distribution, subject to sampling uncertainty. It can help compare learners, but one held-out loss does not establish an asymptotic convergence rate or check an entire inferential theorem.
### Match the evaluation distribution to the required norm
For missing outcomes, held-out outcome losses can be computed only where $R=1$. Their covariates follow $Z\mid R=1$, while the capstone's norm averages over all eligible $Z$. These distributions need not agree. Under $\pi_0\ge c_0>0$, write $e(Z)=\widehat m(Z)-m_0(Z)$. Then
$$
\begin{aligned}
E_Z[e(Z)^2]&\le c_0^{-1}E_Z[\pi_0(Z)e(Z)^2]\\
&=\frac{P(R=1)}{c_0}E[e(Z)^2\mid R=1].
\end{aligned}
$$
Thus observed-case L2 accuracy can control target-population L2 accuracy under a uniform lower probability bound. The constant worsens as the lower bound decreases. Without such a condition, good average prediction among observed outcomes can hide poor learning in sparsely observed target strata. This is a distribution issue, separate from whether training and validation were split correctly.
Approximation error describes how closely the fitting class can represent the true regression. Estimation error describes learning within that class from a finite sample. For a quadratic truth, a class containing only straight lines leaves a nonvanishing approximation error however many observations arrive. A class containing quadratics can represent that truth, but its fitted coefficients still fluctuate. A flexible class can reduce approximation error while making estimation more demanding; the total error is what the inferential bound uses.
## A concrete local-regression calculation
Suppose $Z$ is one-dimensional on a bounded interval with a design density bounded above and away from zero. Assume a twice continuously differentiable mean with bounded curvature, bounded conditional noise variance, and suitable kernel and boundary conditions. Here $n$ denotes training sample size. A local-linear estimator fits an intercept and slope near a query point $z_0$, giving more weight to observations within bandwidth $h$.
The Taylor expansion
$$
m(z)=m(z_0)+m'(z_0)(z-z_0) +\tfrac12m''(\xi)(z-z_0)^2
$$
shows why fitting a line removes the constant and slope terms: curvature is the first term the local line cannot reproduce. For nearby points $|z-z_0|$ is of order $h$, so the leading bias is of order $h^2$. A neighborhood contains order $nh$ observations, and averaging its noise gives variance of order $1/(nh)$. Shrinking $h$ reduces smoothing bias but increases noise. For the asymptotic calculation, take $h\to0$ and $nh\to\infty$.
The resulting integrated squared error has the leading form
$$
C_bh^4+\frac{C_v}{nh}.
$$
Squaring the bias gives $h^4$, which is why the displayed error uses that power rather than $h^2$. For positive leading constants, differentiate with respect to $h$:
$$
\begin{aligned}
4C_bh^3-\frac{C_v}{nh^2}&=0,\\
h_{\mathrm{opt}}&=\left(\frac{C_v}{4C_bn}\right)^{1/5}.
\end{aligned}
$$
At this bandwidth, both terms have order $n^{-4/5}$. This is an order for **mean integrated squared error** (MISE): integrate squared function error over new covariates, then average over repeated training samples. Its square root has order $n^{-2/5}$, rather than $n^{-4/5}$.
To connect that average statement to a random norm, suppose $E_{\mathrm{train}}[\|\widehat m-m\|_2^2]\le Cn^{-4/5}$. Markov's inequality gives, for any $M>0$,
$$
\Pr\{\|\widehat m-m\|_2>Mn^{-2/5}\}
\le\frac{C}{M^2}.
$$
Choosing $M$ large bounds the tail uniformly in $n$, which is precisely the needed $O_p(n^{-2/5})$ statement. The probability is over training datasets. The rate depends on the design, smoothness, bandwidth, kernel, and moment conditions; a method being called a “smoother” does not supply them.
```{r}
#| label: fig-bandwidth-tradeoff
#| fig-cap: "An illustrative leading-error calculation with both constants set to one. The log scale makes the interior minimum visible; the vertical line marks the minimizing bandwidth. This is a rate diagram, not measured prediction error."
#| fig-alt: "Squared bias increases with bandwidth, variance decreases, and their sum has an interior minimum."
n <- 400; h <- seq(0.04, 0.5, length.out = 200)
bias2 <- h^4; variance <- 1 / (n * h)
matplot(h, cbind(bias2, variance, bias2 + variance), type = "l", log = "y",
lty = c(1, 2, 1),
col = c("#D55E00", "#2F80C8", "#009E73"),
xlab = "Bandwidth h", ylab = "Leading integrated squared error (log scale)")
abline(v = (1 / (4 * n))^(1/5), lty = 3, col = "gray40")
legend("topleft", c("Squared bias", "Variance", "Sum"),
col = c("#D55E00", "#2F80C8", "#009E73"), lty = c(1, 2, 1), bty = "n")
```
## Fit the toy regression and inspect its mistakes
The next example isolates regression learning using the capstone’s mean $m_0(z)=2+z+z^2$ and bounded errors. For this illustration all outcomes are supplied; it is not the missing-outcome estimator. The capstone fits its outcome regression only where outcomes are observed. In `LocalLine`, the design columns are 1 and $z-z_0$, and Gaussian weights decrease with distance from the query point. The fitted intercept estimates the mean at that point because $z-z_0=0$ there.
The grid error averages squared differences from the known synthetic conditional mean, not noisy outcomes. It is a finite-grid approximation over the displayed interior interval, not an exact full-population integrated risk. Real data usually do not supply the true curve for this comparison.
```{r}
set.seed(31)
z <- runif(300, -1, 1)
true_mean <- function(z) 2 + z + z^2
y <- true_mean(z) + runif(300, -1, 1)
grid <- seq(-0.9, 0.9, length.out = 70)
LocalLine <- function(z, y, query, bandwidth) {
vapply(query, function(q) {
fit <- lm.wfit(cbind(1, z - q), y, w = dnorm((z - q) / bandwidth))
unname(fit$coefficients[1])
}, numeric(1))
}
bandwidths <- c(0.05, 0.15, 0.5)
fitted_curves <- sapply(bandwidths, function(h) LocalLine(z, y, grid, h))
data.frame(h = bandwidths,
grid_mean_error = colMeans((fitted_curves - true_mean(grid))^2))
```
```{r}
#| label: fig-local-fits
#| fig-cap: "Local-linear fits to one synthetic dataset. A narrow bandwidth follows more sample noise; a broad bandwidth smooths away the curved signal. The dashed curve is known simulation truth, not an available real-data reference."
#| fig-alt: "Noisy observations and three local-linear curves compared with the known quadratic conditional mean."
plot(z, y, pch = 16, cex = 0.45, col = "gray70",
xlab = "Covariate Z", ylab = "Outcome and fitted conditional mean")
matlines(grid, fitted_curves, lty = 1, lwd = 2,
col = c("#2F80C8", "#D55E00", "#009E73"))
lines(grid, true_mean(grid), lty = 2, lwd = 2)
legend("topleft", c(paste("h =", bandwidths), "Known mean"),
col = c("#2F80C8", "#D55E00", "#009E73", "black"),
lty = c(1, 1, 1, 2), bty = "n", cex = 0.8)
```
A single grid-error comparison depends on the realized sample. To evaluate a bandwidth-selection procedure, repeat the whole fitting and selection process across simulated training samples and use independent validation points. Choosing the best bandwidth against known simulation truth is an oracle comparison, not a feasible procedure for real data.
## Dimension changes the rate argument
In $d$ dimensions, a neighborhood of width $h$ has volume of order $h^d$, so only order $nh^d$ observations contribute locally. For an $s$-smooth regression and an estimator able to exploit that smoothness, a benchmark squared-error calculation is $h^{2s}+1/(nh^d)$. Balancing gives
$$
\begin{aligned}
h&\asymp n^{-1/(2s+d)},\\
\|\widehat m-m\|_2&=O_p(n^{-s/(2s+d)}).
\end{aligned}
$$
These are benchmark rates under appropriate design, smoothness, and moment conditions. A local-constant or local-linear estimator does not automatically exploit arbitrarily high smoothness. Structural assumptions such as sparsity or additivity can change the effective problem; the number of recorded covariates alone does not determine the attainable rate.
For two nuisances with the same benchmark rate, the product exponent is $2s/(2s+d)$. It exceeds $1/2$ exactly when $s>d/2$. With twice-smooth functions ($s=2$):
| $d$ | L2 exponent | Product exponent | Negligible bias? |
|---|---|---|---|
| 1 | $2/5$ | $4/5$ | Yes |
| 4 | $1/4$ | $1/2$ | Boundary; big-O alone is insufficient |
| 5 | $2/9$ | $4/9$ | No guarantee |
This does not prove that inference is impossible in the latter settings. It says these particular norm bounds do not establish negligibility for this construction. Better structure, unequal nuisance rates, or another analysis could change the conclusion.
## Connect learning to the target's remainder
For the missing-outcome mean, the exact bias is $E[(\hat\pi-\pi_0)(\hat m-m_0)/\hat\pi]$. A lower bound on fitted probabilities controls the denominator; Cauchy–Schwarz then bounds it by a constant times the product of L2 errors. A residual-covariance target has a related product, but it must be derived for that different target.
For example, two one-dimensional twice-smooth fits with L2 errors $O_p(n^{-2/5})$ give product $O_p(n^{-4/5})$, which becomes $O_p(n^{-3/10})=o_p(1)$ after root-$n$ scaling. If one nuisance is correctly specified and estimated at rate $O_p(n^{-1/2})$, mere $o_p(1)$ consistency of the other gives a product $o_p(n^{-1/2})$. Thus the nuisance rates need not match. With a fixed number of folds, training size is proportional to total sample size, so these polynomial orders remain the same.
Write the exact remainder, derive its bound, and state rates under actual learning assumptions before controlling the empirical term and variance. Cross-validation targets prediction performance; it does not directly certify the product remainder. Good outcome prediction also cannot supply unsupported observation probabilities or repair a wrong estimand.
## Fit the whole pipeline inside training folds
In an independent-person study, split by person. Inside each training set, fit transformations that require learning, choose features when appropriate, tune hyperparameters using training-only validation, and fit nuisance models. Apply the fitted operations to the held-out units. Inner validation chooses a learner; the outer held-out fold evaluates the contribution for inference. The outcome model's inner validation uses observed training outcomes, while observation-model validation uses the recorded $R$ values. Do not choose the final learner by the held-out target estimate or its apparent significance.
Inspect fitted-probability ranges, tail weights, calibration, held-out prediction loss, and sensitivity to sensible learner choices. Probability calibration concerns agreement between predicted probabilities and observed frequencies; discrimination alone does not assess it. These checks can reveal poor fits but do not verify MAR or an asymptotic rate.
Clipping probabilities imposes a lower denominator bound. If the true probabilities lie inside the imposed bounds, this can stabilize the calculation without excluding the truth. If the lower cutoff exceeds some true probabilities, the clipped fit cannot converge to $\pi_0$ there; the efficient-inference argument using both true nuisances must be reconsidered. Document clipping and its frequency. It does not establish population positivity, and cross-fitting does not remove approximation bias or every instability.
## Exercises
The checkpoint is this page's completion target. Additional practice is optional. Try the checkpoint before opening its hint or solution.
::: {#checkpoint-sp-10 .checkpoint}
### Checkpoint · SP-10 — Derive a nuisance rate and assess its dimension requirement {.unnumbered}
**Task type:** Derivation and interpretation.
Starting from $C_bh^4+C_v/(nh)$ with positive constants, derive the optimal bandwidth order and root-MISE order. Then use the benchmark exponent $s/(2s+d)$ to decide when two equal nuisance rates suffice for a product remainder. Compare $s=2$ with $d=1,4,5$, and explain what is assumed about the estimator and function class.
**Ready to continue when:** you can distinguish squared error from its square root and translate a smoothness/dimension assumption into a sufficient inferential condition.
**Transfer:** If one nuisance has rate n^(-1/2) and the other is merely L2-consistent, what does the product bound imply?
<details class="checkpoint-help">
<summary>Hint</summary>
Differentiate the leading error in h, then compare the sum of the two L2 exponents with 1/2.
</details>
<details class="checkpoint-help">
<summary>Solution outline — open after trying</summary>
The derivative equation is $4C_bh^3=C_v/(nh^2)$, so $h$ has order $n^{-1/5}$. Squared error has order $n^{-4/5}$ and root-MISE order $n^{-2/5}$. For equal rates, $2s/(2s+d)>1/2$ is equivalent to $s>d/2$. With $s=2$, the individual exponents are 2/5, 1/4, and 2/9 for dimensions 1, 4, and 5. Only the first gives strict negligibility from these big-O bounds; the boundary does not guarantee it. These are benchmark statements under suitable smoothness, design, moment, and estimator-order assumptions. A library call or one held-out score does not establish the rate.
</details>
:::
### Additional practice (optional) {.unnumbered}
1. Plot the fitted local-regression curves as well as their errors and identify where oversmoothing loses structure.
2. Compare a correctly specified quadratic regression and a linear regression for a nonlinear synthetic mean; separate approximation error from finite-sample noise.
3. Write a fold diagram for repeated specimens and identify every learned preprocessing step.
## Reading
- [Nonparametric Regression and Classification](https://stat.cmu.edu/~larry/%3Dsml/NonparametricPrediction.pdf), Ryan Tibshirani with Larry Wasserman: local polynomial regression and risk.
<span id="continue-through-the-collection"></span>
