Orthogonality and cross-fitting

Control two different errors while estimating the same missing-outcome mean

Before reading: Missingness, selection, and the observed-data target, Influence functions, and stochastic orders from Large-sample reasoning for methods research.

1 Keep the target fixed while learning nuisances

The target is the eligible-population feature mean \mu=E[Y]. We observe independent records W=(Z,R,RY): Z is recorded for everyone, and R=1 means the outcome is observed. The product RY encodes an unrecorded outcome when R=0; it does not say that person’s outcome equals zero.

Assume an integrable outcome, missing at random (MAR), R\perp Y\mid Z, and positive observation probability over the target population. Then

\begin{aligned} m_0(z)&=E[Y\mid Z=z,R=1],\\ \pi_0(z)&=P(R=1\mid Z=z),\\ \mu&=E[m_0(Z)]. \end{aligned}

The subscript 0 marks the true population functions; hats will mark estimates. MAR connects the observed-outcome regression to the full-population conditional mean. The outer expectation uses the covariate distribution of all eligible people. Neither learning nor cross-fitting supplies this identifying assumption.

Write Pf=E[f(W)] and P_nf=n^{-1}\sum_i f(W_i). For a function of Z, \|f\|_2=\sqrt{E[f(Z)^2]}, averaging over the eligible population. A fitted function is random across training datasets; when we hold a fit fixed, P averages over a fresh observation with that fit unchanged.

Averaging fitted means alone can be sensitive to regression error. An augmented contribution combines a prediction with an observed residual:

\begin{aligned} D_\eta(W)&=m(Z)+\frac{R}{\pi(Z)}\{Y-m(Z)\},\\ \eta&=(m,\pi). \end{aligned}

Every person contributes a prediction m(Z). A person with an observed outcome also contributes a residual, weighted by the inverse observation probability. At the true functions, E[R(Y-m_0)\mid Z]=0, so PD_{\eta_0}=\mu. If m=3, \pi=1/2, and an observed outcome is 5, the contribution is 3+2(5-3)=7; a person with the same covariates and a missing outcome contributes 3. These are estimation contributions, not imputed individual outcomes, so they need not stay within the outcome’s range.

The centered function \phi_0=D_{\eta_0}-\mu is the observed-data influence function derived in From an influence function to an estimator. The mean-zero equation D_\eta-\mu is an estimating function; D_\eta is not a likelihood score. This page explains why estimating its nuisance arguments requires both bias control and careful training/evaluation separation.

2 Orthogonality concerns the supplied nuisance functions

Hold the population law and target fixed. Change only the nuisance arguments supplied to M(\eta)=PD_\eta. Neyman orthogonality means that the first derivative of this map is zero at the true nuisances along the permitted perturbations. This differs from a score-path derivative, where the distribution and target can change, and from independence of two random variables. Here it says that a small nuisance error has no first-order effect on the population estimating equation.

To see the cancellation, fix fitted functions with \widehat\pi\ge c>0. Conditioning on Z gives

\begin{aligned} E[R(Y-\widehat m)\mid Z]&=\pi_0(m_0-\widehat m),\\ E[D_{\widehat\eta}\mid Z]-m_0&=(\widehat m-m_0) \left(1-\frac{\pi_0}{\widehat\pi}\right). \end{aligned}

The first term in the contribution introduces prediction error; the residual correction subtracts the part scaled by \pi_0/\widehat\pi. Averaging the remaining difference gives the exact identity

PD_{\widehat\eta}-\mu =E\left[\frac{\widehat\pi-\pi_0}{\widehat\pi} (\widehat m-m_0)\right].

Write a perturbation as m_0+\varepsilon h_m, \pi_0+\varepsilon h_\pi, keeping probabilities admissible. Its bias is

\varepsilon^2E\left[\frac{h_\pi h_m}{\pi_0+\varepsilon h_\pi}\right].

Under the needed integrability and denominator bounds, dividing by \varepsilon and taking \varepsilon\to0 gives derivative zero. The first surviving change is a product of the two perturbations. By comparison, the regression-only contribution has population change \varepsilon E[h_m], generally first order. Augmentation removes that term. The exact identity shows which product remains; a zero formal derivative alone is not a general rate theorem.

For a numerical check, take \pi_0=0.5, \widehat\pi=0.6, and \widehat m-m_0=0.2 everywhere. The exact bias is (0.1/0.6)0.2=1/30, compared with 0.2 for prediction alone. Halving both errors gives (0.05/0.55)0.1=1/110, approximately a quarter as large. The denominator also changes, so the finite change need not be exactly quadratic.

3 Consistency and root-n inference are different claims

Cauchy–Schwarz yields

\begin{aligned} |PD_{\widehat\eta}-\mu| &\le E\left[\frac{|\widehat\pi-\pi_0|}{c}|\widehat m-m_0|\right]\\ &\le c^{-1}\|\widehat\pi-\pi_0\|_2\|\widehat m-m_0\|_2. \end{aligned}

If either supplied function is exactly correct, this population bias is zero. For learned functions, one nuisance converging to truth can make the bias vanish if the other error stays bounded in the required norm and the denominator remains controlled. With a vanishing empirical error, this gives double-robust consistency. “Double” refers to two opportunities for this consistency property; it does not mean robustness to MAR failure or to arbitrary errors in both functions.

Consistency asks only that the estimation error vanish. A centered root-n limit asks for the nuisance bias to be negligible compared with the leading n^{-1/2} sampling fluctuation. The efficient-inference argument using \phi_0 here assumes both nuisances converge, a product o_p(n^{-1/2}), and the empirical and variance conditions below. These are sufficient conditions for this proof. A different limit with one incorrect nuisance needs its own expansion and variance analysis.

If the two L2 errors are O_p(n^{-a}) and O_p(n^{-b}), their product is O_p(n^{-(a+b)}). Multiplication by \sqrt n leaves order O_p(n^{1/2-a-b}). Thus a+b>1/2 is sufficient through this bound.

L2 exponents (a,b)	Product order	Root-n bias guarantee from the bound
(1/3,1/5)	O_p(n^{-8/15})	Negligible: 8/15>1/2
(1/4,1/4)	O_p(n^{-1/2})	Boundary: big-O alone is insufficient
(1/5,1/5)	O_p(n^{-2/5})	No negligibility guarantee

At the boundary, stronger little-o control of at least one factor can suffice. If the bound is too large, the actual signed expectation might still cancel; failing a sufficient condition is not itself proof of bias. Faster convergence of one nuisance can compensate for slower convergence of the other. Estimating nuisance functions for inference connects these exponents to regression learning.

4 Cross-fitting controls reuse of evaluation noise

Split independent people into a fixed number K of folds, using a split independent of their measurements. Fit nuisance functions outside each validation fold and evaluate the contribution inside it. In a two-fold example:

Contributions being evaluated	Data allowed to train those fits
People in fold A	Fold B
People in fold B	Fold A

Every person is evaluated once and can help train the fit used for other people. Cross-fitting recovers use of the full sample for evaluation; it does not train any one fit on the full sample. Train learned preprocessing, selection, and tuning inside the relevant training set. For repeated specimens, keep an entire person together and adapt the variance argument to the independent unit.

Let I_k be validation fold k, n_k its size, P_{n,k} its average, and \widehat\eta_k the fit trained outside that fold. The subscript k labels the held-out fold, not data used for training. The estimator is

\widehat\mu=\sum_{k=1}^K\frac{n_k}{n} P_{n,k}D_{\widehat\eta_k}.

Fold-size weights make this exactly the average of the n held-out contributions. A simple unweighted average of fold estimates would generally give different person weights when fold sizes differ.

For validation fold k, condition on its training data. Its fitted functions are then fixed, while the validation observations are fresh independent draws. This permits a conditional-variance argument for the fitted-minus-true empirical term. Conditioning on all fitted functions simultaneously would be a different step: the fits use observations from other validation folds, so the full set of contributions is not jointly independent.

Cross-fitting does not identify an unsupported target, make a bad learner consistent, or remove the product bias. For example, constant nuisance errors of order n^{-1/5} can leave a bias of order n^{-2/5}, which grows after root-n scaling. Separated folds do not change that population calculation.

5 Locate the two remainders

Let H_k=D_{\widehat\eta_k}-D_{\eta_0} and w_k=n_k/n. Add and subtract the true contribution, and then its population average. The error decomposes exactly as

\widehat\mu-\mu=(P_n-P)\phi_0 +\sum_k w_k(P_{n,k}-P)H_k +\sum_k w_kPH_k.

The first term averages the same fixed, mean-zero \phi_0 over everyone. With finite nonzero variance it has a CLT. The middle term is a centered empirical remainder: how much the evaluation sample’s average fitted error differs from its fresh-population average. Conditional on training, its scaled variance is at most PH_k^2, so \|H_k\|_2=o_p(1) makes it negligible. The last term is the population bias remainder, the exact product derived above. Cross-fitting does not shrink that product by itself.

To obtain the usual limit, both remainders must be o_p(n^{-1/2}). Then the leading contribution determines the limiting law. A feasible SE still needs consistent contribution-variance estimation and finite nonzero limiting variance; a small computed SE is not evidence that the bias remainder is small.

From an influence function to an estimator completes the derivation, conditional-Chebyshev proof, implementation, and uncertainty argument for this same target. Residual covariance and nuisance-estimation error is a network-analysis transfer with its own gradient and remainder, rather than another name for the missing mean.

6 Exercises

The checkpoint is a completion target. Additional practice is optional.

Checkpoint · SP-11 — Bound the nuisance-bias remainder

Task type: Derivation and rate reasoning.

Fix nuisance fits trained independently of a fresh observation. Under the missing-mean identification assumptions, derive PD_{\widehat\eta}-\mu. Assume \widehat\pi\ge c>0 and square-integrable nuisance errors. Bound the bias using Cauchy–Schwarz. If the L2 errors are O_p(n^{-a}) and O_p(n^{-b}), give a sufficient condition for a negligible root-n bias. Check exponent pairs (1/3,1/5), (1/4,1/4), and (1/5,1/5). Explain which additional random term this calculation does not control.

Ready to continue when: you can derive the exact product remainder, distinguish big-O from little-o, and identify the additional empirical and variance arguments needed for inference.

Transfer: Compare this weighted product with the residual-covariance product in the application page. What role does the denominator bound play?

Hint

Condition on Z: E[R(Y-\widehat m)\mid Z]=\pi_0(m_0-\widehat m). Combine the prediction and residual terms before applying the norm bound.

Solution outline — open after trying

The bias is E[(\widehat\pi-\pi_0)(\widehat m-m_0)/\widehat\pi], bounded by c^{-1} times the two L2 errors. An exponent sum greater than 1/2 guarantees little-o through this bound. The first pair satisfies it, equality at (1/4,1/4) is insufficient from big-O alone, and (1/5,1/5) gives no negligible-bias guarantee. Failing a sufficient bound alone is not proof of actual failure, since a particular inner product may cancel. The fitted-function empirical remainder and variance consistency remain to be controlled. Cross-fitting addresses the former under its own conditions, not the population bias.

Additional practice (optional)

Set one nuisance equal to truth and derive the remaining population bias.
Use bounded perturbation directions to verify the zero first derivative directly.
Draw the fitting/evaluation scheme for repeated specimens from independent people.

--- title: "Orthogonality and cross-fitting" subtitle: "Control two different errors while estimating the same missing-outcome mean" aliases: - "/guides/statistical-background/05-semiparametric-orthogonality.html" - "/guides/statistical-background/06-semiparametric-orthogonality.html" --- **Before reading:** [Missingness, selection, and the observed-data target](../questions-and-design/missingness-and-selection.qmd), [Influence functions](../statistical-models/influence-functions.qmd), and stochastic orders from [Large-sample reasoning for methods research](../estimation-and-inference/large-sample-reasoning.qmd). ## Keep the target fixed while learning nuisances The target is the eligible-population feature mean $\mu=E[Y]$. We observe independent records $W=(Z,R,RY)$: $Z$ is recorded for everyone, and $R=1$ means the outcome is observed. The product $RY$ encodes an unrecorded outcome when $R=0$; it does not say that person's outcome equals zero. Assume an integrable outcome, **missing at random** (MAR), $R\perp Y\mid Z$, and positive observation probability over the target population. Then $$ \begin{aligned} m_0(z)&=E[Y\mid Z=z,R=1],\\ \pi_0(z)&=P(R=1\mid Z=z),\\ \mu&=E[m_0(Z)]. \end{aligned} $$ The subscript 0 marks the true population functions; hats will mark estimates. MAR connects the observed-outcome regression to the full-population conditional mean. The outer expectation uses the covariate distribution of all eligible people. Neither learning nor cross-fitting supplies this identifying assumption. Write $Pf=E[f(W)]$ and $P_nf=n^{-1}\sum_i f(W_i)$. For a function of $Z$, $\|f\|_2=\sqrt{E[f(Z)^2]}$, averaging over the eligible population. A fitted function is random across training datasets; when we hold a fit fixed, $P$ averages over a fresh observation with that fit unchanged. Averaging fitted means alone can be sensitive to regression error. An augmented contribution combines a prediction with an observed residual: $$ \begin{aligned} D_\eta(W)&=m(Z)+\frac{R}{\pi(Z)}\{Y-m(Z)\},\\ \eta&=(m,\pi). \end{aligned} $$ Every person contributes a prediction $m(Z)$. A person with an observed outcome also contributes a residual, weighted by the inverse observation probability. At the true functions, $E[R(Y-m_0)\mid Z]=0$, so $PD_{\eta_0}=\mu$. If $m=3$, $\pi=1/2$, and an observed outcome is 5, the contribution is $3+2(5-3)=7$; a person with the same covariates and a missing outcome contributes 3. These are estimation contributions, not imputed individual outcomes, so they need not stay within the outcome's range. The centered function $\phi_0=D_{\eta_0}-\mu$ is the observed-data influence function derived in [From an influence function to an estimator](influence-function-to-estimator.qmd). The mean-zero equation $D_\eta-\mu$ is an estimating function; $D_\eta$ is not a likelihood score. This page explains why estimating its nuisance arguments requires both bias control and careful training/evaluation separation. ## Orthogonality concerns the supplied nuisance functions Hold the population law and target fixed. Change only the nuisance arguments supplied to $M(\eta)=PD_\eta$. **Neyman orthogonality** means that the first derivative of this map is zero at the true nuisances along the permitted perturbations. This differs from a score-path derivative, where the distribution and target can change, and from independence of two random variables. Here it says that a small nuisance error has no first-order effect on the population estimating equation. To see the cancellation, fix fitted functions with $\widehat\pi\ge c>0$. Conditioning on $Z$ gives $$ \begin{aligned} E[R(Y-\widehat m)\mid Z]&=\pi_0(m_0-\widehat m),\\ E[D_{\widehat\eta}\mid Z]-m_0&=(\widehat m-m_0) \left(1-\frac{\pi_0}{\widehat\pi}\right). \end{aligned} $$ The first term in the contribution introduces prediction error; the residual correction subtracts the part scaled by $\pi_0/\widehat\pi$. Averaging the remaining difference gives the exact identity $$ PD_{\widehat\eta}-\mu =E\left[\frac{\widehat\pi-\pi_0}{\widehat\pi} (\widehat m-m_0)\right]. $$ Write a perturbation as $m_0+\varepsilon h_m$, $\pi_0+\varepsilon h_\pi$, keeping probabilities admissible. Its bias is $$ \varepsilon^2E\left[\frac{h_\pi h_m}{\pi_0+\varepsilon h_\pi}\right]. $$ Under the needed integrability and denominator bounds, dividing by $\varepsilon$ and taking $\varepsilon\to0$ gives derivative zero. The first surviving change is a product of the two perturbations. By comparison, the regression-only contribution has population change $\varepsilon E[h_m]$, generally first order. Augmentation removes that term. The exact identity shows which product remains; a zero formal derivative alone is not a general rate theorem. For a numerical check, take $\pi_0=0.5$, $\widehat\pi=0.6$, and $\widehat m-m_0=0.2$ everywhere. The exact bias is $(0.1/0.6)0.2=1/30$, compared with 0.2 for prediction alone. Halving both errors gives $(0.05/0.55)0.1=1/110$, approximately a quarter as large. The denominator also changes, so the finite change need not be exactly quadratic. ## Consistency and root-n inference are different claims Cauchy–Schwarz yields $$ \begin{aligned} |PD_{\widehat\eta}-\mu| &\le E\left[\frac{|\widehat\pi-\pi_0|}{c}|\widehat m-m_0|\right]\\ &\le c^{-1}\|\widehat\pi-\pi_0\|_2\|\widehat m-m_0\|_2. \end{aligned} $$ If either supplied function is exactly correct, this population bias is zero. For learned functions, one nuisance converging to truth can make the bias vanish if the other error stays bounded in the required norm and the denominator remains controlled. With a vanishing empirical error, this gives **double-robust consistency**. “Double” refers to two opportunities for this consistency property; it does not mean robustness to MAR failure or to arbitrary errors in both functions. Consistency asks only that the estimation error vanish. A centered root-$n$ limit asks for the nuisance bias to be negligible compared with the leading $n^{-1/2}$ sampling fluctuation. The efficient-inference argument using $\phi_0$ here assumes both nuisances converge, a product $o_p(n^{-1/2})$, and the empirical and variance conditions below. These are sufficient conditions for this proof. A different limit with one incorrect nuisance needs its own expansion and variance analysis. If the two L2 errors are $O_p(n^{-a})$ and $O_p(n^{-b})$, their product is $O_p(n^{-(a+b)})$. Multiplication by $\sqrt n$ leaves order $O_p(n^{1/2-a-b})$. Thus $a+b>1/2$ is sufficient through this bound. | L2 exponents $(a,b)$ | Product order | Root-$n$ bias guarantee from the bound | |---|---|---| | $(1/3,1/5)$ | $O_p(n^{-8/15})$ | Negligible: $8/15>1/2$ | | $(1/4,1/4)$ | $O_p(n^{-1/2})$ | Boundary: big-O alone is insufficient | | $(1/5,1/5)$ | $O_p(n^{-2/5})$ | No negligibility guarantee | At the boundary, stronger little-o control of at least one factor can suffice. If the bound is too large, the actual signed expectation might still cancel; failing a sufficient condition is not itself proof of bias. Faster convergence of one nuisance can compensate for slower convergence of the other. [Estimating nuisance functions for inference](nuisance-estimation.qmd) connects these exponents to regression learning. ## Cross-fitting controls reuse of evaluation noise Split independent people into a fixed number $K$ of folds, using a split independent of their measurements. Fit nuisance functions outside each validation fold and evaluate the contribution inside it. In a two-fold example: | Contributions being evaluated | Data allowed to train those fits | |---|---| | People in fold A | Fold B | | People in fold B | Fold A | Every person is evaluated once and can help train the fit used for other people. Cross-fitting recovers use of the full sample for evaluation; it does not train any one fit on the full sample. Train learned preprocessing, selection, and tuning inside the relevant training set. For repeated specimens, keep an entire person together and adapt the variance argument to the independent unit. Let $I_k$ be validation fold $k$, $n_k$ its size, $P_{n,k}$ its average, and $\widehat\eta_k$ the fit trained outside that fold. The subscript $k$ labels the held-out fold, not data used for training. The estimator is $$ \widehat\mu=\sum_{k=1}^K\frac{n_k}{n} P_{n,k}D_{\widehat\eta_k}. $$ Fold-size weights make this exactly the average of the $n$ held-out contributions. A simple unweighted average of fold estimates would generally give different person weights when fold sizes differ. For validation fold $k$, condition on its training data. Its fitted functions are then fixed, while the validation observations are fresh independent draws. This permits a conditional-variance argument for the fitted-minus-true empirical term. Conditioning on all fitted functions simultaneously would be a different step: the fits use observations from other validation folds, so the full set of contributions is not jointly independent. Cross-fitting does not identify an unsupported target, make a bad learner consistent, or remove the product bias. For example, constant nuisance errors of order $n^{-1/5}$ can leave a bias of order $n^{-2/5}$, which grows after root-n scaling. Separated folds do not change that population calculation. ## Locate the two remainders Let $H_k=D_{\widehat\eta_k}-D_{\eta_0}$ and $w_k=n_k/n$. Add and subtract the true contribution, and then its population average. The error decomposes exactly as $$ \widehat\mu-\mu=(P_n-P)\phi_0 +\sum_k w_k(P_{n,k}-P)H_k +\sum_k w_kPH_k. $$ The first term averages the same fixed, mean-zero $\phi_0$ over everyone. With finite nonzero variance it has a CLT. The middle term is a **centered empirical remainder**: how much the evaluation sample's average fitted error differs from its fresh-population average. Conditional on training, its scaled variance is at most $PH_k^2$, so $\|H_k\|_2=o_p(1)$ makes it negligible. The last term is the **population bias remainder**, the exact product derived above. Cross-fitting does not shrink that product by itself. To obtain the usual limit, both remainders must be $o_p(n^{-1/2})$. Then the leading contribution determines the limiting law. A feasible SE still needs consistent contribution-variance estimation and finite nonzero limiting variance; a small computed SE is not evidence that the bias remainder is small. [From an influence function to an estimator](influence-function-to-estimator.qmd) completes the derivation, conditional-Chebyshev proof, implementation, and uncertainty argument for this same target. [Residual covariance and nuisance-estimation error](../analysis-problems/residual-covariance-and-nuisance-error.qmd) is a network-analysis transfer with its own gradient and remainder, rather than another name for the missing mean. ## Exercises The checkpoint is a completion target. Additional practice is optional. ::: {#checkpoint-sp-11 .checkpoint} ### Checkpoint · SP-11 — Bound the nuisance-bias remainder {.unnumbered} **Task type:** Derivation and rate reasoning. Fix nuisance fits trained independently of a fresh observation. Under the missing-mean identification assumptions, derive $PD_{\widehat\eta}-\mu$. Assume $\widehat\pi\ge c>0$ and square-integrable nuisance errors. Bound the bias using Cauchy–Schwarz. If the L2 errors are $O_p(n^{-a})$ and $O_p(n^{-b})$, give a sufficient condition for a negligible root-n bias. Check exponent pairs (1/3,1/5), (1/4,1/4), and (1/5,1/5). Explain which additional random term this calculation does not control. **Ready to continue when:** you can derive the exact product remainder, distinguish big-O from little-o, and identify the additional empirical and variance arguments needed for inference. **Transfer:** Compare this weighted product with the residual-covariance product in the application page. What role does the denominator bound play? <details class="checkpoint-help"> <summary>Hint</summary> Condition on Z: $E[R(Y-\widehat m)\mid Z]=\pi_0(m_0-\widehat m)$. Combine the prediction and residual terms before applying the norm bound. </details> <details class="checkpoint-help"> <summary>Solution outline — open after trying</summary> The bias is $E[(\widehat\pi-\pi_0)(\widehat m-m_0)/\widehat\pi]$, bounded by $c^{-1}$ times the two L2 errors. An exponent sum greater than 1/2 guarantees little-o through this bound. The first pair satisfies it, equality at (1/4,1/4) is insufficient from big-O alone, and (1/5,1/5) gives no negligible-bias guarantee. Failing a sufficient bound alone is not proof of actual failure, since a particular inner product may cancel. The fitted-function empirical remainder and variance consistency remain to be controlled. Cross-fitting addresses the former under its own conditions, not the population bias. </details> ::: ### Additional practice (optional) {.unnumbered} 1. Set one nuisance equal to truth and derive the remaining population bias. 2. Use bounded perturbation directions to verify the zero first derivative directly. 3. Draw the fitting/evaluation scheme for repeated specimens from independent people. ## Related calculations The relocated worked calculation is in [Residual covariance and nuisance-estimation error](../analysis-problems/residual-covariance-and-nuisance-error.qmd).