Conditional expectation and L2 geometry

Residuals, projections, and the calculation behind nuisance adjustment.

Before reading: Expectations, covariance, and basic regression; see the What is your estimator actually estimating? page for the choice of adjustment.

1 The one idea

Much of semiparametric theory is a carefully organized version of a familiar move: remove from a variable the part another variable can predict, then work with what remains. In ordinary linear regression that means subtracting a fitted line. In the settings that matter later, the fitted object is allowed to be any function.

Let X be a measurement and Z recorded covariates, and assume E[X^2]<\infty. The conditional expectation

m_X(Z)=E[X\mid Z]

is a random variable obtained by applying a population prediction function to Z. At a particular covariate value z, m_X(z) is the conditional mean of X. For continuous covariates, this is defined through a conditional distribution; it is not a literal average over observations with exactly the same covariate value in a finite dataset.

For a discrete example, suppose half the people have Z=0 and half have Z=1, with conditional means 1 and 3. Then m_X(Z)=1+2Z. Subtracting 1 in the first stratum and 3 in the second leaves a residual whose mean is zero in each stratum, not just after pooling them.

The residual X-m_X(Z) has the following population property:

E\left[\{X-m_X(Z)\}h(Z)\right]=0.

This holds for every square-integrable function h(Z). It says that the residual is perpendicular to every allowed direction depending only on Z. The meaning of “perpendicular” here comes from an expectation inner product, rather than angles in an observation-level scatterplot.

Figure 1: Conditional expectation is an orthogonal projection. The horizontal line shows part of the space of functions of Z; the residual is perpendicular to the entire space. Any alternative h(Z) adds prediction error to the minimum possible mean squared error. The geometry is schematic, not a scatterplot of observations.

2 Expectations as averages and inner products

For a random variable U, E[U] is its population average. We usually cannot compute it exactly, so we replace it with the sample average

P_n U = \frac{1}{n}\sum_{i=1}^n U_i.

The notation P_n says “take an average.” Under the appropriate LLN, this sample quantity approaches the population expectation for a fixed integrable function of i.i.d. observations. The distinction matters: a population identity involving E need not be an exact identity after replacing the unknown function by a fitted one.

If U and V have finite second moments, define

\langle U,V\rangle = E[UV].

This behaves like a dot product. Its associated squared length is \|U\|^2 = E[U^2]. The collection of random variables with finite second moments is called L_2(P). You do not need functional analysis to use the notation. Keep three translations in mind:

• \langle U,V\rangle = 0 means that U and V are orthogonal.
• Orthogonal random variables are uncorrelated if both have mean zero.
• A projection is the closest element of a set under squared-error loss.

For centered variables, E[UV] is their covariance. An orthogonal residual is therefore uncorrelated with a function of the covariates under the stated moment conditions. Zero covariance is a statement about one expected product; independence is a stronger statement about an entire joint distribution.

Functions that agree with probability one have zero L_2 distance and are treated as the same element of this space. Thus a projection is unique up to equality almost surely. This concerns agreement on a population observation, rather than almost sure convergence of a sequence of estimates.

The sample version is ordinary linear algebra. If u and v are vectors of observed values, n^{-1}u^\top v estimates E[UV]. Exact perpendicularity of two fitted data vectors and population orthogonality of two random variables are related, but distinct, statements.

3 Conditional expectation is a projection

The conditional expectation E[X\mid Z] is the best prediction of X that can use Z, where “best” means smallest population mean squared error. With E[X^2]<\infty, conditional Jensen’s inequality gives E[m_X(Z)^2]\le E[X^2], so the prediction itself is square-integrable. For every square-integrable candidate h(Z),

E\left[\{X-E[X\mid Z]\}^2\right] \leq E\left[\{X-h(Z)\}^2\right].

It is not necessary for E[X\mid Z] to be linear, smooth, or easy to estimate. It is the population object that wins among all functions of Z.

The proof gives the fact we will use. Let m(Z)=E[X\mid Z]. For any h(Z),

\begin{aligned} E[(X-h)^2]&=E[(X-m+m-h)^2]\\ &=E[(X-m)^2]+E[(m-h)^2]+2E[(X-m)(m-h)]. \end{aligned}

The final term is zero. First, conditioning on Z gives

\begin{aligned} E[X-m\mid Z]&=E[X\mid Z]-m(Z)\\ &=0. \end{aligned}

Next, (m-h) is fixed once Z is known, so it can be taken outside that conditional expectation. Finally, the law of total expectation averages the conditional result:

\begin{aligned} E[(X-m)(m-h)]&=E[(m-h)E[X-m\mid Z]]\\ &=0. \end{aligned}

Finite second moments make this product integrable by Cauchy–Schwarz. The extra error from predicting with h is therefore exactly E[(m-h)^2]\ge0. Equality holds precisely when h(Z)=m(Z) with probability one. This is the Pythagorean theorem in L_2(P) and the uniqueness statement behind “best prediction.”

3.1 Check it with data

Here X has a nonlinear conditional mean. A straight line misses the curve; a spline gets closer. Neither fact changes the population definition of m_X.

set.seed(1)
n <- 800
z <- runif(n, -1, 1)
x <- sin(2 * pi * z) + rnorm(n, sd = 0.45)

fit_line <- lm(x ~ z)
fit_spline <- smooth.spline(z, x, df = 10)
x_line <- fitted(fit_line)
x_spline <- predict(fit_spline, z)$y

c(linear_MSE = mean((x - x_line)^2),
  spline_MSE = mean((x - x_spline)^2),
  no_adjustment_MSE = mean((x - mean(x))^2))

       linear_MSE        spline_MSE no_adjustment_MSE 
        0.6541736         0.2248078         0.7259596 

Here the true conditional mean is \sin(2\pi Z) and the irreducible conditional noise variance is 0.45^2. The reported MSEs are training errors, evaluated on the data used to fit each predictor. They illustrate how the chosen fits follow this sample; they do not estimate fresh-population risk without further justification. A sufficiently flexible fit could reduce training error while predicting poorly on new people.

The population projection is defined before choosing an algorithm. To assess a learner’s approximation to it, evaluate prediction error on independent observations or use an appropriate validation scheme. The optional exercises include this comparison.

4 The residual is orthogonal to every function of Z

Set \widetilde X = X-m_X(Z). For any h(Z),

\begin{aligned} E[\widetilde X h(Z)]&=E\left[E\{\widetilde X h(Z)\mid Z\}\right]\\ &=E\left[h(Z)E\{X-m_X(Z)\mid Z\}\right]\\ &=0. \end{aligned}

The first step is the law of total expectation; the second takes h(Z) outside the conditional expectation and uses the conditional mean-zero residual. This is the residualization identity. It contains several familiar special cases, whenever the displayed functions have the required moments.

• Taking h(Z)=1 gives E[\widetilde X]=0.
• Taking h(Z)=Z gives \operatorname{Cov}(\widetilde X,Z)=0.
• Taking h(Z)=Z^2, \sin(Z), or an indicator of a subgroup gives the same result.

In a finite-sample OLS fit, the normal equations make the fitted residual vector orthogonal to the included design columns. At the population level, a best linear predictor satisfies the corresponding finite set of moment equations. The true conditional-expectation residual satisfies the larger collection of equations for every square-integrable function of Z. Estimating a flexible regression aims to approximate this population object; it does not grant all these identities automatically.

Orthogonality also need not remove dependence. Let Z be binary and U an independent, mean-zero, unit-variance variable. For X=Z+(1+Z)U, the conditional mean is Z and the residual is (1+Z)U. Its conditional mean is zero, yet its variance is 1 when Z=0 and 4 when Z=1. The residual’s distribution still depends on Z.

# The residual from a flexible fit is nearly orthogonal to several functions of z.
r <- x - x_spline
c(mean = mean(r),
  z = mean(r * z),
  z_squared = mean(r * z^2),
  sine = mean(r * sin(2 * pi * z)))

         mean             z     z_squared          sine 
 5.027173e-14  1.068541e-13 -6.825678e-06  2.476831e-02 

These are sample products using an estimated function on its own training data. They are a numerical illustration, not a proof of population orthogonality. If a square-integrable fitted function \hat m_X is instead held fixed and evaluated on a fresh observation, conditioning on Z gives

E[(X-\hat m_X(Z))h(Z)]=E[(m_X(Z)-\hat m_X(Z))h(Z)].

Its remaining population moment depends on regression error. Orthogonality and cross-fitting explains how training/evaluation separation and nuisance accuracy enter inference.

5 Conditional covariance and partialling out

For two variables X and Y with finite second moments, write

\psi_{XY}=E\big[\{X-m_X(Z)\} \times\{Y-m_Y(Z)\}\big].

Condition on Z and expand the product:

\begin{aligned} E[(X-m_X)(Y-m_Y)\mid Z]&=E[XY\mid Z]-m_X(Z)m_Y(Z)\\ &=\operatorname{Cov}(X,Y\mid Z). \end{aligned}

The outer expectation therefore averages conditional covariance over the population’s covariate distribution. The analogous residual variances are \psi_{XX}=E[\operatorname{Var}(X\mid Z)] and \psi_{YY}=E[\operatorname{Var}(Y\mid Z)]. When both are finite and positive, their ratio gives a residual correlation:

\rho_{XY\cdot Z} = \frac{\psi_{XY}}{\sqrt{\psi_{XX}\psi_{YY}}}.

The expression is useful in network work: a marginal correlation can be large merely because two genes both change with cell composition, batch, or disease severity. Residualizing against a measured Z asks about association left after that predictable structure is removed.

The order of operations matters: averaging covariances and then standardizing generally differs from averaging within-stratum correlations. Also, positive and negative conditional covariances can cancel in the population average. A zero value does not even force every stratum’s covariance to be zero, much less establish conditional independence.

These are associational targets. Whether adjusting for Z is scientifically appropriate, or supports a causal interpretation, requires the argument in What is your estimator actually estimating?.

6 The partially linear model

A later efficiency calculation uses the partially linear model:

\begin{gathered} Y=\beta X+g(Z)+\varepsilon,\\ E[\varepsilon\mid X,Z]=0. \end{gathered}

Here \beta is the quantity of interest and g is an unknown nuisance function. Taking conditional expectations given Z gives

E[Y\mid Z] = \beta E[X\mid Z] + g(Z).

Subtract this equation from the original one:

Y-m_Y(Z) = \beta\{X-m_X(Z)\}+\varepsilon.

The error term has mean zero conditional on (X,Z), so

\begin{aligned} E[(X-m_X(Z))\varepsilon]&=E[(X-m_X(Z))E[\varepsilon\mid X,Z]]\\ &=0. \end{aligned}

Multiplying the residualized outcome equation by X-m_X(Z) and taking expectations therefore gives the following ratio, provided its denominator is positive:

\beta = \frac{E[\{X-m_X(Z)\}\{Y-m_Y(Z)\}]} {E[\{X-m_X(Z)\}^2]}.

This is Robinson’s partialling-out identity. The numerator is the residual cross moment and the denominator is residual exposure variation, just as in a regression slope. If X=m_X(Z) almost surely, that denominator is zero: changing \beta can be absorbed into the unrestricted function g(Z), so the coefficient cannot be separated from that nuisance. Positive residual variation is the identification condition. Efficiency and the cost of estimating the nuisance functions require the later model-specific arguments.

set.seed(2)
n <- 1000
z <- runif(n)
x <- 1.5 * sin(2 * pi * z) + rnorm(n)
y <- 0.6 * x + sin(2 * pi * z) + rnorm(n)

mx <- predict(smooth.spline(z, x, df = 12), z)$y
my <- predict(smooth.spline(z, y, df = 12), z)$y
x_res <- x - mx; y_res <- y - my

c(true_beta = 0.6,
  naive_linear_adjustment = coef(lm(y ~ x + z))["x"],
  residual_slope = sum(x_res * y_res) / sum(x_res^2))

                true_beta naive_linear_adjustment.x            residual_slope 
                0.6000000                 0.8203026                 0.6245794 

Here the shared sine component remains after linear adjustment for Z, producing a population slope bias. Nonlinearity alone would not imply that bias: the omitted component must also be associated with the residual exposure. Spline residualization removes most of the shared nonlinear component in this example. In a real analysis we must still account for errors in \hat m_X and \hat m_Y; that is the later reason for orthogonal formulas and cross-fitting.

7 Where this connects

• The What is your estimator actually estimating? page asks whether adjustment for Z answers the scientific question. This page shows the population calculation once the adjustment is justified.
• The Estimating equations and asymptotic linearity page treats the residual identity as an equation whose solution is an estimator.
• The Likelihood, scores, and parametric efficiency page shows the same residualization as projection of a score away from a finite-dimensional nuisance.
• The Model restrictions and efficiency page makes the nuisance space infinite-dimensional and derives the efficient influence function.

8 Exercises

Checkpoints are completion targets for this page. Additional practice below is optional. Each checkpoint gives a completion standard, a hint, and a solution outline.

Checkpoint · SP-02 — Prove the projection property

Task type: Proof and interpretation.

Assume E[X^2]<\infty, let m(Z)=E[X\mid Z], and let h(Z) be square-integrable.

1. Prove E[(X-m(Z))h(Z)]=0 using conditional expectation.
2. Derive E[(X-h)^2]=E[(X-m)^2]+E[(m-h)^2].
3. Conclude that m minimizes mean squared error and state exactly when equality holds.
4. Explain why perpendicularity here refers to random variables in L_2(P), rather than a scatterplot of individual observations.

Ready to continue when: you can derive the decomposition, justify the vanishing cross term, and state uniqueness up to almost-sure equality.

Transfer: Use h(Z)=a+bZ with population coefficients. Under what condition can this linear predictor achieve the population minimum over all functions of Z? Distinguish that condition from estimating its coefficients in a finite sample.

Hint

Write X-h=(X-m)+(m-h) and condition the cross term on Z.

Solution outline — open after trying

Because h is a function of Z, E[(X-m)h]=E[hE[X-m\mid Z]]=0. The same argument applies to m-h. Squaring X-h=(X-m)+(m-h) and taking expectations therefore removes the cross term. The remaining extra term E[(m-h)^2] is nonnegative and is zero exactly when h=m almost surely. Finite second moments make the products integrable by Cauchy–Schwarz. This is orthogonality under the inner product E[UV].

Additional practice (optional)

These exercises extend the ideas through further derivations, simulations, or applications. Choose those useful for your work.

1. In the simulation above, replace 1.5\sin(2\pi Z) by a linear function of Z. Compare linear adjustment and spline residualization. When is the simpler adjustment enough?
2. Simulate X = Z + U and Y = Z + U + V, where Z,U,V are independent standard normals. Compare the marginal correlation of X and Y with their residual correlation after adjusting for Z. Explain both numbers.
3. Evaluate the fitted line and spline from the first example on a fresh sample from the same generator. Compare training MSE, validation MSE, and the population noise variance 0.45^2.

9 Reading

• Tsiatis, A. A. (2006). Semiparametric Theory and Missing Data. Springer. Sections 3.1 to 3.3 for the geometry in full generality.
• van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge. Sections 25.1 and 25.2 for Hilbert-space notation used in semiparametric theory.
• Robinson, P. M. (1988). Root-N-consistent semiparametric regression. Econometrica, 56(4), 931 to 954. The partialling-out construction.
• Buja, A., Brown, L., and Berk, R. (2019). Models as approximations, Part I. Statistical Science, 34(4), 523 to 544. A useful account of regression adjustment when a linear model is only an approximation.