Likelihood, scores, and parametric efficiency

Scores, Fisher information, nuisance parameters, and the finite-dimensional version of the semiparametric story.

Before reading: Estimating equations and the projection identity from Conditional expectation and L2 geometry.

1 The one idea

Likelihood theory gives the simplest version of a semiparametric problem. A parameter of interest changes the distribution in one direction; a nuisance parameter changes it in another. The score measures each direction. To estimate the parameter of interest efficiently, keep the part of its score that the nuisance cannot imitate.

In a finite-dimensional model this is ordinary projection of one vector of scores on another. In a semiparametric model, the nuisance has infinitely many directions, but the calculation is the same. This page develops the finite-dimensional version first.

2 Likelihood and score

Suppose one observation record W has density or probability mass function p_\theta(w), indexed by a scalar parameter \theta. Lowercase w denotes a possible value of that record. The log likelihood for a sample is

\ell_n(\theta) = \sum_{i=1}^n \log p_\theta(W_i).

The i.i.d. model makes the joint density a product, and taking logarithms turns it into this sum. For observed data, likelihood is a function of candidate parameter values: it compares how the model describes the same dataset as \theta varies. It is not a probability density over \theta.

The score is the derivative of the log density,

S_\theta(W) = \frac{\partial}{\partial\theta}\log p_\theta(W).

For one observation, S_\theta(W) is a random variable. Its value tells us the local slope of the log density at a candidate parameter. The total sample score is \ell_n'(\theta)=\sum_iS_\theta(W_i); dividing by n gives the average score. An interior, differentiable maximum likelihood estimate satisfies

P_n S_{\hat\theta}(W)=0.

So maximum likelihood is a special case of the estimating-equation framework in the Estimating equations and asymptotic linearity page. Its special feature is that the same derivative that defines the equation also encodes how the distribution changes when \theta changes.

This zero condition need not hold at a boundary or identify the global maximum. For the regular model calculation below, assume the needed differentiability and integrability, a common support, and valid interchanges of derivatives and integrals. Then the population score has mean zero at the truth:

E_\theta[S_\theta(W)] = 0.

To see why, differentiate \int p_\theta(w)\,dw=1:

\begin{aligned} 0&=\int\frac{\partial}{\partial\theta}p_\theta(w)\,dw\\ &=\int p_\theta(w)\frac{\partial}{\partial\theta}\log p_\theta(w)\,dw. \end{aligned}

The score is therefore a mean-zero random variable, just like the estimating functions in Estimating equations and asymptotic linearity.

The expectation here is under the same model value at which the score is evaluated. It is not a statement that an observed sample score vanishes at every candidate value. Parameter-dependent supports can invalidate this calculation: the uniform-endpoint example in Bootstrap and resampling for statistical inference has boundary behavior that these regular score identities do not capture.

3 Normal mean: the complete calculation

Let Y\sim N(\mu,\sigma^2), with \sigma^2 known. Then

\log p_\mu(Y)=-\frac12\log(2\pi\sigma^2) -\frac{(Y-\mu)^2}{2\sigma^2},

so the score is

S_\mu(Y) = \frac{Y-\mu}{\sigma^2}.

The derivative of the quadratic term contributes (Y-\mu)/\sigma^2. Summing the scores and setting them to zero gives \sum_i(Y_i-\hat\mu)=0, hence \hat\mu=\bar Y. The score is a centered observation scaled by the known variance.

The Fisher information is the score’s second moment:

\begin{aligned} I(\mu)&= E[S_\mu(Y)^2]\\ &= \frac{1}{\sigma^2}. \end{aligned}

Equivalently, under the same regularity conditions,

I(\mu)=-E\left[\frac{\partial^2}{\partial\mu^2}\log p_\mu(Y)\right].

This is information per observation. The score variance calculation is \sigma^2/\sigma^4=1/\sigma^2. For n independent observations, total information is nI(\mu), and expected sample curvature is -nI(\mu). Larger information produces a more concentrated sampling distribution for an efficient estimator under this model.

The curvature identity follows by differentiating E_\theta[S_\theta]=0, including the change in the density as well as the change in the score:

0=E_\theta[\partial_\theta S_\theta]+E_\theta[S_\theta^2].

Thus the score equation has derivative expectation A=-I and contribution variance B=I. Substituting into the estimating-equation formula -A^{-1}S gives

\begin{aligned} \operatorname{IF}(Y)&=I(\mu)^{-1}S_\mu(Y)\\ &=Y-\mu, \end{aligned}

and the mean’s first-order variance is I(\mu)^{-1}/n=\sigma^2/n, exact for this sample mean. The score and influence function are different objects: multiplying the score by inverse information supplies the normalization needed for an estimation contribution. If Y=7, \mu=5, and \sigma^2=4, the score is 1/2, the information is 1/4, and the influence contribution is 2. The mean estimator’s SE in a study of n people is 2/\sqrt n.

Figure 1: The sample log likelihood is \ell_n(\mu)-\ell_n(\bar Y)=-n(\mu-\bar Y)^2/(2\sigma^2). Its derivative is the total score \sum_iS_\mu(Y_i)=-n(\mu-\bar Y)/\sigma^2. The maximum and score zero coincide. The curvature is -n/\sigma^2=-4, while information per observation is I=1/\sigma^2=1.

4 The information bound

Why does information set a variance bound? Assume 0<I(\theta)<\infty. For the regular asymptotically linear estimators studied here, a centered, square-integrable influence function \phi must represent a unit change in the target coordinate: E[\phi S_\theta]=1. Cauchy–Schwarz then gives

\begin{aligned} 1&=(E[\phi S_\theta])^2\\ &\le E[\phi^2]E[S_\theta^2],\\ E[\phi^2]&\ge I(\theta)^{-1}. \end{aligned}

The influence variance is the limiting variance of the scaled estimation error; dividing it by n gives the estimator’s first-order variance. The bound is therefore 1/(nI), and equality requires \phi=I^{-1}S_\theta. An estimator attaining it is efficient within this model. A regular maximum likelihood estimator attains it under the conditions supporting its expansion and variance calculation.

The finite-sample Cramér–Rao inequality is related but has a different statement: under its differentiability and unbiasedness conditions, an unbiased estimator has variance at least 1/(nI). The asymptotic result above concerns first-order contributions of regular estimators and allows finite-sample bias. It does not give an exact variance formula for every estimator or every sample size. Model restrictions and efficiency explains the role of regularity along nearby distributions.

Figure 2: The left curves hold \bar Y fixed to compare likelihood shape; they are not probability densities for \mu. The right curves are the exact sampling distributions of the normal sample mean centered at the true mean. In this example \operatorname{Var}(\bar Y)=1/(nI)=\sigma^2/n, so the mean reaches the information bound. Efficiency compares regular estimators within the same model; it does not assert that every estimator has this variance.

Figure 3: For the full sample, nI=4 and the information bound is 1/(nI)=1/4. The full mean attains it. Averaging only the first two observations remains unbiased but has variance 1/2. These exact normal calculations illustrate the bound without changing the model or the available data. More generally, the guide’s efficiency statement concerns first-order variance among regular estimators.

5 Add a nuisance parameter

Now consider a normal linear model,

Y \mid X \sim N(\alpha+\beta X,\sigma^2),

where \beta is the target and \alpha is an unknown nuisance intercept. Take \sigma^2>0 as known for this derivation and assume the required covariate moments exist, with \operatorname{Var}(X)>0. We treat X as random with a law not indexed by (\alpha,\beta), and use the conditional likelihood of Y given X. Expectations average over the population covariates as well as outcomes.

The intercept is a nuisance because our current question concerns the slope; it need not be scientifically unimportant. Write \varepsilon=Y-\alpha-\beta X. Differentiating the conditional normal log density gives

\begin{aligned} S_\beta&= \frac{X\varepsilon}{\sigma^2},\\ S_\alpha&= \frac{\varepsilon}{\sigma^2}. \end{aligned}

Since E[\varepsilon^2\mid X]=\sigma^2, their cross moment is

E[S_\beta S_\alpha] = \frac{E[X]}{\sigma^2}.

If E[X]\ne0, the two scores are correlated: part of a slope change can be imitated by moving the intercept. Seek a coefficient c making S_\beta-cS_\alpha orthogonal to S_\alpha. Solving E[(S_\beta-cS_\alpha)S_\alpha]=0 gives c=E[S_\beta S_\alpha]/E[S_\alpha^2]=E[X]. Subtracting that projection yields

\begin{aligned} S_{\text{eff}} &= S_\beta - \frac{E[S_\beta S_\alpha]}{E[S_\alpha^2]}S_\alpha \\ &= \frac{\{X-E[X]\}\varepsilon}{\sigma^2}. \end{aligned}

This is the efficient score for \beta after accounting for the nuisance intercept. The information for \beta is its squared length,

\begin{aligned} I_{\text{eff}}&= E[S_{\text{eff}}^2]\\ &= \frac{\operatorname{Var}(X)}{\sigma^2}. \end{aligned}

If the intercept were known, slope information would instead be E[X^2]/\sigma^2. Estimating the intercept removes the part (E[X])^2/\sigma^2 that it can imitate. When E[X]=0, the scores are already orthogonal and the information bound is unchanged. An unknown nuisance need not cause a strict first-order information loss in every model.

The remaining slope information depends on variation around the covariate mean. If X were constant, a slope change could be completely absorbed into the intercept and I_{\mathrm{eff}} would be zero. Re-expressing the covariate as X-E[X] makes the separation visible; it does not create additional information or change the fitted slope when an intercept is included.

Figure 4: For a fixed design with sample mean 3 and sample second central moment 4, these are exact log-likelihood differences from the joint maximum (\hat\alpha,\hat\beta). Holding the intercept fixed gives a drop -13(\beta-\hat\beta)^2/2 per observation. Re-fitting it gives \alpha-\hat\alpha=-3(\beta-\hat\beta) and a smaller drop -4(\beta-\hat\beta)^2/2. This explains the information loss in terms of indistinguishable changes in fitted means.

6 Information matrices and the Schur complement

With parameter (\beta,\eta), where \eta is a nuisance vector, partition the Fisher information matrix as

I = \begin{pmatrix} I_{\beta\beta} & I_{\beta\eta} \\ I_{\eta\beta} & I_{\eta\eta} \end{pmatrix}.

The efficient score is

S_{\text{eff}} = S_\beta - I_{\beta\eta}I_{\eta\eta}^{-1}S_\eta,

and its information is

I_{\text{eff}} = I_{\beta\beta}-I_{\beta\eta}I_{\eta\eta}^{-1}I_{\eta\beta}.

To obtain the projection coefficient, solve the moment equation

\begin{aligned} E[(S_\beta-cS_\eta)S_\eta^\top]&=I_{\beta\eta}-cI_{\eta\eta}\\ &=0. \end{aligned}

Assuming I_{\eta\eta} is invertible gives c=I_{\beta\eta}I_{\eta\eta}^{-1}. Expanding the second moment of the residual score gives the displayed Schur complement. It is the amount of information left after nuisance-score projection.

When I_{\mathrm{eff}}>0, the efficient influence function and first-order variance bound are

\begin{aligned} \phi_{\mathrm{eff}}&=I_{\mathrm{eff}}^{-1}S_{\mathrm{eff}},\\ \text{variance bound}&=\frac1{nI_{\mathrm{eff}}}. \end{aligned}

The normalization ensures E[\phi_{\mathrm{eff}}S_\beta]=1 while E[\phi_{\mathrm{eff}}S_\eta]=0. Removing nuisance overlap makes the score shorter, so information decreases or stays the same. Taking its reciprocal then increases or preserves the estimator variance bound. This is why a smaller efficient score is compatible with less precision.

This formula is worth recognizing because the semiparametric version changes only one thing: instead of projecting onto the span of finitely many nuisance scores, project onto a whole nuisance tangent space.

Figure 5: For the preceding regression, I_{\alpha\alpha}=1, I_{\beta\alpha}=3 and I_{\beta\beta}=13. The projection is 3S_\alpha and the orthogonal remainder is S_{\mathrm{eff}}=(X-3)\varepsilon. The triangle is drawn with side lengths in the ratio 3:2:\sqrt{13}; the bars show their squares. Thus the Schur complement measures the information remaining after nuisance projection.

7 Check the projection in a regression

Here the covariate has nonzero mean, so the slope and intercept scores are correlated. Centering X removes that correlation.

set.seed(1)
n <- 1000
x <- rnorm(n, mean = 3, sd = 2)
y <- 1 + 0.7 * x + rnorm(n, sd = 1.5)
fit <- lm(y ~ x)
e <- residuals(fit)

# Scores are scaled by sigma^2; the common scale does not change correlations.
slope_score <- x * e
intercept_score <- e
centered_slope_score <- (x - mean(x)) * e

c(raw_score_correlation = cor(slope_score, intercept_score),
  centered_score_correlation = cor(centered_slope_score, intercept_score),
  slope = coef(fit)["x"])

     raw_score_correlation centered_score_correlation 
                0.81481782                 0.01754632 
                   slope.x 
                0.70482441 

These are fitted score contributions in one dataset. Their empirical correlation involves products such as (X_i-\bar X)e_i^2, whereas the OLS normal equations balance X_i e_i. Exact OLS residual balance therefore does not force these score correlations to vanish. Under this correctly specified homoscedastic generator, the population score orthogonality explains the approximate finite-sample pattern as the sample grows.

8 From a nuisance vector to a nuisance function

In the partially linear model,

Y = \beta X + g(Z) + \varepsilon,

retain the homoscedastic Gaussian outcome-error assumption, \varepsilon\mid X,Z\sim N(0,\sigma^2), for this score derivation. The nuisance is the unknown function g, rather than only an intercept. The conditional-mean model alone needs additional work and may have a different efficiency calculation under heteroscedasticity. Moving g in a direction k(Z) gives a nuisance score proportional to k(Z)\varepsilon. The efficient score for \beta is obtained by projecting X\varepsilon onto all such directions. The answer is

S_{\text{eff}} = \frac{\{X-E[X\mid Z]\}\varepsilon}{\sigma^2}.

Compare this to the intercept example. The constant E[X] has become the conditional expectation E[X\mid Z]. The finite-dimensional span of nuisance scores has become the infinite-dimensional space of functions of Z. The geometry has not changed.

The Conditional expectation and L2 geometry page supplies the residualization identity. The Model restrictions and efficiency page derives the full efficient influence function and explains the efficiency bound.

Figure 6: The marginal mean is E[X]=1. Removing it leaves X-1=\sin(\pi Z)+U, whose conditional mean still follows the orange curve. Removing E[X\mid Z]=1+\sin(\pi Z) leaves U, whose conditional mean is the green zero line. This removes the predictable mean, not necessarily all dependence on Z: the conditional variance of U can still vary with Z. Under the homoscedastic Gaussian outcome-error model used for the score derivation, the efficient score changes from (X-1)\varepsilon/\sigma^2 with an intercept nuisance to U\varepsilon/\sigma^2 with a nuisance function.

9 What likelihood assumes

Likelihood is powerful because it makes a detailed probability model do a great deal of work. It also means that its conclusions are tied to that model.

• The score equation targets the parameter of the specified model. Under misspecification, it generally converges to a pseudo-true value rather than the scientific target. See the What is your estimator actually estimating? page.
• The usual information-matrix standard error uses the equality between score variance and expected curvature. Under misspecification, these differ; the sandwich variance from the Estimating equations and asymptotic linearity page is the appropriate first-order calculation.
• A semiparametric model keeps the pieces of the likelihood that define the target while declining to specify a nuisance distribution or function fully. The price is that the nuisance must be projected out rather than simply estimated with a few coefficients.

10 Exercises

Checkpoints are completion targets for this page. Additional practice below is optional. Each checkpoint gives a completion standard, a hint, and a solution outline.

Checkpoint · SP-07 — Remove a nuisance score and compute the bound

Task type: Derivation and interpretation.

Let Y\mid X,Z\sim N(\beta X+\alpha+\gamma Z,\sigma^2) with known positive \sigma^2. Write C=(1,Z)^\top and assume E[CC^\top] is nonsingular. Assume the required second moments exist and X has positive residual variance after its linear projection onto C.

Derive the target and nuisance scores, remove the nuisance-score projection, and obtain the efficient score, efficient information, EIF, and variance bound. Explain why the residualization here is linear in C rather than automatically X-E[X\mid Z].

Ready to continue when: you can carry the same nuisance projection through to information and estimator variance, with its model assumptions stated.

Transfer: If the nuisance is an unrestricted function g(Z) instead, identify the larger nuisance space and the prediction that replaces aC.

Hint

Let a=E[XC^\top]E[CC^\top]^{-1} and R_X=X-aC. Use E[\varepsilon^2\mid X,Z]=\sigma^2.

Solution outline — open after trying

The scores are S_\beta=X\varepsilon/\sigma^2 and S_\eta=C\varepsilon/\sigma^2. The score projection is aC\varepsilon/\sigma^2, leaving S_{\mathrm{eff}}=R_X\varepsilon/\sigma^2. Its second moment is I_{\mathrm{eff}}=E[R_X^2]/\sigma^2. Therefore the EIF is R_X\varepsilon/E[R_X^2] and the variance bound is \sigma^2/\{nE[R_X^2]\}. The nuisance span here contains the intercept and linear Z score. It does not contain every function of Z, so linear projection need not equal conditional expectation.

Checkpoint · SP-09 — Compare information with known and unknown variance

Task type: Derivation.

For Y\sim N(\mu,\sigma^2), first take \sigma^2>0 as known. Derive the score, information per observation, and variance bound for \mu. Then treat (\mu,\sigma^2) as unknown, derive both scores and their information matrix, and determine whether the efficient bound for \mu changes.

Ready to continue when: you can show why adding a nuisance parameter may leave the target information unchanged, rather than assuming information always decreases strictly.

Transfer: Compare this with the slope/intercept example in the guide, where the scores are generally correlated.

Hint

Use the centered normal third and fourth moments: E[(Y-\mu)^3]=0 and E[(Y-\mu)^4]=3v^2.

Solution outline — open after trying

The scores, differentiating with respect to \mu and \sigma^2, are

\begin{aligned} S_\mu&=\frac{Y-\mu}{\sigma^2},\\ S_{\sigma^2}&=\frac{(Y-\mu)^2-\sigma^2}{2\sigma^4}. \end{aligned}

Their cross moment is zero and the information matrix is diagonal, with entries 1/\sigma^2 and 1/(2\sigma^4). Thus projecting the mean score off the variance score leaves it unchanged. The bound for the mean is \sigma^2/n in either model and its EIF is Y-\mu. Estimating \sigma^2 still affects how finite-sample intervals are constructed; orthogonal scores do not make the variance known.

Additional practice (optional)

These exercises extend the ideas through further derivations, simulations, or applications. Choose those useful for your work.

1. Simulate the regression example with E[X]=0, then with E[X]=5. Compare the correlation between the raw slope and intercept scores. Does the slope estimate change when you center X?
2. Show algebraically that the (\beta,\beta) entry of I^{-1} equals I_{\text{eff}}^{-1} for a scalar target and vector nuisance. This is the matrix form of “removing what the nuisance can imitate.”

11 Reading

• Cox, D. R. and Hinkley, D. V. (1974). Theoretical Statistics. Chapman and Hall. Chapters 2 and 9 for likelihood and nuisance parameters.
• van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge. Chapter 7 for maximum likelihood and information bounds.
• Bickel, P. J., Klaassen, C. A. J., Ritov, Y., and Wellner, J. A. (1993). Efficient and Adaptive Estimation for Semiparametric Models. Johns Hopkins University Press. Chapters 1 and 2 for the parametric geometry that leads into the semiparametric version.
• Tsiatis, A. A. (2006). Semiparametric Theory and Missing Data. Springer. Chapter 3 for efficient scores, information, and their semiparametric extension.