Statistical models, targets, and paths

Define which distributions may change before differentiating the target

Before reading: Large-sample reasoning for methods research, Estimating equations and asymptotic linearity, elementary expectations and derivatives, and the target/model distinction in What is your estimator actually estimating?. The advanced resampling and cluster branches can be revisited after influence functions.

1 Keep the distribution, model, and target separate

Let W be one observation record and P its population distribution. A model \mathcal M is a set of distributions we allow for that record. The actual population law is one member of this set, although we do not know which member. A scalar target is a map \psi:\mathcal M\to\mathbb R: it assigns a population quantity to every allowed law. The map defines the question; \psi(P) is its value at the actual population.

For complete observations of a feature Y, write the mean target as \mu(P)=E_P[Y]. This is the same mean \mu used in the previous guide, now displaying which distribution supplies the expectation. Choosing a sample-mean algorithm does not decide which populations are allowed.

For example, both a normal model and an unrestricted finite-variance model contain the mean as a target. The normal model allows only normal distribution shapes; the larger model permits many other shapes. A partially linear model instead restricts a conditional mean to \beta X+g(Z) while leaving g unspecified. These are restrictions on population laws, rather than names of software routines.

The term nuisance depends on the question. When the mean is the target in a normal model, the variance may be a nuisance parameter. If the variance itself is the question, its role changes. Nuisance quantities can be scientifically important; they are simply not the final quantity being estimated in the current argument.

A low-dimensional target can be studied in a fully nonparametric model. Semiparametric estimation is therefore not limited to partially linear or proportional-hazards regression. The missing-outcome mean later uses a broadly unrestricted observed-data model for W=(Z,R,RY), after identification assumptions connect its functional to the full-population question.

2 Move through the model along a valid path

A path P_t is a one-parameter family in the model with P_0=P. Think of t as a small hypothetical population perturbation, not a sample size or an observation’s value. Different paths change different aspects of the distribution. The path must remain inside the model for sufficiently small t.

If the path has densities p_t relative to a common reference measure and the derivative is justified, its score at zero is

s(W)=\left.\frac{\partial}{\partial t}\log p_t(W)\right|_{t=0}.

The score describes a relative density change: \partial_t p_t|_0=p\,s. Positive values of s increase probability locally and negative values decrease it. Differentiating the total probability \int p_t=1, with a justified interchange, gives E_P[s]=0. Probability added in some places must be removed elsewhere.

A concrete construction begins with a bounded, mean-zero function s and sets

dP_t=(1+ts)dP.

This means reweighting the original law by 1+ts. Its total mass is E_P[1+ts]=1. If |s|\le K<\infty, choosing |t|K<1 keeps every weight positive. Finally,

\begin{aligned} \left.\partial_t\log p_t(W)\right|_0&=\left.\frac{s(W)}{1+ts(W)}\right|_0\\ &=s(W). \end{aligned}

Thus we have checked normalization, nonnegativity, and the score itself. In an unrestricted model this gives a valid path. A normal-shape or symmetry restriction imposes further requirements; a normalized density need not preserve them.

For three equally likely feature values, the construction is concrete:

Y	Original probability	Score s(Y)	Path probability
0	1/3	-1	(1-t)/3
2	1/3	0	1/3
4	1/3	1	(1+t)/3

For |t|<1, the probabilities are positive and sum to one. At t=0.3, they are approximately 0.233, 0.333, and 0.433: probability moves from 0 to 4 while the middle probability stays fixed. Direct averaging gives

\begin{aligned} \mu(P_t)&=\frac{2+4(1+t)}3\\ &=2+\frac43t. \end{aligned}

The target’s derivative at zero is therefore 4/3. It is a rate of population-mean change per unit perturbation, not the standard error of an estimate.

3 Represent the target’s derivative

Instead of computing the target derivative separately for every path, we seek one function that represents all the permitted directional derivatives. A square-integrable, mean-zero gradient \phi satisfies, for every regular score path under consideration,

\left.\frac{d}{dt}\psi(P_t)\right|_{0}=E_P[\phi(W)s(W)].

The right side is an L_2(P) inner product: multiply the gradient by the path’s score and take a population expectation. The score specifies which change we are making; the gradient describes the target’s sensitivity to that change. A representation must work for all allowed directions, not just one convenient path.

For the mean and the bounded-score construction, ordinary expectation algebra gives

\begin{aligned} \mu(P_t)&=E_P[Y(1+ts)],\\ \left.\frac{d}{dt}\mu(P_t)\right|_0&=E_P[Ys]\\ &=E_P[(Y-\mu)s]. \end{aligned}

The last equality uses E_P[s]=0. Thus \phi(Y)=Y-\mu represents the mean derivative. In the three-point example, E[(Y-2)s(Y)]=(2+0+2)/3=4/3, agreeing with the direct calculation. Integrability of Y suffices for this bounded-direction derivative; finite variance makes Y-\mu a square-integrable gradient suitable for the usual CLT argument.

The sample mean has the separate identity \bar Y-\mu=P_n(Y-\mu). This is why the same centered function appears both in a population derivative and in a sampling expansion. For a general target, the derivative calculation alone does not supply an estimator expansion, its negligible remainder, or a feasible SE. Those require their own arguments.

A point-contamination path (1-\varepsilon)P+\varepsilon\delta_y instead places a little mass at a chosen value y. For a continuous law it introduces an atom where P assigns zero mass, so it is generally not a smooth dominated score path through P. It can also leave a restricted model. Influence functions uses this convenient derivative for elementary unrestricted functionals, then relates it to the score-path representation above.

4 What a tangent space records

The tangent space \mathcal T\subseteq L_2^0(P) records the model’s permitted first-order directions. The notation L_2^0(P) means functions with finite second moment and expectation zero; its inner product is E_P[fg] and its norm is \|f\|_2=\sqrt{E_P[f^2]}.

To form \mathcal T, take linear combinations of regular path scores and include their L_2 limits. This is the closed linear span. “Closed” matters because approximating a direction increasingly well should not exclude its limiting direction. In the unrestricted model, bounded mean-zero scores are dense in L_2^0(P), giving \mathcal T=L_2^0(P). Restrictions can remove directions and leave a smaller space.

For example, if the population probability of a binary stratum Z is known and fixed by the model, a path can alter outcomes within strata but cannot change their probability weights. The conditional-expectation restriction on its score is then E[s\mid Z]=0. Model restrictions and efficiency will use this example to show how an excluded direction changes the efficient gradient.

Under the appropriate linearity and continuity assumptions, the target derivative has a unique representing element inside the closed tangent space. This is its canonical gradient, or efficient influence function. A representation outside that space may contain an extra component orthogonal to every permitted score, contributing zero to every derivative inner product. Model restrictions and efficiency explains why such a component can add sampling variance.

The general Hilbert-space representation theorem is useful mathematical background. Proving its existence statement is an optional extension; the completion task here is to construct and differentiate an actual valid path.

5 Exercises

The checkpoint is a completion target. Additional practice is optional.

Checkpoint · SP-03 — Build a valid path and calculate its score

Task type: Proof and interpretation.

In an unrestricted model, let s(W) be bounded, with E[s]=0, and define dP_t=(1+ts)dP. Assume E[|Y|]<\infty for the mean target.

Check normalization and nonnegativity for sufficiently small t, calculate the log-density score at zero, and derive the path derivative of E_t[Y]. Explain why an arbitrary path of this form may not stay in a restricted statistical model.

Ready to continue when: you can verify that a proposed perturbation is a probability distribution and relate its score to the target derivative.

Transfer: For a symmetric location model, can an arbitrary mean-zero s change the nuisance density while preserving symmetry? What restriction would its direction need?

Hint

Bound |t|\|s\|_\infty by a number below one, then use E[s]=0.

Solution outline — open after trying

The total mass is E[1+ts]=1 and 1+ts>0 if |t|\|s\|_\infty<1 (a zero score gives a constant path). The log-density derivative is s/(1+ts), equal to s at zero. Since s is bounded and Y integrable, E_t[Y]=E[Y]+tE[Ys], so the mean derivative is E[(Y-\mu)s]. A restricted model only permits paths preserving its restrictions; normalization alone does not establish membership.

Additional practice (optional)

Verify the three-point table’s normalization, nonnegativity, score, and mean derivative.
Give a bounded mean-zero direction that changes the probabilities of a symmetric three-point distribution asymmetrically. Why does that path leave a symmetric model?
Compare a distributional derivative with the sample-mean expansion. Which extra argument supplies a sampling distribution?

--- title: "Statistical models, targets, and paths" subtitle: "Define which distributions may change before differentiating the target" --- **Before reading:** [Large-sample reasoning for methods research](../estimation-and-inference/large-sample-reasoning.qmd), [Estimating equations and asymptotic linearity](../estimation-and-inference/estimating-equations.qmd), elementary expectations and derivatives, and the target/model distinction in [What is your estimator actually estimating?](../questions-and-design/estimands-and-misspecification.qmd). The advanced resampling and cluster branches can be revisited after influence functions. ## Keep the distribution, model, and target separate Let $W$ be one observation record and $P$ its population distribution. A model $\mathcal M$ is a set of distributions we allow for that record. The actual population law is one member of this set, although we do not know which member. A scalar target is a map $\psi:\mathcal M\to\mathbb R$: it assigns a population quantity to every allowed law. The map defines the question; $\psi(P)$ is its value at the actual population. For complete observations of a feature $Y$, write the mean target as $\mu(P)=E_P[Y]$. This is the same mean $\mu$ used in the previous guide, now displaying which distribution supplies the expectation. Choosing a sample-mean algorithm does not decide which populations are allowed. For example, both a normal model and an unrestricted finite-variance model contain the mean as a target. The normal model allows only normal distribution shapes; the larger model permits many other shapes. A partially linear model instead restricts a conditional mean to $\beta X+g(Z)$ while leaving $g$ unspecified. These are restrictions on population laws, rather than names of software routines. The term **nuisance** depends on the question. When the mean is the target in a normal model, the variance may be a nuisance parameter. If the variance itself is the question, its role changes. Nuisance quantities can be scientifically important; they are simply not the final quantity being estimated in the current argument. A low-dimensional target can be studied in a fully nonparametric model. Semiparametric estimation is therefore not limited to partially linear or proportional-hazards regression. The missing-outcome mean later uses a broadly unrestricted **observed-data** model for $W=(Z,R,RY)$, after identification assumptions connect its functional to the full-population question. ## Move through the model along a valid path A path $P_t$ is a one-parameter family in the model with $P_0=P$. Think of $t$ as a small hypothetical population perturbation, not a sample size or an observation's value. Different paths change different aspects of the distribution. The path must remain inside the model for sufficiently small $t$. If the path has densities $p_t$ relative to a common reference measure and the derivative is justified, its **score** at zero is $$ s(W)=\left.\frac{\partial}{\partial t}\log p_t(W)\right|_{t=0}. $$ The score describes a *relative* density change: $\partial_t p_t|_0=p\,s$. Positive values of $s$ increase probability locally and negative values decrease it. Differentiating the total probability $\int p_t=1$, with a justified interchange, gives $E_P[s]=0$. Probability added in some places must be removed elsewhere. A concrete construction begins with a bounded, mean-zero function $s$ and sets $$ dP_t=(1+ts)dP. $$ This means reweighting the original law by $1+ts$. Its total mass is $E_P[1+ts]=1$. If $|s|\le K<\infty$, choosing $|t|K<1$ keeps every weight positive. Finally, $$ \begin{aligned} \left.\partial_t\log p_t(W)\right|_0&=\left.\frac{s(W)}{1+ts(W)}\right|_0\\ &=s(W). \end{aligned} $$ Thus we have checked normalization, nonnegativity, and the score itself. In an unrestricted model this gives a valid path. A normal-shape or symmetry restriction imposes further requirements; a normalized density need not preserve them. For three equally likely feature values, the construction is concrete: | $Y$ | Original probability | Score $s(Y)$ | Path probability | |---|---|---|---| | 0 | $1/3$ | $-1$ | $(1-t)/3$ | | 2 | $1/3$ | $0$ | $1/3$ | | 4 | $1/3$ | $1$ | $(1+t)/3$ | For $|t|<1$, the probabilities are positive and sum to one. At $t=0.3$, they are approximately $0.233$, $0.333$, and $0.433$: probability moves from 0 to 4 while the middle probability stays fixed. Direct averaging gives $$ \begin{aligned} \mu(P_t)&=\frac{2+4(1+t)}3\\ &=2+\frac43t. \end{aligned} $$ The target's derivative at zero is therefore $4/3$. It is a rate of population-mean change per unit perturbation, not the standard error of an estimate. ## Represent the target’s derivative Instead of computing the target derivative separately for every path, we seek one function that represents all the permitted directional derivatives. A square-integrable, mean-zero **gradient** $\phi$ satisfies, for every regular score path under consideration, $$ \left.\frac{d}{dt}\psi(P_t)\right|_{0}=E_P[\phi(W)s(W)]. $$ The right side is an $L_2(P)$ inner product: multiply the gradient by the path's score and take a population expectation. The score specifies which change we are making; the gradient describes the target's sensitivity to that change. A representation must work for all allowed directions, not just one convenient path. For the mean and the bounded-score construction, ordinary expectation algebra gives $$ \begin{aligned} \mu(P_t)&=E_P[Y(1+ts)],\\ \left.\frac{d}{dt}\mu(P_t)\right|_0&=E_P[Ys]\\ &=E_P[(Y-\mu)s]. \end{aligned} $$ The last equality uses $E_P[s]=0$. Thus $\phi(Y)=Y-\mu$ represents the mean derivative. In the three-point example, $E[(Y-2)s(Y)]=(2+0+2)/3=4/3$, agreeing with the direct calculation. Integrability of $Y$ suffices for this bounded-direction derivative; finite variance makes $Y-\mu$ a square-integrable gradient suitable for the usual CLT argument. The sample mean has the separate identity $\bar Y-\mu=P_n(Y-\mu)$. This is why the same centered function appears both in a population derivative and in a sampling expansion. For a general target, the derivative calculation alone does not supply an estimator expansion, its negligible remainder, or a feasible SE. Those require their own arguments. A point-contamination path $(1-\varepsilon)P+\varepsilon\delta_y$ instead places a little mass at a chosen value $y$. For a continuous law it introduces an atom where $P$ assigns zero mass, so it is generally not a smooth dominated score path through $P$. It can also leave a restricted model. [Influence functions](influence-functions.qmd) uses this convenient derivative for elementary unrestricted functionals, then relates it to the score-path representation above. ## What a tangent space records The tangent space $\mathcal T\subseteq L_2^0(P)$ records the model's permitted first-order directions. The notation $L_2^0(P)$ means functions with finite second moment and expectation zero; its inner product is $E_P[fg]$ and its norm is $\|f\|_2=\sqrt{E_P[f^2]}$. To form $\mathcal T$, take linear combinations of regular path scores and include their $L_2$ limits. This is the **closed linear span**. “Closed” matters because approximating a direction increasingly well should not exclude its limiting direction. In the unrestricted model, bounded mean-zero scores are dense in $L_2^0(P)$, giving $\mathcal T=L_2^0(P)$. Restrictions can remove directions and leave a smaller space. For example, if the population probability of a binary stratum $Z$ is known and fixed by the model, a path can alter outcomes within strata but cannot change their probability weights. The conditional-expectation restriction on its score is then $E[s\mid Z]=0$. [Model restrictions and efficiency](models-and-efficiency.qmd) will use this example to show how an excluded direction changes the efficient gradient. Under the appropriate linearity and continuity assumptions, the target derivative has a unique representing element **inside** the closed tangent space. This is its canonical gradient, or efficient influence function. A representation outside that space may contain an extra component orthogonal to every permitted score, contributing zero to every derivative inner product. [Model restrictions and efficiency](models-and-efficiency.qmd) explains why such a component can add sampling variance. The general Hilbert-space representation theorem is useful mathematical background. Proving its existence statement is an optional extension; the completion task here is to construct and differentiate an actual valid path. ## Exercises The checkpoint is a completion target. Additional practice is optional. ::: {#checkpoint-sp-03 .checkpoint} ### Checkpoint · SP-03 — Build a valid path and calculate its score {.unnumbered} **Task type:** Proof and interpretation. In an unrestricted model, let $s(W)$ be bounded, with $E[s]=0$, and define $dP_t=(1+ts)dP$. Assume $E[|Y|]<\infty$ for the mean target. Check normalization and nonnegativity for sufficiently small $t$, calculate the log-density score at zero, and derive the path derivative of $E_t[Y]$. Explain why an arbitrary path of this form may not stay in a restricted statistical model. **Ready to continue when:** you can verify that a proposed perturbation is a probability distribution and relate its score to the target derivative. **Transfer:** For a symmetric location model, can an arbitrary mean-zero $s$ change the nuisance density while preserving symmetry? What restriction would its direction need? <details class="checkpoint-help"> <summary>Hint</summary> Bound $|t|\|s\|_\infty$ by a number below one, then use $E[s]=0$. </details> <details class="checkpoint-help"> <summary>Solution outline — open after trying</summary> The total mass is $E[1+ts]=1$ and $1+ts>0$ if $|t|\|s\|_\infty<1$ (a zero score gives a constant path). The log-density derivative is $s/(1+ts)$, equal to $s$ at zero. Since $s$ is bounded and $Y$ integrable, $E_t[Y]=E[Y]+tE[Ys]$, so the mean derivative is $E[(Y-\mu)s]$. A restricted model only permits paths preserving its restrictions; normalization alone does not establish membership. </details> ::: ### Additional practice (optional) {.unnumbered} 1. Verify the three-point table’s normalization, nonnegativity, score, and mean derivative. 2. Give a bounded mean-zero direction that changes the probabilities of a symmetric three-point distribution asymmetrically. Why does that path leave a symmetric model? 3. Compare a distributional derivative with the sample-mean expansion. Which extra argument supplies a sampling distribution?