Connect a target’s local derivative to an estimator’s first-order error

Before reading: Statistical models, targets, and paths, Taylor expansions in Large-sample reasoning for methods research, and Estimating equations and asymptotic linearity.

1 A derivative and an estimator expansion

There are two connected uses of an influence function in this collection. For a functional T(F), it describes a first-order change when the distribution changes. For a regular asymptotically linear estimator, it is the observation-level contribution in \hat\theta-\theta\approx n^{-1}\sum_i\phi(W_i). Here W_i denotes one complete observation record. Matching the derivative to the estimator requires the appropriate model, regularity, and a negligible remainder; a numerical derivative alone does not prove that an estimator has this expansion.

The Estimating equations and asymptotic linearity page derives such contributions from a sample equation. Here we first study functionals themselves, then use a mean, variance, and ratio as worked examples. The correlation calculation at the end is an optional transfer example. The Jackknife pseudo-values page is a related application branch.

2 Quantities as functionals of a distribution

A statistic is usually written as a function of a dataset, but many can also be written as a function of a distribution. Here F denotes the population law, the object called P on earlier pages, and \int a\,dF means its expectation of a. The mean is \mu(F)=\int y\,dF(y), and the variance is \sigma^2(F)=\int(y-\mu(F))^2\,dF(y). A functional T maps a distribution to a number.

At the population law, T(F) is the target. At the empirical law F_N, which puts probability 1/N on each of the N observations, it is a plug-in estimate: \mu(F_N)=\bar Y. The target stays fixed across repeated studies; F_N and T(F_N) change with the sample. This page uses N for sample size, equivalent to the n used in the preceding guide; F_N is the empirical distribution called P_n there.

This viewpoint pays off because we can now ask how T changes when the distribution changes slightly, which is a derivative.

3 Definition

Put a small probability mass \varepsilon at a chosen point x, while reducing the original law’s weight to 1-\varepsilon:

F_\varepsilon=(1-\varepsilon)F+\varepsilon\delta_x.

The point mass \delta_x assigns probability one to x. Thus F_\varepsilon remains a probability distribution, and x is held fixed while \varepsilon changes. The contamination influence function is the rate at which the target changes per unit of this small replacement weight:

\operatorname{IF}(x; T, F) = \lim_{\varepsilon \downarrow 0} \frac{T(F_\varepsilon) - T(F)}{\varepsilon}.

The derivative is one-sided because negative probability at a new point is not generally allowed. When a first-order approximation is justified, a small weight \varepsilon changes the target by approximately \varepsilon\operatorname{IF}(x;T,F). The IF itself is a slope, not the finite change caused by adding an observation with full weight.

For the elementary differentiable functionals below, the derivative extends linearly over mixture directions, with integrability and the required interchanges. Its centered representation satisfies E_F[\operatorname{IF}(Y;T,F)]=0. Replacing some of F by F itself leaves the law unchanged, which explains that centering. Replacing it by another law G gives derivative \int\operatorname{IF}(x;T,F)\,dG(x). A pointwise directional derivative alone would not guarantee this linear representation for an arbitrary functional.

Two elementary influence functions and an association example:

Functional Influence function at F
Mean \mu x - \mu
Variance \sigma^2 (x - \mu)^2 - \sigma^2
Correlation \rho \tilde x \tilde y - \tfrac{\rho}{2}(\tilde x^2 + \tilde y^2), with \tilde x = (x - \mu_x)/\sigma_x, \tilde y = (y - \mu_y)/\sigma_y

The calculations below explain the mean and variance entries and a ratio rule. The correlation formula is a later optional transfer built from moments and that same differentiation rule.

3.1 Mean: differentiate the reweighted expectation

For \mu=E_F[Y], contaminate at y. Then

\mu(F_\varepsilon)=(1-\varepsilon)\mu+\varepsilon y.

Subtracting \mu and dividing by \varepsilon gives \operatorname{IF}(y;\mu,F)=y-\mu, exactly. A high value increases the mean and a low value decreases it, with sensitivity proportional to its distance from the population mean. Its average is zero. An integrable outcome defines this derivative; finite variance is needed to use it in a finite-variance CLT.

Along a bounded-score path from the previous page, the mean derivative is E[(Y-\mu)s]. Thus the contamination calculation and score-path representation agree for this functional.

3.2 Variance: account for the changing mean

Write \sigma^2=E[Y^2]-\mu^2. Under contamination at y, the second moment changes at rate y^2-E[Y^2], and the mean changes at rate y-\mu. The chain rule therefore gives

\begin{aligned} \operatorname{IF}(y;\sigma^2,F) &=y^2-E[Y^2]-2\mu(y-\mu)\\ &=(y-\mu)^2-\sigma^2. \end{aligned}

Starting from the second raw moment alone would miss the correction -2\mu(y-\mu). If we instead differentiate the central-moment expression E[(Y-\mu)^2], the contribution from differentiating its center is -2E[Y-\mu](y-\mu)=0; the same final answer results. The final expression has mean zero because E[(Y-\mu)^2]=\sigma^2. Along a score path, differentiating the same two moments gives E[\{(Y-\mu)^2-\sigma^2\}s].

Finite second moments define the variance and this directional derivative. A finite fourth moment is a convenient sufficient condition for the variance influence function to have finite second moment, which is what its CLT needs.

3.3 Ratio: combine two mean derivatives

For an observation (U,V), let a=E[U], b=E[V]\ne0, and r=a/b. Contamination at (u,v) changes the two means at rates u-a and v-b. The quotient rule gives

\begin{aligned} \phi_r(u,v)&=\frac{u-a}{b}-\frac{a(v-b)}{b^2}\\ &=\frac{u-rv}{b}. \end{aligned}

Its mean is zero because E[U-rV]=a-rb=0. With finite second moments, its variance is

\operatorname{Var}(\phi_r)=\frac{\operatorname{Var}(U)+r^2\operatorname{Var}(V)}{b^2} -\frac{2r\operatorname{Cov}(U,V)}{b^2}.

The covariance belongs in the calculation because numerator and denominator are estimated from the same records. For \hat r=\bar U/\bar V, the exact error identity is (\bar U-r\bar V)/\bar V. The LLN keeps \bar V near the nonzero b; replacing its inverse by 1/b leaves a second-order product of mean-estimation errors. This supplies the first-order contribution \phi_r under the stated moment conditions. Averaging individual ratios U_i/V_i generally targets a different functional and may even be undefined when some denominators are zero.

4 The von Mises expansion

When the functional admits a linear derivative with a controlled remainder in a stated norm, a von Mises expansion takes the form below.1 A point-contamination slope alone does not establish that property. For two distributions F and G,

T(G)=T(F) +\int\operatorname{IF}(x;T,F)\,dG(x) +R_2(G,F),

where R_2 collects the error after the linear term. A genuinely quadratic bound needs additional smoothness and an appropriate notion of closeness; the notation does not supply that bound. This distinction matters for a fitted or empirical distribution.

To connect this distributional expansion to sampling, take G=F_N. The linear term is an average of fixed, mean-zero functions of the i.i.d. observations. Write \phi_0(W)=\operatorname{IF}(W;T,F) and R_N=R_2(F_N,F). The needed statement is

\begin{aligned} T(F_N)-T(F)&=P_N\phi_0+R_N,\\ R_N&=o_p(N^{-1/2}). \end{aligned}

With 0<E[\phi_0^2]<\infty, a CLT handles \sqrt N P_N\phi_0, and Slutsky removes \sqrt N R_N=o_p(1).2 The limiting variance of the scaled error is E[\phi_0^2]; the estimator’s first-order variance is E[\phi_0^2]/N. A feasible SE is the sample SD of estimated influence values divided by \sqrt N, provided their variance estimate is consistent. The derivative calculation does not by itself prove either the remainder rate or variance consistency.

For the mean, the remainder is exactly zero: \bar Y-\mu=P_N(Y-\mu). For the plug-in variance \widehat{\sigma}^2=N^{-1}\sum_i(Y_i-\bar Y)^2, the divisor is N, as required by the empirical-law definition; the sample variance s_Y^2 uses N-1. An exact calculation gives

\widehat{\sigma}^2-\sigma^2=P_N\{(Y-\mu)^2-\sigma^2\}-(\bar Y-\mu)^2.

The mean error is O_p(N^{-1/2}), so its square is O_p(N^{-1})=o_p(N^{-1/2}). This verifies the remainder rate rather than assuming it from the word “derivative.” A CLT for the leading variance contribution additionally uses its fourth-moment condition. The ratio example similarly uses a nonzero limiting denominator and product-rate control.

5 From a derivative to a model-relative influence function

Two applications explain why these calculations matter.

The first is robustness. An unbounded IF means local first-order sensitivity is not uniformly bounded over possible contamination points. The mean’s y-\mu grows without bound as y moves outward. A median has a bounded IF under the positive-density conditions on the quantile page. This compares local sensitivity; it does not establish every aspect of an estimator’s robustness, and a correlation statistic remains in its allowed range despite an unbounded derivative. Hampel’s (1974) robustness perspective helps explain the different association measures discussed in Network inference from omics data.

The second is efficiency. Under the required differentiability assumptions in the unrestricted model, the square-integrable mean-zero gradient is unique and determines the regular-estimation variance bound. In a restricted model, only derivatives along its permitted score directions must be represented. Two gradients can therefore differ in directions that all those scores are orthogonal to. A point-contamination calculation that leaves the restricted model does not identify its canonical gradient automatically.

Model restrictions and efficiency explains the projection that selects the efficient representation. Constructing an estimator that attains it still requires the expansion and variance arguments. Orthogonality and cross-fitting then studies the additional errors introduced by learned nuisance functions.

The quantile checkpoint SP-06 connects a path derivative to a nonsmooth sample-root expansion. The From an influence function to an estimator page builds an estimator from its EIF and proves its expansion.

6 Exercises

Checkpoints are completion targets for this page. Additional practice below is optional. Each checkpoint gives a completion standard, a hint, and a solution outline.

Checkpoint · SP-04 — Derive mean and variance influence functions

Task type: Derivation.

Assume E[Y^4]<\infty. Starting from P_\varepsilon=(1-\varepsilon)P+\varepsilon\delta_y, derive influence functions for \mu=E[Y] and \sigma^2=E[(Y-\mu)^2]. Account for the fact that the mean changes inside the variance. Check that each influence function has expectation zero, and verify its derivative representation along a bounded, mean-zero-score path.

Ready to continue when: you can differentiate both functionals, center their contributions, and explain the moment assumptions.

Transfer: Find the influence function of E[(Y-c)^2] for a fixed known c. Which derivative term disappears?

Hint

Use \sigma^2=E[Y^2]-\mu^2 and differentiate both terms. For a smooth path, dE_t[a(Y)]/dt|_0=E[a(Y)s(Y)].

Solution outline — open after trying

The contaminated mean has derivative y-\mu. The second moment has derivative y^2-E[Y^2]. Thus the variance derivative is y^2-E[Y^2]-2\mu(y-\mu)=(y-\mu)^2-\sigma^2. Both contributions have expectation zero. Along a bounded-score path, the mean derivative is E[(Y-\mu)s] and the variance derivative is E[\{(Y-\mu)^2-\sigma^2\}s], using E[s]=0. A finite second moment defines the variance; a finite fourth moment ensures its influence function has finite variance, as needed for the usual CLT-based inference.

Checkpoint · SP-05 — Differentiate a ratio of means

Task type: Derivation and interpretation.

For an i.i.d. observation W=(U,V) with finite second moments and b=E[V]\ne0, define a=E[U] and r=a/b. Derive its influence function, check centering, and give the first-order variance of \bar U/\bar V. State why estimating a ratio is different from averaging the individual ratios U_i/V_i.

Ready to continue when: you can combine the two mean contributions, state the denominator condition, and interpret the correct population target.

Transfer: Take V=1 almost surely and recover the sample-mean influence function.

Hint

The derivative of a/b in direction (da,db) is da/b-a\,db/b^2.

Solution outline — open after trying

Substitution gives

\begin{aligned} \phi_r(U,V)&=\frac{U-a}{b}-\frac{a(V-b)}{b^2}\\ &=\frac{U-rV}{b}. \end{aligned}

Its mean is zero. The first-order estimator variance is \operatorname{Var}(U-rV)/(nb^2); plug-in influence values provide a standard error. By the LLN, \bar V stays away from zero with probability tending to one. The target is E[U]/E[V], whereas averaging U/V targets E[U/V] when that expectation exists; the two generally differ.

Additional practice (optional)

These exercises extend the ideas through further derivations, simulations, or applications. Choose those useful for your work.

  1. Verify numerically that the sample average of \operatorname{IF}(X_i; T, F_N) is zero for the correlation, and explain why it is exactly zero for the formula in this page.
  2. In the mixture computation in Association targets, pooling, and imperfect adjustment, set shift = 0 and multiply the patients’ values of gene a by 2. Is the remainder zero? Is it the same in both groups? Does within-group centering change it? Does within-group standardization? Then multiply both genes by 2 and explain the change using the influence-function formula.
  3. Compute the IF of Spearman’s correlation numerically at a few points and compare its shape to Pearson’s. Which one is bounded?

7 Compute one numerically (optional correlation transfer)

The definition is also an algorithm. Put weight (1-\varepsilon)/N on each observation and \varepsilon on the new point, compute the weighted statistic, and difference. Compare with the formula.

Code 
set.seed(1)
n <- 500
x <- rnorm(n); y <- 0.6 * x + 0.8 * rnorm(n)

# Weighted correlation of (x, y) with weights w
WCor <- function(x, y, w) {
  cv <- cov.wt(cbind(x, y), wt = w, method = "ML")$cov
  cv[1, 2] / sqrt(cv[1, 1] * cv[2, 2])
}

# Numerical IF of the correlation at the empirical distribution, evaluated at a new point
NumericIF <- function(x0, y0, eps = 1e-4) {
  w <- c(rep((1 - eps) / n, n), eps)
  (WCor(c(x, x0), c(y, y0), w) - cor(x, y)) / eps
}

# Analytic IF, standardized with divisor N (the scaling of the functional at the empirical distribution)
SdN <- function(v) sqrt(mean((v - mean(v))^2))
AnalyticIF <- function(x0, y0) {
  r <- cor(x, y)
  xt <- (x0 - mean(x)) / SdN(x); yt <- (y0 - mean(y)) / SdN(y)
  xt * yt - r / 2 * (xt^2 + yt^2)
}

pts <- rbind(c(0, 0), c(2, 2), c(2, -2), c(-1, 3))
cbind(pts, numeric = apply(pts, 1, function(p) NumericIF(p[1], p[2])),
      analytic = apply(pts, 1, function(p) AnalyticIF(p[1], p[2])))
                 numeric      analytic
[1,]  0  0 -0.0007942872 -0.0007942872
[2,]  2  2  1.7040853368  1.7047466919
[3,]  2 -2 -5.9163710249 -5.9186147672
[4,] -1  3 -5.7475178557 -5.7507604588

The discrepancy is finite-difference error. Reducing eps over a suitable range improves the derivative approximation; extremely small steps can instead magnify floating-point cancellation. The analytic formula standardizes with divisor N, as the functional at F_N and cov.wt with method = "ML" do. The signs describe local changes: a point far along the diagonal pulls this correlation up, a point off the diagonal pulls it down, and a point near the center changes it little.

The picture below (Figure 1) shows the same calculation as a curve: add a point with a small weight \varepsilon, watch the correlation of the sample, and read the influence function off the slope at \varepsilon = 0.

Three places to add a point one sample of 500 -2 -2 0 0 2 2 gene A gene B (2, 2) (0, 0) (2, −2) Each added point gets a small weight ε, and the sample keeps weight 1 − ε. How the correlation responds change in the correlation as the added point gains weight -0.3 -0.2 -0.1 0 0.1 0 0.025 0.05 weight ε on the added point change in the correlation (2, 2) slope 1.70 (0, 0) slope 0.00 (2, −2) slope −5.92 Dashed: the tangent at weight 0. Its slope is the influence function at that point.
Figure 1: The influence function as a slope. Left: the sample of 500 from the code above, with three places to add a point (vermilion rings). Right: the correlation of the sample with a point added at weight \varepsilon, minus the correlation without it, for each of the three points. Near \varepsilon = 0 each curve is nearly a straight line, and its slope is the influence function at that point: 1.70 at (2, 2), 0.00 at (0, 0), and -5.92 at (2, -2). Farther out the curves bend away from their tangent lines (dashed), and that bending is what the remainder in the von Mises expansion collects.

8 Reading

Hines et al., Demystifying statistical learning based on efficient influence functions, develops the derivative-to-estimator connection.

Back to top

Footnotes

  1. A Taylor expansion approximates a function near a point by a polynomial: the value at the point, plus the slope times the distance from the point, plus a curvature term times the distance squared, and so on. Keeping only the slope term is the first-order approximation, and everything after it is the remainder. For example, e^x \approx 1 + x for small x, with a remainder of about x^2/2.

    Here the function is T, the point is F, the distance is the change from F to G, and the slope is the influence function. The remainder R_2(G, F) collects the curvature terms. It is small when G is close to F and can be large when G is far from F, a phenomenon explored in the network-analysis application.↩︎

  2. Smoothness of a population functional and control of an empirical remainder are separate requirements. The sampling argument here needs \sqrt N R_N\xrightarrow{p}0; the remainder must vanish faster than the leading sampling fluctuation. Finite second moments suffice for the mean CLT. Finite fourth moments provide a convenient sufficient condition for variance and correlation expansions, with nonzero marginal variances for correlation. A support endpoint has a different local behavior and rate; the support-endpoint counterexample in Bootstrap and resampling for statistical inference makes this precise.

    The condition matters because the asymptotic normality above, and every standard error built from the influence function, rest on it. When it fails, a standard error built from the influence function need not describe the estimator’s actual variability.↩︎