What is your estimator actually estimating?

Keep the target, model, and procedure’s limit distinct

Before reading: Means, conditional means, covariance, and basic regression. The population and observation record are introduced in From a research question to an identified estimand.

1 Write the target before choosing the procedure

An estimand is a quantity we intend to learn about a stated population. An estimator is a rule applied to a random sample; an estimate is its numerical output. The probability limit of an estimator describes what that rule approaches as the sample grows. The target and the limit need not be the same quantity.

Keep one example in view. We want the mean measured feature level \mu=E[Y] among eligible people. Covariates Z are recorded for everyone, and R indicates whether the feature is observed. The record is W=(Z,R,RY); when R=0, the product is an encoding of a missing contribution, not an observed zero. The observed-outcome mean targets E[Y\mid R=1], which can differ from E[Y].

The estimand is fixed by the population and question, not by which estimator is convenient. An estimator is random before a sample is drawn; after observing the data, its output is one estimate. Its probability limit is another population quantity, determined by the rule and sampling process. Consistency for a target means that this limit equals that target under the stated conditions.

2 A model is a set of possible distributions

A model specifies which data-generating distributions are allowed. It is separate from the fitting algorithm. A parametric model uses finitely many coordinates, such as a normal mean and variance. A nonparametric model leaves the distribution broadly unrestricted, subject to stated conditions. Semiparametric work combines a low-dimensional target with distributional features that are not described by a finite parameter vector.

For the missing mean, the unknown regression m(z)=E[Y\mid Z=z,R=1] and observation probability \pi(z)=P(R=1\mid Z=z) will be nuisance functions: population objects needed to estimate the target, rather than the final quantity reported. Naming a regression learner does not define these objects or establish their identifying assumptions.

A parameter can also be an estimand. For example, a mean is a functional on distributions, and a linear projection coefficient can be defined outside a linear conditional-mean model. What matters is the domain on which the quantity is defined and the interpretation claimed for it. A causal interpretation needs a separate identifying argument.

3 A precise answer to a different question

Suppose half the eligible people have Z=0 and half have Z=1. Their feature means are 0 and 4. Outcomes are observed with probabilities 0.8 and 0.2, respectively, independently of the outcome within each stratum.

The observed fraction is 0.5(0.8)+0.5(0.2)=0.5. Among observed people, the probability of Z=1 is 0.5(0.2)/0.5=0.2 by Bayes’ rule; the other stratum has weight 0.8. Selection has changed the averaging weights, without changing either stratum’s mean.

Quantity	Population weights on the strata	Value
Eligible-population mean	1/2,1/2	2
Mean among observed outcomes	0.8,0.2	0.8

The unweighted observed-outcome mean can consistently estimate 0.8. If our target is the eligible-population mean, the procedure has asymptotic discrepancy 0.8-2=-1.2. More independent people can make its standard error smaller without removing that discrepancy. If the scientific question instead concerns the observed subset, 0.8 may be the intended target.

This is a target mismatch. It should be distinguished from finite-sample bias, which concerns the expectation of an estimator at a particular sample size. It should also be distinguished from nonidentification: two full populations can sometimes generate the same observed record while having different target means.

3.1 Why that is the procedure’s limit

The complete-case mean is a ratio,

T_n=\frac{n^{-1}\sum_iR_iY_i}{n^{-1}\sum_iR_i}.

For independent identically distributed records, finite required expectations, and P(R=1)>0, the law of large numbers makes the numerator and denominator approach their population averages. Their ratio approaches

\begin{aligned} \mu^\star&=\frac{E[RY]}{P(R=1)}\\ &=E[Y\mid R=1]. \end{aligned}

The denominator is nonzero with probability approaching one. This explains the limit rather than assuming that a familiar sample mean must estimate the desired mean. Large-sample reasoning for methods research later develops the convergence argument formally.

Now separate the error around the limit from the mismatch:

T_n-\mu=(T_n-\mu^\star)+(\mu^\star-\mu).

The first term gets smaller as the estimator learns its own target. The second stays at −1.2 in this example. Even a perfectly estimated SE for the first term cannot remove the second. Whether we can instead identify and estimate the full mean depends on the observation assumptions, addressed on the next missingness page.

4 A working regression has a population meaning

For a nonconstant scalar X with suitable moments, the least-squares slope with an intercept targets

\beta^*=\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(X)}.

This is a legitimate projection functional even if the conditional mean is curved. Under a stated model Y=\beta X+\gamma Z+\varepsilon with E[\varepsilon\mid X,Z]=0,

\beta^*=\beta+\gamma\frac{\operatorname{Cov}(X,Z)}{\operatorname{Var}(X)}.

The coefficient at fixed Z and the marginal projection slope answer different questions. Calling the second “biased” requires naming the first as the intended target. Neither is automatically a causal effect. The detailed pooling and imperfect-adjustment calculations appear in Association targets, pooling, and imperfect adjustment, where they serve a network-analysis question.

To see where the projection formula comes from, minimize the population squared error E[(Y-a-bX)^2]. The intercept condition gives a=E[Y]-bE[X]. The slope condition then gives \operatorname{Cov}(X,Y)-b\operatorname{Var}(X)=0. These conditions define the best linear approximation, whether or not the true conditional mean is linear. Under the displayed richer mean model, expanding \operatorname{Cov}(X,Y) gives the stated omitted-variable term.

There are therefore two separate possibilities. The reduced regression can describe the observed X–Y relation adequately but its coefficient answers a different question from the slope at fixed Z. Or its conditional-mean model can be wrong and its fitted coefficient still approach a well-defined projection. In either case, first specify which interpretation is wanted; then assess uncertainty for that quantity.

5 Definition, identification, and estimation have different jobs

Definition states what is wanted. Identification connects that quantity to the observable distribution under assumptions. Estimation supplies a sample procedure for the identified functional. Inference studies its uncertainty and approximation errors.

For the running mean, missing at random, positive observation probabilities over the target support, and covariates recorded for everyone identify E[Y] as E[m(Z)]. Missingness, selection, and the observed-data target works through that claim. Later guides construct an estimator and show when its uncertainty can be justified.

A formula for a discrepancy can suggest a diagnostic only when its relevant ingredients are observable or identified. A test on the observed data cannot generally verify missing at random or rule out unmeasured confounding. Non-rejection does not certify that a working assumption is true.

6 Exercises

The checkpoint is a completion target. Additional practice is optional.

Checkpoint · DES-03 — Separate a target from a procedure’s limit

Task type: Interpretation and calculation.

Use the two-stratum example to calculate the eligible-population and observed-outcome means. Name the estimand and probability limit when the observed-outcome mean is used to answer the full-population question. Explain what increasing the sample size changes. Then define a question for which the observed-outcome mean is the intended target.

Ready to continue when: you can distinguish target mismatch, sampling uncertainty, finite-sample bias, and nonidentification.

Transfer: If the conditional mean is nonlinear, can a linear projection coefficient still be a well-defined estimand?

Hint

First calculate the distribution of Z among observed outcomes; then keep the scientific question fixed while evaluating the procedure.

Solution outline — open after trying

The eligible mean is 2. Selection changes the stratum weights to 0.8 and 0.2, giving an observed-outcome mean 0.8. The procedure converges to 0.8 under the stated sampling conditions, so its asymptotic discrepancy for the target 2 is -1.2. More independent people reduce sampling variation around that limit. A question explicitly about the observed subset can instead target 0.8. Whether the full mean is identified requires assumptions about the unobserved outcomes. A linear projection is meaningful outside a linear conditional-mean model when its moments and denominator are defined.

Additional practice (optional)

Change the observation probabilities while keeping the eligible population fixed. Which target stays the same?
Derive the least-squares slope limit under the displayed two-variable conditional-mean model.
Specify a standardized group contrast and explain why its target distribution must be named.

--- title: "What is your estimator actually estimating?" subtitle: "Keep the target, model, and procedure’s limit distinct" aliases: - "/guides/statistical-background/01-estimands.html" --- **Before reading:** Means, conditional means, covariance, and basic regression. The population and observation record are introduced in [From a research question to an identified estimand](question-to-estimand.qmd). ## Write the target before choosing the procedure An estimand is a quantity we intend to learn about a stated population. An estimator is a rule applied to a random sample; an estimate is its numerical output. The probability limit of an estimator describes what that rule approaches as the sample grows. The target and the limit need not be the same quantity. Keep one example in view. We want the mean measured feature level $\mu=E[Y]$ among eligible people. Covariates $Z$ are recorded for everyone, and $R$ indicates whether the feature is observed. The record is $W=(Z,R,RY)$; when $R=0$, the product is an encoding of a missing contribution, not an observed zero. The observed-outcome mean targets $E[Y\mid R=1]$, which can differ from $E[Y]$. The estimand is fixed by the population and question, not by which estimator is convenient. An estimator is random before a sample is drawn; after observing the data, its output is one estimate. Its probability limit is another population quantity, determined by the rule and sampling process. Consistency for a target means that this limit equals that target under the stated conditions. ## A model is a set of possible distributions A model specifies which data-generating distributions are allowed. It is separate from the fitting algorithm. A parametric model uses finitely many coordinates, such as a normal mean and variance. A nonparametric model leaves the distribution broadly unrestricted, subject to stated conditions. Semiparametric work combines a low-dimensional target with distributional features that are not described by a finite parameter vector. For the missing mean, the unknown regression $m(z)=E[Y\mid Z=z,R=1]$ and observation probability $\pi(z)=P(R=1\mid Z=z)$ will be nuisance functions: population objects needed to estimate the target, rather than the final quantity reported. Naming a regression learner does not define these objects or establish their identifying assumptions. A parameter can also be an estimand. For example, a mean is a functional on distributions, and a linear projection coefficient can be defined outside a linear conditional-mean model. What matters is the domain on which the quantity is defined and the interpretation claimed for it. A causal interpretation needs a separate identifying argument. ## A precise answer to a different question Suppose half the eligible people have $Z=0$ and half have $Z=1$. Their feature means are 0 and 4. Outcomes are observed with probabilities 0.8 and 0.2, respectively, independently of the outcome within each stratum. The observed fraction is $0.5(0.8)+0.5(0.2)=0.5$. Among observed people, the probability of Z=1 is $0.5(0.2)/0.5=0.2$ by Bayes' rule; the other stratum has weight 0.8. Selection has changed the averaging weights, without changing either stratum's mean. | Quantity | Population weights on the strata | Value | |---|---|---| | Eligible-population mean | $1/2,1/2$ | $2$ | | Mean among observed outcomes | $0.8,0.2$ | $0.8$ | The unweighted observed-outcome mean can consistently estimate 0.8. If our target is the eligible-population mean, the procedure has asymptotic discrepancy $0.8-2=-1.2$. More independent people can make its standard error smaller without removing that discrepancy. If the scientific question instead concerns the observed subset, 0.8 may be the intended target. This is a target mismatch. It should be distinguished from finite-sample bias, which concerns the expectation of an estimator at a particular sample size. It should also be distinguished from nonidentification: two full populations can sometimes generate the same observed record while having different target means. ### Why that is the procedure's limit The complete-case mean is a ratio, $$ T_n=\frac{n^{-1}\sum_iR_iY_i}{n^{-1}\sum_iR_i}. $$ For independent identically distributed records, finite required expectations, and $P(R=1)>0$, the law of large numbers makes the numerator and denominator approach their population averages. Their ratio approaches $$ \begin{aligned} \mu^\star&=\frac{E[RY]}{P(R=1)}\\ &=E[Y\mid R=1]. \end{aligned} $$ The denominator is nonzero with probability approaching one. This explains the limit rather than assuming that a familiar sample mean must estimate the desired mean. [Large-sample reasoning for methods research](../estimation-and-inference/large-sample-reasoning.qmd) later develops the convergence argument formally. Now separate the error around the limit from the mismatch: $$ T_n-\mu=(T_n-\mu^\star)+(\mu^\star-\mu). $$ The first term gets smaller as the estimator learns its own target. The second stays at −1.2 in this example. Even a perfectly estimated SE for the first term cannot remove the second. Whether we can instead identify and estimate the full mean depends on the observation assumptions, addressed on the next missingness page. ## A working regression has a population meaning For a nonconstant scalar $X$ with suitable moments, the least-squares slope with an intercept targets $$ \beta^*=\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(X)}. $$ This is a legitimate projection functional even if the conditional mean is curved. Under a stated model $Y=\beta X+\gamma Z+\varepsilon$ with $E[\varepsilon\mid X,Z]=0$, $$ \beta^*=\beta+\gamma\frac{\operatorname{Cov}(X,Z)}{\operatorname{Var}(X)}. $$ The coefficient at fixed $Z$ and the marginal projection slope answer different questions. Calling the second “biased” requires naming the first as the intended target. Neither is automatically a causal effect. The detailed pooling and imperfect-adjustment calculations appear in [Association targets, pooling, and imperfect adjustment](../analysis-problems/association-targets-and-bias.qmd), where they serve a network-analysis question. To see where the projection formula comes from, minimize the population squared error $E[(Y-a-bX)^2]$. The intercept condition gives $a=E[Y]-bE[X]$. The slope condition then gives $\operatorname{Cov}(X,Y)-b\operatorname{Var}(X)=0$. These conditions define the best linear approximation, whether or not the true conditional mean is linear. Under the displayed richer mean model, expanding $\operatorname{Cov}(X,Y)$ gives the stated omitted-variable term. There are therefore two separate possibilities. The reduced regression can describe the observed X–Y relation adequately but its coefficient answers a different question from the slope at fixed Z. Or its conditional-mean model can be wrong and its fitted coefficient still approach a well-defined projection. In either case, first specify which interpretation is wanted; then assess uncertainty for that quantity. ## Definition, identification, and estimation have different jobs Definition states what is wanted. Identification connects that quantity to the observable distribution under assumptions. Estimation supplies a sample procedure for the identified functional. Inference studies its uncertainty and approximation errors. For the running mean, missing at random, positive observation probabilities over the target support, and covariates recorded for everyone identify $E[Y]$ as $E[m(Z)]$. [Missingness, selection, and the observed-data target](missingness-and-selection.qmd) works through that claim. Later guides construct an estimator and show when its uncertainty can be justified. A formula for a discrepancy can suggest a diagnostic only when its relevant ingredients are observable or identified. A test on the observed data cannot generally verify missing at random or rule out unmeasured confounding. Non-rejection does not certify that a working assumption is true. ## Exercises The checkpoint is a completion target. Additional practice is optional. ::: {#checkpoint-des-03 .checkpoint} ### Checkpoint · DES-03 — Separate a target from a procedure’s limit {.unnumbered} **Task type:** Interpretation and calculation. Use the two-stratum example to calculate the eligible-population and observed-outcome means. Name the estimand and probability limit when the observed-outcome mean is used to answer the full-population question. Explain what increasing the sample size changes. Then define a question for which the observed-outcome mean is the intended target. **Ready to continue when:** you can distinguish target mismatch, sampling uncertainty, finite-sample bias, and nonidentification. **Transfer:** If the conditional mean is nonlinear, can a linear projection coefficient still be a well-defined estimand? <details class="checkpoint-help"> <summary>Hint</summary> First calculate the distribution of Z among observed outcomes; then keep the scientific question fixed while evaluating the procedure. </details> <details class="checkpoint-help"> <summary>Solution outline — open after trying</summary> The eligible mean is 2. Selection changes the stratum weights to 0.8 and 0.2, giving an observed-outcome mean 0.8. The procedure converges to 0.8 under the stated sampling conditions, so its asymptotic discrepancy for the target 2 is -1.2. More independent people reduce sampling variation around that limit. A question explicitly about the observed subset can instead target 0.8. Whether the full mean is identified requires assumptions about the unobserved outcomes. A linear projection is meaningful outside a linear conditional-mean model when its moments and denominator are defined. </details> ::: ### Additional practice (optional) {.unnumbered} 1. Change the observation probabilities while keeping the eligible population fixed. Which target stays the same? 2. Derive the least-squares slope limit under the displayed two-variable conditional-mean model. 3. Specify a standardized group contrast and explain why its target distribution must be named. ## Related calculations The relocated worked calculation is in [Association targets, pooling, and imperfect adjustment](../analysis-problems/association-targets-and-bias.qmd).