Missingness, selection, and the observed-data target

Identify the population mean before repairing an incomplete column.

Before reading: From a research question to an identified estimand, the independent unit, and elementary conditional means. For discrete Z, the law of total expectation simply averages the stratum means using population stratum probabilities; the geometric projection theory is introduced later.

1 A missing outcome is an observation process

Let Y be one outcome or feature measurement on a specified scale, Z covariates available for every eligible unit, and R an indicator that Y is observed. The observed record is often written W=(Z,R,RY). The indicator is essential: RY=0 when R=0 encodes an unrecorded outcome, not evidence that the outcome equals zero. In R, keep unobserved outcomes as NA and form contributions only for observed rows.

The target here is \mu=E[Y] in the full eligible population. The mean among complete cases is E[Y\mid R=1], a different target. Whether the latter answers the scientific question is not determined by how few values are missing.

Missing outcomes, missing covariates, assay non-detection, and selection of whole people create different observed-data problems. A method developed for one pattern should not be transferred solely because both datasets contain NA.

2 A simple selection calculation

Suppose half the eligible people have Z=0 and half Z=1. The outcome means are 0 and 4. Observation probabilities are 0.8 and 0.2. Selection on Z changes the mix of complete cases even if observation is independent of the outcome within each stratum.

Code

pz <- c(0.5, 0.5); pi_z <- c(0.8, 0.2); m <- c(0, 4)
pz_observed <- pz * pi_z / sum(pz * pi_z)
c(full_mean = sum(pz * m), complete_case_mean = sum(pz_observed * m))

         full_mean complete_case_mean 
               2.0                0.8

The full mean is 2 and the complete-case mean is 0.8. No within-stratum outcome-dependent missingness was needed to create the discrepancy.

This reuses the example from What is your estimator actually estimating?. That page establishes the naive procedure’s limit. Here the next question is what lets us learn the full mean from incomplete observations.

3 Identification under missing at random

Assume E[|Y|]<\infty, R\perp Y\mid Z (missing at random for this outcome), and \pi(Z)=P(R=1\mid Z)>0 almost surely over the target population. Also assume Z is recorded for that population. Then

\begin{aligned} m(Z)&=E[Y\mid Z,R=1]\\ &=E[Y\mid Z],\\ \mu&=E[m(Z)]\\ &=E\left[\frac{RY}{\pi(Z)}\right]. \end{aligned}

The first equality is the identifying restriction. The outer expectation must use the target covariate distribution, not only the complete cases. Positive observation probability supplies outcome information in each needed stratum. For simple variance and remainder arguments, stronger conditions such as \pi\ge c>0 and suitable moments are often imposed; mere positivity for identification is weaker.

Work through the weighted identity by conditioning on Z:

\begin{aligned} E\left[\frac{RY}{\pi(Z)}\,\middle|\,Z\right]&=\frac{P(R=1\mid Z)}{\pi(Z)}E[Y\mid Z,R=1]\\ &=m(Z). \end{aligned}

The denominator cancels the chance of observing that stratum’s outcome. Averaging m(Z) over all eligible people’s covariates then gives the full mean. For this mean target, equality of the relevant conditional means is sufficient; the usual MAR assumption equates the full conditional outcome distributions and is stronger. It is still an assumption about outcomes we do not observe.

Small positive observation probability and zero probability cause different problems. If \pi(z)=0.01, the stratum is represented in the population observed-data law, but a finite study may contain very few outcomes and unstable large weights. If \pi(z)=0 for a positive-mass target stratum, unrestricted identification fails. Truncating weights can trade bias against variation, but cannot create outcomes in that absent stratum. An empty finite-sample cell also does not by itself prove a population structural zero.

Neither MAR nor an unobserved outcome distribution can generally be verified from the observed data alone. Compare mechanisms scientifically, inspect patterns, and assess sensitivity. A model that predicts R well does not prove MAR.

4 A self-contained weighted example

The next example uses known observation probabilities because the simulation supplies them. It should move the weighted mean toward 2, while the complete-case mean remains near 0.8. In a real study the observation probabilities often have to be estimated and their errors accounted for.

Code

set.seed(41)
n <- 2000
z <- rbinom(n, 1, 0.5); prob <- 0.8 - 0.6 * z
full_y <- 4 * z + rnorm(n)
r <- rbinom(n, 1, prob)
y <- ifelse(r == 1, full_y, NA_real_)
contribution <- numeric(n)
contribution[r == 1] <- y[r == 1] / prob[r == 1]
weight <- numeric(n)
weight[r == 1] <- 1 / prob[r == 1]
c(complete_case = mean(y[r == 1]),
  known_weight_mean = mean(contribution),
  normalized_weight_mean = sum(contribution) / sum(weight), truth = 2)

         complete_case      known_weight_mean normalized_weight_mean 
             0.7912569              1.9026191              1.9678026 
                 truth 
             2.0000000

The Z=0 outcomes receive weight 1/0.8=1.25; the less frequently observed Z=1 outcomes receive weight 1/0.2=5. The weights restore the target’s representation in expectation. The contribution average divides by all n eligible people, including those with an unrecorded outcome.

Dividing that weighted sum by the observed count would instead approach \mu/P(R=1)=4 here. It is not the same normalization. The normalized weighted ratio divides by the sum of the observation weights; that denominator divided by n approaches E[R/\pi(Z)]=1, so the ratio also consistently targets \mu. It need not equal the contribution average in a finite sample and has its own variance calculation. Identification supplies the population identity; it does not make every sample implementation equivalent.

5 Non-detection is not automatically MAR

If a feature is observed only when its concentration exceeds a detection limit, R depends on the missing Y even after conditioning on routinely measured covariates. MAR is then generally unsupported. A recorded zero may mean a real zero, an unobserved value below a threshold, or a processing convention; inspect the assay record.

A censoring or measurement model can exploit a known limit under additional assumptions. Multiple imputation also requires a model and appropriate uncertainty propagation; generating several completed tables does not by itself resolve nonidentification. A deterministic fill value is a scientific/modeling choice, not the observed truth.

When an outcome is known to lie in [L,U], a simple assumption-light bound is

\begin{aligned} E[Y]&\ge E[RY]+L P(R=0),\\ E[Y]&\le E[RY]+U P(R=0). \end{aligned}

Here RY is the observed contribution with zero assigned only to that algebraic contribution when missing. The bounds do not assert that the missing outcomes are zero. Better measurement information can tighten them; an observed-data point estimate outside its identifying assumptions should not be presented as more certain than the information supports.

When a population stratum has no observed outcomes, keep its missing mean explicit. With eligible weights one half each, mean zero at Z=0, and no observed outcomes at Z=1, the latter stratum could have mean 4 or 8. Both full populations generate the same observed record, but their full means are 2 and 4. We know the stratum’s weight because Z is recorded for everyone; its outcome mean remains unknown.

If the assay establishes that every missing value lies in [L,d], with d<U, replace U by d in the upper bound. This uses actual measurement information. It is unavailable for records missing for unrelated technical or selection reasons. These are population identification bounds; estimating their endpoints adds sampling uncertainty, as developed in Sensitivity analysis for assumptions and research decisions.

6 Selection and covariate missingness need separate targets

Let S indicate enrollment or inclusion in the dataset, and reserve R for outcome observation within it. Correcting missing outcomes among S=1 people does not by itself generalize to S=0 people. If only included people have Z recorded, the full eligible covariate distribution may itself be unknown. Generalizing to that population can require external information, sampling weights, or selection assumptions. MAR for outcomes among included people does not solve that separate problem.

Missing covariates change which nuisance functions can be evaluated. Either define an observed-data pattern-specific analysis with a justified target, or supply assumptions and methods for the joint missing-data process. Dropping every row with any missing feature can dramatically change the population and the analysis family in high-dimensional omics.

7 Connect identification to construction

The From an influence function to an estimator page continues the same full-population mean question with a richer continuous covariate example. It combines regression and observation models, derives the influence function, and states the stronger conditions needed for normal inference. “Either model correct” is a consistency property for that construction, not a substitute for MAR and positivity.

8 Exercises

Try each checkpoint before opening its hint or solution.

Checkpoint · MIS-01 — Identify the target under an observation process

Task type: Derivation and diagnosis.

The two-stratum example has eligible-population mean 2 and complete-case mean 0.8, as calculated in What is your estimator actually estimating?. Derive why E[RY/\pi(Z)]=2 under the stated MAR and positivity assumptions, and explain how the inverse probabilities restore the eligible-population weights. Then set the observation probability at Z=1 to zero. Show why two different missing-stratum means give the same observed distribution but different population means. Explain what happens if missing outcomes are silently filled with zero and then treated as observed.

Ready to continue when: you can distinguish a complete-case target, an identified full-population target, and a genuinely unidentified component.

Transfer: Which extra information would let you tighten bounds for outcomes below a known detection limit?

Hint

Condition on Z to calculate E[RY/\pi(Z)\mid Z]. Separate the observation indicator from the outcome value, and identify the step that requires a positive observation probability.

Solution outline — open after trying

MAR gives E[RY\mid Z]=\pi(Z)E[Y\mid Z]. With \pi(Z)>0, division and the law of total expectation give E[RY/\pi(Z)]=E[Y]=2. Numerically, the stratum contributions are 0.5(0.8)(0)/0.8=0 and 0.5(0.2)(4)/0.2=2. Inverse weighting cancels the unequal observation probabilities rather than preserving the complete-case stratum mix.

If \pi(1)=0, that division is unavailable in the second stratum. Choosing its unobserved mean as 4 or 8 leaves the observed distribution unchanged but changes the full mean from 2 to 4. Filling missing outcomes with zero and treating them as observed targets E[RY] rather than E[Y]; in the original positive-probability example this is 0.5(0.2)(4)=0.4. It hides the missing component rather than identifying it.

Additional practice (optional)

Simulate an observation probability that also depends on the unobserved outcome and assess the MAR-weighted estimator.
Compute the bounds when 30% of outcomes are missing and all outcomes lie in [0,5]. State what observed quantity is still needed.
List separately the assumptions for selection into your cohort and missing outcomes within it.

9 Reading

Kennedy, review of efficient functional estimation, includes the missing-outcome mean as an observed-data functional.
Rubin (1976), Inference and missing data, for foundational distinctions about missing-data mechanisms.

--- title: "Missingness, selection, and the observed-data target" subtitle: "Identify the population mean before repairing an incomplete column." guide-status: "Available" --- **Before reading:** [From a research question to an identified estimand](question-to-estimand.qmd), the independent unit, and elementary conditional means. For discrete Z, the law of total expectation simply averages the stratum means using population stratum probabilities; the geometric projection theory is introduced later. ## A missing outcome is an observation process Let $Y$ be one outcome or feature measurement on a specified scale, $Z$ covariates available for every eligible unit, and $R$ an indicator that $Y$ is observed. The observed record is often written $W=(Z,R,RY)$. The indicator is essential: $RY=0$ when $R=0$ encodes an unrecorded outcome, not evidence that the outcome equals zero. In R, keep unobserved outcomes as `NA` and form contributions only for observed rows. The target here is $\mu=E[Y]$ in the full eligible population. The mean among complete cases is $E[Y\mid R=1]$, a different target. Whether the latter answers the scientific question is not determined by how few values are missing. Missing outcomes, missing covariates, assay non-detection, and selection of whole people create different observed-data problems. A method developed for one pattern should not be transferred solely because both datasets contain `NA`. ## A simple selection calculation Suppose half the eligible people have $Z=0$ and half $Z=1$. The outcome means are 0 and 4. Observation probabilities are 0.8 and 0.2. Selection on $Z$ changes the mix of complete cases even if observation is independent of the outcome within each stratum. ```{r} pz <- c(0.5, 0.5); pi_z <- c(0.8, 0.2); m <- c(0, 4) pz_observed <- pz * pi_z / sum(pz * pi_z) c(full_mean = sum(pz * m), complete_case_mean = sum(pz_observed * m)) ``` The full mean is 2 and the complete-case mean is 0.8. No within-stratum outcome-dependent missingness was needed to create the discrepancy. <span id="fig-observation-selection"></span> This reuses the example from [What is your estimator actually estimating?](estimands-and-misspecification.qmd). That page establishes the naive procedure's limit. Here the next question is what lets us learn the full mean from incomplete observations. ## Identification under missing at random Assume $E[|Y|]<\infty$, $R\perp Y\mid Z$ (missing at random for this outcome), and $\pi(Z)=P(R=1\mid Z)>0$ almost surely over the target population. Also assume $Z$ is recorded for that population. Then $$ \begin{aligned} m(Z)&=E[Y\mid Z,R=1]\\ &=E[Y\mid Z],\\ \mu&=E[m(Z)]\\ &=E\left[\frac{RY}{\pi(Z)}\right]. \end{aligned} $$ The first equality is the identifying restriction. The outer expectation must use the target covariate distribution, not only the complete cases. Positive observation probability supplies outcome information in each needed stratum. For simple variance and remainder arguments, stronger conditions such as $\pi\ge c>0$ and suitable moments are often imposed; mere positivity for identification is weaker. Work through the weighted identity by conditioning on Z: $$ \begin{aligned} E\left[\frac{RY}{\pi(Z)}\,\middle|\,Z\right]&=\frac{P(R=1\mid Z)}{\pi(Z)}E[Y\mid Z,R=1]\\ &=m(Z). \end{aligned} $$ The denominator cancels the chance of observing that stratum's outcome. Averaging m(Z) over **all** eligible people's covariates then gives the full mean. For this mean target, equality of the relevant conditional means is sufficient; the usual MAR assumption equates the full conditional outcome distributions and is stronger. It is still an assumption about outcomes we do not observe. Small positive observation probability and zero probability cause different problems. If $\pi(z)=0.01$, the stratum is represented in the population observed-data law, but a finite study may contain very few outcomes and unstable large weights. If $\pi(z)=0$ for a positive-mass target stratum, unrestricted identification fails. Truncating weights can trade bias against variation, but cannot create outcomes in that absent stratum. An empty finite-sample cell also does not by itself prove a population structural zero. Neither MAR nor an unobserved outcome distribution can generally be verified from the observed data alone. Compare mechanisms scientifically, inspect patterns, and assess sensitivity. A model that predicts $R$ well does not prove MAR. ## A self-contained weighted example The next example uses known observation probabilities because the simulation supplies them. It should move the weighted mean toward 2, while the complete-case mean remains near 0.8. In a real study the observation probabilities often have to be estimated and their errors accounted for. ```{r} set.seed(41) n <- 2000 z <- rbinom(n, 1, 0.5); prob <- 0.8 - 0.6 * z full_y <- 4 * z + rnorm(n) r <- rbinom(n, 1, prob) y <- ifelse(r == 1, full_y, NA_real_) contribution <- numeric(n) contribution[r == 1] <- y[r == 1] / prob[r == 1] weight <- numeric(n) weight[r == 1] <- 1 / prob[r == 1] c(complete_case = mean(y[r == 1]), known_weight_mean = mean(contribution), normalized_weight_mean = sum(contribution) / sum(weight), truth = 2) ``` The Z=0 outcomes receive weight $1/0.8=1.25$; the less frequently observed Z=1 outcomes receive weight $1/0.2=5$. The weights restore the target's representation in expectation. The contribution average divides by **all n eligible people**, including those with an unrecorded outcome. Dividing that weighted sum by the observed count would instead approach $\mu/P(R=1)=4$ here. It is not the same normalization. The normalized weighted ratio divides by the sum of the observation weights; that denominator divided by n approaches $E[R/\pi(Z)]=1$, so the ratio also consistently targets $\mu$. It need not equal the contribution average in a finite sample and has its own variance calculation. Identification supplies the population identity; it does not make every sample implementation equivalent. ## Non-detection is not automatically MAR If a feature is observed only when its concentration exceeds a detection limit, $R$ depends on the missing $Y$ even after conditioning on routinely measured covariates. MAR is then generally unsupported. A recorded zero may mean a real zero, an unobserved value below a threshold, or a processing convention; inspect the assay record. A censoring or measurement model can exploit a known limit under additional assumptions. Multiple imputation also requires a model and appropriate uncertainty propagation; generating several completed tables does not by itself resolve nonidentification. A deterministic fill value is a scientific/modeling choice, not the observed truth. When an outcome is known to lie in $[L,U]$, a simple assumption-light bound is $$ \begin{aligned} E[Y]&\ge E[RY]+L P(R=0),\\ E[Y]&\le E[RY]+U P(R=0). \end{aligned} $$ Here $RY$ is the observed contribution with zero assigned only to that algebraic contribution when missing. The bounds do not assert that the missing outcomes are zero. Better measurement information can tighten them; an observed-data point estimate outside its identifying assumptions should not be presented as more certain than the information supports. When a population stratum has no observed outcomes, keep its missing mean explicit. With eligible weights one half each, mean zero at Z=0, and no observed outcomes at Z=1, the latter stratum could have mean 4 or 8. Both full populations generate the same observed record, but their full means are 2 and 4. We know the stratum's weight because Z is recorded for everyone; its outcome mean remains unknown. If the assay establishes that every missing value lies in $[L,d]$, with $d<U$, replace U by d in the upper bound. This uses actual measurement information. It is unavailable for records missing for unrelated technical or selection reasons. These are population identification bounds; estimating their endpoints adds sampling uncertainty, as developed in [Sensitivity analysis for assumptions and research decisions](../method-development/sensitivity-analysis.qmd). ## Selection and covariate missingness need separate targets Let S indicate enrollment or inclusion in the dataset, and reserve R for outcome observation within it. Correcting missing outcomes among S=1 people does not by itself generalize to S=0 people. If only included people have $Z$ recorded, the full eligible covariate distribution may itself be unknown. Generalizing to that population can require external information, sampling weights, or selection assumptions. MAR for outcomes among included people does not solve that separate problem. Missing covariates change which nuisance functions can be evaluated. Either define an observed-data pattern-specific analysis with a justified target, or supply assumptions and methods for the joint missing-data process. Dropping every row with any missing feature can dramatically change the population and the analysis family in high-dimensional omics. ## Connect identification to construction The [From an influence function to an estimator](../semiparametric-methods/influence-function-to-estimator.qmd) page continues the same full-population mean question with a richer continuous covariate example. It combines regression and observation models, derives the influence function, and states the stronger conditions needed for normal inference. “Either model correct” is a consistency property for that construction, not a substitute for MAR and positivity. ## Exercises Try each checkpoint before opening its hint or solution. ::: {#checkpoint-mis-01 .checkpoint} ### Checkpoint · MIS-01 — Identify the target under an observation process {.unnumbered} **Task type:** Derivation and diagnosis. The two-stratum example has eligible-population mean 2 and complete-case mean 0.8, as calculated in [What is your estimator actually estimating?](estimands-and-misspecification.qmd). Derive why $E[RY/\pi(Z)]=2$ under the stated MAR and positivity assumptions, and explain how the inverse probabilities restore the eligible-population weights. Then set the observation probability at $Z=1$ to zero. Show why two different missing-stratum means give the same observed distribution but different population means. Explain what happens if missing outcomes are silently filled with zero and then treated as observed. **Ready to continue when:** you can distinguish a complete-case target, an identified full-population target, and a genuinely unidentified component. **Transfer:** Which extra information would let you tighten bounds for outcomes below a known detection limit? <details class="checkpoint-help"> <summary>Hint</summary> Condition on $Z$ to calculate $E[RY/\pi(Z)\mid Z]$. Separate the observation indicator from the outcome value, and identify the step that requires a positive observation probability. </details> <details class="checkpoint-help"> <summary>Solution outline — open after trying</summary> MAR gives $E[RY\mid Z]=\pi(Z)E[Y\mid Z]$. With $\pi(Z)>0$, division and the law of total expectation give $E[RY/\pi(Z)]=E[Y]=2$. Numerically, the stratum contributions are $0.5(0.8)(0)/0.8=0$ and $0.5(0.2)(4)/0.2=2$. Inverse weighting cancels the unequal observation probabilities rather than preserving the complete-case stratum mix. If $\pi(1)=0$, that division is unavailable in the second stratum. Choosing its unobserved mean as 4 or 8 leaves the observed distribution unchanged but changes the full mean from 2 to 4. Filling missing outcomes with zero and treating them as observed targets $E[RY]$ rather than $E[Y]$; in the original positive-probability example this is $0.5(0.2)(4)=0.4$. It hides the missing component rather than identifying it. </details> ::: ### Additional practice (optional) {.unnumbered} 1. Simulate an observation probability that also depends on the unobserved outcome and assess the MAR-weighted estimator. 2. Compute the bounds when 30% of outcomes are missing and all outcomes lie in [0,5]. State what observed quantity is still needed. 3. List separately the assumptions for selection into your cohort and missing outcomes within it. ## Reading - Kennedy, [review of efficient functional estimation](https://arxiv.org/html/2203.06469v2), includes the missing-outcome mean as an observed-data functional. - Rubin (1976), [Inference and missing data](https://doi.org/10.1093/biomet/63.3.581), for foundational distinctions about missing-data mechanisms. <span id="continue-through-the-collection"></span>