Sensitivity analysis for assumptions and research decisions

Keep the target visible while probing what the data cannot settle

Before reading: From a research question to an identified estimand, Missingness, selection, and the observed-data target, and Influence functions for the variance derivation.

1 Ask which conclusion depends on which assumption

Sensitivity analysis describes how an answer changes across explicit departures or choices. Start with a fixed scientific target and name what is being varied. A different missing-outcome assumption, a different estimator of the same target, and a different outcome transformation are different exercises.

A useful record contains the target, baseline assumptions, departure parameter or alternative procedure, rationale for its range, expected effect, and uncertainty calculation. Choose scientifically defensible ranges before inspecting which ones preserve significance. Stability across several convenient fits is evidence about those fits, not verification of assumptions the data cannot test.

The group’s published manuscripts provide application case studies. When reading one, reconstruct its target and identify the assumptions that a sensitivity analysis would probe. The calculations here use synthetic or exact teaching examples so that their scope is visible.

2 First distinguish three kinds of change

Change	What is being varied?	What to keep visible
Allow a missing-outcome mean to differ from an observed mean	Identification assumption for the same full-population mean	The departure parameter and its scientific range
Change a nuisance learner or tuning parameter	Estimation procedure for a fixed functional	Training rules, diagnostics, remainder and variance conditions
Change log offsets, reference taxa, adjustment variables in a residual edge, or target population	Often the estimand itself	A separate definition and interpretation for each target
Cap inverse-probability weights	A regularized estimator	Whether it still estimates the original target and what bias remains

A preprocessing choice can also affect measurement validity or nuisance fitting. Write the actual functional rather than assigning every choice to a category by its name. The Preprocessing and the quantity being estimated page gives examples of changed scales and biological adjustment.

3 An observed law can support different full means

Let Y\in[0,1] be an outcome, R indicate that it is observed, and suppose independent people supply W=(R,RY). Write

\begin{aligned} p&=P(R=1),\\ q&=E[Y\mid R=1]. \end{aligned}

The observed data identify p and q when p>0. They do not identify q_0=E[Y\mid R=0] without more information. The full-population target is

\begin{aligned} \mu&=E[Y]\\ &=pq+(1-p)q_0. \end{aligned}

With p=0.6 and q=0.65, missing means 0.2 and 0.8 imply full means 0.47 and 0.71. Both worlds have the same observed outcome distribution if we hold that distribution fixed; the missing outcomes differ. More observations from the same observation process can estimate p,q precisely without choosing between those worlds.

Here is an explicit construction. In both worlds, draw R\sim\operatorname{Bernoulli}(0.6) and draw observed outcomes from Y\mid R=1\sim\operatorname{Bernoulli}(0.65). For the people with R=0, choose Bernoulli(0.2) in the first world and Bernoulli(0.8) in the second. An observed record has probability 0.39 of containing Y=1, probability 0.21 of containing Y=0, and probability 0.40 of containing a missing outcome, in either world. No feature of those observed records distinguishes the two missing-outcome means. This is nonidentification, not a small-sample problem.

In an omics study, the analogous issue is whether unobserved intensities have systematically different distributions from observed ones, for example because low abundance affects detection. A more flexible regression of the observed values can estimate their distribution better; it cannot reveal the unobserved distribution without further information or assumptions.

4 Parameterize a departure instead of filling in certainty

A simple pattern-mixture sensitivity parameter is

\begin{aligned} \delta&=E[Y\mid R=0]-E[Y\mid R=1],\\ \mu_\delta&=q+(1-p)\delta. \end{aligned}

At \delta=0, the two outcome means agree; that is a mean restriction, weaker than full independence. For bounded outcomes the population-compatible range is -q\leq\delta\leq1-q. This mathematical range is not necessarily a scientifically plausible range. Use external knowledge, assay limits, or substantive arguments to justify a narrower range, and state how it was chosen.

The units matter. Here \delta=-0.2 means the missing mean is 0.2 outcome units below the observed mean. With 40% missing, it lowers the full-population mean by 0.4(0.2)=0.08. The factor 1-p explains why the same assumption departure has more influence when more outcomes are missing. This linear pattern is specific to the chosen mean target and additive departure; a log-scale departure or odds-ratio restriction describes a different assumption.

Choosing a value of \delta supplies the missing information needed to identify this mean. Varying it over a justified range asks which conclusions survive those different assumptions. It does not estimate \delta from the observed records or declare that every chosen value is equally likely.

With covariates Z, a natural extension defines \delta(Z) as the missing-versus-observed conditional mean difference. For the same eligible-population target,

\mu_\delta=E\{m(Z)+(1-\pi(Z))\delta(Z)\},

where m(Z)=E[Y\mid Z,R=1] and \pi(Z)=P(R=1\mid Z). The outer average uses the covariate distribution of all eligible people. At each Z, the correction is the missing fraction times the specified conditional mean departure. MAR supplies one route to the zero-departure identification formula, with positivity on the target support. A chosen nonzero departure is an additional assumption, not something identified by fitting m more flexibly. These pattern-mixture ideas are discussed in the National Research Council sensitivity-analysis chapter and Carpenter and Smuk.

5 Quantify sampling uncertainty conditional on the departure

In the no-covariate example, estimate p by the observed fraction and q by the observed outcome mean. Hold \delta fixed as part of the assumed identification model. Different samples change both estimated quantities, so both contribute to uncertainty in \hat\mu_\delta.

First write q=E[RY]/p. The numerator is a population mean and the denominator is an observation probability. The quotient rule gives its influence function:

\begin{aligned} \phi_q(W) &=\frac{RY-pq}{p}-\frac{q(R-p)}p\\ &=\frac R p(Y-q). \end{aligned}

The influence function for p is R-p. Differentiating q+(1-p)\delta therefore gives

D_\delta(W)=\frac{R}{p}(Y-q)-\delta(R-p).

The first term is the ratio-mean contribution for q; the second differentiates (1-p)\delta. It is mean zero. We require independent people, 0<p<1, fixed interior-compatible \delta, and finite nonzero contribution variance for the usual normal approximation. The bounded teaching outcome supplies finite moments. If the departure is estimated from external data, its additional uncertainty and dependence need to be included.

This also yields a useful variance check. Let \sigma_1^2=\operatorname{Var}(Y\mid R=1). The two terms in D_\delta are uncorrelated because E[Y-q\mid R=1]=0. Hence

\begin{aligned} \operatorname{Var}(D_\delta) &=\frac{\sigma_1^2}{p}+\delta^2p(1-p),\\ \operatorname{SE}(\hat\mu_\delta) &\approx\sqrt{\operatorname{Var}(D_\delta)/n}. \end{aligned}

The first component is uncertainty in the observed mean; the second comes from the random observed fraction. At zero departure only the first remains. For binary outcomes, \sigma_1^2=q(1-q). With p=0.6,q=0.65,\delta=-0.375, and n=400, the population formula gives an SE of about 0.032. It quantifies sampling uncertainty if that departure is correct, without resolving whether the missing mean really differs by −0.375.

The implementation never multiplies a missing outcome by zero; it computes the residual term only for observed people.

Code

SensitivityMean <- function(r, y, delta) {
  stopifnot(length(r) == length(y), all(r %in% 0:1),
            all(is.finite(y[r == 1])), all(y[r == 1] >= 0 & y[r == 1] <= 1),
            all(is.finite(delta)))
  n <- length(r); p <- mean(r); stopifnot(n >= 2, p > 0, p < 1)
  q <- mean(y[r == 1]); residual <- numeric(n)
  residual[r == 1] <- (y[r == 1]-q)/p
  estimate <- q+(1-p)*delta
  SE <- vapply(delta, function(d) sd(residual-d*(r-p))/sqrt(n), numeric(1))
  data.frame(delta, missing_mean = q+delta, estimate, SE,
             lower = estimate-qnorm(0.975)*SE, upper = estimate+qnorm(0.975)*SE)
}
set.seed(3201)
n <- 400; r <- rbinom(n, 1, 0.6); y <- rep(NA_real_, n)
y[r == 1] <- rbinom(sum(r), 1, 0.65)
# A prespecified teaching range; scientific applications need their own justification.
curve_result <- SensitivityMean(r, y, delta = seq(-0.5, 0.25, length.out = 101))
stopifnot(all(curve_result$missing_mean >= 0 & curve_result$missing_mean <= 1))
p_hat <- mean(r); q_hat <- mean(y[r == 1])
c(observed_fraction = p_hat, observed_mean = q_hat,
  point_estimate_crossing_half = (0.5-q_hat)/(1-p_hat))

           observed_fraction                observed_mean 
                   0.6600000                    0.6212121 
point_estimate_crossing_half 
                  -0.3565062

Code

s <- curve_result
plot(s$delta, s$estimate, type = "n", ylim = range(c(s$lower, s$upper, 0.5)),
     xlab = "Assumed missing minus observed mean (delta)", ylab = "Full-population mean")
polygon(c(s$delta, rev(s$delta)), c(s$lower, rev(s$upper)),
        col = "#2F80C830", border = NA)
lines(s$delta, s$estimate, col = "#2F80C8", lwd = 2)
abline(h = 0.5, lty = 2); abline(v = 0, lty = 3)

An increasing sensitivity curve crosses a horizontal benchmark, with pointwise sampling intervals and a vertical zero-departure reference. — Figure 1: One synthetic observed dataset (400 people). The line shows the full-mean estimate as the assumed missing-minus-observed mean difference changes. Shading gives pointwise 95% Wald intervals conditional on each fixed departure. The benchmark 0.5 and departure range are teaching choices, not empirical evidence for a particular missingness law.

The curve’s movement describes assumption sensitivity; its interval width describes sampling uncertainty for a specified departure. The shading is not a simultaneous band, a probability distribution over \delta, or an estimate of the missing-data law. Wald intervals may also leave the outcome’s allowed range in small samples; clipping their endpoints does not establish coverage.

All points on the curve use the same people and the same estimates of p and q; they are not 101 independent studies. With increasing n, the pointwise intervals shrink around the population curve, but the curve still moves by (1-p)\delta across departures. Better sampling precision does not remove uncertainty about the identification assumption.

The point-estimate tipping value for a benchmark c is (c-\widehat q)/(1-\widehat p). An interval endpoint crossing c is another criterion with another tipping value. Report which is used, the units, and whether it lies in the justified range. Searching for the departure that preserves a small p-value reverses the purpose of the analysis.

For the population values above and c=0.5, the tipping departure is -0.375, giving a missing mean 0.65-0.375=0.275. The point estimate stays above 0.5 for departures larger than this value. Requiring the lower interval endpoint to exceed 0.5 is a stronger statement and crosses at a different departure. A tipping point helps interpret a scientific range; it does not decide that range for us.

If an independent validation study estimates \delta with SE s_\delta, the delta method adds approximately (1-p)^2s_\delta^2 to the variance based on the main study. Shared observations require the covariance term as well. This is a different calculation from scanning fixed assumed values; externally estimated uncertainty must be propagated rather than silently treated as fixed.

6 Partial identification can be the honest endpoint

Without a missing-outcome mean restriction, boundedness gives the sharp population bounds

\begin{aligned} pq&\leq\mu\\ &\leq pq+(1-p). \end{aligned}

At p=0.6,q=0.65 these are [0.39,0.79]. Every interior value can be produced by a compatible missing mean. These are identification bounds at known population quantities, not a confidence interval. Plugging estimated p,q into the endpoints adds sampling uncertainty; confidence sets for partially identified targets require a separate construction.

The bounds are sharp because every value in them is attained by some allowed q_0\in[0,1]. Their width is 1-p=0.4. Even an infinite sample from this observation process would leave that width unchanged. Since the interval straddles 0.5, boundedness alone cannot determine which side of the benchmark the full mean lies on.

6.1 Add sampling uncertainty to the bounds

There is a simple conservative large-sample construction for this example. Let X_L=RY and X_U=RY+1-R, with RY defined as zero for a missing record. Then L=E[X_L] and U=E[X_U] are the lower and upper identification bounds. Estimate each by its sample mean and estimate its SE by its sample SD divided by \sqrt n.

Use a lower one-sided bound for L and an upper one-sided bound for U, each with error probability \alpha/2. Their union-bound construction is

\begin{aligned} a_L&=\hat L-z_{1-\alpha/2}\widehat{\operatorname{SE}}_L,\\ a_U&=\hat U+z_{1-\alpha/2}\widehat{\operatorname{SE}}_U,\\ C_n&=[a_L,a_U]. \end{aligned}

Under the i.i.d. CLTs and nonzero endpoint variances, this covers the entire population identification interval with asymptotic probability at least 1-\alpha. It is conservative and uses sampling approximations, not an exact finite-sample theorem. The endpoints can be correlated; the union bound does not require their independence.

Code

observed_value <- numeric(n)
observed_value[r == 1] <- y[r == 1]
lower_contribution <- observed_value
upper_contribution <- observed_value + 1 - r
L_hat <- mean(lower_contribution); U_hat <- mean(upper_contribution)
c(estimated_L = L_hat, estimated_U = U_hat,
  confidence_lower = L_hat - qnorm(0.975)*sd(lower_contribution)/sqrt(n),
  confidence_upper = U_hat + qnorm(0.975)*sd(upper_contribution)/sqrt(n))

     estimated_L      estimated_U confidence_lower confidence_upper 
       0.4100000        0.7500000        0.3617409        0.7924876

This wider confidence set expresses sampling uncertainty around the identified set. The pointwise intervals on the sensitivity curve instead condition on particular departures that select particular means. Neither should be labeled simply “the uncertainty interval” without saying what it covers.

The same reasoning exposes a support gap in an adjusted comparison. Suppose a target population places mass 0.3 in stratum Z=1, but group 1 supplies no outcomes there. If its supported mean at Z=0 is 0.5 and outcomes are in [0,1], its target-standardized mean is between 0.35 and 0.65. Restricting the population to Z=0 identifies a different target. Weight clipping or additional bootstrap draws do not identify the original unsupported mean.

7 Use diagnostics without treating them as assumption tests

Plot observation probabilities, group support, tail weights, missingness by assay/batch, and data exclusions. Compare justified nuisance learners with identical target definitions and training rules. A held-out loss can diagnose prediction, but does not test MAR against all MNAR alternatives or verify an unobserved stratum mean.

For an omics analysis, also label changes to normalization references, detection rules, cell-composition adjustment, log offsets, and feature families. Explain whether each change probes measurement assumptions, estimator stability, or a different biological target. Inspect effect magnitudes and intervals, not just discovery counts. Stability can coexist with a shared modeling error across all procedures.

8 Report a sensitivity argument someone can inspect

State the original estimand and baseline identification argument; define the departure and its plausible range; retain all planned scenarios; report the estimates and appropriately conditional uncertainty; and explain which substantive conclusion changes. Preserve unsupported or failed fits as part of the record. A useful conclusion is specific: “The estimated mean exceeds 0.5 for this stated range of missing-outcome mean differences.” It does not describe the assumptions as verified.

Use Bootstrap and resampling for statistical inference for uncertainty under a justified observation/sampling model. Use Evaluating a statistical method to assess a procedure against known targets in a simulation. Neither task replaces an identification sensitivity argument.

9 Exercises

Try each checkpoint before opening its hint or solution.

Checkpoint · SENS-01 — Separate unidentified variation from sampling uncertainty

Task type: Derivation and computation.

For Y in [0,1], p=0.6 and observed mean q=0.65, derive the full-mean formula under missing-minus-observed mean difference δ. Calculate the means when missing means are 0.2 and 0.8, the no-assumption bounds, and the point-estimate departure at which the full mean equals 0.5. Derive D_δ for a fixed δ and distinguish its SE from uncertainty about δ.

Ready to continue when: you can construct compatible missing-data worlds, calculate a sensitivity curve, and label the identification and sampling uncertainties separately.

Transfer: How would uncertainty in a departure estimated from an independent validation study enter the variance argument?

Hint

Write the full mean as a mixture; differentiate q as a ratio and p as a mean.

Solution outline — open after trying

μ_δ=q+(1−p)δ. Missing means 0.2 and 0.8 correspond to δ=−0.45 and 0.15, giving μ=0.47 and 0.71. Boundedness alone gives [0.39,0.79]. The mean reaches 0.5 at δ=(0.5−0.65)/0.4=−0.375, within the mathematical compatible range [−0.65,0.35]. D_δ=R(Y−q)/p−δ(R−p). Its SD divided by sqrt(n) measures sampling variation conditional on the fixed departure under the stated conditions. It does not identify δ or assign probabilities to its values.

Checkpoint · SENS-02 — Name what each robustness check changes

Task type: Interpretation and diagnosis.

A workflow changes the nuisance learner, caps weights, excludes an unsupported stratum, changes a log offset, and allows nonzero δ for missing outcomes. Identify which choices can preserve the original functional, which can change the target, and which add an identification assumption. For a target with unsupported mass 0.3, supported mean 0.5, and outcomes in [0,1], derive its mean bounds. Explain what more bootstrap replicates can establish.

Ready to continue when: you can keep the estimand explicit while distinguishing an estimator choice, an assumption departure, and a changed population or measurement.

Transfer: Would stable conclusions across ten learners that all impose the same wrong missingness assumption verify that assumption?

Hint

Write the population functional after each change rather than classifying the operation by its name.

Solution outline — open after trying

A new learner can estimate the same nuisance object, with rate/variance conditions rechecked. Capping weights regularizes an estimator and may leave bias for the original target. Excluding a stratum changes the target population; changing a log offset changes the measured outcome and its mean target. A specified δ supplies a missing-outcome mean restriction for the same full-population mean. The support-gap bounds are 0.7(0.5)+0.3[0,1]=[0.35,0.65]. More resamples improve conditional Monte Carlo precision, not identification of the unsupported mean.

Additional practice (optional)

Find separately the departure where the estimate crosses 0.5 and where each pointwise interval endpoint crosses it. Explain the different conclusions.
Specify a scientifically motivated δ(Z) for an assay example and identify which parts are informed by observed data or external evidence.
Read a published group manuscript and make a table of target, key assumption, proposed departure, plausible range, and conclusion to assess.

10 Reading

National Research Council, principles and methods of sensitivity analyses: explicit departures from missing-data assumptions.
Carpenter and Smuk (2021), Missing data: a statistical framework for practice.

11 Continue through the guide

Use the group’s published manuscripts as case studies, and the Reporting a statistical methods study page to state the sensitivity result with its scope.

--- title: "Sensitivity analysis for assumptions and research decisions" subtitle: "Keep the target visible while probing what the data cannot settle" guide-status: "Available" --- **Before reading:** [From a research question to an identified estimand](../questions-and-design/question-to-estimand.qmd), [Missingness, selection, and the observed-data target](../questions-and-design/missingness-and-selection.qmd), and [Influence functions](../statistical-models/influence-functions.qmd) for the variance derivation. ## Ask which conclusion depends on which assumption Sensitivity analysis describes how an answer changes across explicit departures or choices. Start with a fixed scientific target and name what is being varied. A different missing-outcome assumption, a different estimator of the same target, and a different outcome transformation are different exercises. A useful record contains the target, baseline assumptions, departure parameter or alternative procedure, rationale for its range, expected effect, and uncertainty calculation. Choose scientifically defensible ranges before inspecting which ones preserve significance. Stability across several convenient fits is evidence about those fits, not verification of assumptions the data cannot test. The group’s [published manuscripts](../../publications.qmd) provide application case studies. When reading one, reconstruct its target and identify the assumptions that a sensitivity analysis would probe. The calculations here use synthetic or exact teaching examples so that their scope is visible. ## First distinguish three kinds of change ::: {.table-responsive} | Change | What is being varied? | What to keep visible | |---|---|---| | Allow a missing-outcome mean to differ from an observed mean | Identification assumption for the same full-population mean | The departure parameter and its scientific range | | Change a nuisance learner or tuning parameter | Estimation procedure for a fixed functional | Training rules, diagnostics, remainder and variance conditions | | Change log offsets, reference taxa, adjustment variables in a residual edge, or target population | Often the estimand itself | A separate definition and interpretation for each target | | Cap inverse-probability weights | A regularized estimator | Whether it still estimates the original target and what bias remains | ::: A preprocessing choice can also affect measurement validity or nuisance fitting. Write the actual functional rather than assigning every choice to a category by its name. The [Preprocessing and the quantity being estimated](../data-and-measurement/preprocessing-and-estimands.qmd) page gives examples of changed scales and biological adjustment. ## An observed law can support different full means Let $Y\in[0,1]$ be an outcome, $R$ indicate that it is observed, and suppose independent people supply $W=(R,RY)$. Write $$ \begin{aligned} p&=P(R=1),\\ q&=E[Y\mid R=1]. \end{aligned} $$ The observed data identify $p$ and $q$ when $p>0$. They do not identify $q_0=E[Y\mid R=0]$ without more information. The full-population target is $$ \begin{aligned} \mu&=E[Y]\\ &=pq+(1-p)q_0. \end{aligned} $$ With $p=0.6$ and $q=0.65$, missing means 0.2 and 0.8 imply full means 0.47 and 0.71. Both worlds have the same observed outcome distribution if we hold that distribution fixed; the missing outcomes differ. More observations from the same observation process can estimate $p,q$ precisely without choosing between those worlds. <span id="fig-sensitivity-worlds"></span> Here is an explicit construction. In both worlds, draw $R\sim\operatorname{Bernoulli}(0.6)$ and draw observed outcomes from $Y\mid R=1\sim\operatorname{Bernoulli}(0.65)$. For the people with R=0, choose Bernoulli(0.2) in the first world and Bernoulli(0.8) in the second. An observed record has probability 0.39 of containing Y=1, probability 0.21 of containing Y=0, and probability 0.40 of containing a missing outcome, in **either** world. No feature of those observed records distinguishes the two missing-outcome means. This is nonidentification, not a small-sample problem. In an omics study, the analogous issue is whether unobserved intensities have systematically different distributions from observed ones, for example because low abundance affects detection. A more flexible regression of the observed values can estimate their distribution better; it cannot reveal the unobserved distribution without further information or assumptions. ## Parameterize a departure instead of filling in certainty A simple pattern-mixture sensitivity parameter is $$ \begin{aligned} \delta&=E[Y\mid R=0]-E[Y\mid R=1],\\ \mu_\delta&=q+(1-p)\delta. \end{aligned} $$ At $\delta=0$, the two outcome means agree; that is a mean restriction, weaker than full independence. For bounded outcomes the population-compatible range is $-q\leq\delta\leq1-q$. This mathematical range is not necessarily a scientifically plausible range. Use external knowledge, assay limits, or substantive arguments to justify a narrower range, and state how it was chosen. The units matter. Here $\delta=-0.2$ means the missing mean is 0.2 outcome units below the observed mean. With 40% missing, it lowers the full-population mean by $0.4(0.2)=0.08$. The factor $1-p$ explains why the same assumption departure has more influence when more outcomes are missing. This linear pattern is specific to the chosen mean target and additive departure; a log-scale departure or odds-ratio restriction describes a different assumption. Choosing a value of $\delta$ supplies the missing information needed to identify this mean. Varying it over a justified range asks which conclusions survive those different assumptions. It does not estimate $\delta$ from the observed records or declare that every chosen value is equally likely. With covariates $Z$, a natural extension defines $\delta(Z)$ as the missing-versus-observed conditional mean difference. For the same eligible-population target, $$ \mu_\delta=E\{m(Z)+(1-\pi(Z))\delta(Z)\}, $$ where $m(Z)=E[Y\mid Z,R=1]$ and $\pi(Z)=P(R=1\mid Z)$. The outer average uses the covariate distribution of all eligible people. At each Z, the correction is the missing fraction times the specified conditional mean departure. MAR supplies one route to the zero-departure identification formula, with positivity on the target support. A chosen nonzero departure is an additional assumption, not something identified by fitting m more flexibly. These pattern-mixture ideas are discussed in the [National Research Council sensitivity-analysis chapter](https://www.nationalacademies.org/read/12955/chapter/7) and [Carpenter and Smuk](https://doi.org/10.1002/bimj.202000196). ## Quantify sampling uncertainty conditional on the departure In the no-covariate example, estimate p by the observed fraction and q by the observed outcome mean. Hold $\delta$ fixed as part of the assumed identification model. Different samples change both estimated quantities, so both contribute to uncertainty in $\hat\mu_\delta$. First write $q=E[RY]/p$. The numerator is a population mean and the denominator is an observation probability. The quotient rule gives its influence function: $$ \begin{aligned} \phi_q(W) &=\frac{RY-pq}{p}-\frac{q(R-p)}p\\ &=\frac R p(Y-q). \end{aligned} $$ The influence function for p is $R-p$. Differentiating $q+(1-p)\delta$ therefore gives $$ D_\delta(W)=\frac{R}{p}(Y-q)-\delta(R-p). $$ The first term is the ratio-mean contribution for $q$; the second differentiates $(1-p)\delta$. It is mean zero. We require independent people, $0<p<1$, fixed interior-compatible $\delta$, and finite nonzero contribution variance for the usual normal approximation. The bounded teaching outcome supplies finite moments. If the departure is estimated from external data, its additional uncertainty and dependence need to be included. This also yields a useful variance check. Let $\sigma_1^2=\operatorname{Var}(Y\mid R=1)$. The two terms in $D_\delta$ are uncorrelated because $E[Y-q\mid R=1]=0$. Hence $$ \begin{aligned} \operatorname{Var}(D_\delta) &=\frac{\sigma_1^2}{p}+\delta^2p(1-p),\\ \operatorname{SE}(\hat\mu_\delta) &\approx\sqrt{\operatorname{Var}(D_\delta)/n}. \end{aligned} $$ The first component is uncertainty in the observed mean; the second comes from the random observed fraction. At zero departure only the first remains. For binary outcomes, $\sigma_1^2=q(1-q)$. With $p=0.6,q=0.65,\delta=-0.375$, and n=400, the population formula gives an SE of about 0.032. It quantifies sampling uncertainty **if that departure is correct**, without resolving whether the missing mean really differs by −0.375. The implementation never multiplies a missing outcome by zero; it computes the residual term only for observed people. ```{r} SensitivityMean <- function(r, y, delta) { stopifnot(length(r) == length(y), all(r %in% 0:1), all(is.finite(y[r == 1])), all(y[r == 1] >= 0 & y[r == 1] <= 1), all(is.finite(delta))) n <- length(r); p <- mean(r); stopifnot(n >= 2, p > 0, p < 1) q <- mean(y[r == 1]); residual <- numeric(n) residual[r == 1] <- (y[r == 1]-q)/p estimate <- q+(1-p)*delta SE <- vapply(delta, function(d) sd(residual-d*(r-p))/sqrt(n), numeric(1)) data.frame(delta, missing_mean = q+delta, estimate, SE, lower = estimate-qnorm(0.975)*SE, upper = estimate+qnorm(0.975)*SE) } set.seed(3201) n <- 400; r <- rbinom(n, 1, 0.6); y <- rep(NA_real_, n) y[r == 1] <- rbinom(sum(r), 1, 0.65) # A prespecified teaching range; scientific applications need their own justification. curve_result <- SensitivityMean(r, y, delta = seq(-0.5, 0.25, length.out = 101)) stopifnot(all(curve_result$missing_mean >= 0 & curve_result$missing_mean <= 1)) p_hat <- mean(r); q_hat <- mean(y[r == 1]) c(observed_fraction = p_hat, observed_mean = q_hat, point_estimate_crossing_half = (0.5-q_hat)/(1-p_hat)) ``` ```{r} #| label: fig-sensitivity-curve #| fig-height: 4.5 #| fig-cap: "One synthetic observed dataset (400 people). The line shows the full-mean estimate as the assumed missing-minus-observed mean difference changes. Shading gives pointwise 95% Wald intervals conditional on each fixed departure. The benchmark 0.5 and departure range are teaching choices, not empirical evidence for a particular missingness law." #| fig-alt: "An increasing sensitivity curve crosses a horizontal benchmark, with pointwise sampling intervals and a vertical zero-departure reference." s <- curve_result plot(s$delta, s$estimate, type = "n", ylim = range(c(s$lower, s$upper, 0.5)), xlab = "Assumed missing minus observed mean (delta)", ylab = "Full-population mean") polygon(c(s$delta, rev(s$delta)), c(s$lower, rev(s$upper)), col = "#2F80C830", border = NA) lines(s$delta, s$estimate, col = "#2F80C8", lwd = 2) abline(h = 0.5, lty = 2); abline(v = 0, lty = 3) ``` The curve's movement describes assumption sensitivity; its interval width describes sampling uncertainty for a specified departure. The shading is not a simultaneous band, a probability distribution over $\delta$, or an estimate of the missing-data law. Wald intervals may also leave the outcome's allowed range in small samples; clipping their endpoints does not establish coverage. All points on the curve use the same people and the same estimates of p and q; they are not 101 independent studies. With increasing n, the pointwise intervals shrink around the population curve, but the curve still moves by $(1-p)\delta$ across departures. Better sampling precision does not remove uncertainty about the identification assumption. The point-estimate tipping value for a benchmark $c$ is $(c-\widehat q)/(1-\widehat p)$. An interval endpoint crossing $c$ is another criterion with another tipping value. Report which is used, the units, and whether it lies in the justified range. Searching for the departure that preserves a small p-value reverses the purpose of the analysis. For the population values above and $c=0.5$, the tipping departure is $-0.375$, giving a missing mean $0.65-0.375=0.275$. The point estimate stays above 0.5 for departures larger than this value. Requiring the **lower interval endpoint** to exceed 0.5 is a stronger statement and crosses at a different departure. A tipping point helps interpret a scientific range; it does not decide that range for us. If an independent validation study estimates $\delta$ with SE $s_\delta$, the delta method adds approximately $(1-p)^2s_\delta^2$ to the variance based on the main study. Shared observations require the covariance term as well. This is a different calculation from scanning fixed assumed values; externally estimated uncertainty must be propagated rather than silently treated as fixed. ## Partial identification can be the honest endpoint Without a missing-outcome mean restriction, boundedness gives the sharp population bounds $$ \begin{aligned} pq&\leq\mu\\ &\leq pq+(1-p). \end{aligned} $$ At $p=0.6,q=0.65$ these are $[0.39,0.79]$. Every interior value can be produced by a compatible missing mean. These are identification bounds at known population quantities, not a confidence interval. Plugging estimated $p,q$ into the endpoints adds sampling uncertainty; confidence sets for partially identified targets require a separate construction. The bounds are **sharp** because every value in them is attained by some allowed $q_0\in[0,1]$. Their width is $1-p=0.4$. Even an infinite sample from this observation process would leave that width unchanged. Since the interval straddles 0.5, boundedness alone cannot determine which side of the benchmark the full mean lies on. ### Add sampling uncertainty to the bounds There is a simple conservative large-sample construction for this example. Let $X_L=RY$ and $X_U=RY+1-R$, with $RY$ defined as zero for a missing record. Then $L=E[X_L]$ and $U=E[X_U]$ are the lower and upper identification bounds. Estimate each by its sample mean and estimate its SE by its sample SD divided by $\sqrt n$. Use a lower one-sided bound for L and an upper one-sided bound for U, each with error probability $\alpha/2$. Their union-bound construction is $$ \begin{aligned} a_L&=\hat L-z_{1-\alpha/2}\widehat{\operatorname{SE}}_L,\\ a_U&=\hat U+z_{1-\alpha/2}\widehat{\operatorname{SE}}_U,\\ C_n&=[a_L,a_U]. \end{aligned} $$ Under the i.i.d. CLTs and nonzero endpoint variances, this covers the entire population identification interval with asymptotic probability at least $1-\alpha$. It is conservative and uses sampling approximations, not an exact finite-sample theorem. The endpoints can be correlated; the union bound does not require their independence. ```{r} observed_value <- numeric(n) observed_value[r == 1] <- y[r == 1] lower_contribution <- observed_value upper_contribution <- observed_value + 1 - r L_hat <- mean(lower_contribution); U_hat <- mean(upper_contribution) c(estimated_L = L_hat, estimated_U = U_hat, confidence_lower = L_hat - qnorm(0.975)*sd(lower_contribution)/sqrt(n), confidence_upper = U_hat + qnorm(0.975)*sd(upper_contribution)/sqrt(n)) ``` This wider confidence set expresses sampling uncertainty **around the identified set**. The pointwise intervals on the sensitivity curve instead condition on particular departures that select particular means. Neither should be labeled simply “the uncertainty interval” without saying what it covers. The same reasoning exposes a support gap in an adjusted comparison. Suppose a target population places mass 0.3 in stratum $Z=1$, but group 1 supplies no outcomes there. If its supported mean at $Z=0$ is 0.5 and outcomes are in $[0,1]$, its target-standardized mean is between 0.35 and 0.65. Restricting the population to $Z=0$ identifies a different target. Weight clipping or additional bootstrap draws do not identify the original unsupported mean. ## Use diagnostics without treating them as assumption tests Plot observation probabilities, group support, tail weights, missingness by assay/batch, and data exclusions. Compare justified nuisance learners with identical target definitions and training rules. A held-out loss can diagnose prediction, but does not test MAR against all MNAR alternatives or verify an unobserved stratum mean. For an omics analysis, also label changes to normalization references, detection rules, cell-composition adjustment, log offsets, and feature families. Explain whether each change probes measurement assumptions, estimator stability, or a different biological target. Inspect effect magnitudes and intervals, not just discovery counts. Stability can coexist with a shared modeling error across all procedures. ## Report a sensitivity argument someone can inspect State the original estimand and baseline identification argument; define the departure and its plausible range; retain all planned scenarios; report the estimates and appropriately conditional uncertainty; and explain which substantive conclusion changes. Preserve unsupported or failed fits as part of the record. A useful conclusion is specific: “The estimated mean exceeds 0.5 for this stated range of missing-outcome mean differences.” It does not describe the assumptions as verified. Use [Bootstrap and resampling for statistical inference](../estimation-and-inference/bootstrap-and-resampling.qmd) for uncertainty under a justified observation/sampling model. Use [Evaluating a statistical method](evaluating-methods.qmd) to assess a procedure against known targets in a simulation. Neither task replaces an identification sensitivity argument. ## Exercises Try each checkpoint before opening its hint or solution. ::: {#checkpoint-sens-01 .checkpoint} ### Checkpoint · SENS-01 — Separate unidentified variation from sampling uncertainty {.unnumbered} **Task type:** Derivation and computation. For Y in [0,1], p=0.6 and observed mean q=0.65, derive the full-mean formula under missing-minus-observed mean difference δ. Calculate the means when missing means are 0.2 and 0.8, the no-assumption bounds, and the point-estimate departure at which the full mean equals 0.5. Derive D_δ for a fixed δ and distinguish its SE from uncertainty about δ. **Ready to continue when:** you can construct compatible missing-data worlds, calculate a sensitivity curve, and label the identification and sampling uncertainties separately. **Transfer:** How would uncertainty in a departure estimated from an independent validation study enter the variance argument? <details class="checkpoint-help"> <summary>Hint</summary> Write the full mean as a mixture; differentiate q as a ratio and p as a mean. </details> <details class="checkpoint-help"> <summary>Solution outline — open after trying</summary> μ_δ=q+(1−p)δ. Missing means 0.2 and 0.8 correspond to δ=−0.45 and 0.15, giving μ=0.47 and 0.71. Boundedness alone gives [0.39,0.79]. The mean reaches 0.5 at δ=(0.5−0.65)/0.4=−0.375, within the mathematical compatible range [−0.65,0.35]. D_δ=R(Y−q)/p−δ(R−p). Its SD divided by sqrt(n) measures sampling variation conditional on the fixed departure under the stated conditions. It does not identify δ or assign probabilities to its values. </details> ::: ::: {#checkpoint-sens-02 .checkpoint} ### Checkpoint · SENS-02 — Name what each robustness check changes {.unnumbered} **Task type:** Interpretation and diagnosis. A workflow changes the nuisance learner, caps weights, excludes an unsupported stratum, changes a log offset, and allows nonzero δ for missing outcomes. Identify which choices can preserve the original functional, which can change the target, and which add an identification assumption. For a target with unsupported mass 0.3, supported mean 0.5, and outcomes in [0,1], derive its mean bounds. Explain what more bootstrap replicates can establish. **Ready to continue when:** you can keep the estimand explicit while distinguishing an estimator choice, an assumption departure, and a changed population or measurement. **Transfer:** Would stable conclusions across ten learners that all impose the same wrong missingness assumption verify that assumption? <details class="checkpoint-help"> <summary>Hint</summary> Write the population functional after each change rather than classifying the operation by its name. </details> <details class="checkpoint-help"> <summary>Solution outline — open after trying</summary> A new learner can estimate the same nuisance object, with rate/variance conditions rechecked. Capping weights regularizes an estimator and may leave bias for the original target. Excluding a stratum changes the target population; changing a log offset changes the measured outcome and its mean target. A specified δ supplies a missing-outcome mean restriction for the same full-population mean. The support-gap bounds are 0.7(0.5)+0.3[0,1]=[0.35,0.65]. More resamples improve conditional Monte Carlo precision, not identification of the unsupported mean. </details> ::: ### Additional practice (optional) {.unnumbered} 1. Find separately the departure where the estimate crosses 0.5 and where each pointwise interval endpoint crosses it. Explain the different conclusions. 2. Specify a scientifically motivated δ(Z) for an assay example and identify which parts are informed by observed data or external evidence. 3. Read a published group manuscript and make a table of target, key assumption, proposed departure, plausible range, and conclusion to assess. ## Reading - National Research Council, [principles and methods of sensitivity analyses](https://www.nationalacademies.org/read/12955/chapter/7): explicit departures from missing-data assumptions. - Carpenter and Smuk (2021), [Missing data: a statistical framework for practice](https://doi.org/10.1002/bimj.202000196). ## Continue through the guide {#continue-through-the-collection} Use the group’s [published manuscripts](../../publications.qmd) as case studies, and the [Reporting a statistical methods study](../research-practice/reporting-methods-studies.qmd) page to state the sensitivity result with its scope.