Understanding an omics dataset

From an assay matrix to a statistical observation.

Before reading: Basic R matrices and data frames; no assay-specific biology is assumed.

1 Start with the observation, not the algorithm

Imagine receiving a matrix called expression_final.csv. The name does not establish what its values measure, whether its columns represent independent people, or whether “final” means normalized, transformed, filtered, or batch-adjusted. Before calculating a correlation, reconstruct the observation that produced one entry.

An omics dataset connects at least four objects: an assay matrix, sample metadata, feature annotations, and a processing record. In a feature-by-specimen matrix, Y_{gi} is the value recorded for feature g in specimen i. The sample table describes the columns; the feature table describes the rows; the processing record explains how the entries were produced. A row of metadata can determine the independent unit, comparison, and covariates needed to interpret a mean contrast or network edge.

Keep three units separate. A person may supply several specimens, such as blood draws at different visits. A specimen may supply several technical measurements, such as aliquots or repeated instrument runs. Repeating a run can help estimate measurement variability. Repeating a visit can reveal within-person change. Neither creates another independently sampled person. The independent unit follows the study design and research question, rather than the number of matrix columns.

For a feature-level analysis with one specimen per person, an observation might be W_i=(A_i,Z_i,Y_{gi}): group, covariates, and a measured feature on a specified scale. With repeated visits it is a whole person record W_i=(A_i,\{Z_{it},Y_{git}\}_{t\in\mathcal T_i}), where \mathcal T_i lists that person’s observed visits. Independence is assumed between these records, while dependence within a record is retained. Treating visit rows as independent changes the variance calculation and can give frequent visitors more weight in the target.

2 What a feature can represent

Assay family	Feature and measurement questions
Gene expression	Gene, transcript, or array probe. Counts or intensity? Which annotation and transform?
Proteomics	Peptide, protein group, or protein. How were peptides summarized? What produces non-detection?
Metabolomics and lipidomics	Peak, annotated compound, or lipid species. Is identity resolved? Is the scale relative or calibrated?
Microbiome	Taxon, sequence variant, or functional feature. Relative to which total or reference, and at what resolution?
Methylation	Probe, genomic site, or region. What scale and region summary were supplied?

These are starting questions, not interchangeable analysis recipes. A feature is the object the pipeline reports. Its annotation is evidence about what that object represents. Two peaks can come from one molecule, and one protein-group row can combine evidence compatible with several proteins. A gene symbol or compound name does not establish one-to-one molecular identity. Retain the original feature key and annotation version even when a convenient display name is added.

The numerical appearance also needs explanation. Integer-looking values need not be raw counts; a fraction need not be a calibrated molecular proportion. Read the assay protocol and processing record before choosing a likelihood or interpreting a fold change. Gene expression data, for people who have never seen any develops one assay in detail. The other assay primers provide parallel specializations of these same measurement questions.

3 Match the three tables by identifiers

The next toy dataset deliberately supplies metadata out of matrix order. Predict whether matching by row position will assign the right group to every specimen. The correct answer should depend on IDs, not the order in a spreadsheet.

Code

assay <- matrix(c(10, 20, 30, 12, 22, 32, 80, 90, 100, 82, 92, 102),
                nrow = 3,
                dimnames = list(c("gene_1", "gene_2", "gene_3"),
                                c("s1", "s2", "s3", "s4")))
samples <- data.frame(specimen = c("s3", "s1", "s4", "s2"),
                      person = c("p2", "p1", "p2", "p1"),
                      group = c("patient", "control", "patient", "control"),
                      visit = c(1, 1, 2, 2))
features <- data.frame(feature = rownames(assay),
                       annotation_version = "synthetic-v1")
stopifnot(!anyDuplicated(samples$specimen),
          setequal(colnames(assay), samples$specimen))
aligned <- samples[match(colnames(assay), samples$specimen), ]
stopifnot(identical(colnames(assay), aligned$specimen))
aligned

  specimen person   group visit
2       s1     p1 control     1
4       s2     p1 control     2
1       s3     p2 patient     1
3       s4     p2 patient     2

match() returns the metadata row corresponding to each matrix column ID. Here the required row order is 2, 4, 1, 3. After matching, specimens s1 and s2 are controls from p1, while s3 and s4 are patients’ specimens from p2. The final identical() check verifies the resulting order, rather than merely verifying that the same IDs occur somewhere.

The first feature has values 10, 12, 80, and 82. Correct labels give a specimen-average difference 81-11=70. Attaching the original metadata by row position instead labels s1 and s3 as patients, giving 45-47=-2. A join error can reverse a comparison without changing a single assay value.

There are four specimens but only two people. Group and person are perfectly aligned in this tiny example: it demonstrates a join, not credible population-level disease variability. More genes do not supply additional independently sampled people.

For a real import, check unique matrix column IDs and unique metadata specimen keys, equal intended ID sets, matching dimensions, and explicit order after the join. setequal() ignores duplicates, and match() returns the first matching row; neither alone rules out a duplicate-key problem. Check feature keys as well. A one-to-many annotation join can duplicate assay rows even when the sample join is correct. Multi-omics layers need separate specimen alignment; matching person IDs alone can accidentally pair different visits or tissues.

4 Separate observation from interpretation

A recorded number is a measurement under a protocol. It can reflect biological abundance, technical sampling, extraction efficiency, annotation, and preprocessing. An idealized response equation makes the distinction concrete:

Y_{gi}=k_{gi}X_{gi}+b_{gi}+\varepsilon_{gi}.

Here X_{gi} is a biological quantity of interest, k_{gi} a response or recovery factor, b_{gi} background, and \varepsilon_{gi} residual measurement error. This is a teaching model, not a universal assay law. Even when two groups have the same X, their mean measured Y can differ if recovery or background differs. Conversely, a relative normalization can hide a common increase in several absolute quantities.

For a contrast E[Y_g\mid A=1]-E[Y_g\mid A=0], A labels the two groups and Y_g must be defined precisely: raw count, normalized log-expression, protein intensity, or another object. The expectation averages over people in the stated group. A group mean on a specified measured scale is a valid descriptive target. Interpreting it as a contrast in latent abundance X_g needs an additional measurement argument. A well-estimated measured contrast does not by itself establish that connection.

For a network, the target additionally specifies the population over which features co-vary and the variables adjusted for. Cell-composition changes can produce associations even if expression within each cell type is stable. A measured association can be a useful descriptive target without being evidence of direct regulation.

5 A first-contact audit

Record the source and version, feature definition, matrix orientation, specimen and person IDs, assay scale, preprocessing, exclusions, and missing-value conventions. Check that every retained column has one metadata record and that annotation joins do not multiply features unexpectedly. Tabulate specimens per independent unit and group by technical run. Inspect distributions and missingness before deciding whether a transformation is needed.

For zeros and missing entries, ask what event produced the code. A zero read count can result from finite sampling, a signal can fall below a detection threshold, and an entire specimen can fail QC. These mechanisms lead to different statistical observations. For example, a non-detection flag under a known threshold supplies an inequality about a latent signal; NA with no recorded reason may supply less information. Preserve flags and limits alongside values instead of collapsing them prematurely.

Keep uncertainty in the record. If the source does not say whether zeros indicate absence, non-detection, or processing, state that gap and investigate it. Do not infer a missingness mechanism from the numeric code alone. Preserve the original inputs while producing a separately documented analysis-ready dataset.

The Bioconductor SummarizedExperiment documentation provides a concrete implementation of keeping assays, sample metadata, and feature annotations coordinated. The base-R example teaches the same identifier alignment without requiring a package installation.

6 Exercises

The checkpoint is this page’s completion target. Additional practice is optional. Try the checkpoint before opening its hint or solution.

Checkpoint · OM-02 — Reconstruct the statistical observation

Task type: Interpretation and data checking.

You receive 120 columns, representing three visits from each of 40 people, and two assay layers with different column orders. Describe an independent observation for a person-level analysis. Write the checks needed to align the layers and metadata. State which additional information is needed before interpreting a feature contrast as an absolute-abundance difference.

Ready to continue when: you can reconstruct the sampling unit and identify a scale/measurement gap before fitting a model.

Transfer: If one layer is available only at baseline, which proposed cross-layer comparisons are still supported?

Hint

Count people separately from specimens; match keys separately in each layer.

Solution outline — open after trying

The independent observation is a person’s full visit record under an independent-person sampling model. Verify unique specimen keys, common intended specimens, correct person and visit mappings, and explicit column alignment for each assay. Check whether layers cover the same visits; a shared person ID is insufficient to match specimens. A contrast concerns the supplied scale until calibration and measurement assumptions establish an absolute-abundance interpretation. Record unknown processing and non-detection conventions.

Additional practice (optional)

Reorder samples randomly and show that ID matching gives the same aligned result.
Introduce a duplicate specimen ID and explain why match() alone does not protect against a one-to-many join.
For an assay you use, write a one-page measurement and metadata dictionary.

7 Reading

SummarizedExperiment vignette: coordinated assays and metadata.

--- title: "Understanding an omics dataset" subtitle: "From an assay matrix to a statistical observation." guide-status: "Available" --- **Before reading:** Basic R matrices and data frames; no assay-specific biology is assumed. ## Start with the observation, not the algorithm Imagine receiving a matrix called `expression_final.csv`. The name does not establish what its values measure, whether its columns represent independent people, or whether “final” means normalized, transformed, filtered, or batch-adjusted. Before calculating a correlation, reconstruct the observation that produced one entry. An omics dataset connects at least four objects: an assay matrix, sample metadata, feature annotations, and a processing record. In a feature-by-specimen matrix, $Y_{gi}$ is the value recorded for feature $g$ in specimen $i$. The sample table describes the columns; the feature table describes the rows; the processing record explains how the entries were produced. A row of metadata can determine the independent unit, comparison, and covariates needed to interpret a mean contrast or network edge. Keep three units separate. A **person** may supply several **specimens**, such as blood draws at different visits. A specimen may supply several **technical measurements**, such as aliquots or repeated instrument runs. Repeating a run can help estimate measurement variability. Repeating a visit can reveal within-person change. Neither creates another independently sampled person. The independent unit follows the study design and research question, rather than the number of matrix columns. For a feature-level analysis with one specimen per person, an observation might be $W_i=(A_i,Z_i,Y_{gi})$: group, covariates, and a measured feature on a specified scale. With repeated visits it is a whole person record $W_i=(A_i,\{Z_{it},Y_{git}\}_{t\in\mathcal T_i})$, where $\mathcal T_i$ lists that person's observed visits. Independence is assumed between these records, while dependence within a record is retained. Treating visit rows as independent changes the variance calculation and can give frequent visitors more weight in the target. ## What a feature can represent ::: {.table-responsive} | Assay family | Feature and measurement questions | |---|---| | Gene expression | Gene, transcript, or array probe. Counts or intensity? Which annotation and transform? | | Proteomics | Peptide, protein group, or protein. How were peptides summarized? What produces non-detection? | | Metabolomics and lipidomics | Peak, annotated compound, or lipid species. Is identity resolved? Is the scale relative or calibrated? | | Microbiome | Taxon, sequence variant, or functional feature. Relative to which total or reference, and at what resolution? | | Methylation | Probe, genomic site, or region. What scale and region summary were supplied? | ::: These are starting questions, not interchangeable analysis recipes. A **feature** is the object the pipeline reports. Its annotation is evidence about what that object represents. Two peaks can come from one molecule, and one protein-group row can combine evidence compatible with several proteins. A gene symbol or compound name does not establish one-to-one molecular identity. Retain the original feature key and annotation version even when a convenient display name is added. The numerical appearance also needs explanation. Integer-looking values need not be raw counts; a fraction need not be a calibrated molecular proportion. Read the assay protocol and processing record before choosing a likelihood or interpreting a fold change. [Gene expression data, for people who have never seen any](gene-expression.qmd) develops one assay in detail. The other assay primers provide parallel specializations of these same measurement questions. ## Match the three tables by identifiers The next toy dataset deliberately supplies metadata out of matrix order. Predict whether matching by row position will assign the right group to every specimen. The correct answer should depend on IDs, not the order in a spreadsheet. ```{r} assay <- matrix(c(10, 20, 30, 12, 22, 32, 80, 90, 100, 82, 92, 102), nrow = 3, dimnames = list(c("gene_1", "gene_2", "gene_3"), c("s1", "s2", "s3", "s4"))) samples <- data.frame(specimen = c("s3", "s1", "s4", "s2"), person = c("p2", "p1", "p2", "p1"), group = c("patient", "control", "patient", "control"), visit = c(1, 1, 2, 2)) features <- data.frame(feature = rownames(assay), annotation_version = "synthetic-v1") stopifnot(!anyDuplicated(samples$specimen), setequal(colnames(assay), samples$specimen)) aligned <- samples[match(colnames(assay), samples$specimen), ] stopifnot(identical(colnames(assay), aligned$specimen)) aligned ``` <span id="fig-specimens-people"></span> `match()` returns the metadata row corresponding to each matrix column ID. Here the required row order is 2, 4, 1, 3. After matching, specimens `s1` and `s2` are controls from `p1`, while `s3` and `s4` are patients' specimens from `p2`. The final `identical()` check verifies the resulting order, rather than merely verifying that the same IDs occur somewhere. The first feature has values 10, 12, 80, and 82. Correct labels give a specimen-average difference $81-11=70$. Attaching the original metadata by row position instead labels `s1` and `s3` as patients, giving $45-47=-2$. A join error can reverse a comparison without changing a single assay value. There are four specimens but only two people. Group and person are perfectly aligned in this tiny example: it demonstrates a join, not credible population-level disease variability. More genes do not supply additional independently sampled people. For a real import, check unique matrix column IDs and unique metadata specimen keys, equal intended ID sets, matching dimensions, and explicit order after the join. `setequal()` ignores duplicates, and `match()` returns the first matching row; neither alone rules out a duplicate-key problem. Check feature keys as well. A one-to-many annotation join can duplicate assay rows even when the sample join is correct. Multi-omics layers need separate specimen alignment; matching person IDs alone can accidentally pair different visits or tissues. ## Separate observation from interpretation A recorded number is a measurement under a protocol. It can reflect biological abundance, technical sampling, extraction efficiency, annotation, and preprocessing. An idealized response equation makes the distinction concrete: $$ Y_{gi}=k_{gi}X_{gi}+b_{gi}+\varepsilon_{gi}. $$ Here $X_{gi}$ is a biological quantity of interest, $k_{gi}$ a response or recovery factor, $b_{gi}$ background, and $\varepsilon_{gi}$ residual measurement error. This is a teaching model, not a universal assay law. Even when two groups have the same $X$, their mean measured $Y$ can differ if recovery or background differs. Conversely, a relative normalization can hide a common increase in several absolute quantities. For a contrast $E[Y_g\mid A=1]-E[Y_g\mid A=0]$, $A$ labels the two groups and $Y_g$ must be defined precisely: raw count, normalized log-expression, protein intensity, or another object. The expectation averages over people in the stated group. A group mean on a specified measured scale is a valid descriptive target. Interpreting it as a contrast in latent abundance $X_g$ needs an additional measurement argument. A well-estimated measured contrast does not by itself establish that connection. For a network, the target additionally specifies the population over which features co-vary and the variables adjusted for. Cell-composition changes can produce associations even if expression within each cell type is stable. A measured association can be a useful descriptive target without being evidence of direct regulation. ## A first-contact audit Record the source and version, feature definition, matrix orientation, specimen and person IDs, assay scale, preprocessing, exclusions, and missing-value conventions. Check that every retained column has one metadata record and that annotation joins do not multiply features unexpectedly. Tabulate specimens per independent unit and group by technical run. Inspect distributions and missingness before deciding whether a transformation is needed. For zeros and missing entries, ask what event produced the code. A zero read count can result from finite sampling, a signal can fall below a detection threshold, and an entire specimen can fail QC. These mechanisms lead to different statistical observations. For example, a non-detection flag under a known threshold supplies an inequality about a latent signal; `NA` with no recorded reason may supply less information. Preserve flags and limits alongside values instead of collapsing them prematurely. Keep uncertainty in the record. If the source does not say whether zeros indicate absence, non-detection, or processing, state that gap and investigate it. Do not infer a missingness mechanism from the numeric code alone. Preserve the original inputs while producing a separately documented analysis-ready dataset. The Bioconductor [SummarizedExperiment documentation](https://www.bioconductor.org/packages/release/bioc/vignettes/SummarizedExperiment/inst/doc/SummarizedExperiment.html) provides a concrete implementation of keeping assays, sample metadata, and feature annotations coordinated. The base-R example teaches the same identifier alignment without requiring a package installation. ## Exercises The checkpoint is this page's completion target. Additional practice is optional. Try the checkpoint before opening its hint or solution. ::: {#checkpoint-om-02 .checkpoint} ### Checkpoint · OM-02 — Reconstruct the statistical observation {.unnumbered} **Task type:** Interpretation and data checking. You receive 120 columns, representing three visits from each of 40 people, and two assay layers with different column orders. Describe an independent observation for a person-level analysis. Write the checks needed to align the layers and metadata. State which additional information is needed before interpreting a feature contrast as an absolute-abundance difference. **Ready to continue when:** you can reconstruct the sampling unit and identify a scale/measurement gap before fitting a model. **Transfer:** If one layer is available only at baseline, which proposed cross-layer comparisons are still supported? <details class="checkpoint-help"> <summary>Hint</summary> Count people separately from specimens; match keys separately in each layer. </details> <details class="checkpoint-help"> <summary>Solution outline — open after trying</summary> The independent observation is a person's full visit record under an independent-person sampling model. Verify unique specimen keys, common intended specimens, correct person and visit mappings, and explicit column alignment for each assay. Check whether layers cover the same visits; a shared person ID is insufficient to match specimens. A contrast concerns the supplied scale until calibration and measurement assumptions establish an absolute-abundance interpretation. Record unknown processing and non-detection conventions. </details> ::: ### Additional practice (optional) {.unnumbered} 1. Reorder `samples` randomly and show that ID matching gives the same aligned result. 2. Introduce a duplicate specimen ID and explain why `match()` alone does not protect against a one-to-many join. 3. For an assay you use, write a one-page measurement and metadata dictionary. ## Reading - [SummarizedExperiment vignette](https://www.bioconductor.org/packages/release/bioc/vignettes/SummarizedExperiment/inst/doc/SummarizedExperiment.html): coordinated assays and metadata. <span id="continue-through-the-collection"></span>