Data provenance and an analysis-ready dataset

Identifiers, transformations, exclusions, and a visible target population

Before reading: Read Understanding an omics dataset; the examples use base R.

1 An analysis matrix needs a history

A numerical matrix alone does not tell you where measurements came from, which people they represent, or why some samples disappeared. Provenance connects an acquired object to its analysis version and makes those choices inspectable.

For each input, record its source and accession or institutional identifier, release/version, acquisition date, access conditions, file identity/checksum, assay and units, reference genome or feature-annotation version where relevant, and the code that reads it. Use actual identifiers from the source; a synthetic example should explicitly say its identifiers are invented.

For each derived object, record its parent inputs, transformation code and version, settings, inclusion/exclusion decisions, and output identity. Preserve the distinction between a person, specimen, aliquot, assay run, and matrix column. Their relations may be one-to-many. The Understanding an omics dataset page explains these objects.

Provenance has two parts. File identity tells us which objects were used. Data meaning tells us what their rows, columns, labels, and values represent, and why those objects answer the question. An unchanged file can have misinterpreted group labels. A correctly labeled assay can belong to an earlier preprocessing release. The research record needs both kinds of information.

2 Align by identifiers and check cardinality

The following toy data have invented specimen and person identifiers, and arbitrary synthetic measurement units. Matrix columns are specimens; two specimens belong to the same person. The contract is one metadata record per matrix column, keyed by specimen. Person ID is a grouping variable, not a unique specimen key.

Code

assay <- matrix(c(2, 4, 3, 5, 8, 9), nrow = 2,
  dimnames = list(c("feature_A", "feature_B"), c("S1", "S2", "S3")))
metadata <- data.frame(specimen = c("S3", "S1", "S2"),
                       person = c("P2", "P1", "P1"),
                       usable = c(TRUE, TRUE, FALSE),
                       reason = c("retained", "retained", "prespecified assay QC"))
AlignSpecimens <- function(assay, metadata) {
  stopifnot(is.matrix(assay), is.numeric(assay),
            all(c("specimen", "person", "usable", "reason") %in% names(metadata)),
            !is.null(colnames(assay)), !anyNA(colnames(assay)),
            all(nzchar(trimws(colnames(assay)))), !anyDuplicated(colnames(assay)),
            !anyNA(metadata$specimen), !anyDuplicated(metadata$specimen),
            all(nzchar(trimws(metadata$specimen))),
            !anyNA(metadata$person), all(nzchar(trimws(metadata$person))),
            is.logical(metadata$usable), !anyNA(metadata$usable),
            setequal(colnames(assay), metadata$specimen))
  aligned <- metadata[match(colnames(assay), metadata$specimen), , drop = FALSE]
  stopifnot(identical(colnames(assay), aligned$specimen))
  aligned
}
aligned <- AlignSpecimens(assay, metadata)
aligned

  specimen person usable                reason
2       S1     P1   TRUE              retained
3       S2     P1  FALSE prespecified assay QC
1       S3     P2   TRUE              retained

Code

# An intentional error is caught before a join can multiply observations.
bad_metadata <- rbind(metadata, metadata[1, ])
tryCatch(AlignSpecimens(assay, bad_metadata),
         error = function(e) "Rejected: duplicate specimen key")

[1] "Rejected: duplicate specimen key"

setequal deliberately requires the same specimen set in this teaching contract. If unmatched records are expected, use an explicit policy and output both unmatched sets before continuing. An inner join that silently drops unmatched records conceals an exclusion decision. A duplicated feature-to-symbol map can similarly multiply rows: retain stable feature identifiers and define how one-to-many annotations will be handled rather than silently summing or deduplicating them.

2.1 Work through the matching operation

The assay order is S1, S2, S3; the metadata order is S3, S1, S2. Matching produces indices 2, 3, 1 and reorders the metadata to the assay. Comparing only row counts would miss a positional assignment error:

Code

data.frame(matrix_specimen = colnames(assay),
  positional_metadata_key = metadata$specimen,
  positional_person = metadata$person, matched_person = aligned$person)

  matrix_specimen positional_metadata_key positional_person matched_person
1              S1                      S3                P2             P1
2              S2                      S1                P1             P1
3              S3                      S2                P1             P2

There are no missing values in the wrong positional assignment. The mistake is that S1 is attributed to P2 and S3 to P1. A numerical fit could succeed while relating measurements to the wrong people. The final identical() assertion checks ordering after key matching.

2.2 State the relationship before joining

For a key k with a_k rows in one table and b_k in the other, an ordinary match-based join can produce a_kb_k rows. Duplicated specimen annotations can therefore duplicate measurements. Joining two visit tables by person alone can pair every visit with every other visit for that person. Those extra rows are not additional independent observations.

Use specimen IDs for specimen metadata, run IDs for assay runs, or a person–visit key only when the protocol guarantees one record per combination. Multiple aliquots at a visit need another identifier. Represent valid one-to-many relationships explicitly, with an aggregation or analysis rule; keeping the first match silently chooses a record.

For a planned assay intersection, report setdiff(first_keys, second_keys) and setdiff(second_keys, first_keys) before applying the inclusion rule. If the assays use different identifiers, first establish a documented crosswalk at the intended person, visit, or specimen level. Equal-looking strings alone do not establish that relationship.

3 Retain exclusions and denominators

Code

decision_log <- aligned[c("specimen", "person", "usable", "reason")]
keep <- aligned$usable
analysis_assay <- assay[, keep, drop = FALSE]
analysis_metadata <- aligned[keep, , drop = FALSE]
stopifnot(identical(colnames(analysis_assay), analysis_metadata$specimen))
attrition <- data.frame(
  stage = c("Acquired", "Retained after QC"),
  specimens = c(ncol(assay), ncol(analysis_assay)),
  people = c(length(unique(aligned$person)),
             length(unique(analysis_metadata$person))))
attrition

              stage specimens people
1          Acquired         3      2
2 Retained after QC         2      2

Code

decision_log

  specimen person usable                reason
2       S1     P1   TRUE              retained
3       S2     P1  FALSE prespecified assay QC
1       S3     P2   TRUE              retained

Removing a specimen does not always remove a person; losing every specimen from a person does. Here specimens fall from three to two while people remain two. Keep both denominators. In a real study, define whether a person must have both visits, a particular assay, or complete covariates to enter the final analysis. A complete-case multi-omics intersection can select a different population from any individual assay cohort.

QC exclusions can also depend on biology. Explain whether the estimand concerns all eligible people or the retained subset, and what assumptions would connect the two. An audit trail makes selection visible; it does not make selection ignorable. See Missingness, selection, and the observed-data target.

Retain separate reason flags if a specimen fails several criteria. If a table assigns one primary reason so that removed counts add up, state the assignment rule. Otherwise overlapping reasons can be double-counted. Record exclusions at each actual stage, rather than reconstructing the flow from the retained sample alone.

3.1 Counts and weighting answer different questions

Before exclusions, averaging matrix columns weights people by their specimen counts. P1 contributes two specimens and P2 one. For feature A:

Code

c(specimen_average = mean(assay["feature_A", ]),
  person_average = mean(tapply(assay["feature_A", ], aligned$person, mean)),
  retained_specimen_average = mean(analysis_assay["feature_A", ]))

         specimen_average            person_average retained_specimen_average 
                 4.333333                  5.250000                  5.000000

The acquired specimen average is 13/3\approx4.33. Averaging within each person first gives (2.5+8)/2=5.25. After QC the retained values are 2 and 8, giving 5. These summaries answer different questions. Define whether the analysis averages people, visits, specimens, or scheduled-time contrasts. QC can change both who remains and the weights among people who remain. An unchanged person count does not imply an unchanged target or weighting.

4 Use file checks alongside semantic checks

A checksum helps determine whether a stored file changed. It does not tell you that the matrix uses the right units, that labels are correct, or that the scientific target is identified. The example uses tools::md5sum only for local change detection, not as a security guarantee.

Code

record_dir <- tempfile("provenance-"); dir.create(record_dir)
input_files <- file.path(record_dir, c("synthetic-assay.csv", "synthetic-metadata.csv"))
write.csv(assay, input_files[1])
write.csv(metadata, input_files[2], row.names = FALSE)
manifest <- data.frame(file = basename(input_files),
  role = c("acquired assay", "specimen metadata"),
  source = "synthetic teaching example", version = "1",
  checksum = unname(tools::md5sum(input_files)))
manifest

                    file              role                     source version
1    synthetic-assay.csv    acquired assay synthetic teaching example       1
2 synthetic-metadata.csv specimen metadata synthetic teaching example       1
                          checksum
1 0404810db3b07a81d095a3b5863a1b22
2 080659f5cb9e674dda9ba75467718dc7

Code

stopifnot(identical(unname(tools::md5sum(input_files)), manifest$checksum))

Semantic checks should encode the intended data contract: unique keys at the stated unit; expected feature and specimen sets; allowed missingness; measurement scales; valid visit order; group/batch cross-tabulations; and appropriate ranges. Choose checks that would expose plausible mistakes. A count matrix might require nonnegative integer entries, while normalized log expression generally should not be tested as counts.

For a derived assay, also record the parent objects, code revision, filtering settings, and output identity. Its checksum cannot explain why S2 disappeared; the decision log can. A decision log without exact parent versions cannot establish which values were filtered. Preserve acquired versions when releases change and generate a new derived version rather than relabeling the old one.

5 Deliver an analysis-ready object with context

Retain the assay, aligned metadata, feature annotations, processing settings, decision log, and manifest together or with explicit cross-references. State the independent sampling unit and any clustering identifiers. Document which objects contain outcomes only on observed units and which covariates are available for the whole eligible population.

A useful handoff lets the next analyst answer: Which population do these columns represent? Which scale is measured? How did this feature label arise? Why is an eligible participant absent? Which code produced this version? If these cannot be answered, the object is numerically convenient but not yet ready for a defensible analysis.

The teaching object can keep its aligned components together:

Code

analysis_object <- list(
  assay = analysis_assay, specimens = analysis_metadata,
  decisions = decision_log, input_manifest = manifest,
  description = list(scale = "arbitrary synthetic measurement units",
    matrix_column = "specimen", independent_unit = "person",
    inclusion = "usable specimens; both original people retain a specimen"))
stopifnot(identical(colnames(analysis_object$assay),
                    analysis_object$specimens$specimen))
names(analysis_object)

[1] "assay"          "specimens"      "decisions"      "input_manifest"
[5] "description"

That packaging helps maintain alignment, but does not choose the weighting rule. Record that rule with the analysis configuration. A matrix-aware container such as SummarizedExperiment coordinates subsetting of assays and metadata; it still depends on correctly constructed identifiers and annotations at entry.

6 Exercises

Try each checkpoint before opening its hint or solution.

Checkpoint · PROV-01 — Audit a join and its population consequences

Task type: Data checking and interpretation.

A study acquires two visits per person. A metadata join duplicates some specimens, then an inner join to a second assay drops every specimen from 12 people. Specify the keys and cardinality checks, the two unmatched-record reports, an attrition table, and the decisions needed before calling the result analysis-ready. Explain whether checksums alone would reveal either problem.

Using the three-specimen example, also calculate the specimen-weighted average and the equally weighted person average. Explain why retaining the same number of people after QC does not by itself preserve the weights or target.

Ready to continue when: you can reconstruct the joins and exclusions and distinguish the retained-data target from the original eligible-population target.

Transfer: How would a many-to-one feature-annotation mapping alter your checks compared with a one-to-one specimen join?

Hint

Count records at specimen and person levels before and after each transformation.

Solution outline — open after trying

Require unique specimen keys where a one-to-one specimen join is intended; retain person/visit links separately. Report records present only in the first and only in the second assay. Stop on unexpected duplication, and log any intentional one-to-many relationship. Tabulate specimens and distinct people at acquisition, each assay, their intersection, and final analysis, with reasons. Decide whether the target is the intersection population or the full eligible population and state the selection assumptions for the latter. Checksums establish file identity/change, not valid join semantics or ignorability. In the toy data the specimen average is 13/3, the equally weighted person average is 5.25, and the retained average is 5. Changing specimen contributions can change weighting without removing a person.

Additional practice (optional)

Extend the example with a person who loses all specimens and show how both attrition columns change.
Write a data contract for a methylation proportion or metabolite-intensity matrix; explain why a raw-count assertion would be inappropriate.

7 Reading

Morgan et al., SummarizedExperiment vignette: coordinating assays and metadata.
Wilson et al., Good enough practices in scientific computing.

--- title: "Data provenance and an analysis-ready dataset" subtitle: "Identifiers, transformations, exclusions, and a visible target population" guide-status: "Available" --- **Before reading:** Read [Understanding an omics dataset](../data-and-measurement/understanding-omics-data.qmd); the examples use base R. ## An analysis matrix needs a history A numerical matrix alone does not tell you where measurements came from, which people they represent, or why some samples disappeared. Provenance connects an acquired object to its analysis version and makes those choices inspectable. For each input, record its source and accession or institutional identifier, release/version, acquisition date, access conditions, file identity/checksum, assay and units, reference genome or feature-annotation version where relevant, and the code that reads it. Use actual identifiers from the source; a synthetic example should explicitly say its identifiers are invented. For each derived object, record its parent inputs, transformation code and version, settings, inclusion/exclusion decisions, and output identity. Preserve the distinction between a person, specimen, aliquot, assay run, and matrix column. Their relations may be one-to-many. The [Understanding an omics dataset](../data-and-measurement/understanding-omics-data.qmd) page explains these objects. Provenance has two parts. **File identity** tells us which objects were used. **Data meaning** tells us what their rows, columns, labels, and values represent, and why those objects answer the question. An unchanged file can have misinterpreted group labels. A correctly labeled assay can belong to an earlier preprocessing release. The research record needs both kinds of information. ## Align by identifiers and check cardinality The following toy data have **invented specimen and person identifiers**, and arbitrary synthetic measurement units. Matrix columns are specimens; two specimens belong to the same person. The contract is one metadata record per matrix column, keyed by specimen. Person ID is a grouping variable, not a unique specimen key. ```{r} assay <- matrix(c(2, 4, 3, 5, 8, 9), nrow = 2, dimnames = list(c("feature_A", "feature_B"), c("S1", "S2", "S3"))) metadata <- data.frame(specimen = c("S3", "S1", "S2"), person = c("P2", "P1", "P1"), usable = c(TRUE, TRUE, FALSE), reason = c("retained", "retained", "prespecified assay QC")) AlignSpecimens <- function(assay, metadata) { stopifnot(is.matrix(assay), is.numeric(assay), all(c("specimen", "person", "usable", "reason") %in% names(metadata)), !is.null(colnames(assay)), !anyNA(colnames(assay)), all(nzchar(trimws(colnames(assay)))), !anyDuplicated(colnames(assay)), !anyNA(metadata$specimen), !anyDuplicated(metadata$specimen), all(nzchar(trimws(metadata$specimen))), !anyNA(metadata$person), all(nzchar(trimws(metadata$person))), is.logical(metadata$usable), !anyNA(metadata$usable), setequal(colnames(assay), metadata$specimen)) aligned <- metadata[match(colnames(assay), metadata$specimen), , drop = FALSE] stopifnot(identical(colnames(assay), aligned$specimen)) aligned } aligned <- AlignSpecimens(assay, metadata) aligned # An intentional error is caught before a join can multiply observations. bad_metadata <- rbind(metadata, metadata[1, ]) tryCatch(AlignSpecimens(assay, bad_metadata), error = function(e) "Rejected: duplicate specimen key") ``` `setequal` deliberately requires the same specimen set in this teaching contract. If unmatched records are expected, use an explicit policy and output both unmatched sets before continuing. An inner join that silently drops unmatched records conceals an exclusion decision. A duplicated feature-to-symbol map can similarly multiply rows: retain stable feature identifiers and define how one-to-many annotations will be handled rather than silently summing or deduplicating them. ### Work through the matching operation The assay order is S1, S2, S3; the metadata order is S3, S1, S2. Matching produces indices 2, 3, 1 and reorders the metadata to the assay. Comparing only row counts would miss a positional assignment error: ```{r} data.frame(matrix_specimen = colnames(assay), positional_metadata_key = metadata$specimen, positional_person = metadata$person, matched_person = aligned$person) ``` There are no missing values in the wrong positional assignment. The mistake is that S1 is attributed to P2 and S3 to P1. A numerical fit could succeed while relating measurements to the wrong people. The final `identical()` assertion checks ordering after key matching. ### State the relationship before joining For a key k with $a_k$ rows in one table and $b_k$ in the other, an ordinary match-based join can produce $a_kb_k$ rows. Duplicated specimen annotations can therefore duplicate measurements. Joining two visit tables by person alone can pair every visit with every other visit for that person. Those extra rows are not additional independent observations. Use specimen IDs for specimen metadata, run IDs for assay runs, or a person–visit key only when the protocol guarantees one record per combination. Multiple aliquots at a visit need another identifier. Represent valid one-to-many relationships explicitly, with an aggregation or analysis rule; keeping the first match silently chooses a record. For a planned assay intersection, report `setdiff(first_keys, second_keys)` and `setdiff(second_keys, first_keys)` before applying the inclusion rule. If the assays use different identifiers, first establish a documented crosswalk at the intended person, visit, or specimen level. Equal-looking strings alone do not establish that relationship. ## Retain exclusions and denominators ```{r} decision_log <- aligned[c("specimen", "person", "usable", "reason")] keep <- aligned$usable analysis_assay <- assay[, keep, drop = FALSE] analysis_metadata <- aligned[keep, , drop = FALSE] stopifnot(identical(colnames(analysis_assay), analysis_metadata$specimen)) attrition <- data.frame( stage = c("Acquired", "Retained after QC"), specimens = c(ncol(assay), ncol(analysis_assay)), people = c(length(unique(aligned$person)), length(unique(analysis_metadata$person)))) attrition decision_log ``` Removing a specimen does not always remove a person; losing every specimen from a person does. Here specimens fall from three to two while people remain two. Keep both denominators. In a real study, define whether a person must have both visits, a particular assay, or complete covariates to enter the final analysis. A complete-case multi-omics intersection can select a different population from any individual assay cohort. QC exclusions can also depend on biology. Explain whether the estimand concerns all eligible people or the retained subset, and what assumptions would connect the two. An audit trail makes selection visible; it does not make selection ignorable. See [Missingness, selection, and the observed-data target](../questions-and-design/missingness-and-selection.qmd). Retain separate reason flags if a specimen fails several criteria. If a table assigns one primary reason so that removed counts add up, state the assignment rule. Otherwise overlapping reasons can be double-counted. Record exclusions at each actual stage, rather than reconstructing the flow from the retained sample alone. ### Counts and weighting answer different questions Before exclusions, averaging matrix columns weights people by their specimen counts. P1 contributes two specimens and P2 one. For feature A: ```{r} c(specimen_average = mean(assay["feature_A", ]), person_average = mean(tapply(assay["feature_A", ], aligned$person, mean)), retained_specimen_average = mean(analysis_assay["feature_A", ])) ``` The acquired specimen average is $13/3\approx4.33$. Averaging within each person first gives $(2.5+8)/2=5.25$. After QC the retained values are 2 and 8, giving 5. These summaries answer different questions. Define whether the analysis averages people, visits, specimens, or scheduled-time contrasts. QC can change both who remains and the weights among people who remain. An unchanged person count does not imply an unchanged target or weighting. ## Use file checks alongside semantic checks A checksum helps determine whether a stored file changed. It does not tell you that the matrix uses the right units, that labels are correct, or that the scientific target is identified. The example uses `tools::md5sum` only for local change detection, not as a security guarantee. ```{r} record_dir <- tempfile("provenance-"); dir.create(record_dir) input_files <- file.path(record_dir, c("synthetic-assay.csv", "synthetic-metadata.csv")) write.csv(assay, input_files[1]) write.csv(metadata, input_files[2], row.names = FALSE) manifest <- data.frame(file = basename(input_files), role = c("acquired assay", "specimen metadata"), source = "synthetic teaching example", version = "1", checksum = unname(tools::md5sum(input_files))) manifest stopifnot(identical(unname(tools::md5sum(input_files)), manifest$checksum)) ``` Semantic checks should encode the intended data contract: unique keys at the stated unit; expected feature and specimen sets; allowed missingness; measurement scales; valid visit order; group/batch cross-tabulations; and appropriate ranges. Choose checks that would expose plausible mistakes. A count matrix might require nonnegative integer entries, while normalized log expression generally should not be tested as counts. For a derived assay, also record the parent objects, code revision, filtering settings, and output identity. Its checksum cannot explain why S2 disappeared; the decision log can. A decision log without exact parent versions cannot establish which values were filtered. Preserve acquired versions when releases change and generate a new derived version rather than relabeling the old one. ## Deliver an analysis-ready object with context Retain the assay, aligned metadata, feature annotations, processing settings, decision log, and manifest together or with explicit cross-references. State the independent sampling unit and any clustering identifiers. Document which objects contain outcomes only on observed units and which covariates are available for the whole eligible population. A useful handoff lets the next analyst answer: Which population do these columns represent? Which scale is measured? How did this feature label arise? Why is an eligible participant absent? Which code produced this version? If these cannot be answered, the object is numerically convenient but not yet ready for a defensible analysis. The teaching object can keep its aligned components together: ```{r} analysis_object <- list( assay = analysis_assay, specimens = analysis_metadata, decisions = decision_log, input_manifest = manifest, description = list(scale = "arbitrary synthetic measurement units", matrix_column = "specimen", independent_unit = "person", inclusion = "usable specimens; both original people retain a specimen")) stopifnot(identical(colnames(analysis_object$assay), analysis_object$specimens$specimen)) names(analysis_object) ``` That packaging helps maintain alignment, but does not choose the weighting rule. Record that rule with the analysis configuration. A matrix-aware container such as `SummarizedExperiment` coordinates subsetting of assays and metadata; it still depends on correctly constructed identifiers and annotations at entry. ## Exercises Try each checkpoint before opening its hint or solution. ::: {#checkpoint-prov-01 .checkpoint} ### Checkpoint · PROV-01 — Audit a join and its population consequences {.unnumbered} **Task type:** Data checking and interpretation. A study acquires two visits per person. A metadata join duplicates some specimens, then an inner join to a second assay drops every specimen from 12 people. Specify the keys and cardinality checks, the two unmatched-record reports, an attrition table, and the decisions needed before calling the result analysis-ready. Explain whether checksums alone would reveal either problem. Using the three-specimen example, also calculate the specimen-weighted average and the equally weighted person average. Explain why retaining the same number of people after QC does not by itself preserve the weights or target. **Ready to continue when:** you can reconstruct the joins and exclusions and distinguish the retained-data target from the original eligible-population target. **Transfer:** How would a many-to-one feature-annotation mapping alter your checks compared with a one-to-one specimen join? <details class="checkpoint-help"> <summary>Hint</summary> Count records at specimen and person levels before and after each transformation. </details> <details class="checkpoint-help"> <summary>Solution outline — open after trying</summary> Require unique specimen keys where a one-to-one specimen join is intended; retain person/visit links separately. Report records present only in the first and only in the second assay. Stop on unexpected duplication, and log any intentional one-to-many relationship. Tabulate specimens and distinct people at acquisition, each assay, their intersection, and final analysis, with reasons. Decide whether the target is the intersection population or the full eligible population and state the selection assumptions for the latter. Checksums establish file identity/change, not valid join semantics or ignorability. In the toy data the specimen average is 13/3, the equally weighted person average is 5.25, and the retained average is 5. Changing specimen contributions can change weighting without removing a person. </details> ::: ### Additional practice (optional) {.unnumbered} 1. Extend the example with a person who loses all specimens and show how both attrition columns change. 2. Write a data contract for a methylation proportion or metabolite-intensity matrix; explain why a raw-count assertion would be inappropriate. ## Reading - Morgan et al., [SummarizedExperiment vignette](https://www.bioconductor.org/packages/release/bioc/vignettes/SummarizedExperiment/inst/doc/SummarizedExperiment.html): coordinating assays and metadata. - Wilson et al., [Good enough practices in scientific computing](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005510).