Methods
Key terms and definitions. In the past decade, researchers have
formalized many concepts related to external validity in the context of
single-site studies.20-23 This has resulted in clearer
definitions for “study population”, “target population”,
“generalizability”, and “transportability.” The study population is
the population supplying outcome, exposure, and covariate data for the
analysis. The target population is a specific population in which
researchers hope to estimate the effect of exposure on outcome, which
may or may not be the study population. Generalizability is the extent
to which a study population can be used to estimate a given effect in a
specific target population it was sampled from (see Figure 1A ).
Transportability is the extent to which a study population can be used
to estimate a given effect in a specific target population it wasnot sampled from ( see Figure 1B ). This
formalization has also resulted in the conceptualization of “target
validity”, defined as how well a given study estimates a treatment
effect in a specific target population. Target validity takes into
account internal validity, for the study population, and external
validity, for the specific target population. This formalization has
also helped clarify understanding of the connection between external
validity and effect measure modifiers (i.e. variables that are
associated with different treatment effect estimates on the scale of
interest)24 as well as ways to use analytic methods to
“balance” effect measure modifiers between study and target
populations.25, 26
Distributed networks and external validity. The structure of
distributed data networks (presented in Figure 2 ) makes these
terms even more important. Unlike most epidemiologic research, there are
multiple study populations – in fact, there are at least as many study
populations as there are nodes in the distributed data network. Because
of random error or differing degrees of confounding, effect estimates at
the various nodes may not agree with one another. Of course, different
distributions of effect measure modifiers at each node can also lead to
differing effect estimates. Each node may represent a separate
geographical region, a different insurance provider, or an entirely
different type of data source (e.g. commercial claims vs electronic
health records). It is entirely possible that an estimate in one node
may not be externally valid, or may have poor target validity, for the
population represented by another node. This is not a major concern if
researchers report only node-level estimates and make few comparisons
across nodes. The estimate of the treatment effect within each node that
makes the fewest assumptions will generally be the estimate yielded by
implementing the research study within that node.
If researchers want to combine results from the various nodes into one
“network-wide” estimate and 95% confidence interval, which is usually
the case, or assess whether different nodes exhibited different amounts
of confounding, however, external validity becomes an important
consideration when planning research. The target population for the
combined network-level analysis affects the analytic strategies that can
be used, both at individual sites and when combining results. Whether
the findings of a given node can generalize to the whole network becomes
an important consideration, as does the extent to which node-level
estimates can be transported to one another or to external target
populations. Finally, the network needs to consider whether their
combined estimate has target validity for the population of interest.
Specific networks and external validity. We assessed how three
major distributed data networks that specifically focus on creating
network-wide estimates (Sentinel, CNODES, and PCORnet) address external
validity in routine aggregate data projects. Researchers from each
network provided background information on its purpose and general
structure and answered questions related to external validity, target
populations of interest, generalizability, transportability, and target
validity.