Methods
Key terms and definitions. In the past decade, researchers have formalized many concepts related to external validity in the context of single-site studies.20-23 This has resulted in clearer definitions for “study population”, “target population”, “generalizability”, and “transportability.” The study population is the population supplying outcome, exposure, and covariate data for the analysis. The target population is a specific population in which researchers hope to estimate the effect of exposure on outcome, which may or may not be the study population. Generalizability is the extent to which a study population can be used to estimate a given effect in a specific target population it was sampled from (see Figure 1A ). Transportability is the extent to which a study population can be used to estimate a given effect in a specific target population it wasnot sampled from ( see Figure 1B ). This formalization has also resulted in the conceptualization of “target validity”, defined as how well a given study estimates a treatment effect in a specific target population. Target validity takes into account internal validity, for the study population, and external validity, for the specific target population. This formalization has also helped clarify understanding of the connection between external validity and effect measure modifiers (i.e. variables that are associated with different treatment effect estimates on the scale of interest)24 as well as ways to use analytic methods to “balance” effect measure modifiers between study and target populations.25, 26
Distributed networks and external validity. The structure of distributed data networks (presented in Figure 2 ) makes these terms even more important. Unlike most epidemiologic research, there are multiple study populations – in fact, there are at least as many study populations as there are nodes in the distributed data network. Because of random error or differing degrees of confounding, effect estimates at the various nodes may not agree with one another. Of course, different distributions of effect measure modifiers at each node can also lead to differing effect estimates. Each node may represent a separate geographical region, a different insurance provider, or an entirely different type of data source (e.g. commercial claims vs electronic health records). It is entirely possible that an estimate in one node may not be externally valid, or may have poor target validity, for the population represented by another node. This is not a major concern if researchers report only node-level estimates and make few comparisons across nodes. The estimate of the treatment effect within each node that makes the fewest assumptions will generally be the estimate yielded by implementing the research study within that node.
If researchers want to combine results from the various nodes into one “network-wide” estimate and 95% confidence interval, which is usually the case, or assess whether different nodes exhibited different amounts of confounding, however, external validity becomes an important consideration when planning research. The target population for the combined network-level analysis affects the analytic strategies that can be used, both at individual sites and when combining results. Whether the findings of a given node can generalize to the whole network becomes an important consideration, as does the extent to which node-level estimates can be transported to one another or to external target populations. Finally, the network needs to consider whether their combined estimate has target validity for the population of interest.
Specific networks and external validity. We assessed how three major distributed data networks that specifically focus on creating network-wide estimates (Sentinel, CNODES, and PCORnet) address external validity in routine aggregate data projects. Researchers from each network provided background information on its purpose and general structure and answered questions related to external validity, target populations of interest, generalizability, transportability, and target validity.