loading page

Quantifying convergence and consistency
  • Nicholas Matiasz,
  • Justin Wood,
  • Alcino Silva
Nicholas Matiasz
University of California Los Angeles
Author Profile
Justin Wood
University of California Los Angeles
Author Profile
Alcino Silva
University of California Los Angeles

Corresponding Author:[email protected]

Author Profile

Abstract

The reproducibility crisis highlights several unresolved issues in science, including the need to develop measures that gauge both the consistency and convergence of data sets. While existing meta-analytic methods quantify the consistency of evidence, they do not quantify its convergence: the extent to which different types of empirical methods have provided evidence to support a hypothesis. To address this gap in meta-analysis, we and colleagues developed a summary metric—the cumulative evidence index (CEI)—which uses Bayesian statistics to quantify the degree of both consistency and convergence of evidence regarding causal hypotheses between two phenomena. Here we outline the CEI’s underlying model, which quantifies the extent to which studies of four types—positive intervention, negative intervention, positive non-intervention, and negative non-intervention—lend credence to any of three types of causal relations: excitatory, inhibitory, or no-connection. Along with p-values and other measures, the CEI can provide a more holistic perspective on a set of evidence by quantitatively expressing epistemic principles that scientists regularly employ qualitatively. The CEI can thus address the reproducibility crisis by formally demonstrating how convergent evidence across multiple study types can yield progress toward scientific consensus, even when an individual type of study fails to yield reproducible results.