Heterogeneity of primary studies in oncology systematic reviews


Background: Systematic reviews compare data across multiple studies to answer a research problem. One important test to perform in systematic reviews is statistical heterogeneity. Statistical heterogeneity measures variance between studies and significant heterogeneity yields a meta-analysis unusable. One new method for presenting heterogeneity graphically to compare study design features and population characteristics is evidence-based mapping (Althuis 2014). The benefit of graphically mapping out heterogeneity is that it allows researchers to properly deal with heterogeneity during meta-analysis.

Methods: A PubMed search of six oncology journals was conducted looking for systematic reviews and meta-analyses. This search strategy was adapted from a previously established search method (Montori 2005). was used to screen manuscripts based on title and abstract. Two coders then independently evaluated the manuscripts for 10 different elements. The evidence-based mapping method heterogeneity was applied to a manuscript chosen by the author. Stata 13.1 was used analyze the coders data and was then searched for trends in heterogeneity use.

Results: The initial PubMed search yielded 337 manuscripts from 6 different journals. Post-screen/coding exclusions left 182 manuscripts across 4 journals for analysis. Of these papers, 50% used varying combinations of heterogeneity tests and of those only 8% have too much heterogeneity to complete the meta-analysis. Of the studies which measured heterogeneity, 25% utilized a random-effects model, 4% utilized a fixed-effects model, and 21% used both. The results from the evidence-based mapping show variance in average patient age, reporting of length of sedation use, study facility, and sedation mode.

Conclusion: It is the impression of this study that the use of quantitative and qualitative heterogeneity measurement tools are underutilized in the four oncology journals evaluated. These assessments should be applied in meta-analyses to reduce the risk of spurious findings being integrated into medical practice. This tool will help determine whether or not a meta-analysis can be performed prior to investing time in said meta-analysis. This is preferable to performing a quantitative measurement of heterogeneity after the fact to indicate whether or not the study analysis is trustworthy.

Keywords: Heterogeneity, Meta-analysis, Systematic Review, Evidence-based mapping; Oncology, Palliative care


Systematic reviews bring together all related empirical evidence based on pre-determined eligibility criteria to answer a research question (Cochrane Handbook of Systematic Reviews of Interventions). This methodology is designed to minimize bias by using an explicit, reproducible approach involving a systematic and comprehensive literature search, an assessment of validity of primary studies, and a systematic presentation and synthesis of findings. Oftentimes, systematic reviews also contain one or more meta-analyses that make use of statistical procedures to summarize the results of primary studies. For example, a systematic review from the Cochrane Library examined the effects of administering antibiotics prior to inserting long-term central venous catheters (CVC), flushing or locking the long-term CVCs with a combined antibiotic and heparin solution, or both, for prevention of Gram positive catheter-associated infections in adults and children receiving cancer treatments (van 2013). Eleven randomized controlled trials involving 828 patients were subjected to meta-analysis, and results indicated no benefit in administering antibiotics before inserting long-term CVCs, though flushing or locking long-term CVCs in combination with antibiotic and heparin solution appeared to reduce Gram positive catheter-related sepsis in patients at risk for neutropenia through chemotherapy or disease.

In such reviews, it is evident that when multiple studies are combined for data synthesis, there will be differences, such as location of testing, drug doses, dosing schedules, follow-up, or ethnicity of participants, to name a few. These differences - classified as clinical diversity (varying participants, interventions, and outcomes studied) and methodological diversity (varying study design and risk of bias) - are known as heterogeneity and are present, to some extent, in all systematic reviews (Gagnier 2013). Addressing this issue is one of the most challenging aspects of the systematic review process (Higgins 2002).

To examine the magnitude of these forms of heterogeneity among a set of primary studies, researchers may evaluate statistical heterogeneity, or the extent to which heterogeneity of study effects is present above that which would be expected by chance alone. If heterogeneity is present, researchers must decide if the primary studies for consideration are too diverse to synthesize or whether follow up analyses, such as meta-regression or sub-group analysis, may be used to explore the effects of these differences on study outcomes.

While much advise has been offered on evaluating heterogeneity, little is known about the ways that systematic reviewers actually address heterogeneity. In fact, what little we know is representative of Cochrane review groups who follow strict guidelines during the review process. Questions, however, remain regarding the practices of systematic reviewers outside of Cochrane review groups, such as researchers in clinical specialties like oncology. We, therefore, examined heterogeneity assessment and analysis practices among meta-analyses in oncology research. We focused on particular methods used to detect heterogeneity and examined the ways in which heterogeneity results informed decision making. We also propose the use of evidence mapping as a tool for evaluating clinical and methodological heterogeneity both for researchers to make informed decisions regarding the distinguishing clinical or methodological features of primary studies and for readers to form conclusions regarding the nature of heterogeneity of studies included in a meta-analysis. We will first briefly acquaint the reader with the process of evidence mapping based on the work of Althuis et al. (Althuis 2014).