Benjamin Djulbegovic

and 13 more

Rationale, aims and objectives 39 Evidence-based medicine (EBM) holds that estimates of effects of health interventions based on 40 high-certainty evidence (CoE) are expected to change less frequently than the effects generated 41 in low CoE studies. However, this foundational principle of EBM has never been empirically 42 tested. 43 Methods 44 We reviewed all systematic reviews and meta-analyses in Cochrane Database of Systematic 45 Reviews from January 2016 through May 2021 (n=3,323). We identified 414(207x2) and 384 46 (192x2) pairs of original and updated Cochrane reviews that assessed CoE and pooled 47 treatment effect estimates. We appraised CoE using the Grading of Recommendations 48 Assessment, Development and Evaluation (GRADE) method. We assessed the difference in 49 effect sizes between the original versus updated reviews as a function of change in CoE, which 50 we report as a ratio of odds ratio (ROR). We compared ROR generated in the studies that 51 changed CoE from very low/low (VL/L) to moderate/high (M/H) vs. MH/H VL/L. We also 52 assessed the heterogeneity and inconsistency (using the tau and I2 statistic), the change in 53 precision of effect estimates (by calculating the ratio of standard errors) (seR), and the absolute 54 deviation in estimates of treatment effects (aROR). 55 Results 56 57 We found that CoE originally appraised as VL/L had 2.1 (95%CI: 1.19 to 4.12; p=0.0091) times 58 higher odds to be changed in the future studies than M/H CoE. However, the effect size was not 59 different when CoE changed from VL/L M/H vs. M/H VL/L [ROR=1.02 (95%CI: 0.74 to 1.39) 60 vs. 1.02 (95%CI: 0.44 to 2.37); p=1 for the between subgroup differences]. aROR was similar 61 between the subgroups [median (IQR):1.12 (1.07 to 1.57) vs 1.21 (1.12 to 2.43)]. We observed 62 large inconsistency (I 2=99%) and imprecision in treatment effects (seR=1.09). 63 Conclusions 64 We provide the first empirical support for a foundational principle of EBM showing that low65 quality evidence changes more often than high CoE. However, the effect size was not different 66 between studies with low vs high CoE. The finding that the effect size did not differ between low 67 and high CoE indicate urgent need to refine current EBM critical appraisal methods
Rationale, aims and objectives New therapies are increasingly approved by regulatory agencies such as the Food and Drug Administration (FDA) and the European Medicines Agency (EMA) based on testing in non-randomized clinical trials. These treatments have typically displayed “dramatic effects” (i.e., effects that are considered large enough to obviate the combined effects of bias and random error). The agencies, however, have not identified how large these effects should be to avoid the need for further testing in randomized controlled trials (RCTs). We investigated the effect size that would circumvent the need for further RCTs testing by the regulatory agencies. We hypothesized that the approval of therapeutic interventions by regulators is based on heuristic decision-making whose accuracy can be best characterized by the application of signal detection theory (SDT). Methods We merged the EMA and FDA database of approvals based on non-RCT comparisons. We excluded duplicate entries between the two databases. We included a total of 134 approvals of drugs and devices based on non-RCTs. We integrated Weber-Fechner law of psychophysics and recognition heuristics within SDT to provide descriptive explanations of the decisions made by the FDA and EMA to approve new treatments based on non-randomized studies without requiring further testing in RCTs. Results Our findings suggest that when the difference between novel treatments and the historical control is at least one logarithm (base 10) of magnitude, the veracity of testing in non-RCTs seems to be established. Conclusion Drug developers and practitioners alike can use the change in one logarithm of effect size as a benchmark to decide if further testing in RCTs should be pursued, or as a guide to interpreting the results reported in non-randomized studies. However, further research would be useful to better characterize the threshold of effect size above which testing in RCTs is not needed.
Rationale Assessing the performance of diagnostic tests requires evaluation of the amount of diagnostic uncertainty the test reduces (i.e. 0% - useless test, 100% - perfect test). Statistical measures currently dominating the evidence-based medicine (EBM) field and particularly meta-analysis (e.g. sensitivity and specificity), cannot explicitly measure this uncertainty reduction. Mutual information (MI), an information theory statistic, is a more appropriate metric for evaluating diagnostic tests as it explicitly quantifies uncertainty and, therefore, facilitates natural interpretation of a test’s value. In this paper, we propose the use of MI as a single measure to express diagnostic test performance and demonstrate how it can be used in meta-analysis of diagnostic test studies. Methods We use two cases from the literature to demonstrate the applicability of MI meta-analysis in assessing diagnostic performance. These cases are: 1) Meta-analysis of studies evaluating ultrasonography (US) to detect endometrial cancer and 2) meta-analysis of studies evaluating magnetic resonance angiography to detect arterial stenosis. Results Results produced by the MI meta-analyses are comparable to the results of meta-analyses based on traditionally used statistical measures. However, the results of MI are easier to understand as it relates directly to the extent of uncertainty a diagnostic test can reduce. For example, a US test diagnosing endometrial cancer is 40% specific and 94% sensitive. The combination of these values is difficult to interpret and may lead to inappropriate assessment (e.g. one could favour the test due to its high sensitivity, ignoring its low specificity). In terms of MI however, the test reduces diagnostic uncertainty by 10%, which is marginal and thus the test is clearly not very useful. Conclusions We have demonstrated the suitability of MI in assessing the performance of diagnostic tests, which can facilitate easier interpretation of the true utility of diagnostic tests.
Rationale, aims and objectives In the US, the reluctance of the federal government to impose a national stay-at-home policy in wake of COVID19 pandemic has left the decision of how to achieve social distancing to individual state governors. We hypothesized that in the absence of formal guidelines, the decision to close a state reflects the classic Weber-Fechner law of psychophysics- the amount by which a stimulus (such as number of cases or deaths) must increase in order to be noticed as a fraction of the intensity of that stimulus. Methods On April 12, 2020 we downloaded data from the New York Times database from all 50 states and the District of Columbia; by that time all but 7 states had issued the stay-at-home orders. We fitted the Weber-Fechner logarithmic function by regressing the log2 of cases and deaths respectively against the daily counts. We also conducted Cox regression analysis to determine if the probability of issuing the stay-at-home order increases proportionally as the number of cases or deaths increases. Results We found that the decision to issue the state-at-home order reflects the Weber-Fechner law. Both the number of infections (p=<0.0001; R2=0.79) and deaths (p<0.0001; R2=0.63) were significantly associated with the decision to issue the stay-at-home orders. The results indicate that for each doubling of infections or deaths, an additional 4 to 6 states will issue stay-at-home orders. Cox regression showed that when the number of deaths reached 256 and the number of infected people were over 16,000 the probability of issuing “stay-at-home” order was close to 100%. We found no difference in decision-making according to the political affiliation; the results remain unchanged on July 16,2020. Conclusions when there are not clearly articulated rules to follow, decision-makers resort to simple heuristics, in this case one consistent with the Weber-Fechner law.