The regulators have accepted non-randomized studies as the basis for licensing of new treatments
A notion that that large treatment effects can sometimes obviate the need for data from RCTs has also been accepted by the FDA and the EMA. The agencies established special pathways-the EMA’s PRIME (Priority Medicines) and Adaptive Pathways programs and the FDA’s “Breakthrough Therapy Designation” programs - designed to support approval of drugs that demonstrate substantial improvement over the existing therapies, which may not require further testing in RCTs. 14 We recently analyzed drug approvals by the EMA and FDA based on non-randomized drug comparisons and confirmed that larger effect sizes in non-randomized studies are associated with higher rates of licensing approval.15,16 Overall, we found that between 7 to 10% of drug approvals are based on non-RCT comparisons and that between 2% and 4% of these approvals displayed “dramatic” effects15,16, defined as relative risks (RR)>28, RR ≥5 17, or RR≥10.9
Although the probability of approval increased with larger effect sizes, we could not identify a specific threshold of effect size above which the regulatory agencies would always grant licensing approval.15,16 Similarly, to date, the agencies have not formally quantified the effect size necessary to preclude the need for additional testing in RCTs before granting licensing approval. Thus, the treatment effect size that is dramatic enough to convince the FDA and EMA that the treatment differences observed are real and free of bias and random error and hence can be approved without RCTs remain unclear. The proposed definitions of dramatic effects- (RR)>28, RR ≥5 17, or RR≥109- have not been theoretically or empirically justified; instead, they represent a decision rule based onheuristics - powerful, rule-of-thumb, decision-making strategies that are often more accurate than complex statistical models.10,11,18
The heuristic theory of decision-making has been linked to SDT12,13 and the threshold model of decision-making12 to show how seemingly unrelated theories in different disciplines can lead to discovery of new relationships and explanations. Here, we argue that when the direct, empirical evaluation of a treatment effect is not possible, an alternative approach is to employ SDT to define the circumstances under which the “signal” (e.g., treatment effect) is credible and reproducibly detected to allow approval of new drugs without further testing in RCTs.19,20 21
We rely on the generalizability of SDT to account for two heuristics that we believe influence the FDA and the EMA’s approval of new treatments: 1) the likelihood of approval without testing in RCTs will increase if the difference in treatment effect between the experimental and control arm is at least one logarithm of magnitude (i.e., reflecting heuristic based on the Weber-Fechner law ) 2021 22; 2) the specific threshold of effect size above which the agencies will not require further RCTs will reflect heuristic known as recognition heuristic .23 We focus on the effect size heuristics, but also discuss the heuristic related to the use of p-values. Recently, there has been an increasing attempt to modify a century-old inferential rule-of-thumb to reject the null hypothesis at p≤0.0524; some authors vehemently oppose a hard cut-off for p-values, 25 while others propose a new heuristic rule of p≤0.005 24 as an acceptable evidentiary standard against a null hypothesis.