Discussion
We sought to address one of the most important clinical question in
contemporary clinical research: how large of an effect size is large
enough to allow approval of treatments without further testing in RCTs?
To address this question, we illustrate the application of the signal
detection and heuristic theory of decision-making to interpret the
effect size that regulatory agencies may use to approve treatment
without further testing in RCTs. We propose that two heuristics can
explain the agencies’ decision-making: first, signal isrecognized as large, and second, the magnitude of that
signal is assessed via the Weber-Fechner law. Our findings suggest that
when the difference between novel treatments and historical controls is
at least one logarithm of magnitude, the veracity of testing in non-RCTs
seems to be established.38 These findings based on the
convergence of the Weber-Fechner and recognition heuristics agree with
the heuristic rule suggested by Glasziou et al9:
further RCTs may not be necessary when RR of experimental treatment is ≥
10 in comparison with control.
Theories of decision making are divided into those that deal with
‘large-’ or ‘small’- world phenomena.39 In a small
world, time constraint is not an issue, decision-makers have access to
the best available evidence – ideally from well-designed and powered
RCTs – regarding all competing management alternatives, consequences
and probabilities. Signal detection theory is a prototype of the small
world theories and is a normative theory that provides a framework for
how people “should” or “ought to” make their decisions and
draw inferences. (This is also known as the theory of“ought”). 40-42
In contrast, in a ‘large’ or real-world context, decision-makers are
typically under time constraints, with limited knowledge about the
complete set of alternatives, consequences, and probabilities. This
means that making rational inferences requires adaptation to
environment/context (adaptive or ecological rationality ) and
respecting epistemological, environmental and computationalconstraints of human brains.11,40 Because
finding the optimum solution to a given problem can be resource and
computationally intensive, adaptive behaviors typically rely onsatisficing (finding a good enough solution), rather than
striving to find a “perfect” solution (via optimising/maximizing
procedures). 40,43 The principle behind satisficing is
that there must exist a point (threshold) at which obtaining more
information or engaging in more computation becomes overly costly and
thereby detrimental. Identifying this threshold, at which a
decision-maker should stop searching for more information, is often
accomplished by using “heuristics”11 for
implementation of bounded rationality.44 The heuristic
theory of decision-making is a descriptive theory, which helps explain
how people actually make their decisions (also known as theory of“is”). 40-42 Surprisingly, simple
heuristic-based inferential and decision-making strategies are often
more accurate than more complex statistical models (the phenomenon known
as “less-is-more”).11
Recently, we12 and others13,36integrated small-world SDT with heuristics decision-making to show how
connecting apparently unrelated theories in different disciplines likely
leads to discovery of new relationships. In this paper, we extend the
theory integration program45 to the application of the
Weber-Fechner law and recognition heuristics in order to provide
descriptive explanations of the decisions made by the FDA and EMA to
approve new treatments based on non-RCT studies without further testing
in RCTs. By integrating heuristic reasoning with SDT, it is sometimes
possible to derive “ought” rule from “is”observations.40-42,46 That is, if had observed high
discriminability of Weber-Fechner, or recognition heuristic, we may then
argue that these empirically derived observations may, in turn, be
normatively used by drug developers and practitioners alike: one log
effect size magnitude could serve as a benchmark to decide if further
testing in RCTs should be pursued, or as a guide in interpretation of
the results reported in non-RCT studies.
Throughout this study, we found some support for “one logarithm of
treatment magnitude” rule, but we should acknowledge the study
limitations. First, the strength of evidence supporting high accuracy
related to the decision to pursue further RCTs based on the one log
effect size is moderate. Second, as discussed in the papers leading to
this one15,16, in addition to effect size, other
factors play a role in the decision to grant licensing approval; these
seem to include issues such as approval for rare diseases where few
effective treatment exists, risk tolerance in the attempt to strike a
balance between failing to approve effective drugs and approving
ineffective or dangerous drugs 47, political pressures
like conflict of interest, feasibility of undertaking of RCTs, small
sample sizes and bias in the assessment of control event rates, as
outlined above.
Nevertheless, it is clear that the larger the effect size, the higher
the probability that treatments will be approved without further testing
in RCTs. 15,16 When integration of multiple factors
are difficult people resort to heuristics, which are often defining
characteristics of psychology of decision-making. 10However, one of the reasons that we could not provide more definitive
evidence related to the specific effect size above which drugs should be
approved based solely on non-RCT data is that our database, even most
comprehensive to date, is relatively small (n=134). Mere exposure to
“dramatic effects” does not account for the mechanism of recognition
heuristic.35 Rather, repeated experience and
internalization of the rule is required for the ease of retrieval to
rely on recognition for making inferences from memory about the
phenomenon of interest.35 We suspect that as
databases- and experience- with approval of drugs based on non-RCTs
increase, regulators and practicing physicians will encounter many more
instances that will help improve the quality of recognition memory and
the use of the methods described here will be more applicable.