Discussion
Although the ANT fMRI task has been widely used in developmental and
clinical neuroscience studies, the test-retest reliability of the three
attention network contrasts has yet to receive sufficient
research attention. Moreover, whether the attention-related intrinsic
brain network components, defined by the resting-state fMRI, could show
acceptable reliability remains elusive. The present study investigates
the test-retest reliability of the attention network test from the
perspective of intrinsic network organization. Our results demonstrate
that the reproducibility of the group activation map of the
three-attention network component was small-to-moderate between sessions
and between healthy and Parkinson’s groups. The test-retest reliability
of alerting, orienting, and executive contrast, defined as the
difference between conditions, is worse than estimates of specific
conditions. In addition, dorsal attention, ventral attention, visual,
and somatomotor networks show acceptable reliability for the congruent
and incongruent conditions.
As the cornerstone of fMRI methodology, reliability has recently
attracted widespread attention and discussion. The reliability estimates
for commonly used fMRI tasks are less satisfactory than researchers
implicitly believe (Bennett & Miller, 2010; Elliott et al. ,
2020). It was estimated that the ICC estimates for most fMRI task
contrasts ranged between 0.33 and 0.66 (Bennett & Miller, 2010). A
recent, more extensive meta-analysis reported that the mean ICC for 90
experiments was 0.397 (Elliott et al. , 2020). Elliott et al.
(2020) also demonstrated that the test-retest reliabilities of activity
in a priori regions of interest were poor, ranging between 0.067 and
0.485. Our results are consistent with those meta-analyses. First, the
spatial overlap of the group activation map was small-to-moderate
between sessions and between healthy and Parkinson’s groups, with dice
coefficients ranging from 0.16 to 0.59. Second, the ICC estimates for
the three attention network contrasts were also poor, with the median
estimate around 0.25. The reliability of fMRI measurements depends on
signal-to-noise ratio and task design. Block design fMRI commonly shows
better reliability than single-trial design. For example, a recent study
reported good-to-excellent overlap (Dice coefficients: 0.54 - 0.82) for
most contrast of a finger-tapping fMRI task, which adopted a typical
block design (Wuthrich et al. , 2023). The ANT fMRI task used a
jittered single-trial design, which might explain the low reliability of
the attention network contrasts.
The three attention network contrasts of the ANT task are commonly used
to quantify the efficiency of attention components in individual
difference or group-difference studies (Madhyastha et al. , 2015;
Neufang et al. , 2015; Boord et al. , 2017; Yang et
al. , 2021). However, as indicated by the present study, the reliability
of the three attention network contrasts questioned the practice of
using them as effective individualized biomarkers. This rationale
applies to fMRI contrast, too (Infantolino et al. , 2018).
In contrast to the low
reliability of the three attention network contrasts, separate task
conditions showed better reliability in the present study. The congruent
condition and incongruent condition comprised a typical example: ICCs in
the dorsal attention, ventral attention, visual, and somatomotor network
ranged between 0.5 and 0.75, with a median of approximately 0.7.
However, the ICC of executive contrast had a median of 0.2 in
attention-related networks. As expected, a severe collinearity between
congruent and incongruent conditions was as high as 0.8 . Our
results on the reliability of the contrast map are consistent with
recent findings. For example, Elliott and colleagues have
discussed the phenomenon in a large meta-analysis (Elliott et
al., 2020). Infantolino et al. (2018) examined the reliability
of a face- and
shape-matching task. Although their study revealed robust amygdala
activation of faces over shapes, the reliability of the activation
difference between faces and shapes was nearly zero. Reward-positive
difference scores, computed between reward and non-reward conditions,
show poor reliability due to the high correlation between the two
conditions (Clayson et al., 2021). Thus, using executive
contrast to quantify the individual difference in executive control is
not a good idea because subtraction has canceled the interested
variance. We suggest using separate condition estimates directly if the
study interest is individual or group differences.
Our results on the reliability of intrinsic networks also support the
direct usage of separate condition estimates. If the three attention
networks of ANT measure attention ability, the reliability estimates of
the intrinsic network should vary according to relevance to attention.
However, there was generally comparable low reliability in the seven
intrinsic networks for the alerting, orienting, and executive contrast.
Our results showed that the reliability of alerting, orienting, and
executive networks did not differ among the seven networks, suggesting
no unique correspondence between attention networks and intrinsic
networks from the reliability perspective. The finding coheres with a
recent study adopting the spatial regression approach (Markett et
al. , 2022). Although attention networks overlapped with multiple
intrinsic networks, their study did not yield unique correspondence
between attention networks and intrinsic networks (Markett et
al. , 2022). However, the reliability of congruent and incongruent
conditions exhibited attention relevance from the perspective of
intrinsic organization. Precisely, the ICCs of the congruent and
incongruent conditions followed the order: dorsal attention, ventral
attention, visual, somatomotor, frontoparietal, default, and limbic
network. Our results on subcortical structures lead to the same
conclusion. The ICC of the condition is generally higher than the ICC of
the ANT network obtained by differencing. Thus, it is reasonable to use
separate condition estimates in individual studies as they were reliable
and theoretically meaningful.
The present study only reports reliability estimates for the original
design developed by Fan et al. (2005) with a healthy adult and
Parkinson’s sample. However, there are several fMRI variants of ANT. To
facilitate examining the interaction among the three attention networks,
Fan and collaborators designed the ANT-R, which manipulated cue-targe
interval and cue validity (Fan et al. , 2009). The fMRI version of
ANT-R has been used to study attention dysfunction in autism (Fanet al. , 2012). Xuan et al. (2016) also used the fMRI version of
ANT-R to elucidate the interactive attentional networks. In addition, a
child-friendly fMRI version has also been widely used (Hao et
al. , 2021). The child-friendly version used fish stimuli instead of
arrow stimuli. In addition, the interval between cue and target was
fixed to be 450ms in the child-friendly fMRI version. The original
design of Fan et al. (2005) used a random interval with a mean of
2800ms, ranging from 300ms to 11800ms. The dataset used by the present
study also followed this design. It should be noted that the cue-target
interval in the original design was the longest. Fan et al. (2012) and
Xuan et al. (2016) randomly chose the cue-target interval from 0, 400,
and 800ms for each trial. Because of attention relapse or inhibition of
return, the long cue-target interval can make the spatial cue useless,
which might explain why the present study’s orientation network achieved
the weakest spatial activation.
In conclusion, the present study manifests imperfect reliability
estimates for the three network contrasts using a sample of healthy
adults and Parkison patients. We advocate that separate condition
estimates be used directly in individual difference studies as they are
more reliable and theoretically meaningful from the perspective of
intrinsic organization. Future research efforts must study the
difference between different versions of ANT fMRI tasks to facilitate
clinical and developmental neuroscience studies on attention
dysfunction.