Discussion
Although the ANT fMRI task has been widely used in developmental and clinical neuroscience studies, the test-retest reliability of the three attention network contrasts has yet to receive sufficient research attention. Moreover, whether the attention-related intrinsic brain network components, defined by the resting-state fMRI, could show acceptable reliability remains elusive. The present study investigates the test-retest reliability of the attention network test from the perspective of intrinsic network organization. Our results demonstrate that the reproducibility of the group activation map of the three-attention network component was small-to-moderate between sessions and between healthy and Parkinson’s groups. The test-retest reliability of alerting, orienting, and executive contrast, defined as the difference between conditions, is worse than estimates of specific conditions. In addition, dorsal attention, ventral attention, visual, and somatomotor networks show acceptable reliability for the congruent and incongruent conditions.
As the cornerstone of fMRI methodology, reliability has recently attracted widespread attention and discussion. The reliability estimates for commonly used fMRI tasks are less satisfactory than researchers implicitly believe (Bennett & Miller, 2010; Elliott et al. , 2020). It was estimated that the ICC estimates for most fMRI task contrasts ranged between 0.33 and 0.66 (Bennett & Miller, 2010). A recent, more extensive meta-analysis reported that the mean ICC for 90 experiments was 0.397 (Elliott et al. , 2020). Elliott et al. (2020) also demonstrated that the test-retest reliabilities of activity in a priori regions of interest were poor, ranging between 0.067 and 0.485. Our results are consistent with those meta-analyses. First, the spatial overlap of the group activation map was small-to-moderate between sessions and between healthy and Parkinson’s groups, with dice coefficients ranging from 0.16 to 0.59. Second, the ICC estimates for the three attention network contrasts were also poor, with the median estimate around 0.25. The reliability of fMRI measurements depends on signal-to-noise ratio and task design. Block design fMRI commonly shows better reliability than single-trial design. For example, a recent study reported good-to-excellent overlap (Dice coefficients: 0.54 - 0.82) for most contrast of a finger-tapping fMRI task, which adopted a typical block design (Wuthrich et al. , 2023). The ANT fMRI task used a jittered single-trial design, which might explain the low reliability of the attention network contrasts.
The three attention network contrasts of the ANT task are commonly used to quantify the efficiency of attention components in individual difference or group-difference studies (Madhyastha et al. , 2015; Neufang et al. , 2015; Boord et al. , 2017; Yang et al. , 2021). However, as indicated by the present study, the reliability of the three attention network contrasts questioned the practice of using them as effective individualized biomarkers. This rationale applies to fMRI contrast, too (Infantolino et al. , 2018). In contrast to the low reliability of the three attention network contrasts, separate task conditions showed better reliability in the present study. The congruent condition and incongruent condition comprised a typical example: ICCs in the dorsal attention, ventral attention, visual, and somatomotor network ranged between 0.5 and 0.75, with a median of approximately 0.7. However, the ICC of executive contrast had a median of 0.2 in attention-related networks. As expected, a severe collinearity between congruent and incongruent conditions was as high as 0.8 . Our results on the reliability of the contrast map are consistent with recent findings. For example, Elliott and colleagues have discussed the phenomenon in a large meta-analysis (Elliott et al., 2020). Infantolino et al. (2018) examined the reliability of a face- and shape-matching task. Although their study revealed robust amygdala activation of faces over shapes, the reliability of the activation difference between faces and shapes was nearly zero. Reward-positive difference scores, computed between reward and non-reward conditions, show poor reliability due to the high correlation between the two conditions (Clayson et al., 2021). Thus, using executive contrast to quantify the individual difference in executive control is not a good idea because subtraction has canceled the interested variance. We suggest using separate condition estimates directly if the study interest is individual or group differences.
Our results on the reliability of intrinsic networks also support the direct usage of separate condition estimates. If the three attention networks of ANT measure attention ability, the reliability estimates of the intrinsic network should vary according to relevance to attention. However, there was generally comparable low reliability in the seven intrinsic networks for the alerting, orienting, and executive contrast. Our results showed that the reliability of alerting, orienting, and executive networks did not differ among the seven networks, suggesting no unique correspondence between attention networks and intrinsic networks from the reliability perspective. The finding coheres with a recent study adopting the spatial regression approach (Markett et al. , 2022). Although attention networks overlapped with multiple intrinsic networks, their study did not yield unique correspondence between attention networks and intrinsic networks (Markett et al. , 2022). However, the reliability of congruent and incongruent conditions exhibited attention relevance from the perspective of intrinsic organization. Precisely, the ICCs of the congruent and incongruent conditions followed the order: dorsal attention, ventral attention, visual, somatomotor, frontoparietal, default, and limbic network. Our results on subcortical structures lead to the same conclusion. The ICC of the condition is generally higher than the ICC of the ANT network obtained by differencing. Thus, it is reasonable to use separate condition estimates in individual studies as they were reliable and theoretically meaningful.
The present study only reports reliability estimates for the original design developed by Fan et al. (2005) with a healthy adult and Parkinson’s sample. However, there are several fMRI variants of ANT. To facilitate examining the interaction among the three attention networks, Fan and collaborators designed the ANT-R, which manipulated cue-targe interval and cue validity (Fan et al. , 2009). The fMRI version of ANT-R has been used to study attention dysfunction in autism (Fanet al. , 2012). Xuan et al. (2016) also used the fMRI version of ANT-R to elucidate the interactive attentional networks. In addition, a child-friendly fMRI version has also been widely used (Hao et al. , 2021). The child-friendly version used fish stimuli instead of arrow stimuli. In addition, the interval between cue and target was fixed to be 450ms in the child-friendly fMRI version. The original design of Fan et al. (2005) used a random interval with a mean of 2800ms, ranging from 300ms to 11800ms. The dataset used by the present study also followed this design. It should be noted that the cue-target interval in the original design was the longest. Fan et al. (2012) and Xuan et al. (2016) randomly chose the cue-target interval from 0, 400, and 800ms for each trial. Because of attention relapse or inhibition of return, the long cue-target interval can make the spatial cue useless, which might explain why the present study’s orientation network achieved the weakest spatial activation.
In conclusion, the present study manifests imperfect reliability estimates for the three network contrasts using a sample of healthy adults and Parkison patients. We advocate that separate condition estimates be used directly in individual difference studies as they are more reliable and theoretically meaningful from the perspective of intrinsic organization. Future research efforts must study the difference between different versions of ANT fMRI tasks to facilitate clinical and developmental neuroscience studies on attention dysfunction.