Overview and results of our procedure:
First, the HBN data set was rated by 4 neuroimaging experts to create a "gold standard" subset of data. Next, the 3D MRI scans were converted into 2D axial brain slices, which were loaded onto braindr (https://braindr.us), a web application to crowdsource the quality ratings. When taking the average quality rating for each slice, an ROC analysis resulted in an area under the curve (AUC) of 0.95. In an effort to remove unreliable raters, the ratings were aggregated by the XGBoost algorithm, and the resulting ROC of the left out data resulted in an AUC of 0.97. Finally, the 2D brain slices and the XGBoosted ratings were used to train the top layer of the pretrained VGG16 network to predict the XGBoosted ratings. In an ROC analysis on left out data, the AUC of these predictions was 0.99.