Datasets containing class noise present significant challenges to
accurate classification, thus requiring classifiers that can refuse to
classify noisy instances. We demonstrate the inability of the popular
confidence-thresholding rejection method to learn from relationships
between input features and not-at-random class noise. To take advantage
of these relationships, we propose a novel null-labelling scheme based
on iterative re-training with relabelled datasets that uses a classifier
to learn to reject instances that are likely to be misclassified. We
demonstrate the ability of null-labelling to achieve a significantly
better tradeoff between classification error and coverage than the
confidence-thresholding method. Models generated by the null-labelling
scheme have the added advantage of interpretability, in that they are
able to identify features correlated with class noise. We also unify
prior theories for combining and evaluating sets of rejecting
classifiers.