Abstract
Learning by ignoring, which identifies less important things and
excludes them from the learning process, is broadly practiced in human
learning and has shown ubiquitous effectiveness. There has been
psychological studies showing that learning to ignore certain things is
a powerful tool for helping people focus. In this paper, we explore
whether this useful human learning methodology can be borrowed to
improve machine learning. We propose a novel machine learning framework
referred to as learning by ignoring (LBI). Our framework automatically
identifies pretraining data examples that have large domain shift from
the target distribution by learning an ignoring variable for each
example and excludes them from the pretraining process. We formulate LBI
as a three-level optimization framework where three learning stages are
involved: pretraining by minimizing the losses weighed by ignoring
variables; finetuning; updating the ignoring variables by minimizing the
validation loss. A gradient-based algorithm is developed to efficiently
solve the three-level optimization problem in LBI. Experiments on
various datasets demonstrate the effectiveness of our framework.