Abstract
A signal-dependent, correlation-based pruning algorithm is proposed to
sparsify inter-layer weight matrices of a Multilayer Perceptron (MLP).
The method measures correlations of node outputs for an input or hidden
layer. The nodes are partitioned, accordingly. The nodes of a partition
with relatively higher correlations are bundled to be the inputs of a
node in the next layer. Such partitioning improves subspace
representation of nodes in the network. The numerical performances for
various MLP architectures and input (training) signal statistics for the
two-class classification problem are presented. The results provide
insights on the relationships between signal statistics, node and layer
behavior, network dimension, depth, sparsity, and system performance. We
show convincing evidence in the paper that the model design should track
input statistics and transformations through the building blocks to
sparsify the network for improved performance and computational
efficiency. The proposed pruning method may also be used to design a
self-reconfiguring network architecture with weight and node sparsities.