MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks

Ariel Gordon
Ofir Nachum
Bo Chen
Tien-Ju Yang
Edward Choi
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

Abstract

We present MorphNet, an approach to automate the design of neural network structures. MorphNet iteratively shrinks and expands a network, shrinking via a resource-weighted sparsifying regularizer on activations and expanding via a uniform multiplicative factor on all layers. In contrast to previous approaches, our method is scalable to large networks, adaptable to specific resource constraints (e.g. the number of floating-point operations per inference), and capable of increasing the network's performance. When applied to standard network architectures on a wide variety of datasets, our approach discovers novel structures in each domain, obtaining higher performance while respecting the resource constraint.