Methods and Analysis of The First Competition in Predicting Generalization of Deep Learning

Yiding Jiang
Parth Natekar
Manik Sharma
Sumukh K. Aithal
Dhruva Kashyap
Natarajan Subramanyam
Carlos Lassance
Daniel M. Roy
Gintare Karolina Dziugaite
Suriya Gunasekar
Isabelle Guyon
Pierre Foret
Scott Yak i
Samy Bengio
Proceedings of the NeurIPS 2020 Competition and Demonstration Track, PMLR (2021)

Abstract

Deep learning has been recently successfully applied to an ever larger number of problems, ranging from pattern recognition to complex decision making. However, several concerns have been raised, including guarantees of good generalization, which is of foremost importance. Despite numerous attempts, conventional statistical learning approaches fall short of providing a satisfactory explanation on why deep learning works. In a competition hosted at the Thirty-Fourth Conference on Neural Information Processing Systems (NeurIPS 2020), we invited the community to design robust and general complexity measures that can accurately predict the generalization of models. In this paper, we describe the competition design, the protocols, and the solutions of the top-three teams at the competition in details. In addition, we discuss the outcomes, common failure modes, and potential future directions for the competition.

Research Areas