N°25-96: Understanding The Virtue of Complexity

Recent papers have challenged certain aspects of the "virtue of complexity" described by Kelly et al. (2024b) (KMZ) and related work. These challenges ultimately have little bearing on the theoretical arguments or empirical findings of KMZ. They do, however, provide a valuable opportunity to better understand the nuanced behavior of complex models. In addition to responding to recent challenges, we provide detailed discussions of how complex models learn in small samples, the roles of "nominal" and "effective" complexity, the unique effects of implicit regularization, and the importance of limits to learning. We then present new empirical and theoretical analyses that expand on KMZ. Finally, we introduce and demonstrate the virtue of ensemble complexity.