CAM Colloquium 2021-22 Student-Selected Speaker : Lexing Ying (Mathematics, Stanford) - The Sobolev Regularization Effect of Stochastic Gradient Descent

Description

Abstract:
The multiplicative structure of parameters and input data in the first layer of neural networks is explored to build connection between the landscape of the loss function with respect to parameters and the landscape of the model function with respect to input data. By this connection, it is shown that flat minima regularize the gradient of the model function, which explains the good generalization performance of flat minima. Then, we go beyond the flatness and consider high-order moments of the gradient noise, and show that Stochastic Gradient Descent (SGD) tends to impose constraints on these moments by a linear stability analysis of SGD around global minima. Together with the multiplicative structure, we identify the Sobolev regularization effect of SGD, i.e. SGD regularizes the Sobolev seminorms of the model function with respect to the input data. Finally, bounds for generalization error and adversarial robustness are provided for solutions found by SGD under assumptions of the data distribution. Joint work with Chao Ma.

Bio:
Lexing Ying is Professor of Mathematics at Stanford University since 2012. Prior to that, he was a professor at the University of Texas at Austin from 2006 to 2012. His research focuses on computational mathematics and scientific computing. He received his Ph.D. from New York University and was a postdoctoral scholar at California Institute of Technology from 2004 to 2006. He is a recipient of the Sloan Research Fellowship (2007), the National Science Foundation CAREER Award (2009), the Feng Kang Prize of Scientific Computing (2011), the James H. Wilkinson Prize in Numerical Analysis and Scientific Computing from SIAM (2013), and the Silver Morningside Medal in Applied Mathematics (2016).

Zoom Link Access:
The link is emailed to the CAM Seminar listserv the week of the talk. If you are not on the listserv, please contact Erika Fowler-Decatur to request the link.