YINS Industry Seminar

“Dynamics and scaling laws in deep learning”

Speaker: Yasaman Bahri
Research Scientist at Google Research, Brain Team

Talk summary: This talk will discuss some topics in supervised deep learning from a scientist’s lens, describing efforts that seek to bridge theory and experiment. First, I’ll briefly review some of what is known about the learning dynamics of deep neural networks, including highlights from the large-width limit of neural networks. I’ll then focus on efforts to understand empirical “scaling laws” for the performance of neural networks. Recent empirical work has found that the test loss often follows smooth power laws as a function of basic variables such as model size and dataset size. I will discuss our work seeking to connect and understand some of these scaling laws. We introduce “variance-limited” and “resolution-limited” scaling regimes to distinguish the origin of the power-law behavior. As an illustration, we investigate the case of teacher-student random feature models where we can study the problem exactly. I’ll close with a few empirical observations about task properties and scaling exponents.   

To participate:
Join from PC, Mac, Linux, iOS or Android: https://yale.zoom.us/j/98819118892
      Or Telephone：203-432-9666 (2-ZOOM if on-campus) or 646 568 7788
      Meeting ID: 988 1911 8892
      International numbers available: https://yale.zoom.us/u/aeF0HRmJcA   

Speaker bio: Yasaman Bahri is a Research Scientist at Google Research on the Brain team. In the past several years, she has been working at the boundary of machine learning and the physical sciences, with a particular focus on bridging theoretical and empirical understanding in deep learning. Her doctoral work is in the area of theoretical condensed matter physics, specifically on quantum many-body systems. She received her Ph.D. in Physics from UC Berkeley (2017) as well as earlier B.A. degrees from UC Berkeley in Physics and Math. She is a recipient of the 2020 Rising Stars Award in EECS and the NSF Graduate Fellowship and has co-organized two ICML workshops. 

.