Polskie Stowarzyszenie Sztucznej Inteligencji, przy współpracy z AI Bay, Zatoką Sztucznej Inteligencji, zaprasza na seminarium naukowe organizowane w formie zdalnej
Kto: dr Stanisław Jastrzębski. Molecule.oneCo: Early Training in Deep Neural Networks: Unpacking Simplicity Bias and SGD Implicit Regularization Effects
Kiedy: 23 marca 2023, 17.00
Jak: zdalnie – Zoom (osoby zapisane dostaną link w wiadomości email)
Abstrakt: The early phase of training of deep neural networks holds many mysteries. For example, using a large learning rate in the early phase of training is critical for achieving good final performance of the model. I will describe our work on understanding these effects. First, I will describe how the simplicity bias emerges in the beginning of training. Second, I will propose that the mechanism by which using a large learning rate improves generalization is by regularizing the local curvature from the outset of training. This mechanism is connected to the fact that the early learning dynamics are chaotic due to large local curvature. To corroborate this mechanism, we have designed an explicit regularizer that enables training well-generalizing networks using a small learning rate.
Bio: Stanislaw Jastrzebski serves as the CTO and Chief Scientist at Molecule.one, a startup speeding up drug discovery by making organic chemistry more predictable. He is passionate about improving the fundamental aspects of deep learning and applying it to automate scientific discovery. He completed his postdoctoral training at New York University. His PhD thesis was based on work on foundations of deep learning done during research visits at MILA (with Yoshua Bengio) and the University of Edinburgh (with Amos Storkey). He received his PhD from Jagiellonian University, advised by Jacek Tabor. He gained industrial experience at Google, Microsoft and Palantir. He has published at leading ML venues (NeurIPS, ICLR, ICML, JMLR). He is also actively contributing to the machine learning community as an Area Chair (most recently for NeurIPS 2023) and as an Action Editor for TMLR.
Zapraszamy
Prof. Jacek Rumiński
Przewodniczący Rady Naukowej PSSI