AI3 Seminar
Meir Feder
Professor, School of Electrical Engineering
Jokel Chair in Information Theory, School of Electrical and Computer Engineering
Tel-Aviv University
Information-Theoretic Framework for Understanding Modern Machine-Learning
Abstract:
Information Theory views learning as universal prediction under log loss, characterized through regret bounds. Unlike the classical results that considered ``small'' model classes and provided uniform regret, the proposed framework provides non-uniform, model dependent bounds utilizing an effective notion of architecture-based model complexity. This complexity is defined by the probability mass or volume of the set of all models in the vicinity of the target model \theta_0, in an informational distance. This volume might be hard to evaluate, yet by local analysis it is related to spectral properties of the expected Hessian or the Fisher Information Matrix at \theta_0, leading to tractable approximations. We argue that successful architectures possess a broad complexity range, enabling learning in highly over-parameterized model classes. The framework sheds light on the role of inductive biases, the effectiveness of stochastic gradient descent (SGD) algorithm, and phenomena such as flat minima. It unifies online, batch, supervised, and generative settings, and applies across the stochastic-realizable and agnostic regimes. Moreover, it provides insights into the success of modern machine-learning architectures, such as deep neural networks and transformers, suggesting that their broad complexity range naturally arises from their layered structure. These insights open the door to the design of alternative architectures with potentially comparable or even superior performance.
Biography:
Meir Feder received the Sc.D. degree in Electrical Engineering and Ocean Engineering in 1987 from the Massachusetts Institute of Technology (MIT) and the Woods Hole Oceanographic Institution (WHOI). After being a Research Associate and a Lecturer at MIT, he joined the School of Electrical Engineering, Tel-Aviv University in 1990, where he is the Jokel Chaired Professor and the former founding head of Tel-Aviv university center for Artificial intelligence and Data science (TAD). Parallel to his academic career, he is closely involved with the high-tech industry: he founded 5 companies, among them Peach Networks (Acq: MSFT) and Amimon (Acq:LON.VTC). Recently, with his renewed interest in machine learning and AI, he co-founded Run:ai (Acq:NVDA), a virtualization, orchestration, and acceleration platform for AI infrastructure, acquired by Nvidia to support its GPU cloud operation.
Prof. Feder received several academic and professional awards including the IEEE Information Theory Society best paper award, the Padovani lectureship, the creative thinking award of the Israeli Defense Forces, and the Research Prize of the Israeli Electronic Industry, awarded by the President of Israel. For the technology he developed in Amimon, he received the 2020 Scientific and Engineering Award of the Academy of Motion Picture Arts and Sciences (OSCAR) and was announced the principal inventor of the technology that attained the 73rd Engineering Emmy Award of the Television Academy.
Location: NCS120
