AI Innovation Institute

AI3 Seminar

Meir Feder

Professor, School of Electrical Engineering
Jokel Chair in Information Theory, School of Electrical and Computer Engineering
Tel-Aviv University

Information-Theoretic Framework for Understanding Modern Machine-Learning

Abstract:

Information Theory views learning as universal prediction under log loss, characterized through regret bounds. Unlike the classical results that considered ``small'' model classes and provided uniform regret, the proposed framework provides non-uniform, model dependent bounds utilizing an effective notion of architecture-based model complexity. This complexity is defined by the probability mass or volume of the set of all models in the vicinity of the target model \theta_0, in an informational distance. This volume might be hard to evaluate, yet by local analysis it is related to spectral properties of the expected Hessian or the Fisher Information Matrix at \theta_0, leading to tractable approximations. We argue that successful architectures possess a broad complexity range, enabling learning in highly over-parameterized model classes. The framework sheds light on the role of inductive biases, the effectiveness of stochastic gradient descent (SGD) algorithm, and phenomena such as flat minima. It unifies online, batch, supervised, and generative settings, and applies across the stochastic-realizable and agnostic regimes. Moreover, it provides insights into the success of modern machine-learning architectures, such as deep neural networks and transformers, suggesting that their broad complexity range naturally arises from their layered structure. These insights open the door to the design of alternative architectures with potentially comparable or even superior performance.

Biography:

Meir Feder received the Sc.D. degree in Electrical Engineering and Ocean Engineering in 1987 from the Massachusetts Institute of Technology (MIT) and the Woods Hole Oceanographic Institution (WHOI). After being a Research Associate and a Lecturer at MIT, he joined the School of Electrical Engineering, Tel-Aviv University in 1990, where he is the Jokel Chaired Professor and the former founding head of Tel-Aviv university center for Artificial intelligence and Data science (TAD). Parallel to his academic career, he is closely involved with the high-tech industry: he founded 5 companies, among them Peach Networks (Acq: MSFT) and Amimon (Acq:LON.VTC). Recently, with his renewed interest in machine learning and AI, he co-founded Run:ai (Acq:NVDA), a virtualization, orchestration, and acceleration platform for AI infrastructure, acquired by Nvidia to support its GPU cloud operation.

Prof. Feder received several academic and professional awards including the IEEE Information Theory Society best paper award, the Padovani lectureship, the creative thinking award of the Israeli Defense Forces, and the Research Prize of the Israeli Electronic Industry, awarded by the President of Israel. For the technology he developed in Amimon, he received the 2020 Scientific and Engineering Award of the Academy of Motion Picture Arts and Sciences (OSCAR) and was announced the principal inventor of the technology that attained the 73rd Engineering Emmy Award of the Television Academy.

Location: NCS120

Chat with Sociology faculty as they share their paths to StonyBrook-what inspired their careers, what led them to teaching,and the experiences that shaped their academic journey.

Dr. Yongjun Zhang

Assistant Professor of Sociology, Departments of Sociology and AAAS

Join this opportunity to talk to Yongjun Zhang about his new interest in the following responsible usage of AI in addressing climate and health issues. Lunch will be served.

Location: SBS Level 4- Sociology Reading Room

View more event information

Over the past decade, researchers in neuroscience, psychology and artificial intelligence have come together to build advanced computer models that mimic how our brain processes what we see. These models are designed to closely copy the brain's visual system, all the way to a key area called the inferior temporal cortex, which plays an important role in recognizing objects.

Because these computer models can be fully observed, scientists can use them to make detailed predictions about how the brain works -- something older, more theoretical models could not do.

Dr. James DiCarlo's work explores whether these computer digital twin models of the brain could help guide safe, non- invasive ways to infl uence brain activity. In his talk, he explains how such a model could be used to design specific patterns of light. When this carefully designed light is added to what the eye naturally sees, it can precisely influence activity in groups of neurons in the inferior temporal cortex.

Since neural activity in this visual brain area may be connected to emotional states like anxiety, this research could eventually open the door to non-invasive approaches that may benefit mental well-being in the future.

Speaker: James J. DiCarlo, MD, PhD, Peter de Florez Professor, MIT Brain and Cognitive Sciences, and Director, MIT Siegel Family Quest for Intelligence

Location: Staller Center Main Stage

The event will be livestreamed at stonybrook.edu/live

The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning.

ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.

ICLR is one of the fastest growing artificial intelligence conferences in the world. Participants at ICLR span a wide range of backgrounds, from academic and industrial researchers, to entrepreneurs and engineers, to graduate students and postdocs.

The rapidly developing field of deep learning is concerned with questions surrounding how we can best learn meaningful and useful representations of data. ICLR takes a broad view of the field and includes topics such as feature learning, metric learning, compositional modeling, structured prediction, reinforcement learning, and issues regarding large-scale learning and non-convex optimization.

A non-exhaustive list of relevant topics explored at the conference include:

• Unsupervised, Semi-supervised, and Supervised Representation Learning
• Representation Learning for Planning and Reinforcement Learning
• Metric Learning and Kernel Learning
• Sparse Coding and Dimensionality Expansion
• Hierarchical Models

• Optimization for Representation Learning
• Learning Representations of Outputs or States
• Implementation Issues, Parallelization, Software Platforms, Hardware
• Applications in Vision, Audio, Speech, Natural Language Processing, Robotics, Neuroscience, or Any Other Field