Abstract: Language is not just something we generate, but something humans use to interact with the world around them. Indeed, today's conversational AI agents speak fluently, but often treat language as prediction rather than interaction, producing responses that sound correct while failing to recognize when requests are ungrounded, impossible, or misunderstood. My research asks what it would take for multimodal agents to take the next step and use language effectively to take actions in grounded contexts and in this talk, I argue that many challenges in multimodal LLM design, including alignment, hallucination, and adaptation, are due to a lack of pragmatics: an understanding of the implicit context behind the implied actions of the words in the query. From visual understanding, to automatic speech recognition, to hallucination detection, I will demonstrate that incorporating pragmatic/contextual reasoning substantially improves agent behavior, and that pragmatic reasoning will drive a necessary shift in how we build multimodal conversational agents that can see, listen, act, and speak in context.


Bio: David M. Chan, Ph.D., is a postdoctoral scholar at the University of California, Berkeley, specializing in multimodal conversational AI. His research focuses on developing scalable AI systems that move beyond language prediction toward grounded language use, integrating vision, audio, and language to enable pragmatic interaction, improve AI-human collaboration, and reduce hallucinations in generative models. Beyond academia, he has worked with leading organizations including Amazon, Google, and NASA, to build and deploy safe, efficient, and accessible machine learning systems. He is also the developer and maintainer of TSNE-CUDA, an open-source tool for high-dimensional data visualization, used by over 40,000 researchers in fields ranging from biomedical technology to industrial manufacturing. David holds a Ph.D. and M.Sc. in Computer Science from UC Berkeley, where he was a graduate fellow with the Center for Technology, Society & Policy (CTSP), and dual B.Sc. degrees in Computer Science and Mathematics from the University of Denver.

Location: NCS 120
TITLE: Sampling Using Langevin Diffusions Beyond the Worst-Case by Andrej Risteski of CMU


ABSTRACT: Many tasks involving generative models involve being able to sample from distributions parametrized as p(x) = e^{-f(x)}/Z where Z is the normalizing constant, for some function f whose values and gradients we can query. This mode of access to f is natural -- for instance sampling from posteriors in latent-variable models. Classical results show that a natural random walk, Langevin diffusion, mixes rapidly when f is convex. Unfortunately, even in simple examples, the applications listed above will entail working with functions f that are nonconvex.

We exhibit instances where Langevin diffusion (combined with other tools) can provably be shown to mix rapidly in instances of relevance in practice: distributions p that are multimodal, as well as distributions p that have a natural manifold structure on their level sets. 


The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning.



ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.

ICLR is one of the fastest growing artificial intelligence conferences in the world. Participants at ICLR span a wide range of backgrounds, from academic and industrial researchers, to entrepreneurs and engineers, to graduate students and postdocs.

The rapidly developing field of deep learning is concerned with questions surrounding how we can best learn meaningful and useful representations of data. ICLR takes a broad view of the field and includes topics such as feature learning, metric learning, compositional modeling, structured prediction, reinforcement learning, and issues regarding large-scale learning and non-convex optimization.

A non-exhaustive list of relevant topics explored at the conference include:



  • Unsupervised, Semi-supervised, and Supervised Representation Learning
  • Representation Learning for Planning and Reinforcement Learning
  • Metric Learning and Kernel Learning
  • Sparse Coding and Dimensionality Expansion
  • Hierarchical Models

  • Optimization for Representation Learning
  • Learning Representations of Outputs or States
  • Implementation Issues, Parallelization, Software Platforms, Hardware
  • Applications in Vision, Audio, Speech, Natural Language Processing, Robotics, Neuroscience, or Any Other Field


For more information or registration, please visit the official website.
Predicting Subjective Attributes in Visual Data - Zijun Wei

ABSTRACT: Recent progress in deep neural networks has revolutionized many computer vision tasks such as image classification, detection and segmentation. However, in addition to excelling in tasks that predict well-defined objective information, human-centered artificial intelligence systems should also be able to model subjective attributes, as defined by human perceptual behavior, that goes beyond the pure physical content of visual data. Example subjective tasks are the prediction of spatial or temporal regions that are interesting to humans (e.g., attract attention or are visually pleasing) and the recognition of subjective attributes (e.g., visually elicited sentiments). Better models for these tasks will improve the human-computer interaction experience in various applications. This thesis investigates several approaches to address the challenges in predicting those subjective attributes in visual data over a diverse set of tasks. I first present a novel framework for real-time automatic photo composition. The framework consists of a cost-effective data collection workflow, an efficient model training pipeline and a lightweight module to account for personalized preferences. Then I develop a novel and general algorithm to detect interesting segments in sequential data, which can be naturally applied to video summarization tasks. Furthermore, I propose methods that learn to represent sentiments elicited by images, in an unsupervised manner, using linguistic features extracted from large scale Web data. To conclude this thesis, I introduce a human-vision-inspired image classification algorithm that also predicts spatial visual attention even though no attention data was used for training it.  
IACS and the Dept of Ecology and Evolution invite you to a PRODiG+ Fellowship Seminar.

Computational Approaches to Understanding and Forecasting Biodiversity Responses to Global Change

Summary: Advancing biodiversity forecasting requires developing computational approaches that better represent ecological complexity. In this talk, I will discuss how my research integrates field observations, predictive modeling, machine learning, and spatiotemporally explicit environmental data to develop more mechanistic and reliable forecasting tools.

Speaker: Dr. Anna Thonis, a postdoctoral researcher in NYU's Winchell Lab and Founder and Co-Chair of the IUCN SSC Anoline Lizard Specialist Group, studies how anthropogenic change reshapes the distributions of reptiles. Combining field and quantitative modeling methods, her work focuses on the ecology and conservation of Puerto Rican and urban Anolis lizards. Her website can be accessed here.

Location: IACS Seminar Room, and via Zoom.

Zoom: https://stonybrook.zoom.us/j/97023584426?pwd=HrnrYeYJfVUhaS8iOKjWF3Tbh8HYS1.1&jst=2
ID: 97023584426
Passcode: 241762
The overall purpose of this seminar is to bring together people with interests in Computer Vision theory and techniques and to examine current research issues. This course will be appropriate for people who already took a Computer Vision graduate course or already had research experience in Computer Vision. To enroll in this course, you must either: (1) be in the PhD program or (2) receive permission from the instructors.

Each seminar will consist of multiple short talks (around 10 minutes) by multiple people. Students can register for 1 credit for CSE 656. Registered students must attend and present a minimum of 2 or 3 talks. Everyone else is welcome to attend. Fill in https://forms.gle/pCVXovgfMfQwGqG38 to subscribe to our mailing list for further announcement.
This symposium will highlight how artificial intelligence (AI) can assist in dementia detection, research and clinical care. For example, the use of robotics to assist with dementia care therapy is truly inspirational and cutting-edge for clinicians, trainees and the community at large, including assisted living facilities. The symposium will also focus on the role of AI in early detection of dementia and in identifying characteristics associated with future cognitive decline.

Learn more and register at https://cme.stonybrookmedicine.edu/continuing-medical-education/conferences/233/alzheimers-symposium-ai-the-future-of-dementia-care-2024/11/15/2024

The Provost's Office is excited to invite you to join in responding to an extraordinary opportunity to enhance our academic and research capabilities in AI at Stony Brook. SUNY recently made funding available to support the creation of departments of AI and Society at its universities. Stony Brook is well-positioned to seize this opportunity to build upon our interdisciplinary strengths in AI.

The office is hosting a forum on Friday, Nov. 15, from 11:30 a.m. to 1:30 p.m., in Ballroom A, SAC. You are invited to attend to learn more about this opportunity and to help us generate ideas to build a compelling proposal for Stony Brook to submit to SUNY. Lunch will be provided.

Please click here to RSVP as soon as possible.

This funding will support innovation in our curriculum, allowing us to create programs that explore the social and societal impact of AI alongside the technological advancements led by researchers in engineering and scientific disciplines.

We believe we can make a significant impact through this SUNY program and look forward to your participation in this initiative.