Abstract: Capturing the spatio-temporal (4D) dynamics of humans has been a long standing research problem in computer vision and graphics. Synthesizing photorealistic human avatars has broad applications, ranging from immersive telepresence in AR/VR and the movie industry, to enriching the education and healthcare systems. Earlier approaches relied on hand-engineered models that use a small amount of data from one or more subjects. With the advent of neural networks, training on large datasets enhanced the output visual quality. Currently, the combination of neural networks with graphics techniques has achieved natural-looking human animation. However, most approaches are identity-specific, trained only on a single identity, and use only one modality.

In this dissertation, we address the problem of learning neural representations of humans in a holistic way. Given that the video data in the real world include multiple modalities (e.g., audio and video) and multiple identities, we develop multi-modal and multi-identity representations. First, we propose to reconstruct the 4D face geometry of humans by leveraging both audio and video information. In this way, the network produces accurate lip shapes and is robust to cases when either modality is insufficient. Next, we introduce a NeRF-based representation for audio-driven human face animation that achieves high-quality lip synchronization for cinematic content. Since humans communicate with their full body, combining body pose, hand gestures, and facial expressions, we extend the network to capture full-body human motion for multiple identities simultaneously. In order to better disentangle identity and non-identity specific information, we subsequently study non-linear interactions between latent factors of variation, and propose a specific multiplicative module. In this way, we learn a multi-identity NeRF that robustly animates human faces under novel expressions and achieves a significant decrease in the total training time. Similarly, we propose a multi-identity Gaussian splatting representation for human bodies, by constructing a high-order tensor. Assuming a low-rank structure, we learn a tensor decomposition that leads to a significant decrease in the total number of learnable parameters, as well as to a robust animation under novel poses. Last but not least, we propose to jointly synthesize audio and visual outputs from just text input. Given the recent rise of large language models, coupling text with natural-looking avatars can enhance the overall interaction between a human and an AI system.

Location: NCS 220 or Zoom


To truly understand human language, we must look at words in the context of the human generating the language. Factors such as demographics, personality, modes of communication, and emotional states have shown to play a crucial role in NLP models pre-LLMs era. Steps of mathematically defining the inclusion of human context in language modeling and more will be discussed with Nikita Soni, a PhD student at Stony Brook University co-advised by H. Andrew Schwartz and Niranjan Balasubramanian. She is the lead organizer of the workshop on human-centered large language modeling.

Please register for the STEM Speaker Series Zoom event here

Please RSVP for the STEM Speaker Series in-person event here
How Language Makes us Smart (without Big Data) presented by Charles Yang

Abstract: Language provides the glue that combines simpler concepts into complex ones. To study how language guides conceptual development, we need precise accounts of how rules are learned from the child's linguistic experience, which is extremely limited in comparison to the amount of data available to current machine learning methods. In this talk, I discuss a mathematical model of inductive generalization, which enables language learning with very small amount of data. Such a view of learning has strong implications for the cross-cultural/linguistic variation of development. As a case study, I show that Hong Kong children learning Cantonese, which has a relatively simpler formal counting system, develop understanding of symbolic numbers a full year ahead of English-learning children in the United States, which is precisely predictable from the learning model. The new conception of learning adds another wrinkle to the eternal question of how language and thought are related to each other.

Bio: Charles Yang studied at the MIT AI lab and now teaches linguistics, computer science and psychology and directs the Program in Cognitive Science at the University of Pennsylvania. He is the author of several books: The Price of Linguistic Productivity (2016 MIT Press) won the Leonard Bloomfield Award from the Linguistic Society of America. His honors include a Guggenheim fellowship.
Stony Brook University Northern California Alumni Chapter - Institute for AI-Driven Discovery and Innovation Panel

Join us for a Northern California Alumni and Friends luncheon followed by a panel discussion, celebrating the Institute for AI-Driven Discovery and Innovation, moderated by Fotis Sotiropoulos, Dean, College of Engineering and Applied Sciences.

Panel Discussion with:
Richard Bravman '78, Chief Strategy Officer, Affinity Solutions
Jalal Mahmud, PhD '08, Master Inventor, IBM Watson
Reza Raji '86, CEO, Xenio Systems
Andrew Protter, PhD '83, Chief Scientific Officer, Auansa Inc.

Moderated by:
Fotis Sotiropoulos, Dean, College of Engineering and Applied Sciences

Click here for more information and to register.
Abstract: As we enter the AI era, domain scientists face a critical question: What can we do to harness AI effectively for scientific discovery? AI has demonstrated remarkable capabilities, from accelerating simulations to uncovering hidden patterns in complex datasets. While these advancements offer unprecedented opportunities, they also raise concerns--AI models often function as black boxes, making it difficult to connect their outputs to established scientific principles. This lack of interpretability can undermine trust and limit adoption, particularly in fields like meteorology where physical understanding is critical.
In this talk, I will explore how interpretable AI can bridge this gap, highlighting its potential to generate explicit, physically meaningful equations rather than opaque neural networks. Through four case studies from my lab, I will showcase how interpretable AI can enhance scientific understanding:
  1. Satellite Precipitation Retrieval: Using AI-based approaches to interpret precipitation retrieval algorithms from AMSU data, we identified critical microwave channels (89 and 150 GHz) that directly link to physical processes in the atmosphere.
  2. Quantitative Precipitation Estimation (QPE): By applying symbolic regression models to polarimetric radar data, we derived mathematical expressions that outperform traditional Z-R relationships and existing QPE algorithms, offering new insights into rainfall microphysics.
  3. Tornado Probability Prediction: Leveraging reinforcement learning-based symbolic deep learning models, we developed interpretable equations that outperform the traditional Significant Tornado Parameter (STP) index, providing a clearer understanding of the relationships between key atmospheric variables and tornado risk.
  4. Domain-Aware Symbolic Regression for Scientific Equations: In our latest work, we introduced a symbolic regression framework that incorporates domain-specific symbol priors extracted from thousands of scientific publications. By encoding common mathematical structures--such as the prevalence of trigonometric functions in physics or logarithmic forms in biology--into a tree-structured reinforcement learning model, we improved both the accuracy and interpretability of discovered equations. This approach accelerates convergence, enforces physical plausibility, and reveals new governing relationships in climate and geophysical data.
Through these examples, I hope to spark discussion on the evolving role of domain scientists in the AI era and inspire new ways to integrate AI with physical understanding in atmospheric research.

IACS Seminar Speaker: Yixin Wen, University of Florida

Location: IACS Seminar Room or Zoom

Join Zoom Meeting: https://stonybrook.zoom.us/j/97596399106?pwd=0PBvElFLqov3biO6OlQxSWLWudkIuH.1
Meeting ID: 975 9639 9106
Passcode: 096213
How to Succeed in Language Design Without Really Trying presented by Professor Brian Kernighan

ABSTRACT: Why do some languages succeed while others fall by the wayside? I've helped create nearly a dozen languages (mostly small) over the years; a handful are still in widespread use, while others have languished or simply disappeared. I've also been present at the creation of several other languages, including some really major ones. In this talk I'll give my humble, but correct, opinion on factors that affect success and failure, and try to offer some insight into what to do if you're trying to design a new language yourself, and why that might be a good thing.

BIO: Brian Kernighan received a PhD in electrical engineering from Princeton in 1969. He joined the Computer Science department at Princeton in 2000, after many years at Bell Labs. He is a co-creator of several programming languages, including AWK and AMPL, and of a number of tools for document preparation. He is the co-author of a dozen books and some technical papers, and holds 5 patents.
He is a member of the National Academy of Engineering and of the American Academy of Arts and Sciences. His research areas include programming languages, tools and interfaces that make computers easier to use, often for non-specialist users. He has also written two books on technology for
non-technical audiences: Understanding the Digital World in 2017 and Millions, Billions, Zillions: Defending Yourself in a World of Too Many Numbers, published in 2018. His most recent book, Unix: A History and a Memoir, was published in October 2019.
Abstract: Human gaze behavior is a fundamental cue for understanding social intent, human-machine interaction, and cognitive processes. This thesis addresses the challenges of gaze target estimation (GTE), also known as gaze following, by developing a holistic understanding of gaze in complex environments.

The first part of this work improves GTE performance by introducing Patch-level Distribution Prediction (PDP). Unlike traditional models that rely on strict pixel-wise regression, PDP models gaze as a distribution over patches, which better accounts for annotation variance and bridges the gap between target location and in/out-of-frame prediction. To address the laborious nature of data labeling, the second part presents GCDR, the first semi-supervised method for gaze following. By prompting large Visual Question Answering (VQA) models to generate initial Grad-CAM heatmaps and refining them with a diffusion model, this method achieves high performance with significantly fewer human annotations. The third part expands the applicability of GTE to multi-camera environments. By introducing the Multi-View Gaze Target (MVGT) dataset, along with two novel frameworks for integrating information between multiple views and predicting the gaze target across views, we explore a new direction that overcomes single-view limitations such as face occlusion and out-of-view targets.

Building on these foundations, the final part of this thesis proposes a new direction toward semantic social gaze understanding using next-generation multimodal Large Language Models (LLMs). Rather than focusing solely on geometric gaze target localization, we aim to enrich gaze prediction with semantic and relational interpretation in complex social scenes. To this end, we will leverage existing gaze following datasets to derive social gaze supervision, including mutual gaze and shared attention, and obtain aligned language descriptions of scene-level gaze behaviors. This proposed work will enable the model to not only locate gaze targets but also predict structured social gaze relations among individuals, meanwhile generating a concise natural-language summary describing the dominant gaze interactions. By integrating spatial gaze estimation, social relation reasoning, and language-based scene understanding within a unified multimodal model, this work takes an important step toward a holistic understanding of human gaze behavior in real-world environments.

Speaker: Qiaomu Miao
Join Zoom Meeting https://stonybrook.zoom.us/j/511954566?pwd=d3hGdHlGZVJISnV3d2pRcnBRbFBJZz09 (ID: 511954566, password: fxnmj4) Join by phone (US) +1 646-876-9923 Join using SIP 511954566@zoomcrc.com (passcode: 542040) Joining instructions: https://www.google.com/url?q=https://applications.zoom.us/addon/invitation/detail?meetingUuid%3DDBgsLRhCTQCGwD9DAxf7mw%253D%253D%26signature%3D02e2f3393481f8bc18cbb5520e174a6b8ceee7e9e58c9dca9d0aa4db9a63279a&sa=D&usg=AOvVaw1e3Mi7F_8TdXl5XstjsrIx Password: fxnmj4