University Libraries Presents: The Library AI Club is a welcoming space for students, faculty, and staff to explore AI in a supportive, low-pressure environment. Meeting every two weeks, the club features discussions, collaborative projects, guest speakers, and hands-on experiments. Join us to learn, share ideas, and engage with AI responsibly and creatively. We'd love to see you at an upcoming meeting! Location: Melville Library, Scholarly Communication Seminar Room
Abstract: Human gaze behavior is a fundamental cue for understanding social intent, human-machine interaction, and cognitive processes. This thesis addresses the challenges of gaze target estimation (GTE), also known as gaze following, by developing a holistic understanding of gaze in complex environments.

The first part of this work improves GTE performance by introducing Patch-level Distribution Prediction (PDP). Unlike traditional models that rely on strict pixel-wise regression, PDP models gaze as a distribution over patches, which better accounts for annotation variance and bridges the gap between target location and in/out-of-frame prediction. To address the laborious nature of data labeling, the second part presents GCDR, the first semi-supervised method for gaze following. By prompting large Visual Question Answering (VQA) models to generate initial Grad-CAM heatmaps and refining them with a diffusion model, this method achieves high performance with significantly fewer human annotations. The third part expands the applicability of GTE to multi-camera environments. By introducing the Multi-View Gaze Target (MVGT) dataset, along with two novel frameworks for integrating information between multiple views and predicting the gaze target across views, we explore a new direction that overcomes single-view limitations such as face occlusion and out-of-view targets.

Building on these foundations, the final part of this thesis proposes a new direction toward semantic social gaze understanding using next-generation multimodal Large Language Models (LLMs). Rather than focusing solely on geometric gaze target localization, we aim to enrich gaze prediction with semantic and relational interpretation in complex social scenes. To this end, we will leverage existing gaze following datasets to derive social gaze supervision, including mutual gaze and shared attention, and obtain aligned language descriptions of scene-level gaze behaviors. This proposed work will enable the model to not only locate gaze targets but also predict structured social gaze relations among individuals, meanwhile generating a concise natural-language summary describing the dominant gaze interactions. By integrating spatial gaze estimation, social relation reasoning, and language-based scene understanding within a unified multimodal model, this work takes an important step toward a holistic understanding of human gaze behavior in real-world environments.

Speaker: Qiaomu Miao

Abstract:

Many real world complex problems are multi-step reasoning tasks. These range from analytic tasks such as answering questions to automation tasks where agents complete tasks on behalf of users.. Evaluation, datasets, and models for such tasks can be unreliable for multiple reasons. (i) Datasets often have annotation artifacts and biases, allowing models to take reasoning shortcuts. Such shortcuts can allow models to make effective guesses -- or, in a sense, cheat -- to achieve high performance without any multi-step reasoning. This issue is further exacerbated for complex tasks because as the number of the required reasoning steps increases, so do the avenues for bypassing those steps. (ii) Models trained on such dataset/s learn to solve the task by taking reasoning shortcuts instead of proper multi-step reasoning. As a result, these models are not robust (reliable) when evaluated in an out-of-distribution evaluation setting. (iii) Lastly, recent works have shown that language models can solve complex multi-step tasks by producing a step-by-step explanation without any training. However, these methods often hallucinate factually incorrect (i.e., unreliable) explanations when posed with knowledge-intensive tasks.

I address these challenges by carefully characterizing the requirements of robust multi-step reasoning and designing reliable evaluation datasets and training methods that necessitate thorough multi-step reasoning. In DiRe, I first formalize and introduce Disconnected Reasoning, i.e., reasoning that allows models to arrive at the correct answer by bypassing necessary reasoning steps, and use this formalization to measure how much multi-step reasoning a model does on a dataset. In MuSiQue, I built a multi-step reasoning dataset for QA from scratch that avoids cheatability via disconnected reasoning, providing a more reliable evaluation. In TeaBReaC, I developed a synthetically generated multi-step QA pretraining dataset designed to force models to avoid disconnected reasoning and learn reliable multi-step reasoning. In IRCoT, I address the reliability of model-generated multi-step reasoning chains by interleaving models' step-by-step reasoning with a step-by-step retrieval from an external corpus, resulting in more factually correct reasoning. Finally, in AppWorld, I built a multi-step reasoning dataset that requires highly interactive problem-solving in an environment carefully designed to ensure models need thorough reasoning to succeed.
Speaker: Harsh Trivedi

Location: NCS 220 or Zoom

https://stonybrook.zoom.us/j/99096379762?pwd=zYCJZQVxRuZd9BboscO4nlodCwsKBr.1

The Della Pietra Lecture Series is pleased to present this lecture by Scott Aaronson:

Abstract: I'll survey some areas where I think theoretical computer science, math, and statistics can potentially contribute to the urgent quest to align powerful AI with humane values. These areas include: the watermarking of AI outputs, mechanistic interpretability (including Paul Christiano's No-Coincidence Principle, and succinct digests of the training process to aid interpretability), and theoretical guarantees for out-of-distribution generalization.

Speaker: Scott Aaronson is Schlumberger Chair of Computer Science at the University of Texas at Austin, and founding director of its Quantum Information Center. He received his bachelor's from Cornell University and his PhD from UC Berkeley. Aaronson's research has focused mainly on the capabilities and limits of quantum computers. His first book, Quantum Computing Since Democritus, was published in 2013 by Cambridge University Press. He received the National Science Foundation's Alan T. Waterman Award, the United States PECASE Award, the Tomassoni-Chisesi Prize in Physics, and the ACM Prize in Computing, and is a Fellow of the ACM and the AAAS and a member of the National Academy of Sciences. He blogs at Shtetl-Optimized, https://www.scottaaronson.com/blog.

Location: Della Pietra Family Auditorium (SCGP 103)

Over the past decade, researchers in neuroscience, psychology and artificial intelligence have come together to build advanced computer models that mimic how our brain processes what we see. These models are designed to closely copy the brain's visual system, all the way to a key area called the inferior temporal cortex, which plays an important role in recognizing objects.

Because these computer models can be fully observed, scientists can use them to make detailed predictions about how the brain works -- something older, more theoretical models could not do.

Dr. James DiCarlo's work explores whether these computer digital twin models of the brain could help guide safe, non- invasive ways to infl uence brain activity. In his talk, he explains how such a model could be used to design specific patterns of light. When this carefully designed light is added to what the eye naturally sees, it can precisely influence activity in groups of neurons in the inferior temporal cortex.

Since neural activity in this visual brain area may be connected to emotional states like anxiety, this research could eventually open the door to non-invasive approaches that may benefit mental well-being in the future.

Speaker: James J. DiCarlo, MD, PhD, Peter de Florez Professor, MIT Brain and Cognitive Sciences, and Director, MIT Siegel Family Quest for Intelligence

Location: Staller Center Main Stage

The event will be livestreamed at stonybrook.edu/live

Abstract: As intelligent systems become more integrated into human environments, fostering trustworthy human-AI collaboration presents a pressing challenge. In this talk, I examine the interplay between an agent's performance and social dynamics in shaping trust in human-AI interactions. My approach combines testbed development, behavioral prototyping, and user study design to create controlled experimental setups that capture real-world interaction complexities, such as ambiguity, multi-agent dynamics, and conflicting goals.

I illustrate this with a recent VR study on multi-user interaction with an autonomous vehicle (AV). Moving beyond dyadic interactions, the study probes human perspectives from the roles of a pedestrian, driver, and AV passenger, all interacting with the AV simultaneously at an ambiguous all-way stop sign intersection. We compare interactions with efficient and prosocial AV behavior strategies, revealing diverging trust perceptions and preferences across user roles. These insights inform a broader research trajectory focused on balancing performance with social considerations in designing trustworthy human-AI collaborations.

Bio: JiHyun Jeong is a postdoctoral researcher at Cornell University working on human-computer interaction and human-robot interaction. Her research develops prototypes and methods to explore performance and social factors that influence collaboration and trust between humans and artificial agents. She holds a Ph.D. and MPS in Information Science from Cornell University, and a BSc in Computer Science and Engineering from Korea University. She is a recipient of an honorable mention for best paper at DIS.

Zoom: https://stonybrook.zoom.us/j/98738234619?pwd=djJFQXBWbkpmblZDT25zNlVMYWpCQT09

Meeting ID: 987 3823 4619
Passcode: 474618

Abstract: Large Language Model (LLM) agents have demonstrated remarkable generalization capabilities across multi-domain tasks. Existing agent tuning approaches typically employ supervised finetuning on entire expert trajectories. However, behavior-cloning of full trajectories can introduce expert bias and weaken generalization to states not covered by the expert data. Additionally, critical steps--such as planning, complex reasoning for intermediate subtasks, and strategic decision-making--are essential to success in agent tasks, so learning these steps is the key to improving LLM agents. For more effective and efficient agent tuning, we propose ATLAS that identifies the critical steps in expert trajectories and finetunes LLMs solely on these steps with reduced costs. By steering the training's focus to a few critical steps, our method mitigates the risk of overfitting entire trajectories and promotes generalization across different environments and tasks. In extensive experiments, an LLM finetuned on only 30% critical steps selected by ATLAS outperforms the LLM finetuned on all steps and recent open-source LLM agents. ATLAS maintains and improves base LLM skills as generalist agents interacting with diverse environments.

Speaker: Bijoy

Location: Old Computer Science Building - CS2311

Submit an abstract celebrating research, new discoveries and achievements in medicine and science!

We encourage faculty, nurse practitioners, post-doctoral fellows, fellows, residents, medical students, graduate students and undergraduate students to submit an abstract. Original research, case reports and case series are welcome.

Abstract submission deadline: FEBRUARY 7, 2025

For more details, visit here.

University Libraries Presents:
Join librarian Christine Fena for an interactive workshop that invites you to explore AI tools first hand, not just as users, but as critical investigators.
Through playful experimentation and collaborative discovery, you'll uncover inherent biases, probe algorithmic flaws, and gain a deeper understanding of AI's limitations and societal impacts.

RSVP on SBEngaged

Location: Melville Library, Central Reading Room, Lab B