Abstract: Recent advances in multimodal AI have enabled models to perceive, reason about, and generate content across vision and language. However, impressive performance does not necessarily imply genuine understanding. When a model correctly answers a visual question, how do we know it actually used the visual evidence? When it generates a plausible response, how can we trust that response? And beyond traditional vision-language tasks, what new forms of computation, reasoning, and interaction might multimodal intelligence enable?
In this talk, I will explore these questions through a series of our research efforts that investigate both the limitations and emerging possibilities of multimodal AI. First, I will discuss methods for improving visual grounding and mitigating hallucinations in vision-language models, including approaches that anticipate potential failures before generation. Next, I will examine the gap between benchmark performance and genuine visual understanding, highlighting systematic evaluations that uncover hidden weaknesses in perception and reasoning. Finally, I will explore new ways of leveraging multimodal representations for efficient information processing, examine the challenges of preserving reasoning fidelity across modalities, and discuss opportunities for extending multimodal intelligence to interactive, long-horizon tasks.
Through these studies, I hope to deepen our understanding of multimodal intelligence and explore new directions toward building more reliable and versatile AI systems.
Bio: Dr. Jiawei (Joe) Zhou is an assistant professor in Data Science, Applied Math and Statistics, and Computer Science at Stony Brook University. He was a research assistant professor at the Toyota Technological Institute at Chicago (TTIC) previously, and he obtained a PhD degree in computer science from Harvard University.
Dr. Zhou's expertise spans in NLP and ML applications, especially generative models with Transformers like large language models (LLMs) and multimodal models such as vision language models (VLMs). His work encompasses technical research such as decoding algorithms and broader applications of trustworthy generation from different sources of information.
His early contributions to executable semantic parsing for efficient conversation systems were recognized with an ACL'22 Outstanding Paper Award, and recently he was awarded with an Amazon Research Award and IEEE ASRU'25 Best Student Paper Award. He maintains keen research interests in NLP and multimodal AI, including evaluation, efficiency, advanced reasoning with reinforcement learning, alignment and safety, novel agentic applications, and next generation of automation such as model self-improvements.
Location: NCS 120