AI Innovation Institute

Abstract : Humans reason about everyday situations by making commonsense-based inferences, derived both from explicitly stated information and implicit, unstated knowledge. In this thesis, I investigate whether NLP models have different aspects of causal knowledge about events and how to improve their understanding of narratives and plans.
Answering questions about why people perform actions in a narrative can test whether NLP systems contain and can effectively apply causal knowledge about events. I introduce TellMeWhy, a dataset concerning why characters in short narratives perform the actions described. An evaluation of then SOTA finetuned models show that they are far worse than humans. To improve models, it is important to understand what aspects of causal knowledge they need and how to best use external sources to inject this knowledge. In KnowWhy, I analyze different ways of injecting knowledge into models, which is difficult since we do not know apriori what type of knowledge will be needed to answer a question, hence requiring a ranking model to pick the most important inference. Results show that this retrieved knowledge helps models of all sizes, thereby improving their understanding of narratives.
Next, I study whether models can reason about causal aspects of plans. I focus on testing whether they understand the underlying causal dependencies reflected in the temporal order of a plan's steps. I introduce CAT-Bench, and find that SOTA models are underwhelming, and that model answers are not consistent across questions about the same step pairs. In their current state, these models cannot yet reliably be used for complex user-facing tasks. I then measure contemporary models' ability to perform user-facing and user-centric plan customization. I introduce the use of semi-symbolic edits in large language model (LLM) based agents and test several multi-LLM-agent architectures for plan customization. While LLMs still lack the ability to understand complex customization hints, my results suggest that LLM-based architectures may be worth exploring further for other customization applications. Finally, I distill complex reasoning capabilities into small language models (SLMs) using synthetic data that reflects a decomposition-then-editing process for plan customization. I demonstrate that explicitly teaching this latent causal reasoning significantly improves the quality of SLM-generated customizations. Overall, my work has improved how well NLP models understand complex reasoning associated with events in different contexts.

Speaker: Yash Kumar Lal

Location: NCS 220 or Zoom https://stonybrook.zoom.us/j/95849648243?pwd=dgPpZtDpgwQrK9z1SaPpNbBifaorzk.1

Learn how to use Stony Brook Google Gemini for uploading files. Using the AI to change tone and how long the text is with a few clicks.

In this session, you will

1. Understand Stony Brook Google Gemini
2. See the New Tone and Length features

Register here.

The biennial conference takes place Sun. Oct 19th through Thu the 23rd, 2025 at the Honolulu Convention Center.
October 19 - 20: Workshops
October 21 - 23: Main Conference
More information can be found here.

Abstract: Modern technologies enable enhanced integrity and privacy guarantees not just for data, but also for computation. This is perhaps most emphatically demonstrated by the steady rise of zero-knowledge proofs, which are short certificates that attest to the correctness of computations (e.g., an age verification check) without revealing any secret inputs (e.g., the birth date on a digital ID). This subtly powerful technology enables anonymous credentials, privacy-preserving machine learning, anonymous blockchains, and much more--making the question of efficient zero-knowledge proofs fundamental to modern secure systems. Echoing Moore's law for computing, zero-knowledge proofs have improved on this front by ten orders of magnitude in the last two decades. In this talk, I will discuss our work on overcoming a key bottleneck that has emerged in this development: memory efficiency.

Speaker: Abhiram Kothapalli is a postdoctoral scholar at the University of California, Berkeley, hosted by Sanjam Garg. He is a recent graduate of Carnegie Mellon University, where he earned his Ph.D. in Computer Science, advised by Bryan Parno. Previously, he was at the University of Illinois at Urbana-Champaign, where he earned his B.S. in Computer Science and B.S. in Mathematics. Kothapalli's research develops cryptographic techniques aimed at scaling expressive privacy and integrity guarantees across the internet.

Location: NCS 120

Public Interest Technology Research and Education by Ellen Zegura, Georgia Tech 2019 * 2020 Shutterstock Distinguished Lecture Series

Hieu Le presents Incorporating Physical Illumination Constraints into Deep Learning Shadow Detection and Removal (PhD Proposal)

Shadows provide useful cues to analyze the scene but also hamper many computer vision algorithms such as image segmentation, object detection or tracking. For those reasons, shadow detection and shadow removal have been well studied topics in computer vision. Early approaches for shadow detection and removal focus on physical illumination models of shadows. These methods can express, identify, and remove shadows in a physically plausible manner. However, these models are often hard to optimize and slow in inference due to reliance on hand-designed image features. On the other hand, recent deep-learning approaches have achieved breakthroughs in performances for both shadow detection and removal. They learn to extract useful features automatically through training while being extremely efficient in computation. However, these models are data-dependent, opaque and ignore the physical aspects of shadows.

We propose to incorporate physical illumination constraints into deep-learning frameworks. Thus the mapping learned by the deep-network closely follows the physics of shadows, enabling the network to systematically and realistically modify shadows in images. For shadow detection, we present a novel GAN framework in which the generator can generate realistic images with attenuated shadows that can be used to train a shadow detector. For shadow removal, we propose a method that uses deep-networks to estimate the unknown parameters for a shadow image formation model that removes shadows. The system outputs shadow-free images in high-quality with no image artifacts and achieves state-of-the-art shadow removal performance. Lastly, we propose a system trained without the need for any shadow-free images in which physical constraints play pivotal roles that enable training the networks.

For Zoom information, please email events@cs.stonybrook.edu.

AI is everywhere and so are the privacy concerns that come with it. At its core, the most common forms of AI we use today are online digital services and thus inherit the usual privacy risks. We'll take a look at indirect prompt injection- a technique that can trick AI tools into revealing or extracting private information as well as techniques being used in academic contexts to manipulate systems and even mislead researchers.

Register here for the online session.