Hieu Le presents Incorporating Physical Illumination Constraints into Deep Learning Shadow Detection and Removal (PhD Proposal)

Shadows provide useful cues to analyze the scene but also hamper many computer vision algorithms such as image segmentation, object detection or tracking. For those reasons, shadow detection and shadow removal have been well studied topics in computer vision. Early approaches for shadow detection and removal focus on physical illumination models of shadows. These methods can express, identify, and remove shadows in a physically plausible manner. However, these models are often hard to optimize and slow in inference due to reliance on hand-designed image features. On the other hand, recent deep-learning approaches have achieved breakthroughs in performances for both shadow detection and removal. They learn to extract useful features automatically through training while being extremely efficient in computation. However, these models are data-dependent, opaque and ignore the physical aspects of shadows.

We propose to incorporate physical illumination constraints into deep-learning frameworks. Thus the mapping learned by the deep-network closely follows the physics of shadows, enabling the network to systematically and realistically modify shadows in images. For shadow detection, we present a novel GAN framework in which the generator can generate realistic images with attenuated shadows that can be used to train a shadow detector. For shadow removal, we propose a method that uses deep-networks to estimate the unknown parameters for a shadow image formation model that removes shadows. The system outputs shadow-free images in high-quality with no image artifacts and achieves state-of-the-art shadow removal performance. Lastly, we propose a system trained without the need for any shadow-free images in which physical constraints play pivotal roles that enable training the networks.

For Zoom information, please email events@cs.stonybrook.edu.

The Department of AI and Society (AIS) at the University at Buffalo is hosting a two-day AI and Society Workshop focused on building AI systems by society, for society. This workshop brings together researchers and community organizers to explore how AI systems can be developed through meaningful collaboration across disciplines.

Topics include:

  • Labor and AI
  • Public services and AI
  • Community-centered AI systems
  • Intersections of humanities, social sciences, arts, and computing

The vision of UB's Department of AI and Society is to create a future where AI systems are built by society, for society. AIS centers community engagement at every stage of AI development through collaboration across disciplines and sectors. AIS was established with a $5 million grant from SUNY, and this workshop is made possible through that support.

Who Should Attend?

  • Researchers
  • Students
  • Community organizers
  • Practitioners interested in AI's societal impact

More about the event

Register here

As AI drives rapid change across professional fields, how do you bring these developments into your classroom? The CELT AI Panel Discussion will gather academic thought leaders to explore how generative AI is reshaping teaching, learning, and the knowledge students need for today's world. Our panelists will share practical strategies for integrating AI-related advancements into course content, highlight both opportunities and challenges, and discuss how educators can help students build critical thinking, ethical awareness, and hands-on experience with emerging AI technologies. Join us to examine how teaching can evolve alongside an AI-transformed society.

Register here.

Abstract: Recent studies have highlighted the vulnerability of Natural Language Processing (NLP) and Vision-Language Models (VLMs) to backdoor attacks, posing significant security risks. Understanding these attack strategies is crucial for assessing model robustness and developing effective defenses. This thesis proposal aims to investigate the vulnerability of language and vision-language models, analyze abnormal behaviors in backdoor-attacked models, and develop defense methods to enhance safety of modern machine learning models at deployment.


We investigate the internal mechanisms of backdoored NLP models, identifying a distinct attention focus drifting phenomenon, where trigger tokens hijack attention regardless of the input context. Through comprehensive qualitative and quantitative analysis, we provide insights into the underlying mechanisms that enable backdoor attacks. Building on these insights, we propose detection methods to differentiate backdoored models from clean ones, through inspecting both the attention distribution and the model predictions. To better understand the vulnerability, we develop advanced backdoor attack strategies targeting language models in classification tasks. For BERT variants, we introduce Trojan Attention Loss (TAL), a novel method that directly manipulates attention patterns to enhance backdoor effectiveness, ensuring stealth and robustness. Vision-Language Models have demonstrated strong performance in recent years. Yet their vulnerability is largely underexplored. We investigate advanced backdoor attack strategies on Vision-Language Models, focusing on image-to-text generation tasks. We demonstrate how backdoors can be embedded in complex multimodal tasks while maintaining semantic integrity under poisoned inputs. Additionally, we propose innovative techniques for injecting backdoors without requiring access to the original training data, expanding the feasibility of real-world attacks.

This proposal provides novel insights into the internal mechanisms of backdoored models, propose effective detection strategies, and develop advanced attack techniques that expose critical vulnerabilities. These findings underscore the urgent need for robust security measures to defend against emerging backdoor threats in deep learning models. The results have been published in top venues including ICLR, ECCV, NAACL, EMNLP, etc.

Speaker: Weimin Lyu


Zoom link: https://stonybrook.zoom.us/j/99880605139?pwd=cfWbRG6n9v3GXEa7OqvXa5cOp5eLBv.1
Meeting ID: 998 8060 5139
Passcode: 843302

Stony Brook University Libraries invites students, faculty, & staff to join a conversation about how AI is transforming the private sector workforce. As AI tools move from experimentation to everyday business use, companies are rethinking roles, skill sets, leadership, and long-term strategy. This discussion-based event will focus on the fast-paced changes and directions at tech companies and their possible impact. This event will be particularly relevant for students preparing for an AI influenced job market and how to position themselves for opportunities in a rapidly evolving professional landscape.

The discussion will be led by Tariq Khan, Senior Director of Private Cloud Solutions at Hewlett Packard Enterprise. Tariq is a technology leader and architect with experience across private cloud, hybrid cloud, and data center platforms. He is responsible for shaping the technology architecture and strategic direction of HPE's Private Cloud offerings across on premises and cloud integrated environments.

Light refreshments will be served.


Location: Melville Library, NRR, Learning Lab

You are cordially invited to attend the biweekly Brookhaven AI Mixer (BAM). BAM includes one short talk on AI research happening at BNL, followed by an open mixer. The first half hour will consist of presentations that will be available via ZOOM, and the second half hour will be for in person only networking.

We meet once a month at noon in CDSD's Training Room (building 725, room 2-124) to learn about interesting AI methods and applications, engage with potential collaborators, prepare for pending FASST funding calls, and build a community of AI for Science at BNL.

Multilayer machine-learning framework for screening catalytic activity and selectivity

Abstract: Machine learning (ML) studies based on quantum chemical datasets have emerged as a powerful tool for accelerating catalyst discovery. However, their potential applications remain limited by high costs, low data quality, and restricted reliability. Here, we report a multilayer binary classification framework (MLBC) for screening catalytic performance in multistep processes with ML models. Key features include low-cost synthetic data generation via kinetic Monte Carlo simulations, high-quality data that capture catalytic behavior under reaction conditions, robust treatment of imbalanced data distributions, and descriptor selection that enhances reliability and interpretability. Using carbon dioxide (CO 2 ) hydrogenation to methanol (CH 3 OH) on copper (Cu)-based catalysts as a case study, the MLBC framework outperforms conventional ML models by demonstrating high reliability and strong generalization in classifying systems with activity and methanol selectivity exceeding those of Cu. In addition, feature analysis reveals that control over the critical transition steps between competing pathways governs activity and selectivity in CO 2 hydrogenation.

In addition to our speaker, we will have a number of CDS staff in attendance with expertise in AI methods and applications including image analysis, foundation models development, and inverse problem solving.

Biography: Dr. Wenjie Liao is a research associate in the Chemistry Division at Brookhaven National Laboratory, where he develops computational and machine-learning approaches to understand and improve heterogeneous catalysts. He earned his Ph.D. from Stony Brook University. His research focuses on identifying active sites, mapping reaction pathways, and predicting catalyst performance, with the goal of accelerating the discovery of more efficient catalysts for energy and chemical transformations.

Location: CDS, Bldg. 725, Training Room

Join ZoomGov Meeting: https://bnl.zoomgov.com/j/1604383624?pwd=ffQ5cUPNxTI7nzClKQO6cnsNbhF9Vf.1

Meeting ID: 160 438 3624
Passcode: 558449

Please Note: Due to a funding shortfall, we are for the time being no longer able to provide pizza and sodas for these events. We will have coffee though, and all are of course welcome to bring their lunch.

Abstract: Human gaze behavior is a fundamental cue for understanding social intent, human-machine interaction, and cognitive processes. This thesis addresses the challenges of gaze target estimation (GTE), also known as gaze following, by developing a holistic understanding of gaze in complex environments.

The first part of this work improves GTE performance by introducing Patch-level Distribution Prediction (PDP). Unlike traditional models that rely on strict pixel-wise regression, PDP models gaze as a distribution over patches, which better accounts for annotation variance and bridges the gap between target location and in/out-of-frame prediction. To address the laborious nature of data labeling, the second part presents GCDR, the first semi-supervised method for gaze following. By prompting large Visual Question Answering (VQA) models to generate initial Grad-CAM heatmaps and refining them with a diffusion model, this method achieves high performance with significantly fewer human annotations. The third part expands the applicability of GTE to multi-camera environments. By introducing the Multi-View Gaze Target (MVGT) dataset, along with two novel frameworks for integrating information between multiple views and predicting the gaze target across views, we explore a new direction that overcomes single-view limitations such as face occlusion and out-of-view targets.

Building on these foundations, the final part of this thesis proposes a new direction toward semantic social gaze understanding using next-generation multimodal Large Language Models (LLMs). Rather than focusing solely on geometric gaze target localization, we aim to enrich gaze prediction with semantic and relational interpretation in complex social scenes. To this end, we will leverage existing gaze following datasets to derive social gaze supervision, including mutual gaze and shared attention, and obtain aligned language descriptions of scene-level gaze behaviors. This proposed work will enable the model to not only locate gaze targets but also predict structured social gaze relations among individuals, meanwhile generating a concise natural-language summary describing the dominant gaze interactions. By integrating spatial gaze estimation, social relation reasoning, and language-based scene understanding within a unified multimodal model, this work takes an important step toward a holistic understanding of human gaze behavior in real-world environments.

Speaker: Qiaomu Miao
18th Annual Engineering Ball Flowerfield, St. James, NY Thursday April, 2nd, 7:00 to 10:00 pm Pick up your tickets in 231 Engineering (Monday - Friday, 10:00 am to 4 pm) Presenting Partner: L3Harris

Abstract: Much like other AI for Science domains, polymer design poses significant challenges. It requires grounding in empirical data and physical laws, precise handling of domain-specific structured representations, and compositional reasoning over multiple interacting constraints--all while working with limited data.

To address these limitations, we introduce PolyBench, a large-scale benchmark comprising over 125K polymer design and analysis tasks grounded in verified experimental and synthetic data. PolyBench includes tasks created from a wide range of data sources and presents diverse structural, property-driven, and synthesis-oriented reasoning problems. Tasks in PolyBench are organized from simple to complex analytical reasoning problems, enabling generalization tests and includes diagnostic probes to evaluate model capabilities. Additionally, to support effective domain alignment, we propose a knowledge-augmented reasoning distillation framework that enriches the dataset with structured chain-of-thought supervision derived from expert-informed reasoning strategies.

Small language models (7B-14B parameters) trained on PolyBench substantially outperform comparably sized baselines and, in many cases, exceed the performance of larger closed-source frontier models on polymer reasoning tasks, while also demonstrating improved transfer to external polymer benchmarks. Last, we conduct a diagnostic study that reveals a compositionality gap: despite strong performance on decomposed sub-questions, models struggle to integrate multiple interacting constraints and intermediate reasoning steps, highlighting fundamental limitations in current scientific language models.

Speaker: Dikshya Mohanty

Location: NCS 115/Online

Zoom: https://stonybrook.zoom.us/j/94746001760?pwd=BCAd8gu7cXLn3PXM6kkbh11V6r0Mr7.1
Meeting ID: 947 4600 1760 Passcode: 987917

Climate Uncertainty, Decision Making, and AI for Earth System Predictability Dr. Nathan Urban, Brookhaven National Laboratory

Bio: Nathan Urban is the group leader of the Optimal Experimental Design & Uncertainty Quantification group in the Applied Mathematics Department at Brookhaven National Laboratory's Computing & Data Sciences directorate (CDS). He holds a Ph.D. in theoretical condensed matter physics from Penn State, and has previously held research positions at Los Alamos National Laboratory, Princeton, and Penn State. His research interests include Bayesian inference and spatiotemporal statistics, probabilistic prediction and forecasting, multi-model / model-form / model structural uncertainty quantification, reduced order modeling, scientific machine learning and hybrid physical-data driven modeling, in-situ/streaming data analysis at scale, information fusion, decision making under uncertainty and optimal experimental design, and integrated multiscale computational frameworks for decision support.

Location: IACS Seminar Room

Lunch will be provided