Abstract: Visual generation is a fundamental problem in computer vision and graphics, with applications ranging from 3D capture to image and video synthesis. Despite rapid progress in neural rendering and generative models, efficiency remains a key bottleneck: high-quality 3D reconstruction often relies on dense multi-view supervision; high-fidelity 3D synthesis requires costly optimization, training, and rendering; and modern image and video generators require substantial computation as the number of tokens grows rapidly for high-resolution generation. This dissertation focuses on efficient visual generation by improving sample efficiency in 3D reconstruction, representation efficiency in 3D generation, and computational efficiency in image and video synthesis. First, we improve the sample efficiency of neural implicit surface reconstruction. We integrate multi-view stereo probability volumes as a geometric regularizer, enabling high-quality sparse-view reconstruction from as few as three input images. Next, we introduce an explicit 3D representation for 3D generation, built from multi-view depth and RGB images with 3D Gaussian features. This design allows the model to use 2D generative priors while enforcing multi-view consistency through epipolar attention. We then address the computational bottleneck in image and video synthesis with importance-based token merging. Our method uses importance signals available during generation to preserve critical information while merging redundant tokens. Finally, we enable efficient mixed-resolution diffusion transformers via phase-aligned attention. This approach stabilizes attention under mixed-resolution token grids and unlocks high-fidelity image and video generation at reduced cost. Taken together, these contributions reduce the data requirements, representational overhead, and computational demands, thereby providing a foundation for high-quality, scalable, and efficient visual generation.

Speaker: Haoyu Wu

Location: NCS 120

Launching a University-Wide AI Innovation Institute:

Last spring, the Office of the Provost led a group of over 30 faculty, staff, and administrators to consider how we can expand and leverage our strengths in AI research and discovery. The resulting recommendation was to launch a university-wide AI Innovation Institute (AI3), which would expand the Institute for AI-driven Discovery and Innovation established in 2018 from a department-level institute within the College of Engineering and Applied Science (CEAS) to the university-wide AI Innovation Institute reporting to the provost.

As a university-wide enterprise, the AI Innovation Institute (AI3) is intended to accelerate, coordinate, and organize AI innovation and education across Stony Brook. The institute will serve to empower the entire university community and beyond, catalyzing core AI research, curriculum innovation, and societal change in the ever-evolving landscape of knowledge work.

The AI Town Hall, led by AI3 Interim Director Skiena, is an open house event that will provide an overview of the major AI initiatives on campus, including the new AI Seed Grant program and Stony Brook's role in New York State's Empire AI program. The session will include time for questions and discussion about the future of AI at Stony Brook.

You are cordially invited to attend the biweekly Brookhaven AI Mixer (BAM). BAM includes one short talk on AI research happening at BNL, followed by an open mixer over coffee and snacks for everyone to network and discuss all things AI. The first half hour will consist of presentations that will be available via ZOOM, and the second half hour will be for in person only networking.

Join us every other Tuesday at noon in CDSD's Training Room (building 725, 2nd floor) to learn about interesting AI methods and applications, engage with potential collaborators, prepare for pending FASST funding calls, and build a community of AI for Science at BNL.

Abstract: The increasing complexity and volume of data from electron microscopy necessitates advanced computational tools for timely and accurate analysis. In this talk, I will present several machine learning (ML) models developed to interpret diverse datasets from transmission electron microscopy (TEM). First, I demonstrate segmentation models for labelling regions of interest from in situ TEM images, such as atomic column positions or reaction sites that allow atomic-level quantitative analysis of data. Second, I introduce a self-supervised CNN model for denoising of low-dose HRTEM images, enabling clearer visualization of atomic features without sacrificing temporal resolution. Finally, a transformer-based model trained to predict copper oxidation states directly from their electron energy loss spectroscopy spectra will be introduced. Together, these projects showcase the power of tailored ML solutions to extract quantitative insights from complex microscopy data.

Biography: Brian Lee is a research associate working for the Electron Microscopy group and Theory and Computation group at the Center for Functional Nanomaterials. Previously, he has received PhD in Mechanical Engineering from Duke University and worked as a postdoc at Purdue University. His research focuses on applying machine learning and simulation techniques for materials science.

Location: CDS, Bldg. 725, Training Room

Join ZoomGov Meeting: https://bnl.zoomgov.com/j/1604383624?pwd=ffQ5cUPNxTI7nzClKQO6cnsNbhF9Vf.1

Meeting ID: 160 438 3624
Passcode: 558449


Abstract: Pre-trained diffusion and flow matching models have made visual generation remarkably powerful, enabling high-fidelity synthesis of images and videos from natural language prompts. However, their behavior is still largely dictated by the pre-training data distribution and likelihood objective, which do not directly encode downstream desiderata such as fine-grained semantic alignment, controllability, or realism. This gap motivates post-training: starting from a base generator and further optimizing it with additional supervision signals derived from human or reward model preferences.This work presents post-training for visual generative models through two complementary case studies. First, Hummingbird addresses the problem of fine-grained contextual alignment in image-text-to-image generation. We introduce a multimodal context evaluator that scores the consistency between rich contextual descriptions and generated images, capturing fine-grained alignment beyond global CLIP similarity. By directly backpropagating these differentiable rewards through the diffusion sampler, Hummingbird substantially improves semantic faithfulness while preserving high visual quality.
Second, PISCES tackles post-training for text-to-video generation, where alignment is inherently semantic-spatio-temporal. We show that naive VLM-based rewards suffer from distributional mismatch and token-level misalignment, leading to reward hacking and suboptimal optimization. PISCES introduces a bi-objective, Optimal Transport (OT)-aligned reward module: distributional OT using Neural Optimal Transport to align text and video embedding distributions, and discrete, partial OT over a spatio-temporal cost matrix to capture semantic alignment at the token level. These rewards are integrated into both direct backpropagation and GRPO-style optimization to post-train state-of-the-art text-to-video generators. Together, Hummingbird and PISCES provide a unified view of how carefully designed visual reward models, coupled with OT-based representation alignment, can reliably improve the downstream behavior of pre-trained image and video generators.

Speaker: Minh Quan Le

Location: NCS 220

Zoom: https://stonybrook.zoom.us/j/94798224254?pwd=CFraer25qnpORbJ14aAVHRwaSJOjJM.1
Date: March 11, 2022
Time: 2:40PM EST

Title: Towards Scalable and Efficient Machine Learning as a Service (MLaaS)

Abstract:
Driven by the explosive growth of big data, the sustained advances of
Machine Learning (ML), and the fast evolving of computer system
techniques, the past few years have witnessed a surging demand for
Machine Learning as a Service (MLaaS). MLaaS is an emerging computing
paradigm that facilitates ML model design, training, inference serving
and provides optimized executions of ML tasks in an automated,
scalable, and efficient manner. In this talk, I will demonstrate how
to integrate ML algorithm research and system research in synergy to
address the pressing challenges in MLaaS. I will first share a story
about how our system experience led to a novel large batching
algorithm design that revolutionizes large-scale training. Then I will
tell another story about how our gradient compression algorithm
research helped us to discover overlooked critical features of modern
ML systems and thereby build a compression-aware distributed ML
system. I will also briefly discuss a promising future of harnessing
serverless computing for MLaaS model inference serving. I will
conclude my talk with a discussion of interdisciplinary research and
future plans.

Bio:
Dr. Feng Yan is an Assistant Professor of Computer Science and
Engineering at University of Nevada, Reno (UNR) and director of the
Intelligent Data and Systems Lab (IDS Lab). Dr. Yan received M.S. and
Ph.D. degrees in Computer Science from the College of William and Mary
and worked at Microsoft Research and HP Labs. Dr. Yan's research
bridges the fields of big data, machine learning, and systems. The
focus of his research is on developing methodologies and building
systems that are automated, high-performing, efficient, robust, and
user-centric. Some of his recent research topics include large-scale
distributed deep learning, machine learning as a service (MLaaS),
federated learning, AutoML, serverless computing, and broad topics in
cloud and HPC. Dr. Yan is also dedicated to interdisciplinary research
and has established fruitful collaborations with domain experts in
areas such as health, physics, geography, material science, mechanical
engineering, civil engineering, and innovated big data and AI-driven
approaches for these domains. Dr. Yan and his team are actively
publishing at the most prestigious venues in computer system area
(such as SOSP, SC, HPDC, USENIX ATC, EuroSys, FAST, VLDB, etc.) and
machine learning area (such as NIPS/NeurIPS, KDD, AAAI, etc.). Dr. Yan
and his students are the recipients of the Best Student Paper Award of
IEEE CLOUD 2018, the Best Paper Award of CLOUD 2019, and the Best
Student Paper Award of ITNG 2021. Dr. Yan is the recipient of the NSF
CAREER Award, the NSF CRII Award, the CSE Best Researcher Award, and
has been nominated for the Regents' Rising Researcher Award. Dr. Yan
serves as Social Media Chair of ACM SIGMETRICS. To learn more
information, please visit Dr. Yan's homepage:
https://www.cse.unr.edu/~fyan.
Time: May 5, 2022, Thursday, 02:00 PM Eastern Time (US and Canada)
Place: New Computer Science (NCS) Room 220, and Zoom

Zoom link: https://stonybrook.zoom.us/j/95948672934?pwd=d3ZDcUJkK3VweFBDVWhIVDhtaFU2Zz09
Meeting ID:  959 4867 2934
Passcode:  082036

Title:  Generative Adversarial Learning using Optimal Transport

Abstract: 

Generative Adversarial Learning (GAL) aims to learn a target distribution in an adversarial manner. A Generative Adversarial Network (GAN) is a concrete implementation of GAL using a discriminator and a generator that play a min-max game. GANs have been used in many machine learning and computer vision applications. However, GANs are known to be hard to train, mainly because a min-max saddle point optimization problem needs to be solved in GAL. In this thesis, I investigate several methods to improve generative adversarial learning using Optimal Transport (OT). 

Previous Wasserstein GANs (WGANs) do not compute the correct Wasserstein distance to train the discriminator. To address this problem, I propose WGAN-TS that uses the L1 transport cost and computes the correct Wasserstein distance to train the discriminator. To ensure the local convergence of WGANs, I propose WGAN-QC that adopts the quadratic transport cost. I prove that WGAN-QC not only computes the correct Wasserstein distance but also converges to a local equilibrium point. To compute the Wasserstein distance over the whole dataset, I propose to use Semi-Discrete Optimal Transport (SDOT) to match noise points and the real images during GAN training. To measure the quality of an SDOT map, I use the Maximum Relative Error (MRE) and the L_1 distance between the target distribution and the transported distribution obtained by an OT map. I propose statistical methods to estimate the MRE and the L_1 distance. I propose an efficient Epoch Gradient Descent algorithm for SDOT (SDOT-EGD). To deal with the 2D special case of GAL, I propose to use OT to learn 2D distributions. In particular, I adopt OT to match persistent diagrams in training a topology-aware GAN and learn density maps in the crowd counting task. Finally, I use OT and the topological maps of the crowd to improve the crowd counting performance and propose a topology-based metric to measure the quality of the crowd density maps.
Nam Nguyen

4-5pm, Dec 17 2020

https://stonybrook.zoom.us/j/94214254415?pwd=K1VoQml4cFdlVW51VW41dWtid2tJdz09



The molecular mechanisms and functions in complex biological systems
currently remain elusive. Recent high-throughput techniques, such as
next-generation sequencing, have generated a wide variety of
multiomics datasets that enable the identification of biological
functions and mechanisms via multiple facets. However, integrating
these large-scale multiomics data and discovering functional insights
are, nevertheless, challenging tasks. To address these challenges,
machine learning has been broadly applied to analyze multiomics. In
particular, multiview learning is more effective than previous
integrative methods for learning data's heterogeneity and revealing
cross-talk patterns. Although it has been applied to various contexts,
such as computer vision and speech recognition, multiview learning has
not yet been widely applied to biological data--specifically,
multiomics data. Therefore, we have developed a framework called
multiview empirical risk minimization (MV-ERM) for unifying multiview
learning methods (Nguyen, et al., PLoS Computational Biology, 2020).
MV-ERM enables potential applications to understand multiomics
including genomics, transcriptomics, and epigenomics, in an aim to
discover the functional and mechanistic interpretations across omics.
Based on MV-ERM, we have developed the following methods:
ManiNetCluster, Varmole and ECMarker.



(1) ManiNetCluster (Nguyen, et al., BMC Genomics, 2019) is a manifold
learning method which simultaneously aligns and clusters gene networks
(e.g., co-expression) to systematically reveal the links of genomic
function between different phenotypes. Specifically, ManiNetCluster
employs manifold alignment to uncover and match local and non-linear
structures among networks, and identifies cross-network functional
links. We demonstrated that ManiNetCluster better aligns the
orthologous genes from their developmental expression profiles across
model organisms than state-of-the-art methods. This indicates the
potential non-linear interactions of evolutionarily conserved genes
across species in development. Furthermore, we applied ManiNetCluster
to time series transcriptome data measured in the green alga
Chlamydomonas reinhardtii to discover the genomic functions linking
various metabolic processes between the light and dark periods of a
diurnally cycling culture;



(2) Varmole (Nguyen, et al., Bioinformatics, 2020) is an interpretable
deep learning method that simultaneously reveals genomic functions and
mechanisms while predicting phenotype from genotype. In particular,
Varmole embeds multi-omic networks into a deep neural network
architecture and prioritizes variants, genes and regulatory linkages
via biological drop-connect without needing prior feature selections.
With an application to schizophonia, we demonstrate that Varmole
provides an effective alternative for recent statistical methods that
associate functional omic data (e.g. gene expression) with genotype
and phenotype and that link variants to individual genes in population
studies such as genome-wide association study;



(3) ECMarker (Jin*, Nguyen*, et al., Bioinformatics, 2020) is an
interpretable and scalable machine learning model that predicts gene
expression biomarkers for disease phenotypes and simultaneously
reveals underlying regulatory mechanisms. Particularly, ECMarker is
built on the integration of semi- and discriminative- restricted
Boltzmann machines, a neural network model for classification allowing
lateral connections at the input gene layer. With application to the
gene expression data of non-small cell lung cancer (NSCLC) patients,
we found that ECMarker not only achieved a relatively high accuracy
for predicting cancer stages but also identified the biomarker genes
and gene networks implying the regulatory mechanisms in lung cancer
development.



Finally, we propose a novel multiview learning method, Malignomics, to
predict phenotypes from heterogeneous multi-omic features. Malignomics
will first align multi-omic features by deep manifold alignment onto a
common latent space, better predicting nonlinear relationships across
omics. This deep alignment aims to preserve both global consistency
and local smoothness across omics and reveal higher-order nonlinear
interactions (i.e., manifolds) among cross-omic features. Second, it
uses these manifold structures to regularize the classifiers for
predicting phenotypes. This manifold-regularization allows
highlighting cross-omic feature manifolds and prioritizing the
features and interactions for the phenotypes. The prioritized
multi-omic features will further reveal underlying phenotypic
functions and mechanisms and thus enhance the biological
interpretation of Malignomics. We will apply Malignomics to
multi-omics data in neuropsychiatric disorders, and prioritize gene
regulatory networks linking risk variants, regulatory elements, and
genes for the disorders. We will also compare Malignomics with the
state-of-the-arts, and investigate how the manifold regulation will
potentially improve understanding of multi-omics functions and
predicting diseases.
Abstract: Humans perceive the world through global structures such as parts, branches, and their spatial arrangement. Most deep learning models, however, operate mainly at the pixel level. This disconnect between local and global understanding limits interpretability and control. In this thesis, we explore topology as a mathematical framework for bridging local predictions and global structure in dense prediction and generation tasks. We first incorporate topological constraints into semantic segmentation to preserve anatomical relationships and improve multi-class consistency. We next develop structure-level uncertainty estimation, producing more interpretable and actionable measures of model error over branches and connections rather than isolated pixels. Then, we introduce a topology-guided diffusion framework for controllable image generation using structural attributes such as object count and connectivity. Finally, we extend image generation to the longitudinal task, where we aim to capture structural changes across timepoints. All these contributions together establish topology as a unifying interface for building dense prediction models that are structurally aware, interpretable, and controllable.

Speaker: Saumya Gupta

Location: NCS 220

Zoom: https://stonybrook.zoom.us/j/97950688136?pwd=NCa3XOsgIaMIsTVlQBQJ11n27NzL8s.1
Meeting ID: 979 5068 8136
Passcode: 941798