Research challenges in using computer vision in robotics systems
Abstract
The past decade has seen a remarkable increase in the level of performance of computer vision techniques, including with the introduction of effective deep learning techniques. Much of this progress is in the form of rapidly increasing performance on standard, curated datasets. However, translating these results into operational vision systems for robotics applications remains a formidable challenge. This talk with explore some of the fundamental questions at the boundary between computer vision and robotics that need to be addressed. This includes introspection/self-awareness of performance, anytime algorithms for computer vision, multi-hypothesis generation, rapid learning and adaptation. The discussion will be illustrated by examples from autonomous air and ground robots.
CSE 600 Talk: Haibin Ling - Computer Vision Research and Applications
Abstract: Having been intensively studied over half a decade, computer vision has evolved as a broad research area and become mature in many applications. In this talk, we will summarize our work in computer vision in both core vision topics and application-oriented ones. In particular, for core vision problems, we will report studies on visual tracking, visual matching and visual detection; for applications, we will describe our work on medical image analysis, intelligent transportation, smart projector systems and preliminary work on material property prediction.
Bio: Haibin Ling received the BS and MS degrees from Peking University in 1997 and 2000, respectively, and the PhD degree from the University of Maryland, College Park, in 2006. From 2000 to 2001, he was an assistant researcher at Microsoft Research Asia. From 2006 to 2007, he worked as a postdoctoral scientist at the University of California Los Angeles. In 2007, he joined Siemens Corporate Research as a research scientist. From 2008 to 2019, he worked as a faculty member of Temple University. In fall 2019, he joined the Department of Computer Science of Stony Brook University where he is currently a SUNY Empire Innovation Professor. His research interests include computer vision, augmented reality, medical image analysis, and human computer interaction. He received the Best Student Paper Award at the ACM UIST in 2003, and the NSF CAREER Award in 2014. He serves as Associate Editor for several journals including IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), Pattern Recognition (PR), and Computer Vision and Image Understanding (CVIU). He has served or will serve as Area Chair for CVPR 2014, 2016, 2019 and 2020.
Abstract: Having been intensively studied over half a decade, computer vision has evolved as a broad research area and become mature in many applications. In this talk, we will summarize our work in computer vision in both core vision topics and application-oriented ones. In particular, for core vision problems, we will report studies on visual tracking, visual matching and visual detection; for applications, we will describe our work on medical image analysis, intelligent transportation, smart projector systems and preliminary work on material property prediction.
Bio: Haibin Ling received the BS and MS degrees from Peking University in 1997 and 2000, respectively, and the PhD degree from the University of Maryland, College Park, in 2006. From 2000 to 2001, he was an assistant researcher at Microsoft Research Asia. From 2006 to 2007, he worked as a postdoctoral scientist at the University of California Los Angeles. In 2007, he joined Siemens Corporate Research as a research scientist. From 2008 to 2019, he worked as a faculty member of Temple University. In fall 2019, he joined the Department of Computer Science of Stony Brook University where he is currently a SUNY Empire Innovation Professor. His research interests include computer vision, augmented reality, medical image analysis, and human computer interaction. He received the Best Student Paper Award at the ACM UIST in 2003, and the NSF CAREER Award in 2014. He serves as Associate Editor for several journals including IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), Pattern Recognition (PR), and Computer Vision and Image Understanding (CVIU). He has served or will serve as Area Chair for CVPR 2014, 2016, 2019 and 2020.
Come learn of the exciting research being done across so many fields using AI! The recipients of AI3's seed awards will present their work in our showcase on November 17, 2025 and we would love to see you there!
The schedule is listed below.
Location: New Computer Science Room 120
Presenters will give a 5-minute talk with 2 minutes for Q & A.
The schedule is listed below.
Location: New Computer Science Room 120
Session 1 - 10:30 AM to 11:45
Kevin Reed, PI, Introducing the AI Techniques in Assessing the Future Changes of Extreme Precipitation and Associated Flood Risks
Co-PIs: Tangnyu Song, Ishrat Dollan
Consultant: Jayesh Rathi
Ruwen Qin, PI, AI-Assisted Analysis of Materials in Recycling Streams
Consultant: Vismay Vora
Giuseppe Gazzola, PI, Using AI to Investigate National Literatures: Italy, France, Spain 1733- 1794
Consultant: Jayesh Rathi
Joseph Lemelin, PI, IAE2^3: AI Ecologies
Co-PIs: Katherine Johnston, Aruna Balasubramanian, Matthew Salzano
Niranjan Balasubramanian, Co-PI, Molecular Foundations for Sustainability: Data Analytics for Sustainable Cellulose Scaffolding Modifications to Remediate Diverse Water Contamination Challenges
PI: Benjamin Hsiao, Co-PI: I. V. Ramakrishnan
Owen Rambow, PI,Achieving Common Ground Through Language and Vision in Mixed-Initiative Human-Machine Communication Via zoom
Co-PI Susan Brennan
Session 2 - 12:30 PM to 1:45
Jack McSweeney, PI, Developing Machine Learning Approaches to Classify Internal Waves
Consultant: Vismay Vora
Eric Josephs, PI, Learning Design Rules to Personalize Precision CRISPR Gene Therapies with Interpretable AI
Consultant: Deboparna Banerjee
Shyam Sharma, PI, Fostering Writing-to-Learn Skills through Critical AI Literacy: A Faculty Development and Student Support Program
Co-PIs: Rose Tirotta-Esposito, Christine Fena
Ritwik Banerjee, PI, A Pragmatic Approach to AI for Digital Media Integrity: Combating Complex Misinformation Through Fallacies and Propaganda
Co-PI: Ruobing Li
Ziyu Shu, Co-PI, Novel Clinical Applications of Deep Image Prior-based CT Image Reconstruction
PI: Xin Qian, Co-PIs: Tiezhi Zhang, Zhaozheng Yin
Prateek Prasanna, Co-PI, An Artificial Intelligence-Driven Clinical Decision Support Tool for the Management of Abdominal Aortic Aneurysm
PI: Apostolos Tassiopoulos, Co-PI's: Mary Saltz, Janos Hajagos, Tahsin Kurc
Presenters will give a 5-minute talk with 2 minutes for Q & A.
The Pittsburgh Supercomputing Center is pleased to present a Machine Learning and Big Data workshop.
This workshop will focus on topics including big data analytics and machine learning with Spark, as well as deep learning.
This will be an IN PERSON event hosted by various satellite sites, there WILL NOT be a direct to desktop option for this event. SBU's Institute for Advanced Computational Science (IACS) is one of those satellite sites!
Location: IACS Conference Room #2
Interested applicants must first have an ACCESS ID. If you don't have the ID, please visit this page to create one: ACCESS USER REGISTRATION.
Once you have an ACCESS ID, please login (see top right here) then register here.
As AI drives rapid change across professional fields, how do you bring these developments into your classroom? The CELT AI Panel Discussion will gather academic thought leaders to explore how generative AI is reshaping teaching, learning, and the knowledge students need for today's world. Our panelists will share practical strategies for integrating AI-related advancements into course content, highlight both opportunities and challenges, and discuss how educators can help students build critical thinking, ethical awareness, and hands-on experience with emerging AI technologies. Join us to examine how teaching can evolve alongside an AI-transformed society.
Register here.
Topic: AI Seminar: Owen Rambow
Time: Mar 17, 2021 10:00 AM Eastern Time (US and Canada)
Join Zoom Meeting
https://stonybrook.zoom.us/j/93614644178?pwd=MzJtVDJYYmU5T1dtMzJiUFMxb0x4dz09
Meeting ID: 936 1464 4178. Passcode: 965936
Natural Language Understanding and Semantic Parsing
(Partly joint work with former colleagues at Elemental Cognition)
Semantic parsing refers to the task of determining the propositional content of language: who did what to whom. It is part of the larger task of natural language understanding (NLU). I will start out by discussing what full NLU means, and argue that we are still far away, as a field, from solving full NLU, or even from knowing how to evaluate it.
In the second part of the talk, I will situate semantic parsing in the context of several other NLU subtasks. Typically, the target representation of semantic parsing uses an ontology (such as PropBank or FrameNet). Semantic parsing includes the subtasks of word sense disambiguation, argument detection, and argument role labeling. I will discuss choices among possible target ontologies. I will justify why we created a new ontology, Hector, based on FrameNet and the lexical resource NOAD, and explain some of its characteristics.
In the third part of the talk, I will present experiments we performed using transformer models. We obtain best results using a two-phase model, in which we first choose the frame, and then, given the frame, choose the arguments. We encode the problem for both tasks using indices in the sentence. While we develop the parser for our new ontology Hector, this approach also beats the state of the art for FrameNet and PropBank parsing.Biography: I am a professor in the Department of Linguistics at Stony Brook University with a joint appointment in IACS.
Time: Mar 17, 2021 10:00 AM Eastern Time (US and Canada)
Join Zoom Meeting
https://stonybrook.zoom.us/j/
Meeting ID: 936 1464 4178. Passcode: 965936
Natural Language Understanding and Semantic Parsing
(Partly joint work with former colleagues at Elemental Cognition)
Semantic parsing refers to the task of determining the propositional content of language: who did what to whom. It is part of the larger task of natural language understanding (NLU). I will start out by discussing what full NLU means, and argue that we are still far away, as a field, from solving full NLU, or even from knowing how to evaluate it.
In the second part of the talk, I will situate semantic parsing in the context of several other NLU subtasks. Typically, the target representation of semantic parsing uses an ontology (such as PropBank or FrameNet). Semantic parsing includes the subtasks of word sense disambiguation, argument detection, and argument role labeling. I will discuss choices among possible target ontologies. I will justify why we created a new ontology, Hector, based on FrameNet and the lexical resource NOAD, and explain some of its characteristics.
In the third part of the talk, I will present experiments we performed using transformer models. We obtain best results using a two-phase model, in which we first choose the frame, and then, given the frame, choose the arguments. We encode the problem for both tasks using indices in the sentence. While we develop the parser for our new ontology Hector, this approach also beats the state of the art for FrameNet and PropBank parsing.Biography: I am a professor in the Department of Linguistics at Stony Brook University with a joint appointment in IACS.
Until recently, I was a research scientist at Elemental Cognition. Elemental Cognition is working on deep natural language understanding.
I got my PhD with Aravind Joshi at the University of Pennsylvania in 1994. I have worked at CoGenTex, and at AT&T Labs -- Research, and for many years I was a research scientist at Columbia University in the Center for Computational Learning Systems.
You are cordially invited to attend the biweekly Brookhaven AI Mixer (BAM). BAM includes one short talk on AI research happening at BNL, followed by an open mixer over coffee and snacks for everyone to network and discuss all things AI. The first half hour will consist of presentations that will be available via ZOOM, and the second half hour will be for in person only networking.
Join us every other Tuesday at noon in CDSD's Training Room (building 725, 2nd floor) to learn about interesting AI methods and applications, engage with potential collaborators, prepare for pending FASST funding calls, and build a community of AI for Science at BNL.
Uncertainty-Aware Adaptation of LLMs for Protein-Protein Interaction Analysis
Abstract: Identification of protein-protein interactions (PPIs) helps derive cellular mechanistic understanding, particularly in the context of complex conditions such as neurodegenerative disorders, metabolic syndromes, and cancer. Large Language Models (LLMs) have demonstrated remarkable potential in predicting protein structures and interactions via automated mining of vast biomedical literature; yet their inherent uncertainty remains a key challenge for deriving reproducible findings, critical for biomedical applications. In this study, we present an uncertainty-aware adaptation of LLMs for PPI analysis, leveraging fine-tuned LLaMA-3 and BioMedGPT models. To enhance prediction reliability, we integrate LoRA ensembles and Bayesian LoRA models for uncertainty quantification (UQ), ensuring confidence- calibrated insights into protein behavior. Our approach achieves competitive performance in PPI identification across diverse disease contexts while addressing model uncertainty, thereby enhancing trustworthiness and reproducibility in computational biology. These findings underscore the potential of uncertainty-aware LLM adaptation for advancing precision medicine and biomedical research.
Biography: Sanket is a research staff member in the Applied Mathematics department within the Computing and Data Sciences Directorate at Brookhaven National Laboratory. Previously, he was the Amalie Emmy Noether Postdoctoral Fellow in the same department. He earned his Ph.D. in Statistics from Michigan State University.. Sanket's research interests span Bayesian statistics, uncertainty quantification (UQ), deep learning, Markov Chain Monte Carlo (MCMC), variational inference, and sparsity methods. He also focuses on dimensionality reduction, surrogate modeling, hybrid physical-data driven models, and active learning, with applications across climate science, materials science, and life sciences.
Location: CDS, Bldg. 725, Training Room
Join ZoomGov Meeting: https://bnl.zoomgov.com/j/1605691898?pwd=xC7GebG7Kvzxa4AjPSIxJw7e9IZtoY.1
Meeting ID: 160 569 1898
Passcode: 303888
Abstract:
Quantifying similarity is a central notion in science and data analysis, pervading everything from phylogenetic trees to the foundation of clustering. Unfortunately, despite being examined and applied for decades, traditional similarity and distance metrics have fundamental drawbacks. The key problem is that all of them are only defined over pairs of objects, so they scale quadratically when one tries to compare N objects. The present explosion in the amount of data available to us requires new ways to process information, and while some current algorithms can handle millions of points, we need alternatives applicable to billions. This is what motivated us to develop a new framework that can compare any number of objects at the same time. With this, we achieve an unprecedented linear scaling when comparing multiple objects. Here we will discuss the main properties of this formalism, along with its applications in drug design and to the analysis of Molecular Dynamics (MD) simulations. Our indices have proven to be incredibly versatile when applied to chemical space exploration and visualization, allowing us to rigorously quantify the chemical diversity of very large molecular libraries. This has led to the creation of several algorithms to sample important regions in chemical space, including a more efficient way of identifying the prevalence of activity cliffs. Additionally, our indices provide a convenient route to sample complex MD trajectories, allowing to identify representative structures very efficiently. Moreover, we can also cluster biological ensembles in a more robust way than with standard algorithms, which has led to our group's work on MDANCE, a very flexible and efficient open-source clustering module. Drop by if you want to know how we clustered one billion molecules!
Speaker:
Assistant Professor, Department of Chemistry and Quantum Theory Project
University of Florida, Gainesville
Website: https://quintana.chem.ufl.edu/
Location:
Laufer Center Lecture Hall 101
Please join University Libraries on March 29 at 1:00 via Zoom as we welcome Dr. Zhang, SUNY Empire Innovation Professor at SBU's Power Lab. This lab is pioneering the research of coordinated networked microgrids (NMs) that can possibly help to restore neighboring distribution grids after a major blackout. That these NMs hold promise to significantly enhance the day-to-day reliability of the power grids, we are proud to host Dr. Zhang as a member of our STEM Speaker Series. Registration required.
https://library.stonybrook.
You are cordially invited to attend the biweekly Brookhaven AI Mixer (BAM). BAM includes one short talk on AI research happening at BNL, followed by an open mixer over coffee and snacks for everyone to network and discuss all things AI. The first half hour will consist of presentations that will be available via ZOOM, and the second half hour will be for in person only networking.
Join us every other Tuesday at noon in CDSD's Training Room (building 725, 2nd floor) to learn about interesting AI methods and applications, engage with potential collaborators, prepare for pending FASST funding calls, and build a community of AI for Science at BNL.
Embodied Intelligence at Scientific User Facilities
Abstract: This presentation explores the active work integrating artificial intelligence and robotics at the National Synchrotron Light Source II, and a perspective for the future. Through various case studies, we highlight the optimization of operations, improved experimental outcomes, and the orchestration of distributed multimodal experiments. This ongoing development includes collaborators from across the light and neutron sources in the DOE complex. We will elaborate on the open-source Bluesky project, and its capabilities to support adaptive and autonomous experiments. Additionally, we will discuss how Bluesky can be integrated with open-source robotic control software to unlock new flexible automation for autonomous scientific research, which scales to new experiments and continues to leverage human ingenuity.
Biography: Dr. Phillip M. Maffettone is an Associate Computational Scientist in the Data Science and Systems Integration Division at NSLS-II. His research focuses on accelerating scientific discovery at user facilities through the integration of robotics, artificial intelligence (AI), and advanced experiment orchestration systems. He leads the N3XTware project, constructing the software architecture for the next 12 beamlines to be built at NSLS-II. Prior to this he built the brain on the world's first mobile robotic scientist at the University of Liverpool, and later spearheaded the machine learning platform for a biotechnology start-up, BigHat Biosciences. He holds a DPhil in Inorganic Chemistry from the University of Oxford and a B.S. in Chemical Engineering from the University at Buffalo.
Location: CDS, Bldg. 725, Training Room
Link: https://bnl.zoomgov.com/j/16049713 31?pwd=nc5CV3cOFrdYxordFieP W07tIDmwYb.1
Meeting ID: 160 497 1331
Passcode: 289875