AI Innovation Institute

Virtual Job Fair for New Stony Brook Graduates & Experienced Alumni Using a platform called Career Fair Plus, participants will be able to schedule 10-minute video meetings with participating employers of interest to them. Recent graduates and alumni can register and learn more about how the fair will be run by registering on Handshake.

Title: Building foundation models for scientific data Seminar

Speaker: Ruben Ohana, Ph.D. and Michael McCabe, Ph.D - Flatiron Institute, New York

Abstract: Foundation models are very large architectures trained on large-scale datasets and can be used to transfer knowledge from a domain to another. Scientific data, particularly numerical simulations of partial differential equations (PDEs), presents unique challenges due to its complexity and the need for domain expertise to assess prediction quality, complicating the building of the first foundation models in this field. In this talk, we will develop our approach of building foundation models for scientific data, highlighting the requirements and expectations for achieving meaningful results. We will also introduce The Well, a comprehensive collection of datasets encompassing multi-scale simulations of fluid dynamics, astrophysics, and biological systems. The Well serves as a foundation for developing models that generalize across diverse physical phenomena, aiming to accelerate scientific discovery through large-scale learning.

Join Zoom Meeting: https://bnl.zoomgov.com/j/1606898802?pwd=GbbPiLGHlEokDskxjeFheMFWfuboxO.1
Meeting ID: 160 689 8802
Passcode: 281575

West Campus - SAC- Student Activities Center - Ballrooms A & B 100 Nicolls Road Stony Brook NY 11794 Job Fair.jpg The Career Center invites Alumni Employers and Job Seekers to the IT/Computer Science Job and Internship Fair this spring. Job Seekers: A job fair is an opportunity for you to present yourself professionally in person to a potential employer, while showcasing your communication skills. Get more information Alumni Employers: Held in both the fall and spring semesters, this event is ideal for employers looking to fill internship, co-op, part-time and full-time opportunities in the field of information technology (i.e. Software Engineering, Network Administration, Web Development, etc.). Register here to recruit top SBU talent.

Abstract:

Recent advances in deep learning have significantly enhanced the capabilities of Natural Language Processing (NLP) and Vision-Language Models (VLMs). However, these advancements come with increased vulnerabilities, notably through backdoor attacks that pose severe security threats. This thesis addresses two critical dimensions of Trustworthy AI and Efficient Multimodal Representation Learning: (1) security through analyzing, detecting, and designing backdoor attacks in NLP and VLMs, and (2) efficiency through advanced multimodal representation methods tailored for clinical and medical imaging applications.

In the first dimension, we explore the internal mechanisms exploited by backdoor attacks, identifying the distinctive phenomenon of attention focus drifting in compromised transformer models, where trigger tokens consistently hijack attention. Leveraging these insights, we propose robust detection frameworks, including the attention-based Trojan detector (AttenTD) and a task-agnostic logit-based detection method (TABDet), achieving effective identification of backdoored NLP models across diverse tasks. We further introduce novel backdoor attack methodologies: the Trojan Attention Loss (TAL), enhancing attack efficiency and stealth through direct attention manipulation, and BadCLM, demonstrating critical vulnerabilities in clinical decision-support systems by effectively compromising clinical language models.

Extending our security exploration to multimodal settings, we investigate backdoor attacks on Vision-Language Models (VLMs), particularly in complex image-to-text generation tasks, proposing innovative techniques (TrojVLM, VLOOD) capable of embedding backdoors without direct access to original training data, thus showcasing practical risks in real-world scenarios.

In the second dimension, we address efficiency and interpretability challenges in clinical and pathology applications. We introduce TCP-LLaVA, the first multimodal large language model (MLLM) designed explicitly for Whole Slide Image (WSI) Visual Question Answering (VQA). Utilizing a novel token compression mechanism inspired by transformer-based models, TCP-LLaVA substantially reduces computational resource consumption while maintaining superior VQA performance across multiple tumor subtypes. Additionally, we present a multimodal transformer model integrating structured Electronic Health Records (EHR) with clinical notes, demonstrating enhanced predictive accuracy and interpretability for in-hospital mortality prediction through integrated gradient-based interpretability methods.

Together, these contributions present a comprehensive approach to ensuring AI models are not only secure against malicious manipulation but also efficient and interpretable for critical clinical applications, underscoring the essential need for trustworthy and effective AI systems.

Speaker: Weimin Lyu

Zoom: https://stonybrook.zoom.us/j/2392326575?pwd=SVQ2VkFXTnZZYmJUMXgvTXBuZWM3UT09

Meeting ID: 239 232 6575
Passcode: 436192

The overall purpose of this seminar is to bring together people with interests in Computer Vision theory and techniques and to examine current research issues. This course will be appropriate for people who already took a Computer Vision graduate course or already had research experience in Computer Vision. To enroll in this course, you must either: (1) be in the PhD program or (2) receive permission from the instructors. Each seminar will consist of multiple short talks (around 15 minutes) by multiple students. Students can register for 1 credit for CSE656. Registered students must attend and present a minimum of 2 talks. Everyone else is welcome to attend. Fill in https://forms.gle/q6UG9ygauLp2a8Po8 to subscribe to our mailing list for further announcement.

When: Thu: 10/28/2021, 10 am
Where: NCS Room 220, or
Zoom: https://stonybrook.zoom.us/j/97978463739?pwd=aVJFVERQa25jYjJrOFZEcWVuSzJLdz09

Deep Surface MeshesPascal FuaEPFLGeometric Deep Learning has recently made striking progress with the advent of Deep Implicit Fields (SDFs). They allow for detailed modeling of watertight surfaces of arbitrary topology while not relying on a 3D Euclidean grid, resulting in a learnable 3D surface parameterization that is not limited in resolution. Unfortunately, they have not yet reached their full potential for applications that require an explicit surface representation in terms of vertices and facets because converting the SDF to such a 3D mesh representation requires a marching-cube algorithm, whose output cannot be easily differentiated with respect to the SDF parameters. In this talk, I will discuss our approach to overcoming this limitation and implementing convolutional neural nets that output complex 3D surface meshes while remaining fully-differentiable and end-to-end trainable. I will also present applications to single view reconstruction, physically-driven Shape optimization, and bio-medical image segmentation.

Bio:

Pascal Fua received an engineering degree from Ecole Polytechnique, Paris, in 1984 and a Ph.D. in Computer Science from the University of Orsay in 1989. He joined EPFL (Swiss Federal Institute of Technology) in 1996 where he is a Professor in the School of Computer and Communication Science and head of the Computer Vision Lab. Before that, he worked at SRI International and at INRIA Sophia-Antipolis as a Computer Scientist. His research interests include shape modeling and motion recovery from images, analysis of microscopy images, and Augmented Reality. He has (co)authored over 300 publications in refereed journals and conferences. He has received several ERC grants. He is an IEEE Fellow and has been an Associate Editor of IEEE journal Transactions for Pattern Analysis and Machine Intelligence. He often serves as program committee member, area chair, and program chair of major vision conferences and has cofounded three spinoff companies. 

Research challenges in using computer vision in robotics systems Abstract The past decade has seen a remarkable increase in the level of performance of computer vision techniques, including with the introduction of effective deep learning techniques. Much of this progress is in the form of rapidly increasing performance on standard, curated datasets. However, translating these results into operational vision systems for robotics applications remains a formidable challenge. This talk with explore some of the fundamental questions at the boundary between computer vision and robotics that need to be addressed. This includes introspection/self-awareness of performance, anytime algorithms for computer vision, multi-hypothesis generation, rapid learning and adaptation. The discussion will be illustrated by examples from autonomous air and ground robots.