Abstract: AI agents are increasingly deployed in complex, interactive environments, yet their runtime remains a major bottleneck for training, evaluation, and real-world use. Typical agent behavior unfolds sequentially, where each action requires an API call that can incur substantial latency. For example, a game of chess between two state-of-the-art agents can take hours. We introduce speculative actions, a lossless acceleration framework for general agentic systems. Inspired by speculative execution in microprocessors and speculative decoding in LLM inference, our method uses faster models to predict likely future actions and executes them in parallel, committing only when predictions match. We evaluate speculative actions across gaming, e-commerce, and web search environments, and additionally study a lossy extension in an operating systems setting. Across domains, we achieve up to 55% next-action prediction accuracy, translating into substantial latency reductions. Finally, we present a cost-latency analysis that formalizes the tradeoff between speculative breadth and time savings. This analysis enables principled tuning and selective branch launching, to ensure multi-branch speculation delivers practical speedups without prohibitive cost growth.

Speaker: Kehan Qi

Location: Old Computer Science Building - CS2311
Research challenges in using computer vision in robotics systems Abstract The past decade has seen a remarkable increase in the level of performance of computer vision techniques, including with the introduction of effective deep learning techniques. Much of this progress is in the form of rapidly increasing performance on standard, curated datasets. However, translating these results into operational vision systems for robotics applications remains a formidable challenge. This talk with explore some of the fundamental questions at the boundary between computer vision and robotics that need to be addressed. This includes introspection/self-awareness of performance, anytime algorithms for computer vision, multi-hypothesis generation, rapid learning and adaptation. The discussion will be illustrated by examples from autonomous air and ground robots.
Abstract: Generative image models trained on massive datasets encode the statistics of our visual world. An off-the-shelf diffusion or flow-matching model can therefore serve as a general-purpose image prior, providing information about images across a wide range of tasks. Harnessing these capabilities, however, requires combining the model at inference time with additional signals or constraints unknown during training. This thesis will develop training-free methods for inference under a generative denoising prior. I first show how sampling from a trained denoiser can be formulated as an optimization problem, which combined with a constraint, can solve tasks ranging from conditional generation and weakly supervised segmentation to combinatorial optimization. I then demonstrate how the inherent properties of denoisers can accelerate inference under such constraints. Replacing gradient descent with an inexact Newton update, based on the symmetry of the denoiser's Jacobian, substantially reduces inference costs without tradeoffs. I also explore a middle ground between model adaptation and fully training-free inference by using the denoiser's robust internal representations to learn constraints from limited labeled data. These methods are applied to gigapixel image domains such as digital histopathology and remote sensing, where generative models can only be trained at a patch scale. To synthesize arbitrarily large images at resolutions unseen during training, I introduce inference-time algorithms that enforce consistency across spatially overlapping patches and image scales. Finally, I propose repurposing inference-time algorithms from sampling tools, to mechanisms for understanding the prior learned by a denoising model. Extending the previously developed techniques, I analyze the Jacobians of generative denoisers, where preliminary results indicate that Jacobian spectra correlates with generative quality. This motivates a Jacobian-spectrum regularization as a way to improve model performance using insights derived from inference-time algorithms.

Speaker: Alexandros Graikos

Location: NCS 220
Abstract: Foundation models are rapidly becoming capable assistants for knowledge work, but their deployment in real settings is limited by three gaps: they do not transfer reliably across environments, their internal reasoning is opaque, and their behavior is hard to control precisely. In this talk, I argue that these limitations are not only about model size -- they are fundamentally about whether learning captures and leverages the underlying structure of the data-generating process. I use causal thinking as a practical lens to model what is invariant, what changes, and what can be intervened on, and I further show how this leads to learning principles that improve trustworthiness.
I will first present methods for learning unifying mechanisms from heterogeneous data, across domains and modalities, to enable reliable transfer and controllable generation. Next, I will show how structured concepts can be recovered even from seemingly unstructured data, by analyzing and improving self-supervised objectives (such as masking and diffusion) through hierarchical latent-variable models. These concept structures can then be used to interpret generative models and support targeted, multi-level edits. Finally, I connect these two threads to generalization beyond the training distribution. I will discuss natural conditions for extrapolation and a compositional generation framework that improves prompt following for novel concept combinations. I will conclude with a brief outlook on self-improving world models and AI-assisted scientific discovery.


Bio: Lingjing Kong is a Ph.D. candidate in the Computer Science Department at Carnegie Mellon University. His research focuses on Causal AI for transferable, interpretable, and controllable systems, with an emphasis on understanding and exploiting the structure of real-world data to make foundation models actionable and more reliable. He develops identification principles and scalable algorithms for learning unified models from heterogeneous data, uncovering hierarchical concept structures in unstructured data (e.g., images and text), and generalizing beyond training support through compositionality and extrapolation. His work has appeared in top ML venues including ICML, NeurIPS, CVPR, ICLR, and EMNLP and has been prototyped and applied in industry.

Location: NCS 120
Ready to stop taking notes and make your virtual meetings work for you? Join our upcoming workshop to master Zoom's built-in AI tools and streamline your workday. You'll learn how to generate instant AI Meeting Summaries to capture key action items effortlessly, use live in-meeting Q&A to ask Zoom AI questions and catch up on missed details in real time, and prompt Zoom to build sleek, professional virtual backgrounds on the fly. Please click the link below to register. We hope to see you there!

https://stonybrookuniversity.co1.qualtrics.com/jfe/form/SV_a9PVlBw0E1Bal1A?source=campus%20announcements&session=100910261600zoom
The overall purpose of this seminar is to bring together people with interests in Computer Vision theory and techniques and to examine current research issues. This course will be appropriate for people who already took a Computer Vision graduate course or already had research experience in Computer Vision. To enroll in this course, you must either: (1) be in the Ph.D. program or (2) receive permission from the instructors. Each seminar will consist of multiple short talks (around 15 minutes) by multiple students. Students can register for 1 credit for CSE656. Registered students must attend and present a minimum of 2 talks. Registered students must attend in person. Up to 3 absences will be excused. Everyone else is welcome to attend. The seminar will be taught by Prof. Chao Chen, chao.chen.1@stonybrook.edu.

Understand Prompting the crucial part to interface with models

Discover how to prompt effectively by exploring the details behind your AI interactions. This isn't just about basic prompting; it's about understanding how to articulate your ideas clearly. We'll showcase a few prompts and how they work. Discover how giving AI the right details can truly boost your productivity and help you reclaim valuable time in your day.

In this session, you will

  1. Utilize AI models effectively
  2. Understanding different prompts
  3. Find out tips that we use with AI

Register: https://stonybrookuniversity.co1.qualtrics.com/jfe/form/SV_dht1o3rNzlZhHka?source=event+manager&session=0805251000ai

Abstract: Generative visual models like Stable Diffusion and Sora generate photorealistic images and videos that are nearly indistinguishable from real ones to a naive observer. However, their grasp of the physical world remains an open question: Do they understand 3D geometry, light, and object interactions, or are they mere pixel parrots of their training data? Through systematic probing, I will demonstrate that these models surprisingly learn fundamental scene properties--intrinsic images such as surface normals, depth, albedo, and shading (à la Barrow & Tenenbaum, 1978)--without explicit supervision, which enables applications like image relighting. But I will also show that this knowledge is insufficient. Careful analysis reveals unexpected failures: inconsistent shadows, multiple vanishing points, and scenes that defy basic physics. All these findings suggest these models excel at local texture synthesis but struggle with global reasoning: a crucial gap between imitation and true understanding. I will then conclude by outlining a path toward generative world models that emulate global and counterfactual reasoning, causality, and physics.

Bio: Anand Bhattad is a Research Assistant Professor at the Toyota Technological Institute at Chicago. He earned his PhD from the University of Illinois Urbana-Champaign in 2024 under the mentorship of David Forsyth. His research interests lie at the intersection of computer vision and computer graphics, with a current focus on understanding the knowledge encoded in generative models. Anand has received Outstanding Reviewer honors at ICCV 2023 and CVPR 2021, and his CVPR 2022 paper was nominated for a Best Paper Award. He actively contributes to the research community by leading workshops at CVPR and ECCV, including Scholars and Big Models: How Can Academics Adapt? (CVPR 2023), CV 20/20: A Retrospective Vision (CVPR 2024), Knowledge in Generative Models (ECCV 2024), and How to Stand Out in the Crowd? (CVPR 2025). For more details, visit https://anandbhattad.github.io/