The Natural Language Processing Reading Group at Stony Brook University meets weekly to discuss recent research papers in NLP and related fields. Join the Google Group here.

Don't Just Fix it in Post: A Science of AI Must Study Training Dynamics


Abstract: What would it mean to have a scientific understanding of AI? Models are not static objects: they are snapshots of time-evolving processes shaped by data, objectives, architectures, and optimization dynamics. Yet much of AI research treats models as fixed artifacts, analyzing behaviors after training rather than asking why they emerge. This position paper argues that a science of AI must move beyond post hoc fixes and study the training dynamics that produce model behavior. Such a science should support progressively stronger forms of understanding: predicting outcomes from early training signals, intervening when trajectories go wrong, and ultimately designing training procedures that more reliably produce desired properties. Scaling laws have made prediction routine for loss; the challenge is extending this success to capabilities, biases, robustness, and safety-relevant behaviors. We articulate requirements for such theories grounded in the history and philosophy of science, examine progress in mechanistic interpretability, fairness, memorization, and simplicity bias, and identify concrete open problems.
Location: NCS 220

Zoom Link: https://stonybrook.zoom.us/j/92942833294?pwd=ysKMaQx7Hp0lkq5nOb3zJ9ivNZVvLv.1
Johannes Hachmann, University of Buffalo Assistant Professor of Chemical Engineering presents Making Machine Learning Work in Chemistry

The use of modern machine learning, informatics and data mining approaches is a relatively new development in the chemical and materials domain. These techniques have been exceedingly successful in other application fields, and since there is no fundamental reason why they should not have a similarly transformative impact on chemical and materials research, there is now a concerted effort by the community to introduce data science in this new context. However, adapting techniques from other application domains for the study of chemical and materials systems requires a substantial rethinking and redevelopment of the existing methods.

In this presentation, we will discuss our work on designing advanced, physics-infused neural network architectures, the fusion of unsupervised clustering with supervised regression for local ensemble models, active and transfer learning techniques, bootstrapping approaches to minimize our training data footprint, methods to increase the applicability domain of data-derived models and automated hyperparameter optimization.

Biosketch: Johannes Hachmann is an Assistant Professor of Chemical Engineering at the University at Buffalo (UB), the Director of the Engineering Science in Data Science graduate program, a Core Member of the UB Computational and Data-Enabled Science and Engineering graduate program, and a Faculty Member of the New York State Center of Excellence in Materials Informatics. He earned a Dipl.-Chem. degree (2004) after undergraduate studies at the universities of Jena and Cambridge, M.Sc. (2007) and Ph.D. (2010) degrees in Chemistry from Cornell University, and he conducted postdoctoral research at Harvard University before joining the UB faculty in 2014. The research of the Hachmann Group fuses (first-principles) molecular and materials modeling with virtual high-throughput screening and modern data science (i.e., the use of database technology, machine learning and informatics) to advance a data-driven discovery and rational design paradigm in the chemical and materials disciplines. One of the centerpieces of the group's efforts is the creation of an open, general-purpose software ecosystem for the data-driven design of chemical systems and the exploration of chemical space. This work was recognized with a 2018 NSF CAREER Award.

Abstract: How do humans learn the sound patterns of their language? Despite a variety of methods and advances in phonotactic learning, there is still a paucity of computational research, methods and data for languages with tones. In this talk, I will explore this question specifically in light of tone languages, where pitch plays a crucial role in distinguishing words' meaning. I provide an implementation of the Bottom-Up Factor Inference Algorithm over Autosegmental Representations (BUFIA-AR), which learns the rules governing possible tone patterns. Using a dataset of Hausa, a West African tone language, the algorithm successfully identifies patterns that are not permitted in the language. These results (i) confirm long-standing linguistic generalizations, (ii) make more specific predictions about exceptional cases, and (iii) reveal previously unnoticed patterns. The results show how mathematical models of sound structure can be brought into dialogue with both linguistic theory and computational learning, highlighting the broader potential of formal approaches to capture human linguistic knowledge.

Bio: Han Li is a fifth-year Ph.D. student in Linguistics department, specializing in computational linguistics under the supervision of Professor Jeff Heinz. Her research focuses on how sound patterns in language can be formally represented and computationally learned, bridging theoretical linguistics and computer science.

Location: Institute for Advanced Computational Science, Seminar Room

Zoom Meeting: https://stonybrook.zoom.us/j/94043459206?pwd=3ra47h8HghOFRfobRBjZaDMyTwialr.1
Meeting ID: 940 4345 9206
Passcode: 332717

Abstract: The remarkable success of large foundational models, such as LLMs and diffusion models, is built on their learning over vast amounts of static data from the Internet. However, human learning and problem-solving are fundamentally interactive processes--humans learn by engaging with their environment, tools, search engine, and feedback loops, iteratively refining their understanding and decisions. This gap between the interactivity of human learning and the static nature of model training raises a critical question: how can we imbue foundational models with the capacity for meaningful interaction?

In this talk, I will explore methods to enhance foundational models by incorporating interaction with the external environment. I will discuss strategies such as leveraging external tools, compilers, function calls to provide dynamic feedback to enhance foundation models. By drawing inspiration from human's interactive learning processes, I demonstrate how interaction-driven learning can lead to models that are not only more accurate but also more adaptable to real-world applications.

This work bridges the gap between static training paradigms and the dynamic, iterative nature of human intelligence, paving the way for a new generation of interactive AI systems.

Bio: Wenhu Chen has been an assistant professor at the Computer Science Department in University of Waterloo and Vector Institute since 2022. He obtained the Canada CIFAR AI Chair Award in 2022 and CIFAR Catalyst Award in 2024. He has worked for Google Deepmind as a part-time research scientist since 2021. Before that, he obtained his PhD from the University of California, Santa Barbara under the supervision of William Wang and Xifeng Yan. His research interest lies in natural language processing, deep learning and multimodal learning. He aims to design models to handle complex reasoning scenarios like math problem-solving, structure knowledge grounding, etc. He is also interested in building more powerful multimodal models to bridge different modalities. He received the Area Chair Award in AACL 2023, the Best Paper Honorable Mention in WACV 2021, the Best Paper Finalist in CVPR 2024, and the UCSB CS Outstanding Dissertation Award in 2021.
Abstract : Humans reason about everyday situations by making commonsense-based inferences, derived both from explicitly stated information and implicit, unstated knowledge. In this thesis, I investigate whether NLP models have different aspects of causal knowledge about events and how to improve their understanding of narratives and plans.
Answering questions about why people perform actions in a narrative can test whether NLP systems contain and can effectively apply causal knowledge about events. I introduce TellMeWhy, a dataset concerning why characters in short narratives perform the actions described. An evaluation of then SOTA finetuned models show that they are far worse than humans. To improve models, it is important to understand what aspects of causal knowledge they need and how to best use external sources to inject this knowledge. In KnowWhy, I analyze different ways of injecting knowledge into models, which is difficult since we do not know apriori what type of knowledge will be needed to answer a question, hence requiring a ranking model to pick the most important inference. Results show that this retrieved knowledge helps models of all sizes, thereby improving their understanding of narratives.
Next, I study whether models can reason about causal aspects of plans. I focus on testing whether they understand the underlying causal dependencies reflected in the temporal order of a plan's steps. I introduce CAT-Bench, and find that SOTA models are underwhelming, and that model answers are not consistent across questions about the same step pairs. In their current state, these models cannot yet reliably be used for complex user-facing tasks. I then measure contemporary models' ability to perform user-facing and user-centric plan customization. I introduce the use of semi-symbolic edits in large language model (LLM) based agents and test several multi-LLM-agent architectures for plan customization. While LLMs still lack the ability to understand complex customization hints, my results suggest that LLM-based architectures may be worth exploring further for other customization applications. Finally, I distill complex reasoning capabilities into small language models (SLMs) using synthetic data that reflects a decomposition-then-editing process for plan customization. I demonstrate that explicitly teaching this latent causal reasoning significantly improves the quality of SLM-generated customizations. Overall, my work has improved how well NLP models understand complex reasoning associated with events in different contexts.

Speaker: Yash Kumar Lal

Location: NCS 220 or Zoom https://stonybrook.zoom.us/j/95849648243?pwd=dgPpZtDpgwQrK9z1SaPpNbBifaorzk.1

University Libraries Presents

What new tools and extra powers are available to you through the library's subscription databases? Join faculty librarian Chris Kretz, Head of Academic Engagement, on a tour of the AI landscape from our database vendors.

Location: Melville Library, Central Reading Room, Lab A, or Online.

Register here to join.

Abstract: Recent work in NLP uses debates between multiple LLMs to arrive at a more accurate conclusion. Earlier chain-of-thought prompting also shows improvements in accuracy when the model is asked to provide step-by-step reasoning in its response. Many publications since have developed strategies to improve the reasoning of model output with the goal of generating a more accurate result. However, even when asked to provide problem solving steps, the content of the reasoning provided by models is not well studied for all tasks and sometimes contains errors or conflicting statements even when the final result is correct. In fact, when evaluated across reasoning tasks, evidence shows that LLMs are not learning how to reason but are instead mimicking relevant solutions from their training sets.
By studying and evaluating the argumentation that LLMs provide, we can determine factors that may benefit or hinder the model's ability to give a complete, cohesive, and thorough answer. While there are signs that LLMs pattern match, finding where, when, and why this fails is valuable, as there may be ways to help the model imitate solutions that are more relevant to the task it is attempting to solve. Determining when pattern matching is not enough could show an area of improvement for future generations of LLMs. This research may separately aid in work on human-(AI)agent and inter-agent interaction. Specifically, frameworks could be used to determine when and why other models or humans are convinced by LLM-generated responses and which argument methods cause other models to change their response. Our current research in systematic versus heuristic cues shows that large language models sometimes present systematic or heuristic reasoning patterns based on prompting. Future research aims to explore other methods of classifying argumentation.

Speaker: Kiera Gross

Joining link: https://meet.google.com/xae-ywpv-udo
The overall purpose of this seminar is to bring together people with interests in Computer Vision theory and techniques and to examine current research issues. This course will be appropriate for people who already took a Computer Vision graduate course or already had research experience in Computer Vision. To enroll in this course, you must either: (1) be in the PhD program or (2) receive permission from the instructors.

Each seminar will consist of multiple short talks (around 10 minutes) by multiple people. Students can register for 1 credit for CSE 656. Registered students must attend and present a minimum of 2 or 3 talks. Everyone else is welcome to attend. Fill in https://forms.gle/pCVXovgfMfQwGqG38 to subscribe to our mailing list for further announcement.
Abstract: Reward hacking, where a reasoning model exploits loopholes in a reward function to achieve high rewards without solving the intended task, poses a significant threat. This behavior may be explicit, i.e. verbalized in the model's chain-of-thought (CoT), or implicit, where the CoT appears benign thus bypasses CoT monitors. To detect implicit reward hacking, we propose TRACE (Truncated Reasoning AUC Evaluation). Our key observation is that hacking occurs when exploiting the loophole is easier than solving the actual task. This means that the model is using less effort than required to achieve high reward. TRACE quantifies effort by measuring how early a model's reasoning becomes sufficient to obtain the reward. We progressively truncate a model's CoT at various lengths, force the model to answer, and estimate the expected reward at each cut-off. A hacking model, which takes a shortcut, will achieve a high expected reward with only a small fraction of its CoT, yielding a large area under the reward-vs-length curve. TRACE achieves over 65% gains over our strongest 72B CoT monitor in math reasoning, and over 30% gains over a 32B monitor in coding. We further show that TRACE can discover unknown loopholes during training. Overall, TRACE offers a scalable unsupervised approach for oversight where current monitoring methods prove ineffective.

Speaker: Dikshya

Location: Old Computer Science Building - CS2311