The Natural Language Processing Reading Group at Stony Brook University meets weekly to discuss recent research papers in NLP and related fields.
Join the Google Group here.

Memorization Dynamics in Knowledge Distillation for Language Models

Abstract: Knowledge Distillation (KD) is increasingly adopted to transfer capabilities from large language models to smaller ones, offering significant improvements in efficiency and utility while often surpassing standard fine-tuning. Beyond performance, KD is also explored as a privacy-preserving mechanism to mitigate the risk of training data leakage. While training data memorization has been extensively studied in standard pre-training and fine-tuning settings, its dynamics in a knowledge distillation setup remain poorly understood. In this work, we study memorization across the KD pipeline using three large language model (LLM) families (Pythia, OLMo-2, Qwen-3) and three datasets (FineWeb, Wikitext, Nemotron-CC-v2). We find: (1) distilled models memorize significantly less training data than standard fine-tuning (reducing memorization by more than 50%); (2) some examples are inherently easier to memorize and account for a large fraction of memorization during distillation (over ~95%); (3) student memorization is predictable prior to distillation using features based on zlib entropy, KL divergence, and perplexity; and (4) while soft and hard distillation have similar overall memorization rates, hard distillation poses a greater risk: it inherits 2.7 times more teacher-specific examples than soft distillation. Overall, we demonstrate that distillation can provide both improved generalization and reduced memorization risks compared to standard fine-tuning.

Location: NCS 220

Zoom Link: https://stonybrook.zoom.us/j/92942833294?pwd=927683
Objectives:
1. Explain the clinical radiology workflow, and highlight how AI is currently in use to impact each step
2. Describe how radiologists interact with the currently available tools, highlighting both positive andnegative examples
3. Offer a brief description of how these tools are approved, validated, and reimbursed
4. Explore the utility of cutting edge AI techniques in diagnostic radiology

Speaker:
Dr. David Payne, MD Neuroradiologist and Assistant Professor, Rush University Medical Centre

Remote Access:
Zoom: https://stonybrook.zoom.us/j/95617197636?pwd=KytzZ2pVRG9SZGpKZUtpNXJISjNjZz09
Meeting ID: 95617197636
Passcode: 924293

Instructors! Identify your students' progress in your course using Intelligent Agent reports in Brightspace.

The Intelligent Agent tool in Brightspace is a great way to view how your students are progressing within your course. This tool gives instructors the ability to choose from a wide array of criteria to determine a student's progress. In addition, the tool allows instructors to send automated emails to students who meet your report criteria for preventative outreach.

In this session, you will:

  1. Learn what Intelligent Agents are
  2. What reports you can generate
  3. How to create an agent
  4. How to schedule a report
  5. How to create automated emails
  6. How to view and export your reports

Register to attend.

Abstract:
Deep neural network (DNN) is a powerful tool for solving image processing and computer vision tasks such as image and video reconstructions, object recognition, and scene understanding, etc. However, DNN have been used for only the digital domain in the imaging pipeline, such as the feature extractor and classifier models after an image is captured and digitized. In this research, we propose a new framework called deep sensing. The proposed framework also models the analog layer to the neural network model and jointly optimizes the parameters in optics and sensor designs of a camera, as well as reconstruction and classification models by the same training strategy.

Bio:
Hajime Nagahara is a professor at D3 Center, Osaka University, since 2024. He received Ph.D. degree in system engineering from Osaka University in 2001. He was a research associate of the Japan Society for the Promotion of Science from 2001 to 2003. He was an assistant professor at the Graduate School of Engineering Science, Osaka University, Japan from 2003 to 2010. He was an associate professor in Faculty of Information Science and Electrical Engineering at Kyushu University from 2010 to 2017. He was a professor at Institute for Datability Science, Osaka University, from 2017 to 2024. He was a visiting associate professor at CREA University of Picardie Jules Verns, France, in 2005. He was a visiting researcher at Columbia University in 2007-2008 and 2016-2017. Computational photography and computer vision are his research areas. He received IPSJ Nagao Special Researcher Award in 2012, ICCP2016 Best Paper Runners-up, and SSII Takagi Award in 2016. He is a program chair for ICCP2019, General Chair for upcoming ACCV2026 Osaka, Associate Editor for IEEE Transactions on Computational Imaging in 2019-2022, Associate Editor for IEEE Transactions of Pattern Recognisiton and Machine Intelligence, and Director of Information Processing Society of Japan in 2022-2024.

Zoom: https://stonybrook.zoom.us/j/4742917886?pwd=YiW3AL5lvPlEaWrd221YfLzYhlgsRK.1
Abstract: Computational pathology has revolutionized cancer diagnosis and research through the analysis of digitized whole slide images (WSIs). However, their giga-pixel size creates two intertwined bottlenecks: computational inefficiency, as prohibitive GPU memory makes standard end-to-end (E2E) training infeasible, and label inefficiency, as expert annotation is tedious and expensive. This dissertation confronts both challenges through novel architectures, training paradigms, and self-supervised learning methods for efficient WSI analysis.
To improve computational efficiency, this dissertation first introduces a locally supervised learning paradigm that enables E2E training on entire WSIs by partitioning a network into gradient-isolated modules, circumventing the memory bottleneck of backpropagation. Second, it presents Prompt-MIL, a parameter-efficient fine-tuning framework training only a few prompts to guide large pre-trained models, reducing trainable parameters, memory, and training time. Third, this work proposes 2DMamba, the first intrinsic Mamba architecture that preserves the crucial 2D spatial structure of images, overcoming the spatial discrepancy in 1D models. Fourth, it presents Locally Bi-directional Mamba (LBMamba), whose hardware-aware local backward scan integrates bi-directional scanning into a single forward pass, improving the throughput-performance trade-off of Mamba models.
To improve label efficiency, this dissertation proposes a precise location based matching strategy for self-supervised dense contrastive learning, which allows a local patch in one augmented view to match multiple overlapping patches in another, producing more accurate correspondences and superior features for dense prediction tasks like segmentation and detection. Additionally, to better scale multi-channel cell imaging modalities, this dissertation introduces ChannelSFormer, a channel-agnostic vision transformer that disentangles spatial and channel-wise reasoning through divided attention and channel class token, enabling effective representation learning across variable channel configurations in both self-supervised and supervised settings.
In summary, this dissertation presents a holistic investigation into the efficiency bottlenecks in computational pathology. Through these combined contributions in model architecture, training paradigms, and self-supervised learning, this work establishes a more scalable, efficient, and powerful computational framework for analyzing giga-pixel pathology images.

Speaker: Jingwei Zhang

Location: NCS 220

Zoom: https://stonybrook.zoom.us/j/93175806292?pwd=xbtxnQyYGoThz5B1DyJxJxPF9lxiJE.1
Meeting ID: 931 7580 6292
Passcode: 314091
Time: May 5, 2022, Thursday, 02:00 PM Eastern Time (US and Canada)
Place: New Computer Science (NCS) Room 220, and Zoom

Zoom link: https://stonybrook.zoom.us/j/95948672934?pwd=d3ZDcUJkK3VweFBDVWhIVDhtaFU2Zz09
Meeting ID:  959 4867 2934
Passcode:  082036

Title:  Generative Adversarial Learning using Optimal Transport

Abstract: 

Generative Adversarial Learning (GAL) aims to learn a target distribution in an adversarial manner. A Generative Adversarial Network (GAN) is a concrete implementation of GAL using a discriminator and a generator that play a min-max game. GANs have been used in many machine learning and computer vision applications. However, GANs are known to be hard to train, mainly because a min-max saddle point optimization problem needs to be solved in GAL. In this thesis, I investigate several methods to improve generative adversarial learning using Optimal Transport (OT). 

Previous Wasserstein GANs (WGANs) do not compute the correct Wasserstein distance to train the discriminator. To address this problem, I propose WGAN-TS that uses the L1 transport cost and computes the correct Wasserstein distance to train the discriminator. To ensure the local convergence of WGANs, I propose WGAN-QC that adopts the quadratic transport cost. I prove that WGAN-QC not only computes the correct Wasserstein distance but also converges to a local equilibrium point. To compute the Wasserstein distance over the whole dataset, I propose to use Semi-Discrete Optimal Transport (SDOT) to match noise points and the real images during GAN training. To measure the quality of an SDOT map, I use the Maximum Relative Error (MRE) and the L_1 distance between the target distribution and the transported distribution obtained by an OT map. I propose statistical methods to estimate the MRE and the L_1 distance. I propose an efficient Epoch Gradient Descent algorithm for SDOT (SDOT-EGD). To deal with the 2D special case of GAL, I propose to use OT to learn 2D distributions. In particular, I adopt OT to match persistent diagrams in training a topology-aware GAN and learn density maps in the crowd counting task. Finally, I use OT and the topological maps of the crowd to improve the crowd counting performance and propose a topology-based metric to measure the quality of the crowd density maps.
2026 MIC Symposium on Multilingualism | Featured Talk

Searching for Common Ground in Human-AI Interaction (What Gen AI Can't Do)

Abstract: For many of us, the alien invasion of LLMs has been unsettling, at best. Who knew that monstrous quantities of text and compute power could simulate human discourse processing? But despite claims about emergent theory-of-mind in reasoning models, LLMs remain stochastic parrots: they do not collaborate, project co-presence, form conceptual pacts, accurately signal commitment to what they say, or accrue common ground. This lack of pragmatic competence is clear from our recent studies of narrative and referential communication by AI vs. humans.

Speaker: Susan Brennan is SUNY Distinguished Professor at Stony Brook University in Cognitive Science within the Department of Psychology, with affiliations in Computer Science and Linguistics. Her research uses eye-tracking, experimental, behavioral, and observational methods to study the psycholinguistics of communication (verbal, nonverbal, and multimodal), adaptive language use, and the human use of technology, including spoken dialogue systems. She currently directs the NSF-funded Research Traineeship project, Detecting and Addressing Bias in Data, Humans, and Institutions, a collaboration among PhD. programs in Cognitive Science, Social & Health Psychology, Computer Science, Applied Math & Statistics, Linguistics, Political Science, Economics, Africana Studies, and Women's Studies. This traineeship supports students' AI and bias-related convergent research projects that include assessing potential for LLVMs (Large Language-Vision Models) to act as interlocutors in spontaneous dialogue as well as collaborating with the Innocence Network about AI tools. She holds a Ph.D. in cognitive psychology from Stanford University, an M.S. in visual studies from MIT (from what is now known as the MIT Media Lab), and a B.A. from Cornell University in cultural anthropology. She has worked in industry at Atari Research, Hewlett-Packard Labs, and Apple Computer, as well as at NSF as a program officer for the Graduate Research Funding Program.

Faculty Response Panel:
  • Jiwon Hwang (Panel Chair; Assistant Professor, Asian and Asian American Studies)
  • Rose Tirotta-Esposito (Director, Center for Excellence in Learning and Teaching)
  • Lav Varshney (Director, AI Innovation Institute)
  • Josh Zhang (Associate Professor, Sociology and Institute for Advanced Computational Science)




Location: Humanities 1006
How Language Makes us Smart (without Big Data) presented by Charles Yang

Abstract: Language provides the glue that combines simpler concepts into complex ones. To study how language guides conceptual development, we need precise accounts of how rules are learned from the child's linguistic experience, which is extremely limited in comparison to the amount of data available to current machine learning methods. In this talk, I discuss a mathematical model of inductive generalization, which enables language learning with very small amount of data. Such a view of learning has strong implications for the cross-cultural/linguistic variation of development. As a case study, I show that Hong Kong children learning Cantonese, which has a relatively simpler formal counting system, develop understanding of symbolic numbers a full year ahead of English-learning children in the United States, which is precisely predictable from the learning model. The new conception of learning adds another wrinkle to the eternal question of how language and thought are related to each other.

Bio: Charles Yang studied at the MIT AI lab and now teaches linguistics, computer science and psychology and directs the Program in Cognitive Science at the University of Pennsylvania. He is the author of several books: The Price of Linguistic Productivity (2016 MIT Press) won the Leonard Bloomfield Award from the Linguistic Society of America. His honors include a Guggenheim fellowship.