Join the Google Group here.
Learning to Evict from Key-Value Cache
Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache. Existing eviction or compression methods reduce cost but rely on heuristics, such as recency or past attention scores, which serve only as indirect proxies for a token's future utility and introduce computational overhead. We reframe KV cache eviction as a reinforcement learning (RL) problem: learning to rank tokens by their predicted usefulness for future decoding. To this end, we introduce KV Policy (KVP), a framework of lightweight per-head RL agents trained on pre-computed generation traces using only key and value vectors. Each agent learns a specialized eviction policy guided by a holistic reward, derived from future utility, that evaluates the quality of the ranking across all cache budgets, requiring no modifications to the underlying LLM or additional inference. Evaluated across two model families on the long-context benchmark RULER (up to 128K tokens) and the multi-turn dialogue benchmark OASST2-4k, KVP significantly outperforms strong baselines. Zero-shot tests on standard downstream tasks (BoolQ, LongBench passage retrieval, GovReport) further show that KVP generalizes beyond its training distribution and to considerably longer sequence lengths. These results demonstrate that learning to predict future token utility is a powerful and scalable paradigm for adaptive KV cache management.
Location: NCS 220
Zoom Link: https://stonybrook.zoom.us/j/92942833294?pwd=ysKMaQx7Hp0lkq5nOb3zJ9ivNZVvLv.1&jst=2
Abstract: Advances in AI and machine learning have enabled new applications and services to interpret and process inputs in previously unthinkable complex environments. Autonomous cars, data analytics, adaptive communication and self-aware software systems are now revolutionizing markets by achieving or exceeding human performance. In this talk, I consider the evolving use of machine learning in security-sensitive contexts and explore why many systems are vulnerable to nonobvious and potentially dangerous manipulation. Here, we examine sensitivity in any application whose misuse might lead to harm--for instance, forcing adaptive network in an unstable state, crashing an autonomous vehicle or bypassing an adult content filter. I explore the use of machine learning in this area particularly in light of recent discoveries in the creation of adversarial samples and defenses against them and posit on future attacks on machine learning. The talk is concluded with a discussion of the technological and societal challenges we face as a result of current and future advances in intelligent computing.
Bio: Patrick McDaniel is the William L. Weiss Professor of Information and Communications Technology and Director of the Institute for Networking and Security Research in the School of Electrical Engineering and Computer Science at the Pennsylvania State University. Professor McDaniel is also a Fellow of the IEEE and ACM and the director of the NSF Frontier Center for Trustworthy Machine Learning. He also served as the program manager and lead scientist for the Army Research Laboratory's Cyber-Security Collaborative Research Alliance from 2013 to 2018. Patrick's research centrally focuses on a wide range of topics in computer and network security and technical public policy. Prior to joining Penn State in 2004, he was a senior research staff member at AT&T Labs-Research.
Yihui Ren
AI Department, Brookhaven National Laboratory
Abstract: Artificial intelligence is rapidly changing how we approach scientific discovery and engineering design. However, realizing the potential of modern AI in scientific domains requires going beyond simply applying models developed for natural language: scientific data have distinct structures, and scientific workflows often require tight integration among learned representations, domain knowledge, simulation, and numerical optimization.
In this talk, I will present two recent efforts that explore complementary directions for building AI systems for science and engineering. First, FM4NPP investigates whether the foundation-model paradigm can scale to experimental nuclear and particle physics. Using more than 11 million particle-collision events, we develop a self-supervised learning framework for sparse detector data and demonstrate scaling up to 188 million parameters. The resulting representations generalize across diverse downstream tasks and enable data-efficient adaptation with lightweight task-specific components.
Second, I will present AutoSizer, an LLM-driven framework for automatic sizing of analog and mixed-signal circuits. Rather than asking an LLM to perform numerical optimization directly, AutoSizer uses the LLM as a reasoning and orchestration layer that interprets circuit behavior, analyzes simulation feedback, and adaptively refines the optimization process.
Together, these studies illustrate two complementary roles for modern AI in scientific workflows: learning reusable representations from complex scientific data and reasoning over computational tools to guide challenging design and optimization problems. I will discuss the lessons from these efforts and opportunities toward more general, scalable, and reusable AI systems for science and engineering.
Short bio: Dr. Yihui (Ray) Ren is a Senior Computational Scientist in AI/ML and the AI-Codesign Group Lead in the Artificial Intelligence Department within Brookhaven National Laboratory's Computing and Data Sciences Directorate. His research focuses on AI for Science, AI-hardware codesign, real-time AI systems, and the evaluation of emerging AI hardware architectures. He also develops advanced AI methodologies for nuclear safeguards applications. Dr. Ren's key contributions include unpaired domain mapping techniques to mitigate domain shift, generative surrogate modeling, object detection and video synthesis using event-based cameras, and the development of AI foundation models such as the Foundation Model for Nuclear and Particle Physics (FM4NPP). Prior to joining Brookhaven National Laboratory in 2018, he was a postdoctoral researcher at Virginia Tech. He received his Ph.D. in Physics from the University of Notre Dame in 2015.
Location: NCS 120
Zoom: https://stonybrook.zoom.us/j/92507585369?pwd=9bw2hsQmLbladVpPRiaWK94gYlwCGQ.1&jst=2
Speaker: Shao-Yu Chang
Zoom: https://stonybrook.zoom.us/j/
What new tools and extra powers are available to you through the library's subscription databases? Join faculty librarian Chris Kretz, Head of Academic Engagement, on a tour of what's available and how you might best use them.
Join the Google Group here.
Bio: Valentina Pyatkin is a postdoctoral researcher at the Allen Institute for AI and the University of Washington, advised by Prof. Hanna Hajishirzi and Prof. Yejin Choi. Additionally, she is part- time affiliated with the ETH AI Center, where she mentors students and works on post-training for the Swiss AI Initiative. She obtained her PhD in Computer Science from Bar Ilan University. Her work has been awarded an ACL Outstanding Paper Award and the ACL Best Theme Paper Award, and has been supported by a Schmidt Sciences Postdoctoral Award. During her doctoral studies, she conducted research internships at Google and the Allen Institute for AI, where she received the AI2 Outstanding Intern of the Year Award.
Location: NCS 120