AI Innovation Institute

Abstract: Modern language agents often need to solve tasks requiring long-horizon, multi-turn interactions, where they retrieve external information, adapt to observations, and answer interdependent queries. Yet, most LLM systems rely on full-context prompting, appending all past turns regardless of their relevance. This leads to un-bounded memory growth, increased computational costs, and degraded reasoning performance on out-of-distribution input lengths due to LLM forgetting the context. We introduce MEM1, an end-to-end reinforcement learning framework that enables agents to operate with constant context size when solving long multi-turn tasks. At each turn, MEM1 updates a compact shared internal state that jointly supports memory consolidation and reasoning. Leveraging reinforcement learning (RL) and rollout trajectory truncation, we train a MEM1 agent to develop internal states that integrate prior memory with new observations from the environment while strategically discarding irrelevant or redundant information. Experiments across three domains, including internal retrieval QA, open-domain web QA, and multi-turn web shopping, show that MEM1-7B improves performance by 3.5x while reducing memory usage by 3.7x compared to Qwen2.5-14B-Instruct on an augmented multi-hop QA dataset with 16 objectives in each task, and generalizes beyond the training horizon. Our results demonstrate the promise of reasoning-driven memory consolidation as a scalable alternative to existing solutions for training long-horizon task-solving agents that involve multiple interactions, where both efficiency and performance are optimized.

Speaker: Yiyang Feng

Location: CS2311

The Tiger Team open house will be Monday, September 20 at high noon by Zoom:

 https://stonybrook.zoom.us/j/99577242078?pwd=T3IzekV6MkFZQXBrMXBSRDNNZ1lLdz09

We live in a new scientific paradigm: the Big Data era, in which a lot of data is available for almost anything. In this new paradigm, the driving force is to use data directly to learn about chemical and physics systems employing artificial intelligence. This paradigm has proven helpful in simulating realistic physical, biological, and chemical models, yielding impressive results. Similarly, the insight gained in these situations can be used to improve our understanding of fundamental processes. In that regard, we want to answer the question: Can a machine learn chemistry? The answer to this question is still debatable, but we will show our ideas and methods to find the answer. We will also discuss our results on predicting atom-diatom reactions and other avenues and work in progress in our group.

Please register for the STEM Speaker Series: Can a Machine learn Chemistry here.

The talk will be exclusively on zoom https://stonybrook.zoom.us/j/7851507944 Speaker: Sooyeon Lee, Rochester Institute of Technology Title: Design and Evaluation of Accessible AI Technologies for Users with Disabilities Abstract: Over one billion people in the world live with some type of disability. Many of them experience barriers in accessing information or using technologies, which can limit social interactions in both physical and digital spaces. In my research, I focus on investigating and designing nonvisual interaction for the community of blind users and non-audio and non-speech interaction for the community of deaf and hard of hearing users. In this talk, I will first present my research investigating nonvisual interaction prototypes for supporting shopping activities for blind users, with an exploration of one-way instructional and two-way conversational interactions and with a variety of form factors and communication modalities through the use of human-computer interaction research methodologies. I will also discuss incorporation of AI technology and its impact on the nonvisual guidance experiences, and further meanings of independence and new ways for designing independence for people with visual impairments. This collaborative work included AI researchers, the community of the blind, and an industry research partner. Additionally, I will discuss my findings and further exciting research opportunities. Secondly, I will overview research projects investigating AI-based applications and tools that support deaf and hard of hearing people's equitable information access and societal participation. This work addresses engagement in online social media spaces, workplace communication, participation in gig work, and interaction with mainstream technology through American Sign Language (ASL) interaction. I will focus on a recent project on users' experiences with AI deep-fake face-transformation technologies to support anonymous participation of deaf and hard of hearing signers in online social media. Lastly, I will discuss my future research directions informed and inspired by this prior and current research. Bio: Sooyeon Lee is a postdoctoral research associate in the Golisano College of Computing and Information Sciences at Rochester Institute of Technology. She received her Ph.D., advised by Dr. John M. Carroll, in Information Sciences and Technology from the College of Information Sciences and Technology at The Pennsylvania State University, and she also conducted design research at Google and Uber. Her research is in the fields of Human-Computer Interaction and Human-AI Interaction with focus on accessibility. She designs, builds, and evaluates new systems and applications that address accessibility barriers. Her work investigates the diversity of users, explores and leverages emerging technologies, and adopts human-centered design and inclusive design approaches in an interdisciplinary research framework. She has multiple publications in top-tier human-computer interaction and computing accessibility journals and conferences, including ACM CHI, CSCW, ASSETS, and TACCESS, and she has received a Best Paper Award Nomination at ASSETS 2021. She has served on Associate Chair for the ACM CHI conference and will serve on Program Committee for ASSETS 2022.

Place:  https://stonybrook.zoom.us/j/99167126152?pwd=TFpEYzM0aFhiOFJxSFJEb1JSS3YyQT09  

Time: 3 PM EST - Dec, 16th, 2020 

Abstract: 

Shadows provide useful cues to analyze visual scenes but also hamper many computer vision algorithms such as image segmentation, object detection, or tracking. For those reasons, shadow detection and shadow removal have been well-studied in computer vision.

Early work on shadow detection and removal focused on physical illumination models of shadows. These methods can express, identify, and remove shadows in a physically plausible manner. However, these models are often hard to optimize and are slow during inference due to their reliance on hand-designed image features. Recently, deep-learning approaches have achieved breakthroughs in performance for both shadow detection and removal. They learn to extract useful features through training while being extremely efficient during inference. However, these models are data-dependent, opaque, and ignore the physical aspects of shadows. Thus they often lack generalization and produce inconsistent results.

We propose incorporating physical illumination constraints of shadows into deep-learning models. These constraints force the networks to more closely follow the physics of shadows, enabling them to systematically and realistically modify shadows in images. For shadow detection, we present a novel Generative Adversarial Network (GAN) based model where the generator learns to generate images with realistic attenuated shadows that can be used to train a shadow detector. For shadow removal, we propose a method that uses deep-networks to estimate the unknown parameters of a shadow image formation model that removes shadows. The system outputs high-quality shadow-free images with little or no image artifacts and achieves state-of-the-art performance in shadow removal when trained on a fully-supervised setting. Moreover, the system is easy to train and constrain since the shadow removal mapping is strictly defined by the simplified illumination model with interpretable parameters. Thus, it can be trained even with a much weaker form of supervision signal. In particular, we show that we can use two sets of patches, shadow and shadow-free, to train our shadow decomposition framework via an adversarial system. These patches are cropped from the shadow images themselves.
Therefore, this is the first deep-learning method for shadow removal that can be trained without any shadow-free images, providing an alternative solution to the paired data dependency issue. The advantage of this training scheme is even more pronounced when tested on a novel domain such as video shadow removal where the method can be fine-tuned on a testing video with only the shadow masks generated by a pre-trained shadow detector and further improves shadow removal results.

How Language Makes us Smart (without Big Data) presented by Charles Yang

Abstract: Language provides the glue that combines simpler concepts into complex ones. To study how language guides conceptual development, we need precise accounts of how rules are learned from the child's linguistic experience, which is extremely limited in comparison to the amount of data available to current machine learning methods. In this talk, I discuss a mathematical model of inductive generalization, which enables language learning with very small amount of data. Such a view of learning has strong implications for the cross-cultural/linguistic variation of development. As a case study, I show that Hong Kong children learning Cantonese, which has a relatively simpler formal counting system, develop understanding of symbolic numbers a full year ahead of English-learning children in the United States, which is precisely predictable from the learning model. The new conception of learning adds another wrinkle to the eternal question of how language and thought are related to each other.

Bio: Charles Yang studied at the MIT AI lab and now teaches linguistics, computer science and psychology and directs the Program in Cognitive Science at the University of Pennsylvania. He is the author of several books: The Price of Linguistic Productivity (2016 MIT Press) won the Leonard Bloomfield Award from the Linguistic Society of America. His honors include a Guggenheim fellowship.

Abstract: Jailbreak attacks circumvent LLMs' built-in safeguards by concealing harmful queries within adversarial prompts. While most existing defenses attempt to mitigate the effects of adversarial prompts, they often prove inadequate as adversarial prompts can take arbitrary, adaptive forms. This paper introduces RobustKV, a novel jailbreak defense that takes a fundamentally different approach by selectively removing critical tokens of harmful queries from key-value (KV) caches. Intuitively, for an adversarial prompt to be effective, its tokens must achieve sufficient `importance' (measured by attention scores), which consequently lowers the importance of tokens in the concealed harmful query. Therefore, by carefully evicting the KVs of low-ranked tokens, RobustKV minimizes the harmful query's presence in the KV cache, thus preventing the LLM from generating informative responses. Extensive evaluation using benchmark datasets and models demonstrates that RobustKV effectively counters state-of-the-art jailbreak attacks while maintaining the LLM's performance on benign queries. Notably, RobustKV creates an interesting effectiveness-evasiveness dilemma for the adversary, leading to its robustness against adaptive attacks.

Speaker: Tanqiu Jiang

Where: NCS 220 and Zoom (https://stonybrook.zoom.us/j/6406956411)