Mansi Sakarvadia

Hello! I am a third-year Department of Energy Computational Science Graduate Fellow and a Computer Science Ph.D. student at the University of Chicago, where I am co-advised by Ian Foster and Kyle Chard.

I develop machine learning interpretability methods. My research aims to systematically reverse engineer neural networks to interpret their weights. For example, much of my work focuses on localizing sources of model failure within weight-space and developing efficient methods to correct model behavior.

Prior to my Ph.D., I completed my Bachelors in Computer Science and Mathematics with a minor in Environmental Science at the University of North Carolina, Chapel Hill.

news

Oct 18, 2024	Presented my poster “Mitigating Memorization in Language Models” at the University of Chicago Communication & Intelligene Symposium.
Oct 3, 2024	Excited to announce that our work on detoxifying LM outputs, “Mind Your Manners: Detoxifying Language Models via Attention Head Intervention”, was accepted to BlackboxNLP 2024 as an extended abstract.
Sep 1, 2024	Had a great time at Lawrence Berekley National Laboratory’s ML and Analytics group this summer working on developing methods to Mitigate Memorization in LMs.
Jul 16, 2024	Presented my poster “Mitigating Memorization in Language Models” at the CSGF Program Review in Washington, DC.
Apr 5, 2024	Excited to announce that I passed my qualifier exams/Master’s defense! Check out a recording of my talk here.

selected publications

Preprint
Mitigating Memorization In Language Models

Mansi Sakarvadia, Aswathy Ajith, Arham Khan, and 6 more authors

2024

Abs arXiv Bib HTML

Language models (LMs) can "memorize" information, i.e., encode training data in their weights in such a way that inference-time queries can lead to verbatim regurgitation of that data. This ability to extract training data can be problematic, for example, when data are private or sensitive. In this work, we investigate methods to mitigate memorization: three regularizer-based, three finetuning-based, and eleven machine unlearning-based methods, with five of the latter being new methods that we introduce. We also introduce TinyMem, a suite of small, computationally-efficient LMs for the rapid development and evaluation of memorization-mitigation methods. We demonstrate that the mitigation methods that we develop using TinyMem can successfully be applied to production-grade LMs, and we determine via experiment that: regularizer-based mitigation methods are slow and ineffective at curbing memorization; fine-tuning-based methods are effective at curbing memorization, but overly expensive, especially for retaining higher accuracies; and unlearning-based methods are faster and more effective, allowing for the precise localization and removal of memorized information from LM weights prior to inference. We show, in particular, that our proposed unlearning method BalancedSubnet outperforms other mitigation methods at removing memorized information while preserving performance on target tasks.
@article{sakarvadia2024Mitigating, title = {Mitigating Memorization In Language Models}, author = {Sakarvadia, Mansi and Ajith, Aswathy and Khan, Arham and Hudson, Nathaniel and Geniesse, Caleb and Chard, Kyle and Yang, Yaoqing and Foster, Ian and Mahoney, Michael W.}, year = {2024}, }
BlackboxNLP
Memory Injections: Correcting Multi-Hop Reasoning Failures during Inference in Transformer-Based Language Models

Mansi Sakarvadia, Aswathy Ajith, Arham Khan, and 5 more authors

2023

Work accepted to BlackBoxNLP 2023.

Abs arXiv Bib

Answering multi-hop reasoning questions requires retrieving and synthesizing information from diverse sources. Large Language Models (LLMs) struggle to perform such reasoning consistently. Here we propose an approach to pinpoint and rectify multi-hop reasoning failures through targeted memory injections on LLM attention heads. First, we analyze the per-layer activations of GPT-2 models in response to single and multi-hop prompts. We then propose a mechanism that allows users to inject pertinent prompt-specific information, which we refer to as "memories," at critical LLM locations during inference. By thus enabling the LLM to incorporate additional relevant information during inference, we enhance the quality of multi-hop prompt completions. We show empirically that a simple, efficient, and targeted memory injection into a key attention layer can often increase the probability of the desired next token in multi-hop tasks, by up to 424%.
@article{sakarvadia2023memory, title = {Memory Injections: Correcting Multi-Hop Reasoning Failures during Inference in Transformer-Based Language Models}, author = {Sakarvadia, Mansi and Ajith, Aswathy and Khan, Arham and Grzenda, Daniel and Hudson, Nathaniel and Bauer, André and Chard, Kyle and Foster, Ian}, year = {2023}, note = {Work accepted to BlackBoxNLP 2023.}, }