Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models
Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, Claudio Mayrink Verdun, David Bau, Samuel Marks
2024-08-05
Summary
This paper discusses the development of a method to better understand the features that language models learn by using board game transcripts. It focuses on improving how we evaluate sparse autoencoders (SAEs), which are models that help make the representations of language models more interpretable.
What's the problem?
Understanding what features language models capture is challenging because there isn't a clear set of 'correct' features to compare against. This makes it hard to evaluate how well SAEs are performing in extracting meaningful information from language model representations. Without a way to measure progress, it's difficult to know if improvements in these models are truly effective.
What's the solution?
The authors propose using chess and Othello game transcripts as a basis for evaluation because these games have clear, interpretable features, such as specific pieces on certain squares. They introduce a new training technique called p-annealing to enhance the performance of SAEs on both traditional metrics and their new supervised metrics based on game transcripts. This allows for a more systematic way to assess how well these models learn and represent meaningful features.
Why it matters?
This research is important because it helps improve our understanding of how language models work, making them more interpretable. By providing a method to evaluate and enhance these models, it can lead to better applications in natural language processing, where knowing what features the model understands can help in tasks like translation, summarization, and more.
Abstract
What latent features are encoded in language model (LM) representations? Recent work on training sparse autoencoders (SAEs) to disentangle interpretable features in LM representations has shown significant promise. However, evaluating the quality of these SAEs is difficult because we lack a ground-truth collection of interpretable features that we expect good SAEs to recover. We thus propose to measure progress in interpretable dictionary learning by working in the setting of LMs trained on chess and Othello transcripts. These settings carry natural collections of interpretable features -- for example, "there is a knight on F3" -- which we leverage into supervised metrics for SAE quality. To guide progress in interpretable dictionary learning, we introduce a new SAE training technique, p-annealing, which improves performance on prior unsupervised metrics as well as our new metrics.