Abstract
Understanding the inner workings of large language models (LLMs) is crucial for advancing their theoretical foundations and real-world applications. While the attention mechanism and multi-layer perceptrons (MLPs) have been studied independently, their interactions remain largely unexplored. This study investigates how attention heads and next-token neurons interact in LLMs to predict new words. We propose a methodology to identify next-token neurons, find prompts that highly activate them, and determine the upstream attention heads responsible. We then generate and evaluate explanations for the activity of these attention heads in an automated manner. Our findings reveal that some attention heads recognize specific contexts relevant to predicting a token and activate a downstream token-predicting neuron accordingly. This mechanism provides a deeper understanding of how attention heads work with MLP neurons to perform next-token prediction. Our approach offers a foundation for further research into the intricate workings of LLMs and their impact on text generation and understanding.- Anthology ID:
- 2024.emnlp-main.930
- Volume:
- Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2024
- Address:
- Miami, Florida, USA
- Editors:
- Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 16681–16697
- Language:
- URL:
- https://aclanthology.org/2024.emnlp-main.930
- DOI:
- 10.18653/v1/2024.emnlp-main.930
- Cite (ACL):
- Clement Neo, Shay B Cohen, and Fazl Barez. 2024. Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16681–16697, Miami, Florida, USA. Association for Computational Linguistics.
- Cite (Informal):
- Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions (Neo et al., EMNLP 2024)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2024.emnlp-main.930.pdf