ANNA: Enhanced Language Representation for Question Answering
Changwook Jun, Hansol Jang, Myoseop Sim, Hyun Kim, Jooyoung Choi, Kyungkoo Min, Kyunghoon Bae
Abstract
Pre-trained language models have brought significant improvements in performance in a variety of natural language processing tasks. Most existing models performing state-of-the-art results have shown their approaches in the separate perspectives of data processing, pre-training tasks, neural network modeling, or fine-tuning. In this paper, we demonstrate how the approaches affect performance individually, and that the language model performs the best results on a specific question answering task when those approaches are jointly considered in pre-training models. In particular, we propose an extended pre-training task, and a new neighbor-aware mechanism that attends neighboring tokens more to capture the richness of context for pre-training language modeling. Our best model achieves new state-of-the-art results of 95.7% F1 and 90.6% EM on SQuAD 1.1 and also outperforms existing pre-trained language models such as RoBERTa, ALBERT, ELECTRA, and XLNet on the SQuAD 2.0 benchmark.- Anthology ID:
- 2022.repl4nlp-1.13
- Volume:
- Proceedings of the 7th Workshop on Representation Learning for NLP
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Spandana Gella, He He, Bodhisattwa Prasad Majumder, Burcu Can, Eleonora Giunchiglia, Samuel Cahyawijaya, Sewon Min, Maximilian Mozes, Xiang Lorraine Li, Isabelle Augenstein, Anna Rogers, Kyunghyun Cho, Edward Grefenstette, Laura Rimell, Chris Dyer
- Venue:
- RepL4NLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 121–132
- Language:
- URL:
- https://aclanthology.org/2022.repl4nlp-1.13
- DOI:
- 10.18653/v1/2022.repl4nlp-1.13
- Cite (ACL):
- Changwook Jun, Hansol Jang, Myoseop Sim, Hyun Kim, Jooyoung Choi, Kyungkoo Min, and Kyunghoon Bae. 2022. ANNA: Enhanced Language Representation for Question Answering. In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 121–132, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- ANNA: Enhanced Language Representation for Question Answering (Jun et al., RepL4NLP 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2022.repl4nlp-1.13.pdf
- Data
- C4, GLUE, SQuAD