ANNA: Enhanced Language Representation for Question Answering
Changwook Jun, Hansol Jang, Myoseop Sim, Hyun Kim, Jooyoung Choi, Kyungkoo Min, Kyunghoon Bae
Abstract
Pre-trained language models have brought significant improvements in performance in a variety of natural language processing tasks. Most existing models performing state-of-the-art results have shown their approaches in the separate perspectives of data processing, pre-training tasks, neural network modeling, or fine-tuning. In this paper, we demonstrate how the approaches affect performance individually, and that the language model performs the best results on a specific question answering task when those approaches are jointly considered in pre-training models. In particular, we propose an extended pre-training task, and a new neighbor-aware mechanism that attends neighboring tokens more to capture the richness of context for pre-training language modeling. Our best model achieves new state-of-the-art results of 95.7% F1 and 90.6% EM on SQuAD 1.1 and also outperforms existing pre-trained language models such as RoBERTa, ALBERT, ELECTRA, and XLNet on the SQuAD 2.0 benchmark.- Anthology ID:
- 2022.repl4nlp-1.13
- Volume:
- Proceedings of the 7th Workshop on Representation Learning for NLP
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Venue:
- RepL4NLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 121–132
- Language:
- URL:
- https://aclanthology.org/2022.repl4nlp-1.13
- DOI:
- 10.18653/v1/2022.repl4nlp-1.13
- Cite (ACL):
- Changwook Jun, Hansol Jang, Myoseop Sim, Hyun Kim, Jooyoung Choi, Kyungkoo Min, and Kyunghoon Bae. 2022. ANNA: Enhanced Language Representation for Question Answering. In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 121–132, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- ANNA: Enhanced Language Representation for Question Answering (Jun et al., RepL4NLP 2022)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2022.repl4nlp-1.13.pdf
- Data
- C4, GLUE, SQuAD