Abstract
Contextualized word embeddings provide better initialization for neural networks that deal with various natural language understanding (NLU) tasks including Question Answering (QA) and more recently, Question Generation(QG). Apart from providing meaningful word representations, pre-trained transformer models (Vaswani et al., 2017), such as BERT (Devlin et al., 2019) also provide self-attentions which encode syntactic information that can be probed for dependency parsing (Hewitt and Manning, 2019) and POStagging (Coenen et al., 2019). In this paper, we show that the information from selfattentions of BERT are useful for language modeling of questions conditioned on paragraph and answer phrases. To control the attention span, we use semi-diagonal mask and utilize a shared model for encoding and decoding, unlike sequence-to-sequence. We further employ copy-mechanism over self-attentions to acheive state-of-the-art results for Question Generation on SQuAD v1.1 (Rajpurkar et al., 2016).- Anthology ID:
- 2020.nlp4convai-1.3
- Volume:
- Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Venue:
- NLP4ConvAI
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 25–31
- Language:
- URL:
- https://aclanthology.org/2020.nlp4convai-1.3
- DOI:
- 10.18653/v1/2020.nlp4convai-1.3
- Cite (ACL):
- Stalin Varanasi, Saadullah Amin, and Guenter Neumann. 2020. CopyBERT: A Unified Approach to Question Generation with Self-Attention. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 25–31, Online. Association for Computational Linguistics.
- Cite (Informal):
- CopyBERT: A Unified Approach to Question Generation with Self-Attention (Varanasi et al., NLP4ConvAI 2020)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/2020.nlp4convai-1.3.pdf