Evaluating the Effectiveness of Efficient Neural Architecture Search for Sentence-Pair Tasks
Ansel MacLaughlin, Jwala Dhamala, Anoop Kumar, Sriram Venkatapathy, Ragav Venkatesan, Rahul Gupta
Abstract
Neural Architecture Search (NAS) methods, which automatically learn entire neural model or individual neural cell architectures, have recently achieved competitive or state-of-the-art (SOTA) performance on variety of natural language processing and computer vision tasks, including language modeling, natural language inference, and image classification. In this work, we explore the applicability of a SOTA NAS algorithm, Efficient Neural Architecture Search (ENAS) (Pham et al., 2018) to two sentence pair tasks, paraphrase detection and semantic textual similarity. We use ENAS to perform a micro-level search and learn a task-optimized RNN cell architecture as a drop-in replacement for an LSTM. We explore the effectiveness of ENAS through experiments on three datasets (MRPC, SICK, STS-B), with two different models (ESIM, BiLSTM-Max), and two sets of embeddings (Glove, BERT). In contrast to prior work applying ENAS to NLP tasks, our results are mixed – we find that ENAS architectures sometimes, but not always, outperform LSTMs and perform similarly to random architecture search.- Anthology ID:
- 2020.insights-1.4
- Volume:
- Proceedings of the First Workshop on Insights from Negative Results in NLP
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Anna Rogers, João Sedoc, Anna Rumshisky
- Venue:
- insights
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 22–31
- Language:
- URL:
- https://aclanthology.org/2020.insights-1.4
- DOI:
- 10.18653/v1/2020.insights-1.4
- Cite (ACL):
- Ansel MacLaughlin, Jwala Dhamala, Anoop Kumar, Sriram Venkatapathy, Ragav Venkatesan, and Rahul Gupta. 2020. Evaluating the Effectiveness of Efficient Neural Architecture Search for Sentence-Pair Tasks. In Proceedings of the First Workshop on Insights from Negative Results in NLP, pages 22–31, Online. Association for Computational Linguistics.
- Cite (Informal):
- Evaluating the Effectiveness of Efficient Neural Architecture Search for Sentence-Pair Tasks (MacLaughlin et al., insights 2020)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2020.insights-1.4.pdf