Abstract
In this study, we examine the ability of contextualized representations of pretrained language model to distinguish whether sequences from instructional articles are plausible or implausible. Towards this end, we compare the BERT, RoBERTa, and DeBERTa models using simple classifiers based on the sentence representations of the [CLS] tokens and perform a detailed analysis by visualizing the representations of the [CLS] tokens of the models. In the experimental results of Subtask A: Multi-Class Classification, DeBERTa exhibits the best performance and produces a more distinguishable representation across different labels. Submitting an ensemble of 10 DeBERTa-based models, our final system achieves an accuracy of 61.4% and is ranked fifth out of models submitted by eight teams. Further in-depth results suggest that the abilities of pretrained language models for the plausibility detection task are more strongly affected by their model structures or attention designs than by their model sizes.- Anthology ID:
- 2022.semeval-1.147
- Volume:
- Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)
- Month:
- July
- Year:
- 2022
- Address:
- Seattle, United States
- Editors:
- Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, Shyam Ratan
- Venue:
- SemEval
- SIG:
- SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1050–1055
- Language:
- URL:
- https://aclanthology.org/2022.semeval-1.147
- DOI:
- 10.18653/v1/2022.semeval-1.147
- Cite (ACL):
- Daewook Kang, Sung-Min Lee, Eunhwan Park, and Seung-Hoon Na. 2022. JBNU-CCLab at SemEval-2022 Task 7: DeBERTa for Identifying Plausible Clarifications in Instructional Texts. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1050–1055, Seattle, United States. Association for Computational Linguistics.
- Cite (Informal):
- JBNU-CCLab at SemEval-2022 Task 7: DeBERTa for Identifying Plausible Clarifications in Instructional Texts (Kang et al., SemEval 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2022.semeval-1.147.pdf