SemEval-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts

Michael Roth, Talita Anthonio, Anna Sauer

Abstract
We describe SemEval-2022 Task 7, a shared task on rating the plausibility of clarifications in instructional texts. The dataset for this task consists of manually clarified how-to guides for which we generated alternative clarifications and collected human plausibility judgements. The task of participating systems was to automatically determine the plausibility of a clarification in the respective context. In total, 21 participants took part in this task, with the best system achieving an accuracy of 68.9%. This report summarizes the results and findings from 8 teams and their system descriptions. Finally, we show in an additional evaluation that predictions by the top participating team make it possible to identify contexts with multiple plausible clarifications with an accuracy of 75.2%.
Anthology ID:
2022.semeval-1.146
Volume:
Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, Shyam Ratan
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
1039–1049
Language:
URL:
https://preview.aclanthology.org/build-pipeline-with-new-library/2022.semeval-1.146/
DOI:
10.18653/v1/2022.semeval-1.146
Bibkey:
Cite (ACL):
Michael Roth, Talita Anthonio, and Anna Sauer. 2022. SemEval-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1039–1049, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
SemEval-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts (Roth et al., SemEval 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/build-pipeline-with-new-library/2022.semeval-1.146.pdf
Video:
 https://preview.aclanthology.org/build-pipeline-with-new-library/2022.semeval-1.146.mp4
Code
 acidann/claire
PDF Search Code Video Fix metadata