SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense

J. A. Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez, Walid Magdy


Abstract
SemEval 2021 Task 7, HaHackathon, was the first shared task to combine the previously separate domains of humor detection and offense detection. We collected 10,000 texts from Twitter and the Kaggle Short Jokes dataset, and had each annotated for humor and offense by 20 annotators aged 18-70. Our subtasks were binary humor detection, prediction of humor and offense ratings, and a novel controversy task: to predict if the variance in the humor ratings was higher than a specific threshold. The subtasks attracted 36-58 submissions, with most of the participants choosing to use pre-trained language models. Many of the highest performing teams also implemented additional optimization techniques, including task-adaptive training and adversarial training. The results suggest that the participating systems are well suited to humor detection, but that humor controversy is a more challenging task. We discuss which models excel in this task, which auxiliary techniques boost their performance, and analyze the errors which were not captured by the best systems.
Anthology ID:
2021.semeval-1.9
Volume:
Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP | SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
105–119
Language:
URL:
https://aclanthology.org/2021.semeval-1.9
DOI:
10.18653/v1/2021.semeval-1.9
Bibkey:
Cite (ACL):
J. A. Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez, and Walid Magdy. 2021. SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 105–119, Online. Association for Computational Linguistics.
Cite (Informal):
SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense (Meaney et al., SemEval 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/2021.semeval-1.9.pdf