News Headline Grouping as a Challenging NLU Task

Philippe Laban, Lucas Bandarkar, Marti A. Hearst

Abstract
Recent progress in Natural Language Understanding (NLU) has seen the latest models outperform human performance on many standard tasks. These impressive results have led the community to introspect on dataset limitations, and iterate on more nuanced challenges. In this paper, we introduce the task of HeadLine Grouping (HLG) and a corresponding dataset (HLGD) consisting of 20,056 pairs of news headlines, each labeled with a binary judgement as to whether the pair belongs within the same group. On HLGD, human annotators achieve high performance of around 0.9 F-1, while current state-of-the art Transformer models only reach 0.75 F-1, opening the path for further improvements. We further propose a novel unsupervised Headline Generator Swap model for the task of HeadLine Grouping that achieves within 3 F-1 of the best supervised model. Finally, we analyze high-performing models with consistency tests, and find that models are not consistent in their predictions, revealing modeling limits of current architectures.
Anthology ID:
2021.naacl-main.255
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3186–3198
Language:
URL:
https://aclanthology.org/2021.naacl-main.255
DOI:
10.18653/v1/2021.naacl-main.255
Bibkey:
Cite (ACL):
Philippe Laban, Lucas Bandarkar, and Marti A. Hearst. 2021. News Headline Grouping as a Challenging NLU Task. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3186–3198, Online. Association for Computational Linguistics.
Cite (Informal):
News Headline Grouping as a Challenging NLU Task (Laban et al., NAACL 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2021.naacl-main.255.pdf
Video:
 https://preview.aclanthology.org/ingestion-script-update/2021.naacl-main.255.mp4
Code
 tingofurro/headline_grouping
Data
HLGD
PDF Search Code Video