Does the Objective Matter? Comparing Training Objectives for Pronoun Resolution
Yordan Yordanov, Oana-Maria Camburu, Vid Kocijan, Thomas Lukasiewicz
Abstract
Hard cases of pronoun resolution have been used as a long-standing benchmark for commonsense reasoning. In the recent literature, pre-trained language models have been used to obtain state-of-the-art results on pronoun resolution. Overall, four categories of training and evaluation objectives have been introduced. The variety of training datasets and pre-trained language models used in these works makes it unclear whether the choice of training objective is critical. In this work, we make a fair comparison of the performance and seed-wise stability of four models that represent the four categories of objectives. Our experiments show that the objective of sequence ranking performs the best in-domain, while the objective of semantic similarity between candidates and pronoun performs the best out-of-domain. We also observe a seed-wise instability of the model using sequence ranking, which is not the case when the other objectives are used.- Anthology ID:
- 2020.emnlp-main.402
- Volume:
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4963–4969
- Language:
- URL:
- https://preview.aclanthology.org/remove-affiliations/2020.emnlp-main.402/
- DOI:
- 10.18653/v1/2020.emnlp-main.402
- Cite (ACL):
- Yordan Yordanov, Oana-Maria Camburu, Vid Kocijan, and Thomas Lukasiewicz. 2020. Does the Objective Matter? Comparing Training Objectives for Pronoun Resolution. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4963–4969, Online. Association for Computational Linguistics.
- Cite (Informal):
- Does the Objective Matter? Comparing Training Objectives for Pronoun Resolution (Yordanov et al., EMNLP 2020)
- PDF:
- https://preview.aclanthology.org/remove-affiliations/2020.emnlp-main.402.pdf
- Code
- YDYordanov/WS-training-objectives
- Data
- WSC, WinoGrande