Abstract
In data ranking applications, pairwise annotation is often more consistent than cardinal annotation for learning ranking models. We examine this in a case study on ranking text passages for argument convincingness. Our task is to choose text passages that provide the highest-quality, most-convincing arguments for opposing sides of a topic. Using data from a deployed system within the Bing search engine, we construct a pairwise-labeled dataset for argument convincingness that is substantially more comprehensive in topical coverage compared to existing public resources. We detail the process of extracting topical passages for queries submitted to a search engine, creating annotated sets of passages aligned to different stances on a topic, and assessing argument convincingness of passages using pairwise annotation. Using a state-of-the-art convincingness model, we evaluate several methods for using pairwise-annotated data examples to train models for ranking passages. Our results show pairwise training outperforms training that regresses to a target score for each passage. Our results also show a simple ‘win-rate’ score is a better regression target than the previously proposed page-rank target. Lastly, addressing the need to filter noisy crowd-sourced annotations when constructing a dataset, we show that filtering for transitivity within pairwise annotations is more effective than filtering based on annotation confidence measures for individual examples.- Anthology ID:
- W19-4517
- Volume:
- Proceedings of the 6th Workshop on Argument Mining
- Month:
- August
- Year:
- 2019
- Address:
- Florence, Italy
- Venue:
- ArgMining
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 146–155
- Language:
- URL:
- https://aclanthology.org/W19-4517
- DOI:
- 10.18653/v1/W19-4517
- Cite (ACL):
- Peter Potash, Adam Ferguson, and Timothy J. Hazen. 2019. Ranking Passages for Argument Convincingness. In Proceedings of the 6th Workshop on Argument Mining, pages 146–155, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- Ranking Passages for Argument Convincingness (Potash et al., ArgMining 2019)
- PDF:
- https://preview.aclanthology.org/remove-xml-comments/W19-4517.pdf