Stance-Aware Re-Ranking for Non-factual Comparative Queries
Jan Heinrich Reimer, Alexander Bondarenko, Maik Fröbe, Matthias Hagen
Abstract
We propose a re-ranking approach to improve the retrieval effectiveness for non-factual comparative queries like ‘Which city is better, London or Paris?’ based on whether the results express a stance towards the comparison objects (London vs. Paris) or not. Applied to the 26 runs submitted to the Touché 2022 task on comparative argument retrieval, our stance-aware re-ranking significantly improves the retrieval effectiveness for all runs when perfect oracle-style stance labels are available. With our most effective practical stance detector based on GPT-3.5 (F₁ of 0.49 on four stance classes), our re-ranking still improves the effectiveness for all runs but only six improvements are significant. Artificially “deteriorating” the oracle-style labels, we further find that an F₁ of 0.90 for stance detection is necessary to significantly improve the retrieval effectiveness for the best run via stance-aware re-ranking.- Anthology ID:
- 2023.argmining-1.5
- Volume:
- Proceedings of the 10th Workshop on Argument Mining
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Milad Alshomary, Chung-Chi Chen, Smaranda Muresan, Joonsuk Park, Julia Romberg
- Venues:
- ArgMining | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 45–51
- Language:
- URL:
- https://aclanthology.org/2023.argmining-1.5
- DOI:
- 10.18653/v1/2023.argmining-1.5
- Cite (ACL):
- Jan Heinrich Reimer, Alexander Bondarenko, Maik Fröbe, and Matthias Hagen. 2023. Stance-Aware Re-Ranking for Non-factual Comparative Queries. In Proceedings of the 10th Workshop on Argument Mining, pages 45–51, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Stance-Aware Re-Ranking for Non-factual Comparative Queries (Reimer et al., ArgMining-WS 2023)
- PDF:
- https://preview.aclanthology.org/ingest-bitext-workshop/2023.argmining-1.5.pdf