What Question Answering can Learn from Trivia Nerds

Jordan Boyd-Graber, Benjamin Börschinger


Abstract
In addition to the traditional task of machines answering questions, question answering (QA) research creates interesting, challenging questions that help systems how to answer questions and reveal the best systems. We argue that creating a QA dataset—and the ubiquitous leaderboard that goes with it—closely resembles running a trivia tournament: you write questions, have agents (either humans or machines) answer the questions, and declare a winner. However, the research community has ignored the hard-learned lessons from decades of the trivia community creating vibrant, fair, and effective question answering competitions. After detailing problems with existing QA datasets, we outline the key lessons—removing ambiguity, discriminating skill, and adjudicating disputes—that can transfer to QA research and how they might be implemented.
Anthology ID:
2020.acl-main.662
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7422–7435
Language:
URL:
https://aclanthology.org/2020.acl-main.662
DOI:
10.18653/v1/2020.acl-main.662
Bibkey:
Cite (ACL):
Jordan Boyd-Graber and Benjamin Börschinger. 2020. What Question Answering can Learn from Trivia Nerds. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7422–7435, Online. Association for Computational Linguistics.
Cite (Informal):
What Question Answering can Learn from Trivia Nerds (Boyd-Graber & Börschinger, ACL 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/2020.acl-main.662.pdf
Video:
 http://slideslive.com/38928685
Data
Natural QuestionsQuizbowlSQuAD