Abstract
It has become a common pattern in our field: One group introduces a language task, exemplified by a dataset, which they argue is challenging enough to serve as a benchmark. They also provide a baseline model for it, which then soon is improved upon by other groups. Often, research efforts then move on, and the pattern repeats itself. What is typically left implicit is the argumentation for why this constitutes progress, and progress towards what. In this paper, we try to step back for a moment from this pattern and work out possible argumentations and their parts.- Anthology ID:
- 2021.acl-short.85
- Volume:
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
- Month:
- August
- Year:
- 2021
- Address:
- Online
- Editors:
- Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
- Venues:
- ACL | IJCNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 670–674
- Language:
- URL:
- https://aclanthology.org/2021.acl-short.85
- DOI:
- 10.18653/v1/2021.acl-short.85
- Cite (ACL):
- David Schlangen. 2021. Targeting the Benchmark: On Methodology in Current Natural Language Processing Research. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 670–674, Online. Association for Computational Linguistics.
- Cite (Informal):
- Targeting the Benchmark: On Methodology in Current Natural Language Processing Research (Schlangen, ACL-IJCNLP 2021)
- PDF:
- https://preview.aclanthology.org/naacl-24-ws-corrections/2021.acl-short.85.pdf
- Data
- Visual Question Answering