Benchmarking: Past, Present and Future

Kenneth Church, Mark Liberman, Valia Kordoni


Abstract
Where have we been, and where are we going? It is easier to talk about the past than the future. These days, benchmarks evolve more bottom up (such as papers with code). There used to be more top-down leadership from government (and industry, in the case of systems, with benchmarks such as SPEC). Going forward, there may be more top-down leadership from organizations like MLPerf and/or influencers like David Ferrucci, who was responsible for IBM’s success with Jeopardy, and has recently written a paper suggesting how the community should think about benchmarking for machine comprehension. Tasks such as reading comprehension become even more interesting as we move beyond English. Multilinguality introduces many challenges, and even more opportunities.
Anthology ID:
2021.bppf-1.1
Volume:
Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future
Month:
Aug
Year:
2021
Address:
Online
Venue:
BPPF
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–7
Language:
URL:
https://aclanthology.org/2021.bppf-1.1
DOI:
10.18653/v1/2021.bppf-1.1
Bibkey:
Cite (ACL):
Kenneth Church, Mark Liberman, and Valia Kordoni. 2021. Benchmarking: Past, Present and Future. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 1–7, Online. Association for Computational Linguistics.
Cite (Informal):
Benchmarking: Past, Present and Future (Church et al., BPPF 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/auto-file-uploads/2021.bppf-1.1.pdf
Code
 kwchurch/benchmarking_past_present_future