Abstract
Evaluation is the central means for assessing, understanding, and communicating about NLP models. In this position paper, we argue evaluation should be more than that: it is a force for driving change, carrying a sociological and political character beyond its technical dimensions. As a force, evaluation’s power arises from its adoption: under our view, evaluation succeeds when it achieves the desired change in the field. Further, by framing evaluation as a force, we consider how it competes with other forces. Under our analysis, we conjecture that the current trajectory of NLP suggests evaluation’s power is waning, in spite of its potential for realizing more pluralistic ambitions in the field. We conclude by discussing the legitimacy of this power, who acquires this power and how it distributes. Ultimately, we hope the research community will more aggressively harness evaluation to drive change.- Anthology ID:
- 2023.findings-acl.522
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2023
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 8227–8239
- Language:
- URL:
- https://aclanthology.org/2023.findings-acl.522
- DOI:
- 10.18653/v1/2023.findings-acl.522
- Cite (ACL):
- Rishi Bommasani. 2023. Evaluation for Change. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8227–8239, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Evaluation for Change (Bommasani, Findings 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2023.findings-acl.522.pdf