On Degrees of Freedom in Defining and Testing Natural Language Understanding

Saku Sugawara, Shun Tsugita

Abstract
Natural language understanding (NLU) studies often exaggerate or underestimate the capabilities of systems, thereby limiting the reproducibility of their findings. These erroneous evaluations can be attributed to the difficulty of defining and testing NLU adequately. In this position paper, we reconsider this challenge by identifying two types of researcher degrees of freedom. We revisit Turing’s original interpretation of the Turing test and reveal that an effective test of NLU does not provide an operational definition; it merely provides inductive evidence that the test subject understands the language sufficiently well to meet stakeholder objectives. In other words, stakeholders are free to arbitrarily define NLU through their objectives. To use the test results as inductive evidence, stakeholders must carefully assess if the interpretation of test scores is valid or not. However, designing and using NLU tests involve other degrees of freedom, such as specifying target skills and defining evaluation metrics. As a result, achieving consensus among stakeholders becomes difficult. To resolve this issue, we propose a validity argument, which is a framework comprising a series of validation criteria across test components. By demonstrating that current practices in NLU studies can be associated with those criteria and organizing them into a comprehensive checklist, we prove that the validity argument can serve as a coherent guideline for designing credible test sets and facilitating scientific communication.
Anthology ID:
2023.findings-acl.861
Volume:
Findings of the Association for Computational Linguistics: ACL 2023
Month:
July
Year:
2023
Address:
Toronto, Canada
Editors:
Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
13625–13649
Language:
URL:
https://aclanthology.org/2023.findings-acl.861
DOI:
10.18653/v1/2023.findings-acl.861
Bibkey:
Cite (ACL):
Saku Sugawara and Shun Tsugita. 2023. On Degrees of Freedom in Defining and Testing Natural Language Understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13625–13649, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
On Degrees of Freedom in Defining and Testing Natural Language Understanding (Sugawara & Tsugita, Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-5/2023.findings-acl.861.pdf
PDF Search