Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems

Hiroki Asano, Tomoya Mizumoto, Kentaro Inui

Abstract
In grammatical error correction (GEC), automatically evaluating system outputs requires gold-standard references, which must be created manually and thus tend to be both expensive and limited in coverage. To address this problem, a reference-less approach has recently emerged; however, previous reference-less metrics that only consider the criterion of grammaticality, have not worked as well as reference-based metrics. This study explores the potential of extending a prior grammaticality-based method to establish a reference-less evaluation method for GEC systems. Further, we empirically show that a reference-less metric that combines fluency and meaning preservation with grammaticality provides a better estimate of manual scores than that of commonly used reference-based metrics. To our knowledge, this is the first study that provides empirical evidence that a reference-less metric can replace reference-based metrics in evaluating GEC systems.
Anthology ID:
I17-2058
Volume:
Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
Month:
November
Year:
2017
Address:
Taipei, Taiwan
Editors:
Greg Kondrak, Taro Watanabe
Venue:
IJCNLP
SIG:
Publisher:
Asian Federation of Natural Language Processing
Note:
Pages:
343–348
Language:
URL:
https://aclanthology.org/I17-2058
DOI:
Bibkey:
Cite (ACL):
Hiroki Asano, Tomoya Mizumoto, and Kentaro Inui. 2017. Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 343–348, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Cite (Informal):
Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems (Asano et al., IJCNLP 2017)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-2/I17-2058.pdf
Data
CoNLL-2014 Shared Task: Grammatical Error Correction
PDF Search