Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation

Takumi Gotou, Ryo Nagata, Masato Mita, Kazuaki Hanawa

Abstract
This paper presents performance measures for grammatical error correction which take into account the difficulty of error correction. To the best of our knowledge, no conventional measure has such functionality despite the fact that some errors are easy to correct and others are not. The main purpose of this work is to provide a way of determining the difficulty of error correction and to motivate researchers in the domain to attack such difficult errors. The performance measures are based on the simple idea that the more systems successfully correct an error, the easier it is considered to be. This paper presents a set of algorithms to implement this idea. It evaluates the performance measures quantitatively and qualitatively on a wide variety of corpora and systems, revealing that they agree with our intuition of correction difficulty. A scorer and difficulty weight data based on the algorithms have been made available on the web.
Anthology ID:
2020.coling-main.188
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
2085–2095
Language:
URL:
https://aclanthology.org/2020.coling-main.188
DOI:
10.18653/v1/2020.coling-main.188
Bibkey:
Cite (ACL):
Takumi Gotou, Ryo Nagata, Masato Mita, and Kazuaki Hanawa. 2020. Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2085–2095, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation (Gotou et al., COLING 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2020.coling-main.188.pdf
Code
 gotutiyan/gts
Data
FCE, JFLEG
PDF Search Code