@inproceedings{asano-etal-2017-reference,
    title = "Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems",
    author = "Asano, Hiroki  and
      Mizumoto, Tomoya  and
      Inui, Kentaro",
    editor = "Kondrak, Greg  and
      Watanabe, Taro",
    booktitle = "Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)",
    month = nov,
    year = "2017",
    address = "Taipei, Taiwan",
    publisher = "Asian Federation of Natural Language Processing",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/I17-2058/",
    pages = "343--348",
    abstract = "In grammatical error correction (GEC), automatically evaluating system outputs requires gold-standard references, which must be created manually and thus tend to be both expensive and limited in coverage. To address this problem, a reference-less approach has recently emerged; however, previous reference-less metrics that only consider the criterion of grammaticality, have not worked as well as reference-based metrics. This study explores the potential of extending a prior grammaticality-based method to establish a reference-less evaluation method for GEC systems. Further, we empirically show that a reference-less metric that combines fluency and meaning preservation with grammaticality provides a better estimate of manual scores than that of commonly used reference-based metrics. To our knowledge, this is the first study that provides empirical evidence that a reference-less metric can replace reference-based metrics in evaluating GEC systems."
}