Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data
Roman Grundkiewicz, Marcin Junczys-Dowmunt, Kenneth Heafield
Abstract
Considerable effort has been made to address the data sparsity problem in neural grammatical error correction. In this work, we propose a simple and surprisingly effective unsupervised synthetic error generation method based on confusion sets extracted from a spellchecker to increase the amount of training data. Synthetic data is used to pre-train a Transformer sequence-to-sequence model, which not only improves over a strong baseline trained on authentic error-annotated data, but also enables the development of a practical GEC system in a scenario where little genuine error-annotated data is available. The developed systems placed first in the BEA19 shared task, achieving 69.47 and 64.24 F0.5 in the restricted and low-resource tracks respectively, both on the W&I+LOCNESS test set. On the popular CoNLL 2014 test set, we report state-of-the-art results of 64.16 M² for the submitted system, and 61.30 M² for the constrained system trained on the NUCLE and Lang-8 data.- Anthology ID:
- W19-4427
- Volume:
- Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications
- Month:
- August
- Year:
- 2019
- Address:
- Florence, Italy
- Editors:
- Helen Yannakoudakis, Ekaterina Kochmar, Claudia Leacock, Nitin Madnani, Ildikó Pilán, Torsten Zesch
- Venue:
- BEA
- SIG:
- SIGEDU
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 252–263
- Language:
- URL:
- https://aclanthology.org/W19-4427
- DOI:
- 10.18653/v1/W19-4427
- Cite (ACL):
- Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Kenneth Heafield. 2019. Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 252–263, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data (Grundkiewicz et al., BEA 2019)
- PDF:
- https://preview.aclanthology.org/corrections-2024-07/W19-4427.pdf
- Code
- grammatical/pretraining-bea2019
- Data
- FCE, JFLEG, WI-LOCNESS