Training a Better Chinese Spelling Correction Model via Prior-knowledge Guided Teacher

Chi Wei, Shaobin Huang, Rongsheng Li, Naiyu Yan, Rui Wang

Abstract
Recent advancements in Chinese Spelling Correction (CSC) predominantly leverage pre-trained language models (PLMs). However, a notable challenge with fine-tuned PLM-based CSC models is their tendency to over-correct, leading to poor generalization for error patterns outside the standard distribution. To address this, we developed a teacher network guided by prior knowledge for distillation learning of CSC models. Unlike traditional teacher networks, which depend on task-related pre-training, our method infuses task-related prior information into the teacher network, offering guidance beyond mere labels to the student network. This strategy significantly enhances the CSC model’s language modeling capabilities, crucial for minimizing over-correction. Importantly, our approach is model-independent and the teacher network does not require task-related pre-training, making it broadly applicable for enhancing various PLM-based CSC models with minimal additional computational resources. Extensive experiments on widely used benchmarks demonstrate that our method achieves new state-of-the-art results. Additionally, we explored the potential of generalizing our method to other non-autoregressive text-generation tasks.
Anthology ID:
2024.findings-acl.806
Volume:
Findings of the Association for Computational Linguistics ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand and virtual meeting
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
13578–13589
Language:
URL:
https://aclanthology.org/2024.findings-acl.806
DOI:
Bibkey:
Cite (ACL):
Chi Wei, Shaobin Huang, Rongsheng Li, Naiyu Yan, and Rui Wang. 2024. Training a Better Chinese Spelling Correction Model via Prior-knowledge Guided Teacher. In Findings of the Association for Computational Linguistics ACL 2024, pages 13578–13589, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
Training a Better Chinese Spelling Correction Model via Prior-knowledge Guided Teacher (Wei et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.findings-acl.806.pdf
PDF Search