@inproceedings{dai-etal-2022-whole,
    title = "{``}Is Whole Word Masking Always Better for {C}hinese {BERT}?{''}: Probing on {C}hinese Grammatical Error Correction",
    author = "Dai, Yong  and
      Li, Linyang  and
      Zhou, Cong  and
      Feng, Zhangyin  and
      Zhao, Enbo  and
      Qiu, Xipeng  and
      Li, Piji  and
      Tang, Duyu",
    editor = "Muresan, Smaranda  and
      Nakov, Preslav  and
      Villavicencio, Aline",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2022",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.findings-acl.1",
    doi = "10.18653/v1/2022.findings-acl.1",
    pages = "1--8",
    abstract = "Whole word masking (WWM), which masks all subwords corresponding to a word at once, makes a better English BERT model. For the Chinese language, however, there is no subword because each token is an atomic character. The meaning of a word in Chinese is different in that a word is a compositional unit consisting of multiple characters. Such difference motivates us to investigate whether WWM leads to better context understanding ability for Chinese BERT. To achieve this, we introduce two probing tasks related to grammatical error correction and ask pretrained models to revise or insert tokens in a masked language modeling manner. We construct a dataset including labels for 19,075 tokens in 10,448 sentences. We train three Chinese BERT models with standard character-level masking (CLM), WWM, and a combination of CLM and WWM, respectively. Our major findings are as follows: First, when one character needs to be inserted or replaced, the model trained with CLM performs the best. Second, when more than one character needs to be handled, WWM is the key to better performance. Finally, when being fine-tuned on sentence-level downstream tasks, models trained with different masking strategies perform comparably.",
}