@inproceedings{li-etal-2020-enhancing,
    title = "Enhancing Pre-trained {C}hinese Character Representation with Word-aligned Attention",
    author = "Li, Yanzeng  and
      Yu, Bowen  and
      Mengge, Xue  and
      Liu, Tingwen",
    editor = "Jurafsky, Dan  and
      Chai, Joyce  and
      Schluter, Natalie  and
      Tetreault, Joel",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2020.acl-main.315/",
    doi = "10.18653/v1/2020.acl-main.315",
    pages = "3442--3448",
    abstract = "Most Chinese pre-trained models take character as the basic unit and learn representation according to character{'}s external contexts, ignoring the semantics expressed in the word, which is the smallest meaningful utterance in Chinese. Hence, we propose a novel word-aligned attention to exploit explicit word information, which is complementary to various character-based Chinese pre-trained language models. Specifically, we devise a pooling mechanism to align the character-level attention to the word level and propose to alleviate the potential issue of segmentation error propagation by multi-source information fusion. As a result, word and character information are explicitly integrated at the fine-tuning procedure. Experimental results on five Chinese NLP benchmark tasks demonstrate that our method achieves significant improvements against BERT, ERNIE and BERT-wwm."
}