@inproceedings{wang-etal-2023-logical,
    title = "Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models",
    author = "Wang, Borui  and
      Huang, Qiuyuan  and
      Deb, Budhaditya  and
      Halfaker, Aaron  and
      Shao, Liqun  and
      McDuff, Daniel  and
      Awadallah, Ahmed Hassan  and
      Radev, Dragomir  and
      Gao, Jianfeng",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest_wac_2008/2023.findings-acl.111/",
    doi = "10.18653/v1/2023.findings-acl.111",
    pages = "1762--1773",
    abstract = "Natural language contains rich logical structures and logical information, and correctly detecting and accurately understanding these logical structures and information underlying natural language texts is very crucial for NLP models' performance on many important NLU and NLG tasks. Existing pre-trained language models based on the transformer architecture mostly adopt a classical design for constructing their input embeddings that ignores the logical structures underlying natural language texts, thus limiting their ability to better capture and encode key logical information in the input sequences. To overcome such limitations, in this paper we first propose a novel approach to construct logic-aware input embeddings for transformer language models through a combination of logic detection, logic mapping and hierarchical logical projections, and then develop a corresponding new modeling paradigm that can upgrade existing transformer language models into logical transformers to boost their performance on different NLU and NLG tasks. Our empirical experiments on four important and challenging NLU and NLG tasks demonstrate that our proposed logical transformer language models can achieve superior performance over their baseline transformer models through a deeper understanding of the logical structures of texts."
}