Efficient Continue Training of Temporal Language Model with Structural Information

Zhaochen Su, Juntao Li, Zikang Zhang, Zihan Zhou, Min Zhang

Abstract
Current language models are mainly trained on snap-shots of data gathered at a particular time, which decreases their capability to generalize over time and model language change. To model the time variable, existing works have explored temporal language models (e.g., TempoBERT) by directly incorporating the timestamp into the training process. While effective to some extent, these methods are limited by the superficial temporal information brought by timestamps, which fails to learn the inherent changes of linguistic components. In this paper, we empirically confirm that the performance of pre-trained language models (PLMs) is closely affiliated with syntactically changed tokens. Based on this observation, we propose a simple yet effective method named Syntax-Guided Temporal Language Model (SG-TLM), which could learn the inherent language changes by capturing an intrinsic relationship between the time prefix and the tokens with salient syntactic change. Experiments on two datasets and three tasks demonstrate that our model outperforms existing temporal language models in both memorization and generalization capabilities. Extensive results further confirm the effectiveness of our approach across different model frameworks, including both encoder-only and decoder-only models (e.g., LLaMA). Our code is available at https://github.com/zhaochen0110/TempoLM.
Anthology ID:
2023.findings-emnlp.418
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6315–6329
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.418
DOI:
10.18653/v1/2023.findings-emnlp.418
Bibkey:
Cite (ACL):
Zhaochen Su, Juntao Li, Zikang Zhang, Zihan Zhou, and Min Zhang. 2023. Efficient Continue Training of Temporal Language Model with Structural Information. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6315–6329, Singapore. Association for Computational Linguistics.
Cite (Informal):
Efficient Continue Training of Temporal Language Model with Structural Information (Su et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/dois-2013-emnlp/2023.findings-emnlp.418.pdf
PDF Search