TILM: Neural Language Models with Evolving Topical Influence
Shubhra Kanti Karmaker Santu, Kalyan Veeramachaneni, Chengxiang Zhai
Abstract
Content of text data are often influenced by contextual factors which often evolve over time (e.g., content of social media are often influenced by topics covered in the major news streams). Existing language models do not consider the influence of such related evolving topics, and thus are not optimal. In this paper, we propose to incorporate such topical-influence into a language model to both improve its accuracy and enable cross-stream analysis of topical influences. Specifically, we propose a novel language model called Topical Influence Language Model (TILM), which is a novel extension of a neural language model to capture the influences on the contents in one text stream by the evolving topics in another related (or possibly same) text stream. Experimental results on six different text stream data comprised of conference paper titles show that the incorporation of evolving topical influence into a language model is beneficial and TILM outperforms multiple baselines in a challenging task of text forecasting. In addition to serving as a language model, TILM further enables interesting analysis of topical influence among multiple text streams.- Anthology ID:
- K19-1073
- Volume:
- Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)
- Month:
- November
- Year:
- 2019
- Address:
- Hong Kong, China
- Editors:
- Mohit Bansal, Aline Villavicencio
- Venue:
- CoNLL
- SIG:
- SIGNLL
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 778–788
- Language:
- URL:
- https://aclanthology.org/K19-1073
- DOI:
- 10.18653/v1/K19-1073
- Cite (ACL):
- Shubhra Kanti Karmaker Santu, Kalyan Veeramachaneni, and Chengxiang Zhai. 2019. TILM: Neural Language Models with Evolving Topical Influence. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 778–788, Hong Kong, China. Association for Computational Linguistics.
- Cite (Informal):
- TILM: Neural Language Models with Evolving Topical Influence (Karmaker Santu et al., CoNLL 2019)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/K19-1073.pdf