PUNR: Pre-training with User Behavior Modeling for News Recommendation

Guangyuan Ma, Hongtao Liu, Xing W, Wanhui Qian, Zhepeng Lv, Qing Yang, Songlin Hu

Abstract
News recommendation aims to predict click behaviors based on user behaviors. How to effectively model the user representations is the key to recommending preferred news. Existing works are mostly focused on improvements in the supervised fine-tuning stage. However, there is still a lack of PLM-based unsupervised pre-training methods optimized for user representations. In this work, we propose an unsupervised pre-training paradigm with two tasks, i.e. user behavior masking and user behavior generation, both towards effective user behavior modeling. Firstly, we introduce the user behavior masking pre-training task to recover the masked user behaviors based on their contextual behaviors. In this way, the model could capture a much stronger and more comprehensive user news reading pattern. Besides, we incorporate a novel auxiliary user behavior generation pre-training task to enhance the user representation vector derived from the user encoder. We use the above pre-trained user modeling encoder to obtain news and user representations in downstream fine-tuning. Evaluations on the real-world news benchmark show significant performance improvements over existing baselines.
Anthology ID:
2023.findings-emnlp.559
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8338–8347
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.559
DOI:
10.18653/v1/2023.findings-emnlp.559
Bibkey:
Cite (ACL):
Guangyuan Ma, Hongtao Liu, Xing W, Wanhui Qian, Zhepeng Lv, Qing Yang, and Songlin Hu. 2023. PUNR: Pre-training with User Behavior Modeling for News Recommendation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8338–8347, Singapore. Association for Computational Linguistics.
Cite (Informal):
PUNR: Pre-training with User Behavior Modeling for News Recommendation (Ma et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2023.findings-emnlp.559.pdf
PDF Search