Discourse Probing of Pretrained Language Models

Fajri Koto, Jey Han Lau, Timothy Baldwin


Abstract
Existing work on probing of pretrained language models (LMs) has predominantly focused on sentence-level syntactic tasks. In this paper, we introduce document-level discourse probing to evaluate the ability of pretrained LMs to capture document-level relations. We experiment with 7 pretrained LMs, 4 languages, and 7 discourse probing tasks, and find BART to be overall the best model at capturing discourse — but only in its encoder, with BERT performing surprisingly well as the baseline model. Across the different models, there are substantial differences in which layers best capture discourse information, and large disparities between models.
Anthology ID:
2021.naacl-main.301
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3849–3864
Language:
URL:
https://aclanthology.org/2021.naacl-main.301
DOI:
10.18653/v1/2021.naacl-main.301
Bibkey:
Cite (ACL):
Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Discourse Probing of Pretrained Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3849–3864, Online. Association for Computational Linguistics.
Cite (Informal):
Discourse Probing of Pretrained Language Models (Koto et al., NAACL 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2021.naacl-main.301.pdf
Video:
 https://preview.aclanthology.org/ingestion-script-update/2021.naacl-main.301.mp4
Code
 fajri91/discourse_probing