Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views
Katerina Margatina, Shuai Wang, Yogarshi Vyas, Neha Anna John, Yassine Benajiba, Miguel Ballesteros
Abstract
Temporal concept drift refers to the problem of data changing over time. In the field of NLP, that would entail that language (e.g. new expressions, meaning shifts) and factual knowledge (e.g. new concepts, updated facts) evolve over time. Focusing on the latter, we benchmark 11 pretrained masked language models (MLMs) on a series of tests designed to evaluate the effect of temporal concept drift, as it is crucial that widely used language models remain up-to-date with the ever-evolving factual updates of the real world. Specifically, we provide a holistic framework that (1) dynamically creates temporal test sets of any time granularity (e.g. month, quarter, year) of factual data from Wikidata, (2) constructs fine-grained splits of tests (e.g. updated, new, unchanged facts) to ensure comprehensive analysis, and (3) evaluates MLMs in three distinct ways (single-token probing, multi-token generation, MLM scoring). In contrast to prior work, our framework aims to unveil how robust an MLM is over time and thus to provide a signal in case it has become outdated, by leveraging multiple views of evaluation.- Anthology ID:
- 2023.eacl-main.211
- Volume:
- Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Andreas Vlachos, Isabelle Augenstein
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2881–2898
- Language:
- URL:
- https://aclanthology.org/2023.eacl-main.211
- DOI:
- 10.18653/v1/2023.eacl-main.211
- Cite (ACL):
- Katerina Margatina, Shuai Wang, Yogarshi Vyas, Neha Anna John, Yassine Benajiba, and Miguel Ballesteros. 2023. Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2881–2898, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views (Margatina et al., EACL 2023)
- PDF:
- https://preview.aclanthology.org/landing_page/2023.eacl-main.211.pdf