Comparing Template-based and Template-free Language Model Probing

Sagi Shaier, Kevin Bennett, Lawrence Hunter, Katharina von der Wense

Abstract
The differences between cloze-task language model (LM) probing with 1) expert-made templates and 2) naturally-occurring text have often been overlooked. Here, we evaluate 16 different LMs on 10 probing English datasets – 4 template-based and 6 template-free – in general and biomedical domains to answer the following research questions: (RQ1) Do model rankings differ between the two approaches? (RQ2) Do models’ absolute scores differ between the two approaches? (RQ3) Do the answers to RQ1 and RQ2 differ between general and domain-specific models? Our findings are: 1) Template-free and template-based approaches often rank models differently, except for the top domain- specific models. 2) Scores decrease by up to 42% Acc@1 when comparing parallel template-free and template-based prompts. 3) Perplexity is negatively correlated with accuracy in the template-free approach, but, counter-intuitively, they are positively correlated for template-based probing. 4) Models tend to predict the same answers frequently across prompts for template-based probing, which is less common when employing template-free techniques.
Anthology ID:
2024.eacl-long.46
Volume:
Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
March
Year:
2024
Address:
St. Julian’s, Malta
Editors:
Yvette Graham, Matthew Purver
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
766–776
Language:
URL:
https://aclanthology.org/2024.eacl-long.46
DOI:
Bibkey:
Cite (ACL):
Sagi Shaier, Kevin Bennett, Lawrence Hunter, and Katharina von der Wense. 2024. Comparing Template-based and Template-free Language Model Probing. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 766–776, St. Julian’s, Malta. Association for Computational Linguistics.
Cite (Informal):
Comparing Template-based and Template-free Language Model Probing (Shaier et al., EACL 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-2/2024.eacl-long.46.pdf
PDF Search