Probing BERT’s ability to encode sentence modality and modal verb sense across varieties of English

Jonas Wagner, Sina Zarrieß

Abstract
In this research, we investigate whether BERT can differentiate between modal verb senses and sentence modalities and whether it performs equally well on different varieties of English. We fit probing classifiers under two conditions: contextualised embeddings of modal verbs and sentence embeddings. We also investigate BERT’s ability to predict masked modal verbs. Additionally, we classify separately for each modal verb to investigate whether BERT encodes different representations of senses for each individual verb. Lastly, we employ classifiers on data from different varieties of English to determine whether non-American English data is an additional hurdle. Results indicate that BERT has different representations for distinct senses for each modal verb, but does not represent modal sense independently from modal verbs. We also show that performance in different varieties of English is not equal, pointing to a necessary shift in the way we train large language models towards more linguistic diversity. We make our annotated dataset of modal sense in different varieties of English available at https://github.com/wagner-jonas/VEM.
Anthology ID:
2023.iwcs-1.3
Volume:
Proceedings of the 15th International Conference on Computational Semantics
Month:
June
Year:
2023
Address:
Nancy, France
Editors:
Maxime Amblard, Ellen Breitholtz
Venue:
IWCS
SIG:
SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
28–38
Language:
URL:
https://aclanthology.org/2023.iwcs-1.3
DOI:
Bibkey:
Cite (ACL):
Jonas Wagner and Sina Zarrieß. 2023. Probing BERT’s ability to encode sentence modality and modal verb sense across varieties of English. In Proceedings of the 15th International Conference on Computational Semantics, pages 28–38, Nancy, France. Association for Computational Linguistics.
Cite (Informal):
Probing BERT’s ability to encode sentence modality and modal verb sense across varieties of English (Wagner & Zarrieß, IWCS 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/landing_page/2023.iwcs-1.3.pdf
PDF Search