Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST!

Frank Wildenburg, Michael Hanna, Sandro Pezzelle

Abstract
In everyday language use, speakers frequently utter and interpret sentences that are semantically underspecified, namely, whose content is insufficient to fully convey their message or interpret them univocally. For example, to interpret the underspecified sentence “Don’t spend too much”, which leaves implicit what (not) to spend, additional linguistic context or outside knowledge is needed. In this work, we propose a novel Dataset of semantically Underspecified Sentences grouped by Type (DUST) and use it to study whether pre-trained language models (LMs) correctly identify and interpret underspecified sentences. We find that newer LMs are reasonably able to identify underspecified sentences when explicitly prompted. However, interpreting them correctly is much harder for any LMs. Our experiments show that when interpreting underspecified sentences, LMs exhibit little uncertainty, contrary to what theoretical accounts of underspecification would predict. Overall, our study reveals limitations in current models’ processing of sentence semantics and highlights the importance of using naturalistic data and communicative scenarios when evaluating LMs’ language capabilities.
Anthology ID:
2024.findings-acl.572
Volume:
Findings of the Association for Computational Linguistics ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand and virtual meeting
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
9598–9613
Language:
URL:
https://aclanthology.org/2024.findings-acl.572
DOI:
Bibkey:
Cite (ACL):
Frank Wildenburg, Michael Hanna, and Sandro Pezzelle. 2024. Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST!. In Findings of the Association for Computational Linguistics ACL 2024, pages 9598–9613, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST! (Wildenburg et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.findings-acl.572.pdf
PDF Search