Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics
Abstract
Scalar adjectives pertain to various domain scales and vary in intensity within each scale (e.g. certain is more intense than likely on the likelihood scale). Scalar implicatures arise from the consideration of alternative statements which could have been made. They can be triggered by scalar adjectives and require listeners to reason pragmatically about them. Some scalar adjectives are more likely to trigger scalar implicatures than others. This phenomenon is referred to as scalar diversity. In this study, we probe different families of Large Language Models such as GPT-4 for their knowledge of the lexical semantics of scalar adjectives and one specific aspect of their pragmatics, namely scalar diversity. We find that they encode rich lexical-semantic information about scalar adjectives. However, the rich lexical-semantic knowledge does not entail a good understanding of scalar diversity. We also compare current models of different sizes and complexities and find that larger models are not always better. Finally, we explain our probing results by leveraging linguistic intuitions and model training objectives.- Anthology ID:
- 2024.lrec-main.1141
- Volume:
- Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
- Month:
- May
- Year:
- 2024
- Address:
- Torino, Italia
- Editors:
- Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
- Venues:
- LREC | COLING
- SIG:
- Publisher:
- ELRA and ICCL
- Note:
- Pages:
- 13033–13049
- Language:
- URL:
- https://aclanthology.org/2024.lrec-main.1141
- DOI:
- Cite (ACL):
- Fangru Lin, Daniel Altshuler, and Janet B. Pierrehumbert. 2024. Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13033–13049, Torino, Italia. ELRA and ICCL.
- Cite (Informal):
- Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics (Lin et al., LREC-COLING 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2024.lrec-main.1141.pdf