SIGA: A Naturalistic NLI Dataset of English Scalar Implicatures with Gradable Adjectives

Rashid Nizamani, Sebastian Schuster, Vera Demberg

Abstract
Many utterances convey meanings that go beyond the literal meaning of a sentence. One class of such meanings is scalar implicatures, a phenomenon by which a speaker conveys the negation of a more informative utterance by producing a less informative utterance. This paper introduces a Natural Language Inference (NLI) dataset designed to investigate the ability of language models to interpret utterances with scalar implicatures. Our dataset is comprised of text extracted from the C4 English text corpus and annotated with both crowd-sourced and expert annotations. We evaluate NLI models based on DeBERTa to investigate 1) whether NLI models can learn to predict pragmatic inferences involving gradable adjectives and 2) whether models generalize to utterances involving unseen adjectives. We find that fine-tuning NLI models on our dataset significantly improves their performance to derive scalar implicatures, both for in-domain and for out-of domain examples. At the same time, we find that the investigated models still perform considerably worse on examples with scalar implicatures than on other types of NLI examples, highlighting that pragmatic inferences still pose challenges for current models.
Anthology ID:
2024.lrec-main.1288
Volume:
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Venues:
LREC | COLING
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
14784–14795
Language:
URL:
https://aclanthology.org/2024.lrec-main.1288
DOI:
Bibkey:
Cite (ACL):
Rashid Nizamani, Sebastian Schuster, and Vera Demberg. 2024. SIGA: A Naturalistic NLI Dataset of English Scalar Implicatures with Gradable Adjectives. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14784–14795, Torino, Italia. ELRA and ICCL.
Cite (Informal):
SIGA: A Naturalistic NLI Dataset of English Scalar Implicatures with Gradable Adjectives (Nizamani et al., LREC-COLING 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-2/2024.lrec-main.1288.pdf
PDF Search