Abstract
Humans tend to strongly agree on ratings on a scale for extreme cases (e.g., a CAT is judged as very concrete), but judgements on mid-scale words exhibit more disagreement. Yet, collected rating norms are heavily exploited across disciplines. Our study focuses on concreteness ratings and (i) implements correlations and supervised classification to identify salient multi-modal characteristics of mid-scale words, and (ii) applies a hard clustering to identify patterns of systematic disagreement across raters. Our results suggest to either fine-tune or filter mid-scale target words before utilising them.- Anthology ID:
- 2023.conll-1.6
- Volume:
- Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Jing Jiang, David Reitter, Shumin Deng
- Venue:
- CoNLL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 70–86
- Language:
- URL:
- https://aclanthology.org/2023.conll-1.6
- DOI:
- 10.18653/v1/2023.conll-1.6
- Cite (ACL):
- Urban Knupleš, Diego Frassinelli, and Sabine Schulte im Walde. 2023. Investigating the Nature of Disagreements on Mid-Scale Ratings: A Case Study on the Abstractness-Concreteness Continuum. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 70–86, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Investigating the Nature of Disagreements on Mid-Scale Ratings: A Case Study on the Abstractness-Concreteness Continuum (Knupleš et al., CoNLL 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2023.conll-1.6.pdf