Human-Model Divergence in the Handling of Vagueness
Elias Stengel-Eskin, Jimena Guallar-Blasco, Benjamin Van Durme
Abstract
While aggregate performance metrics can generate valuable insights at a large scale, their dominance means more complex and nuanced language phenomena, such as vagueness, may be overlooked. Focusing on vague terms (e.g. sunny, cloudy, young, etc.) we inspect the behavior of visually grounded and text-only models, finding systematic divergences from human judgments even when a model’s overall performance is high. To help explain this disparity, we identify two assumptions made by the datasets and models examined and, guided by the philosophy of vagueness, isolate cases where they do not hold.- Anthology ID:
- 2021.unimplicit-1.6
- Volume:
- Proceedings of the 1st Workshop on Understanding Implicit and Underspecified Language
- Month:
- August
- Year:
- 2021
- Address:
- Online
- Editors:
- Michael Roth, Reut Tsarfaty, Yoav Goldberg
- Venue:
- unimplicit
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 43–57
- Language:
- URL:
- https://aclanthology.org/2021.unimplicit-1.6
- DOI:
- 10.18653/v1/2021.unimplicit-1.6
- Cite (ACL):
- Elias Stengel-Eskin, Jimena Guallar-Blasco, and Benjamin Van Durme. 2021. Human-Model Divergence in the Handling of Vagueness. In Proceedings of the 1st Workshop on Understanding Implicit and Underspecified Language, pages 43–57, Online. Association for Computational Linguistics.
- Cite (Informal):
- Human-Model Divergence in the Handling of Vagueness (Stengel-Eskin et al., unimplicit 2021)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2021.unimplicit-1.6.pdf