Abstract
In the task of entity description generation, given a context and a specified entity, a model must describe that entity correctly and in a contextually-relevant way. In this task, as well as broader language generation tasks, the generation of a nonfactual description (factual error) versus an incongruous description (contextual error) is fundamentally different, yet often conflated. We develop an evaluation paradigm that enables us to disentangle these two types of errors in naturally occurring textual contexts. We find that factuality and congruity are often at odds, and that models specifically struggle with accurate descriptions of entities that are less familiar to people. This shortcoming of language models raises concerns around the trustworthiness of such models, since factual errors on less well-known entities are exactly those that a human reader will not recognize.- Anthology ID:
- 2023.acl-long.463
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 8322–8340
- Language:
- URL:
- https://aclanthology.org/2023.acl-long.463
- DOI:
- 10.18653/v1/2023.acl-long.463
- Cite (ACL):
- Navita Goyal, Ani Nenkova, and Hal Daumé III. 2023. Factual or Contextual? Disentangling Error Types in Entity Description Generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8322–8340, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Factual or Contextual? Disentangling Error Types in Entity Description Generation (Goyal et al., ACL 2023)
- PDF:
- https://preview.aclanthology.org/emnlp22-frontmatter/2023.acl-long.463.pdf