Evaluating Factuality in Text Simplification
Ashwin Devaraj, William Sheffield, Byron Wallace, Junyi Jessy Li
Abstract
Automated simplification models aim to make input texts more readable. Such methods have the potential to make complex information accessible to a wider audience, e.g., providing access to recent medical literature which might otherwise be impenetrable for a lay reader. However, such models risk introducing errors into automatically simplified texts, for instance by inserting statements unsupported by the corresponding original text, or by omitting key information. Providing more readable but inaccurate versions of texts may in many cases be worse than providing no such access at all. The problem of factual accuracy (and the lack thereof) has received heightened attention in the context of summarization models, but the factuality of automatically simplified texts has not been investigated. We introduce a taxonomy of errors that we use to analyze both references drawn from standard simplification datasets and state-of-the-art model outputs. We find that errors often appear in both that are not captured by existing evaluation metrics, motivating a need for research into ensuring the factual accuracy of automated simplification models.- Anthology ID:
- 2022.acl-long.506
- Volume:
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Smaranda Muresan, Preslav Nakov, Aline Villavicencio
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 7331–7345
- Language:
- URL:
- https://aclanthology.org/2022.acl-long.506
- DOI:
- 10.18653/v1/2022.acl-long.506
- Award:
- Outstanding Paper
- Cite (ACL):
- Ashwin Devaraj, William Sheffield, Byron Wallace, and Junyi Jessy Li. 2022. Evaluating Factuality in Text Simplification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7331–7345, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Evaluating Factuality in Text Simplification (Devaraj et al., ACL 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.acl-long.506.pdf
- Code
- ashologn/evaluating-factuality-in-text-simplification
- Data
- Newsela, WikiLarge