EditEval: An Instruction-Based Benchmark for Text Improvements
Jane Dwivedi-Yu, Timo Schick, Zhengbao Jiang, Maria Lomeli, Patrick Lewis, Gautier Izacard, Edouard Grave, Sebastian Riedel, Fabio Petroni
Abstract
Evaluation of text generation to date has primarily focused on content created sequentially, rather than improvements on a piece of text. Writing, however, is naturally an iterative and incremental process that requires expertise in different modular skills such as fixing outdated information or making the writing style more consistent. Even so, comprehensive evaluation of a model’s capacity to perform these skills and the ability to edit remains sparse. This work introduces EditEval: An instruction-based, benchmark and evaluation suite that leverages high-quality existing and new datasets in English for the automatic evaluation of editing capabilities, such as making text more cohesive and paraphrasing. We evaluate several pre-trained models, which shows that InstructGPT and PEER on average perform the best, but that most baselines fall below the supervised state-of-the-art, particularly when neutralizing and updating information. Our analysis also shows that commonly used metrics for editing tasks do not always correlate well, and that prompts leading to the strongest performance do not necessarily elicit strong performance across different models. Through the release of this benchmark (code and data available at https://github.com/facebookresearch/EditEval) and a publicly available leaderboard challenge, we hope to unlock future work on developing models more capable of controllable and iterative editing.- Anthology ID:
- 2024.conll-1.7
- Volume:
- Proceedings of the 28th Conference on Computational Natural Language Learning
- Month:
- November
- Year:
- 2024
- Address:
- Miami, FL, USA
- Editors:
- Libby Barak, Malihe Alikhani
- Venue:
- CoNLL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 69–83
- Language:
- URL:
- https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.conll-1.7/
- DOI:
- 10.18653/v1/2024.conll-1.7
- Cite (ACL):
- Jane Dwivedi-Yu, Timo Schick, Zhengbao Jiang, Maria Lomeli, Patrick Lewis, Gautier Izacard, Edouard Grave, Sebastian Riedel, and Fabio Petroni. 2024. EditEval: An Instruction-Based Benchmark for Text Improvements. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 69–83, Miami, FL, USA. Association for Computational Linguistics.
- Cite (Informal):
- EditEval: An Instruction-Based Benchmark for Text Improvements (Dwivedi-Yu et al., CoNLL 2024)
- PDF:
- https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.conll-1.7.pdf