@inproceedings{darcy-etal-2024-aries,
    title = "{ARIES}: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews",
    author = "D{'}Arcy, Mike  and
      Ross, Alexis  and
      Bransom, Erin  and
      Kuehl, Bailey  and
      Bragg, Jonathan  and
      Hope, Tom  and
      Downey, Doug",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.acl-long.377/",
    doi = "10.18653/v1/2024.acl-long.377",
    pages = "6985--7001",
    abstract = "We introduce the task of automatically revising scientific papers based on peer feedback and release ARIES, a dataset of review comments and their corresponding paper edits. The data is drawn from real reviewer-author interactions from computer science, and we provide labels linking each reviewer comment to the specific paper edits made by the author in response. We automatically create a high-precision silver training set, as well as an expert-labeled test set that shows high inter-annotator agreement. In experiments with 10 models covering the state of the art, we find that they struggle even to identify which edits correspond to a comment{---}especially when the relationship between the edit and the comment is indirect and requires reasoning to uncover. We also extensively analyze GPT-4{'}s ability to generate edits given a comment and the original paper. We find that it often succeeds on a superficial level, but tends to rigidly follow the wording of the feedback rather than the underlying intent, and lacks technical details compared to human-written edits."
}