Overview of the DagPap24 Shared Task on Detecting Automatically Generated Scientific Paper
Savvas Chamezopoulos, Drahomira Herrmannova, Anita De Waard, Drahomira Herrmannova, Domenic Rosati, Yury Kashnitsky
Abstract
This paper provides an overview of the 2024 ACL Scholarly Document Processing workshop shared task on the detection of automatically generated scientific papers. Unlike our previous task, which focused on the binary classification of whether scientific passages were machine-generated or not, one likely use case for text generation technology in scientific writing is to intersperse human-written text with passages of machine-generated text. We frame the detection problem as a multiclass span classification task: given an expert of text, label token spans in the text as human-written or machine-generated We shared a dataset containing excerpts from human-written papers as well as artificially generated content collected by Elsevier publishing and editorial teams. As a test set, the participants were provided with a corpus of openly accessible human-written as well as generated papers from the same scientific domains of documents. The shared task saw 457 submissions across 28 participating teams and resulted in three published technical reports. We discuss our findings from the shared task in this overview paper.- Anthology ID:
- 2024.sdp-1.2
- Volume:
- Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024)
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Tirthankar Ghosal, Amanpreet Singh, Anita Waard, Philipp Mayr, Aakanksha Naik, Orion Weller, Yoonjoo Lee, Shannon Shen, Yanxia Qin
- Venues:
- sdp | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 7–11
- Language:
- URL:
- https://aclanthology.org/2024.sdp-1.2
- DOI:
- Cite (ACL):
- Savvas Chamezopoulos, Drahomira Herrmannova, Anita De Waard, Drahomira Herrmannova, Domenic Rosati, and Yury Kashnitsky. 2024. Overview of the DagPap24 Shared Task on Detecting Automatically Generated Scientific Paper. In Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024), pages 7–11, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Overview of the DagPap24 Shared Task on Detecting Automatically Generated Scientific Paper (Chamezopoulos et al., sdp-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.sdp-1.2.pdf