@inproceedings{aubert-b-educhaud-etal-2026-towards,
    title = "Towards Reliable Paper Contributions Annotation in the {ACL} Rolling Review",
    author = "Aubert-B{\textbackslash}'educhaud, Julien  and
      Boudin, Florian  and
      Aizawa, Akiko  and
      Daille, Beatrice  and
      Dufour, Richard",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.178/",
    pages = "3636--3653",
    ISBN = "979-8-89176-395-1",
    abstract = "With the rapid growth of scientific publications, researchers struggle to efficiently assess the relevance of numerous papers. Identifying the types of contributions an article makes can help readers quickly grasp its significance. The ACL Rolling Review (ARR) introduced a typology requiring authors to specify their contributions to improve review quality and fairness. However, the current typology lacks clear definitions and guidance, leading to inconsistent labeling and raising concerns about its reliability.Our re-annotation campaign reveals substantial disagreement between authors and domain experts. Moreover, the predictions of large language models (LLMs), when compared with expert annotations, tend to be close to those provided by the authors. These findings suggest a potential path toward better annotation reliability within the ARR process."
}