@inproceedings{florescu-etal-2025-reprohum,
    title = "{R}epro{H}um {\#}0033-05: Human Evaluation of Factuality from A Multidisciplinary Perspective",
    author = "Florescu, Andra-Maria  and
      Micluța-C{\^a}mpeanu, Marius  and
      Tabusca, Stefana Arina  and
      Dinu, Liviu P",
    editor = "Dhole, Kaustubh  and
      Clinciu, Miruna",
    booktitle = "Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM{\texttwosuperior})",
    month = jul,
    year = "2025",
    address = "Vienna, Austria and virtual meeting",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2025-08/2025.gem-1.53/",
    pages = "583--589",
    ISBN = "979-8-89176-261-9",
    abstract = "The following paper is a joint contribution for the 2025 ReproNLP shared task, part of the ReproHum project. We focused on reproducing the human evaluation based on one criterion, namely, factuality of Scientific Automated Generated Systems from August et al. (2022). In accordance to the ReproHum guidelines, we followed the original study as closely as possible, with two human raters who coded 300 ratings each. Moreover, we had an additional study on two subsets of the dataset based on domain (medicine and physics) in which we employed expert annotators. Our reproduction of the factuality assessment found similar overall rates of factual inaccuracies across models. However, variability and weak agreement with the original model rankings suggest challenges in reliably reproducing results, especially in such cases when results are close."
}