@inproceedings{belz-etal-2023-missing,
    title = "Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in {NLP}",
    author = {Belz, Anya  and
      Thomson, Craig  and
      Reiter, Ehud  and
      Abercrombie, Gavin  and
      Alonso-Moral, Jose M.  and
      Arvan, Mohammad  and
      Braggaar, Anouck  and
      Cieliebak, Mark  and
      Clark, Elizabeth  and
      van Deemter, Kees  and
      Dinkar, Tanvi  and
      Du{\v{s}}ek, Ond{\v{r}}ej  and
      Eger, Steffen  and
      Fang, Qixiang  and
      Gao, Mingqi  and
      Gatt, Albert  and
      Gkatzia, Dimitra  and
      Gonz{\'a}lez-Corbelle, Javier  and
      Hovy, Dirk  and
      H{\"u}rlimann, Manuela  and
      Ito, Takumi  and
      Kelleher, John D.  and
      Klubicka, Filip  and
      Krahmer, Emiel  and
      Lai, Huiyuan  and
      van der Lee, Chris  and
      Li, Yiru  and
      Mahamood, Saad  and
      Mieskes, Margot  and
      van Miltenburg, Emiel  and
      Mosteiro, Pablo  and
      Nissim, Malvina  and
      Parde, Natalie  and
      Pl{\'a}tek, Ond{\v{r}}ej  and
      Rieser, Verena  and
      Ruan, Jie  and
      Tetreault, Joel  and
      Toral, Antonio  and
      Wan, Xiaojun  and
      Wanner, Leo  and
      Watson, Lewis  and
      Yang, Diyi},
    editor = "Tafreshi, Shabnam  and
      Akula, Arjun  and
      Sedoc, Jo{\~a}o  and
      Drozd, Aleksandr  and
      Rogers, Anna  and
      Rumshisky, Anna",
    booktitle = "Proceedings of the Fourth Workshop on Insights from Negative Results in NLP",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.insights-1.1",
    doi = "10.18653/v1/2023.insights-1.1",
    pages = "1--10",
    abstract = "We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which include that just 13{\%} of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, and that all but one of the experiments we selected for reproduction was discovered to have flaws that made the meaningfulness of conducting a reproduction questionable. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach. Our overall (negative) finding that the great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction, paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP.",
}