@inproceedings{schoch-etal-2020-problem,
    title = "{``}This is a Problem, Don{'}t You Agree?{''} Framing and Bias in Human Evaluation for Natural Language Generation",
    author = "Schoch, Stephanie  and
      Yang, Diyi  and
      Ji, Yangfeng",
    editor = "Agarwal, Shubham  and
      Du{\v{s}}ek, Ond{\v{r}}ej  and
      Gehrmann, Sebastian  and
      Gkatzia, Dimitra  and
      Konstas, Ioannis  and
      Van Miltenburg, Emiel  and
      Santhanam, Sashank",
    booktitle = "Proceedings of the 1st Workshop on Evaluating NLG Evaluation",
    month = dec,
    year = "2020",
    address = "Online (Dublin, Ireland)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2020.evalnlgeval-1.2",
    pages = "10--16",
    abstract = "Despite recent efforts reviewing current human evaluation practices for natural language generation (NLG) research, the lack of reported question wording and potential for framing effects or cognitive biases influencing results has been widely overlooked. In this opinion paper, we detail three possible framing effects and cognitive biases that could be imposed on human evaluation in NLG. Based on this, we make a call for increased transparency for human evaluation in NLG and propose the concept of human evaluation statements. We make several recommendations for design details to report that could potentially influence results, such as question wording, and suggest that reporting pertinent design details can help increase comparability across studies as well as reproducibility of results.",
}