@inproceedings{vallejo-etal-2022-evaluating,
    title = "Evaluating the Examiner: The Perils of {P}earson Correlation for Validating Text Similarity Metrics",
    author = "Vallejo, Gisela  and
      Baldwin, Timothy  and
      Frermann, Lea",
    editor = "Parameswaran, Pradeesh  and
      Biggs, Jennifer  and
      Powers, David",
    booktitle = "Proceedings of the 20th Annual Workshop of the Australasian Language Technology Association",
    month = dec,
    year = "2022",
    address = "Adelaide, Australia",
    publisher = "Australasian Language Technology Association",
    url = "https://preview.aclanthology.org/ingest-emnlp/2022.alta-1.18/",
    pages = "130--138",
    abstract = "In recent years, researchers have developed question-answering based approaches to automatically evaluate system summaries, reporting improved validity compared to word overlap-based metrics like ROUGE, in terms of correlation with human ratings of criteria including fluency and hallucination. In this paper, we take a closer look at one particular metric, QuestEval, and ask whether: (1) it can serve as a more general metric for long document similarity assessment; and (2) a single correlation score between metric scores and human ratings, as the currently standard approach, is sufficient for metric validation. We find that correlation scores can be misleading, and that score distributions and outliers should be taken into account. With these caveats in mind, QuestEval can be a promising candidate for long document similarity assessment."
}