@inproceedings{knowles-etal-2024-mslc24,
    title = "{MSLC}24: Further Challenges for Metrics on a Wide Landscape of Translation Quality",
    author = "Knowles, Rebecca  and
      Larkin, Samuel  and
      Lo, Chi-Kiu",
    editor = "Haddow, Barry  and
      Kocmi, Tom  and
      Koehn, Philipp  and
      Monz, Christof",
    booktitle = "Proceedings of the Ninth Conference on Machine Translation",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.wmt-1.34/",
    doi = "10.18653/v1/2024.wmt-1.34",
    pages = "475--491",
    abstract = "In this second edition of the Metric Score Landscape Challenge (MSLC), we examine how automatic metrics for machine translation perform on a wide variety of machine translation output, ranging from very low quality systems to the types of high-quality systems submitted to the General MT shared task at WMT. We also explore metric results on specific types of data, such as empty strings, wrong- or mixed-language text, and more. We raise several alarms about inconsistencies in metric scores, some of which can be resolved by increasingly explicit instructions for metric use, while others highlight technical flaws."
}