@inproceedings{goliakova-etal-2025-metric,
    title = "Metric assessment protocol in the context of answer fluctuation on {MCQ} tasks",
    author = "Goliakova, Ekaterina  and
      Renard, Xavier  and
      Lesot, Marie-Jeanne  and
      Laugel, Thibault  and
      Marsala, Christophe  and
      Detyniecki, Marcin",
    editor = "Dhole, Kaustubh  and
      Clinciu, Miruna",
    booktitle = "Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM{\texttwosuperior})",
    month = jul,
    year = "2025",
    address = "Vienna, Austria and virtual meeting",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/transition-to-people-yaml/2025.gem-1.26/",
    pages = "302--319",
    ISBN = "979-8-89176-261-9",
    abstract = "Using multiple-choice questions (MCQs) has become a standard for assessing LLM capabilities efficiently. A variety of metrics can be employed for this task. However, previous research has not conducted a thorough assessment of them. At the same time, MCQ evaluation suffers from answer fluctuation: models produce different results given slight changes in prompts. We suggest a metric assessment protocol in which evaluation methodologies are analyzed through their connection with fluctuation rates, as well as original performance. Our results show that there is a strong link between existing metrics and the answer changing, even when computed without any additional prompt variants. Highest association on the protocol is demonstrated by a novel metric, worst accuracy."
}