@inproceedings{byun-choi-2025-gen,
    title = "\textit{ {D}-{GEN}}: Automatic Distractor Generation and Evaluation for Reliable Assessment of Generative Models",
    author = "Byun, Grace  and
      Choi, Jinho D.",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.findings-acl.174/",
    pages = "3316--3349",
    ISBN = "979-8-89176-256-5",
    abstract = "Evaluating generative models with open-ended generation is challenging due to inconsistencies in response formats. Multiple-choice (MC) evaluation mitigates this issue, but generating high-quality distractors is time-consuming and labor-intensive. We introduce \textit{D-GEN}, the first open-source distractor generator model that transforms open-ended data into an MC format. To evaluate distractor quality, we propose two novel methods: 1) ranking alignment, ensuring generated distractors retain the discriminatory power of ground-truth distractors, and 2) entropy analysis, comparing model confidence distributions. Our results show that \textit{D-GEN} preserves ranking consistency (Spearman{'}s $\rho$ 0.99, Kendall{'}s $\tau$ 0.94) and closely matches the entropy distribution of ground-truth distractors. Human evaluation further confirms the fluency, coherence, distractiveness, and incorrectness. Our work advances robust and efficient distractor generation with automated evaluation, setting a new standard for MC evaluation."
}