@inproceedings{nikishina-etal-2026-argument,
    title = "Argument-Based Comparative Question Answering Evaluation Benchmark",
    author = "Nikishina, Irina  and
      Anwar, Saba  and
      Dolgov, Nikolay  and
      Manina, Maria  and
      Ignatenko, Daria  and
      Moskvoretskii, Viktor  and
      Shelmanov, Artem  and
      Baldwin, Tim  and
      Biemann, Chris",
    editor = "Elaraby, Mohamed  and
      Hautli-Janisz, Annette  and
      Romberg, Julia  and
      Musi, Elena  and
      Ruggeri, Federico  and
      Lawrence, John",
    booktitle = "Proceedings of the 13th Workshop on Argument Mining and Reasoning",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.argmining-1.6/",
    pages = "43--51",
    ISBN = "979-8-89176-399-9",
    abstract = "Despite the ability of large language models (LLMs) to generate coherent comparative answers, automatic comparative question answering (CQA) remains challenging due to the absence of standardized evaluation criteria and the high resource demands of manual assessment. To address these problems, this paper proposes a comprehensive evaluation framework designed to assess the quality of CQA summaries using LLMs-as-a-Judge. We formulate 15 evaluation criteria for assessing comparative answers generated by various sources, including LLMs, human experts, and prior work. To capture a diverse range of comparative answers, LLM summaries were generated under various prompting scenarios. We evaluate the effectiveness of our framework using both human assessment and LLMs, demonstrating the consistency between automated and manual evaluations. Finally, we fine-tune Llama-3-8B-Instruct on a dataset generated from the best-performing CQA models in our evaluation."
}