@inproceedings{sun-etal-2025-posum,
    title = "{P}o{S}um-Bench: Benchmarking Position Bias in {LLM}-based Conversational Summarization",
    author = "Sun, Xu  and
      Delphin-Poulat, Lionel  and
      Tarnec, Christ{\`e}le  and
      Shimorina, Anastasia",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.404/",
    pages = "7996--8020",
    ISBN = "979-8-89176-332-6",
    abstract = "Large language models (LLMs) are increasingly used for zero-shot conversation summarization, but often exhibit positional bias{---}tending to overemphasize content from the beginning or end of a conversation while neglecting the middle. To address this issue, we introduce PoSum-Bench, a comprehensive benchmark for evaluating positional bias in conversational summarization, featuring diverse English and French conversational datasets spanning formal meetings, casual conversations, and customer service interactions. We propose a novel semantic similarity-based sentence-level metric to quantify the direction and magnitude of positional bias in model-generated summaries, enabling systematic and reference-free evaluation across conversation positions, languages, and conversational contexts.Our benchmark and methodology thus provide the first systematic, cross-lingual framework for reference-free evaluation of positional bias in conversational summarization, laying the groundwork for developing more balanced and unbiased summarization models."
}