@inproceedings{an-etal-2025-transparent,
    title = "Transparent Reference-free Automated Evaluation of Open-Ended User Survey Responses",
    author = "An, Subin  and
      Ji, Yugyeong  and
      Kim, Junyoung  and
      Kook, Heejin  and
      Lu, Yang  and
      Seltzer, Josh",
    editor = "Potdar, Saloni  and
      Rojas-Barahona, Lina  and
      Montella, Sebastien",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2025",
    address = "Suzhou (China)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-luhme/2025.emnlp-industry.65/",
    doi = "10.18653/v1/2025.emnlp-industry.65",
    pages = "963--982",
    ISBN = "979-8-89176-333-3",
    abstract = "Open-ended survey responses provide valuable insights in marketing research, but low-quality responses not only burden researchers with manual filtering but also risk leading to misleading conclusions, underscoring the need for effective evaluation. Existing automatic evaluation methods target LLM-generated text and inadequately assess human-written responses with their distinct characteristics. To address such characteristics, we propose a two-stage evaluation framework specifically designed for human survey responses. First, gibberish filtering removes nonsensical responses. Then, three dimensions{---}effort, relevance, and complete- ness{---}are evaluated using LLM capabilities, grounded in empirical analysis of real-world survey data. Validation on English and Korean datasets shows that our framework not only outperforms existing metrics but also demonstrates high practical applicability for real-world applications such as response quality prediction and response rejection, showing strong correlations with expert assessment."
}