Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant

Joseph Matveyenko, James Liu, John David Parsons, Ryan Brown, Alina I. Palimaru, Vipul Gupta, Prateek Puri


Abstract
Qualitative research emphasizes constructing meaning through iterative engagement with textual data. Traditionally, this human-driven process requires navigating coder fatigue and interpretive drift, thus posing challenges when scaling analysis to larger, more complex datasets. Computational approaches to augment qualitative research have been met with skepticism, partly due to their inability to replicate the nuance, context-awareness, and sophistication of human analysis. LLMs, however, present new opportunities to automate aspects of qualitative analysis while upholding rigor and research quality. In this work, we present and benchmark Muse, an interactive qualitative research system that allows researchers to identify themes and annotate datasets, achieving an inter-rater reliability between Muse and humans of Cohen’s 𝜅 = 0.7 for well-specified codes.
Anthology ID:
2026.acl-industry.131
Volume:
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)
Month:
July
Year:
2026
Address:
San Diego, California, USA
Editors:
Yunyao Li, Georg Rehm, Mei Tu
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1917–1932
Language:
URL:
https://preview.aclanthology.org/ingest-acl/2026.acl-industry.131/
DOI:
Bibkey:
Cite (ACL):
Joseph Matveyenko, James Liu, John David Parsons, Ryan Brown, Alina I. Palimaru, Vipul Gupta, and Prateek Puri. 2026. Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), pages 1917–1932, San Diego, California, USA. Association for Computational Linguistics.
Cite (Informal):
Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant (Matveyenko et al., ACL 2026)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl/2026.acl-industry.131.pdf