Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant
Joseph Matveyenko, James Liu, John David Parsons, Ryan Brown, Alina I. Palimaru, Vipul Gupta, Prateek Puri
Abstract
Qualitative research emphasizes constructing meaning through iterative engagement with textual data. Traditionally, this human-driven process requires navigating coder fatigue and interpretive drift, thus posing challenges when scaling analysis to larger, more complex datasets. Computational approaches to augment qualitative research have been met with skepticism, partly due to their inability to replicate the nuance, context-awareness, and sophistication of human analysis. LLMs, however, present new opportunities to automate aspects of qualitative analysis while upholding rigor and research quality. In this work, we present and benchmark Muse, an interactive qualitative research system that allows researchers to identify themes and annotate datasets, achieving an inter-rater reliability between Muse and humans of Cohen’s 𝜅 = 0.7 for well-specified codes.- Anthology ID:
- 2026.acl-industry.131
- Volume:
- Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)
- Month:
- July
- Year:
- 2026
- Address:
- San Diego, California, USA
- Editors:
- Yunyao Li, Georg Rehm, Mei Tu
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1917–1932
- Language:
- URL:
- https://preview.aclanthology.org/ingest-acl/2026.acl-industry.131/
- DOI:
- Cite (ACL):
- Joseph Matveyenko, James Liu, John David Parsons, Ryan Brown, Alina I. Palimaru, Vipul Gupta, and Prateek Puri. 2026. Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), pages 1917–1932, San Diego, California, USA. Association for Computational Linguistics.
- Cite (Informal):
- Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant (Matveyenko et al., ACL 2026)
- PDF:
- https://preview.aclanthology.org/ingest-acl/2026.acl-industry.131.pdf