@inproceedings{singh-etal-2024-racer,
    title = "{RACER}: An {LLM}-powered Methodology for Scalable Analysis of Semi-structured Mental Health Interviews",
    author = "Singh, Satpreet Harcharan  and
      Jiang, Kevin  and
      Bhasin, Kanchan  and
      Sabharwal, Ashutosh  and
      Moukaddam, Nidal  and
      Patel, Ankit",
    editor = "Peled-Cohen, Lotem  and
      Calderon, Nitay  and
      Lissak, Shir  and
      Reichart, Roi",
    booktitle = "Proceedings of the 1st Workshop on NLP for Science (NLP4Science)",
    month = nov,
    year = "2024",
    address = "Miami, FL, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.nlp4science-1.8/",
    doi = "10.18653/v1/2024.nlp4science-1.8",
    pages = "73--98",
    abstract = "Semi-structured interviews (SSIs) are a commonly employed data-collection method in healthcare research, offering in-depth qualitative insights into subject experiences. Despite their value, manual analysis of SSIs is notoriously time-consuming and labor-intensive, in part due to the difficulty of extracting and categorizing emotional responses, and challenges in scaling human evaluation for large populations. In this study, we develop RACER, a Large Language Model (LLM) based expert-guided automated pipeline that efficiently converts raw interview transcripts into insightful domain-relevant themes and sub-themes. We used RACER to analyze SSIs conducted with 93 healthcare professionals and trainees to assess the broad personal and professional mental health impacts of the COVID-19 crisis. RACER achieves moderately high agreement with two human evaluators (72{\%}), which approaches the human inter-rater agreement (77{\%}). Interestingly, LLMs and humans struggle with similar content involving nuanced emotional, ambivalent/dialectical, and psychological statements. Our study highlights the opportunities and challenges in using LLMs to improve research efficiency and opens new avenues for scalable analysis of SSIs in healthcare research."
}