@inproceedings{wan-etal-2025-noise,
    title = "From Noise to Nuance: Enriching Subjective Data Annotation through Qualitative Analysis",
    author = "Wan, Ruyuan  and
      Wang, Haonan  and
      Huang, Ting-Hao Kenneth  and
      Gao, Jie",
    editor = "Blodgett, Su Lin  and
      Curry, Amanda Cercas  and
      Dev, Sunipa  and
      Li, Siyan  and
      Madaio, Michael  and
      Wang, Jack  and
      Wu, Sherry Tongshuang  and
      Xiao, Ziang  and
      Yang, Diyi",
    booktitle = "Proceedings of the Fourth Workshop on Bridging Human-Computer Interaction and Natural Language Processing (HCI+NLP)",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.hcinlp-1.20/",
    pages = "240--254",
    ISBN = "979-8-89176-353-1",
    abstract = "Subjective data annotation (SDA) plays an important role in many NLP tasks, including sentiment analysis, toxicity detection, and bias identification. Conventional SDA often treats annotator disagreement as noise, overlooking its potential to reveal deeper insights. In contrast, qualitative data analysis (QDA) explicitly engages with diverse positionalities and treats disagreement as a meaningful source of knowledge. In this position paper, we argue that human annotators are a key source of valuable interpretive insights into subjective data beyond surface-level descriptions. Through a comparative analysis of SDA and QDA methodologies, we examine similarities and differences in task nature (e.g., human{'}s role, analysis content, cost, and completion conditions) and practice (annotation schema, annotation workflow, annotator selection, and evaluation). Based on this comparison, we propose five practical recommendations for enabling SDA to capture richer insights. We demonstrate these recommendations in a reinforcement learning from human feedback (RLHF) case study and envision that our interdisciplinary perspective will offer new directions for the field."
}