@inproceedings{zeng-etal-2025-enhanced,
    title = "Enhanced Evaluative Language Annotation through Refined Theoretical Framework and Workflow",
    author = "Zeng, Jiamei  and
      Wang, Haitao  and
      Bunt, Harry  and
      Cao, Xinyu  and
      Cardey, Sylviane  and
      Dong, Min  and
      Hao, Tianyong  and
      Jia, Yangli  and
      Lee, Kiyong  and
      Liao, Shengqing  and
      Pustejovsky, James  and
      Rey, Fran{\c{c}}ois Claude  and
      Romary, Laurent  and
      Zong, Jianfang  and
      Fang, Alex C.",
    editor = "Harry, Bunt",
    booktitle = "Proceedings of the 21st Joint ACL - ISO Workshop on Interoperable Semantic Annotation (ISA-21)",
    month = sep,
    year = "2025",
    address = {D{\"u}sseldorf, Germany},
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/iwcs-25-ingestion/2025.isa-1.8/",
    pages = "76--84",
    ISBN = "979-8-89176-319-7",
    abstract = "As precursor work in preparation for an international standard \textit{ISO/PWI 24617-16 Language resource management {--} Semantic annotation {--} Part 16: Evaluative language}, we aim to test and enhance the reliability of the annotation of subjective evaluation based on Appraisal Theory. We describe a comprehensive three-phase workflow tested on COVID-19 media reports to achieve reliable agreement through progressive training and quality control. Our methodology addresses some of the key challenges through the refinement of targeted guideline refinements and the development of interactive clarification tools, alongside a custom platform that enables the pre-classification of six evaluative categories, systematic annotation review, and organized documentation. We report empirical results that demonstrate substantial improvements from the initial moderate agreement to a strong final consensus. Our research offers both theoretical refinements addressing persistent classification challenges in evaluation and practical solutions for the implementation of the annotation workflow, proposing a replicable methodology for the achievement of reliable annotation consistency in the annotation of evaluative language."
}