Modelling Variability in Human Annotator Simulation

Wen Wu, Wenlin Chen, Chao Zhang, Phil Woodland

Abstract
Human annotator simulation (HAS) serves as a cost-effective substitute for human evaluation tasks such as data annotation and system assessment. It is important to incorporate the variability present in human evaluation into HAS, since it helps capture diverse subjective interpretations and mitigate potential biases and over-representation. This work introduces a novel framework for modelling variability in HAS. Conditional softmax flow (S-CNF) is proposed to model the distribution of subjective human annotations, which leverages diverse human annotations via meta-learning. This enables efficient generation of annotations that exhibit human variability for unlabelled input. In addition, a wide range of evaluation metrics are adopted to assess the capability and efficiency of HAS systems in predicting the aggregated behaviours of human annotators, matching the distribution of human annotations, and simulating the inter-annotator disagreements. Results demonstrate that the proposed method achieves state-of-the-art performance on two real-world human evaluation tasks: emotion recognition and toxic speech detection.
Anthology ID:
2024.findings-acl.67
Volume:
Findings of the Association for Computational Linguistics ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand and virtual meeting
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1139–1157
Language:
URL:
https://aclanthology.org/2024.findings-acl.67
DOI:
Bibkey:
Cite (ACL):
Wen Wu, Wenlin Chen, Chao Zhang, and Phil Woodland. 2024. Modelling Variability in Human Annotator Simulation. In Findings of the Association for Computational Linguistics ACL 2024, pages 1139–1157, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
Modelling Variability in Human Annotator Simulation (Wu et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.findings-acl.67.pdf
PDF Search