@inproceedings{yujie-etal-2024-cost,
    title = "Cost-efficient Crowdsourcing for Span-based Sequence Labeling:Worker Selection and Data Augmentation",
    author = "Yujie, Wang  and
      Chao, Huang  and
      Liner, Yang  and
      Zhixuan, Fang  and
      Yaping, Huang  and
      Yang, Liu  and
      Jingsi, Yu  and
      Erhong, Yang",
    editor = "Sun, Maosong  and
      Liang, Jiye  and
      Han, Xianpei  and
      Liu, Zhiyuan  and
      He, Yulan",
    booktitle = "Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference)",
    month = jul,
    year = "2024",
    address = "Taiyuan, China",
    publisher = "Chinese Information Processing Society of China",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.ccl-1.96/",
    pages = "1239--1256",
    language = "eng",
    abstract = "``This paper introduces a novel crowdsourcing worker selection algorithm, enhancing annotationquality and reducing costs. Unlike previous studies targeting simpler tasks, this study con-tends with the complexities of label interdependencies in sequence labeling. The proposedalgorithm utilizes a Combinatorial Multi-Armed Bandit (CMAB) approach for worker selec-tion, and a cost-effective human feedback mechanism. The challenge of dealing with imbal-anced and small-scale datasets, which hinders offline simulation of worker selection, is tack-led using an innovative data augmentation method termed shifting, expanding, and shrink-ing (SES). Rigorous testing on CoNLL 2003 NER and Chinese OEI datasets showcased thealgorithm{'}s efficiency, with an increase in F1 score up to 100.04{\%} of the expert-only base-line, alongside cost savings up to 65.97{\%}. The paper also encompasses a dataset-independenttest emulating annotation evaluation through a Bernoulli distribution, which still led to animpressive 97.56{\%} F1 score of the expert baseline and 59.88{\%} cost savings. Furthermore,our approach can be seamlessly integrated into Reinforcement Learning from Human Feed-back (RLHF) systems, offering a cost-effective solution for obtaining human feedback. All re-sources, including source code and datasets, are available to the broader research community athttps://github.com/blcuicall/nlp-crowdsourcing.''"
}