@inproceedings{yujie-etal-2024-cost,
    title = "Cost-efficient Crowdsourcing for Span-based Sequence Labeling:Worker Selection and Data Augmentation",
    author = "Wang, Yujie  and
      Huang, Chao  and
      Yang, Liner  and
      Fang, Zhixuan  and
      Huang, Yaping  and
      Liu, Yang  and
      Yu, Jingsi  and
      Yang, Erhong",
    editor = "Maosong, Sun  and
      Jiye, Liang  and
      Xianpei, Han  and
      Zhiyuan, Liu  and
      Yulan, He",
    booktitle = "Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference)",
    month = jul,
    year = "2024",
    address = "Taiyuan, China",
    publisher = "Chinese Information Processing Society of China",
    url = "https://preview.aclanthology.org/author-degibert/2024.ccl-1.96/",
    pages = "1239--1256",
    language = "eng",
    abstract = "``This paper introduces a novel crowdsourcing worker selection algorithm, enhancing annotationquality and reducing costs. Unlike previous studies targeting simpler tasks, this study con-tends with the complexities of label interdependencies in sequence labeling. The proposedalgorithm utilizes a Combinatorial Multi-Armed Bandit (CMAB) approach for worker selec-tion, and a cost-effective human feedback mechanism. The challenge of dealing with imbal-anced and small-scale datasets, which hinders offline simulation of worker selection, is tack-led using an innovative data augmentation method termed shifting, expanding, and shrink-ing (SES). Rigorous testing on CoNLL 2003 NER and Chinese OEI datasets showcased thealgorithm{'}s efficiency, with an increase in F1 score up to 100.04{\%} of the expert-only base-line, alongside cost savings up to 65.97{\%}. The paper also encompasses a dataset-independenttest emulating annotation evaluation through a Bernoulli distribution, which still led to animpressive 97.56{\%} F1 score of the expert baseline and 59.88{\%} cost savings. Furthermore,our approach can be seamlessly integrated into Reinforcement Learning from Human Feed-back (RLHF) systems, offering a cost-effective solution for obtaining human feedback. All re-sources, including source code and datasets, are available to the broader research community athttps://github.com/blcuicall/nlp-crowdsourcing.''"
}