@inproceedings{weber-etal-2021-better,
    title = "It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale {NLU} models",
    author = "Weber, Verena  and
      Piovano, Enrico  and
      Bradford, Melanie",
    editor = "Dragut, Eduard  and
      Li, Yunyao  and
      Popa, Lucian  and
      Vucetic, Slobodan",
    booktitle = "Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances",
    month = jun,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.dash-1.2",
    doi = "10.18653/v1/2021.dash-1.2",
    pages = "8--15",
    abstract = "When a NLU model is updated, new utter- ances must be annotated to be included for training. However, manual annotation is very costly. We evaluate a semi-supervised learning workflow with a human in the loop in a produc- tion environment. The previous NLU model predicts the annotation of the new utterances, a human then reviews the predicted annotation. Only when the NLU prediction is assessed as incorrect the utterance is sent for human anno- tation. Experimental results show that the pro- posed workflow boosts the performance of the NLU model while significantly reducing the annotation volume. Specifically, in our setup, we see improvements of up to 14.16{\%} for a recall-based metric and up to 9.57{\%} for a F1- score based metric, while reducing the annota- tion volume by 97{\%} and overall cost by 60{\%} for each iteration.",
}