LLMs to Replace Crowdsourcing For Parallel Data Creation? The Case of Text Detoxification

Daniil Moskovskiy, Sergey Pletenev, Alexander Panchenko

Abstract
The lack of high-quality training data remains a significant challenge in NLP. Manual annotation methods, such as crowdsourcing, are costly, require intricate task design skills, and, if used incorrectly, may result in poor data quality. From the other hand, LLMs have demonstrated proficiency in many NLP tasks, including zero-shot and few-shot data annotation. However, they often struggle with text detoxification due to alignment constraints and fail to generate the required detoxified text. This work explores the potential of modern open source LLMs to annotate parallel data for text detoxification. Using the recent technique of activation patching, we generate a pseudo-parallel detoxification dataset based on ParaDetox. The detoxification model trained on our generated data shows comparable performance to the original dataset in automatic detoxification evaluation metrics and superior quality in manual evaluation and side-by-side comparisons.
Anthology ID:
2024.findings-emnlp.839
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2024
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
14361–14373
Language:
URL:
https://preview.aclanthology.org/build-pipeline-with-new-library/2024.findings-emnlp.839/
DOI:
10.18653/v1/2024.findings-emnlp.839
Bibkey:
Cite (ACL):
Daniil Moskovskiy, Sergey Pletenev, and Alexander Panchenko. 2024. LLMs to Replace Crowdsourcing For Parallel Data Creation? The Case of Text Detoxification. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14361–14373, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
LLMs to Replace Crowdsourcing For Parallel Data Creation? The Case of Text Detoxification (Moskovskiy et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/build-pipeline-with-new-library/2024.findings-emnlp.839.pdf
PDF Search