Using Information Retrieval Techniques to Automatically Repurpose Existing Dialogue Datasets for Safe Chatbot Development
Tunde Oluwaseyi Ajayi, Gaurav Negi, Mihael Arcan, Paul Buitelaar
Abstract
There has been notable progress in the development of open-domain dialogue systems (chatbots) especially with the rapid advancement of the capabilities of Large Language Models. Chatbots excel at holding conversations in a manner that keeps a user interested and engaged. However, their responses can be unsafe, as they can respond in an offensive manner or offer harmful professional advice. As a way to mitigate this issue, recent work crowdsource datasets with exemplary responses or annotate dialogue safety datasets, which are relatively scarce compared to casual dialogues. Despite the quality of data obtained from crowdsourcing, it can be expensive and time consuming. This work proposes an effective pipeline, using information retrieval, to automatically repurpose existing dialogue datasets for safe chatbot development, as a way to address the aforementioned challenges. We select an existing dialogue dataset, revise its unsafe responses, as a way to obtain a dataset with safer responses to unsafe user inputs. We then fine-tune dialogue models on the original and revised datasets and generate responses to evaluate the safeness of the models.- Anthology ID:
- 2024.safety4convai-1.3
- Volume:
- Proceedings of Safety4ConvAI: The Third Workshop on Safety for Conversational AI @ LREC-COLING 2024
- Month:
- May
- Year:
- 2024
- Address:
- Torino, Italia
- Editors:
- Tanvi Dinkar, Giuseppe Attanasio, Amanda Cercas Curry, Ioannis Konstas, Dirk Hovy, Verena Rieser
- Venues:
- Safety4ConvAI | WS
- SIG:
- Publisher:
- ELRA and ICCL
- Note:
- Pages:
- 16–27
- Language:
- URL:
- https://aclanthology.org/2024.safety4convai-1.3
- DOI:
- Cite (ACL):
- Tunde Oluwaseyi Ajayi, Gaurav Negi, Mihael Arcan, and Paul Buitelaar. 2024. Using Information Retrieval Techniques to Automatically Repurpose Existing Dialogue Datasets for Safe Chatbot Development. In Proceedings of Safety4ConvAI: The Third Workshop on Safety for Conversational AI @ LREC-COLING 2024, pages 16–27, Torino, Italia. ELRA and ICCL.
- Cite (Informal):
- Using Information Retrieval Techniques to Automatically Repurpose Existing Dialogue Datasets for Safe Chatbot Development (Ajayi et al., Safety4ConvAI-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2024.safety4convai-1.3.pdf