Flatness-Aware Gradient Descent for Safe Conversational AI
Leila Khalatbari, Saeid Hosseini, Hossein Sameti, Pascale Fung
Abstract
As generative dialog models become ubiquitous in real-world applications, it is paramount to ensure a harmless generation. There are two major challenges when enforcing safety to open-domain chatbots. Firstly, it is impractical to provide training data reflecting the desired response to all emerging forms of toxicity (generalisation challenge). Secondly, implementing safety features may compromise the quality of the conversation (trade-off challenge). To tackle the challenges, this paper introduces a regularized fine-tuning approach called FlatGD. By employing a safety-tailored loss, we translate better optimization to more safety. To ensure better optimization, FlatGD penalizes sharp trajectories of loss curve, encouraging flatness of the converged local minima. Experimental results on datasets of “BAD” and “prosocial dialog” demonstrate that our model outperforms the current baselines in reducing toxicity while preserving the conversation quality. Moreover, compared to other baselines, FlatGD can better generalize to unseen toxic data.- Anthology ID:
- 2024.trustnlp-1.15
- Volume:
- Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024)
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Anaelia Ovalle, Kai-Wei Chang, Yang Trista Cao, Ninareh Mehrabi, Jieyu Zhao, Aram Galstyan, Jwala Dhamala, Anoop Kumar, Rahul Gupta
- Venues:
- TrustNLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 187–195
- Language:
- URL:
- https://aclanthology.org/2024.trustnlp-1.15
- DOI:
- 10.18653/v1/2024.trustnlp-1.15
- Cite (ACL):
- Leila Khalatbari, Saeid Hosseini, Hossein Sameti, and Pascale Fung. 2024. Flatness-Aware Gradient Descent for Safe Conversational AI. In Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024), pages 187–195, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- Flatness-Aware Gradient Descent for Safe Conversational AI (Khalatbari et al., TrustNLP-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.trustnlp-1.15.pdf