FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models
Kaixin Lan, Tao Fang, Derek Wong, Yabo Xu, Lidia Chao, Cecilia Zhao
Abstract
Pre-trained Language Models (PLMs) have shown impressive results in various Natural Language Generation (NLG) tasks, such as powering chatbots and generating stories. However, an ethical concern arises due to their potential to produce verbatim copies of paragraphs from their training data. This is problematic as PLMs are trained on corpora constructed by human authors. As such, there is a pressing need for research to promote the generation of original content by these models. In this study, we introduce a unique “self-plagiarism” contrastive decoding strategy, aimed at boosting the originality of text produced by PLMs. Our method entails modifying prompts in LLMs to develop an amateur model and a professional model. Specifically, the amateur model is urged to plagiarize using three plagiarism templates we have designed, while the professional model maintains its standard language model status. This strategy employs prompts to stimulate the model’s capacity to identify non-original candidate token combinations and subsequently impose penalties. The application of this strategy is integrated prior to the model’s final layer, ensuring smooth integration with most existing PLMs (T5, GPT, LLaMA) without necessitating further adjustments. Implementing our strategy, we noted a significant decline in non-original sequences comprised of more than three words in the academic AASC dataset and the story-based ROCStories dataset. Source code and scripts will be released after the paper’s acceptance and publication.- Anthology ID:
- 2024.findings-acl.858
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2024
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Lun-Wei Ku, Andre Martins, Vivek Srikumar
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 14432–14447
- Language:
- URL:
- https://aclanthology.org/2024.findings-acl.858
- DOI:
- 10.18653/v1/2024.findings-acl.858
- Cite (ACL):
- Kaixin Lan, Tao Fang, Derek Wong, Yabo Xu, Lidia Chao, and Cecilia Zhao. 2024. FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14432–14447, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models (Lan et al., Findings 2024)
- PDF:
- https://preview.aclanthology.org/autopr/2024.findings-acl.858.pdf