@inproceedings{peng-etal-2020-reducing,
    title = "Reducing Non-Normative Text Generation from Language Models",
    author = "Peng, Xiangyu  and
      Li, Siyan  and
      Frazier, Spencer  and
      Riedl, Mark",
    editor = "Davis, Brian  and
      Graham, Yvette  and
      Kelleher, John  and
      Sripada, Yaji",
    booktitle = "Proceedings of the 13th International Conference on Natural Language Generation",
    month = dec,
    year = "2020",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2020.inlg-1.43/",
    doi = "10.18653/v1/2020.inlg-1.43",
    pages = "374--383",
    abstract = "Large-scale, transformer-based language models such as GPT-2 are pretrained on diverse corpora scraped from the internet. Consequently, they are prone to generating non-normative text (i.e. in violation of social norms). We introduce a technique for fine-tuning GPT-2, using a policy gradient reinforcement learning technique and a normative text classifier to produce reward and punishment values. We evaluate our technique on five data sets using automated and human participant experiments. The normative text classifier is 81-90{\%} accurate when compared to gold-standard human judgements of normative and non-normative generated text. Our normative fine-tuning technique is able to reduce non-normative text by 27-61{\%}, depending on the data set."
}