GAN-LM: Generative Adversarial Network using Language Models for Downstream Applications
Dae Yon Hwang, Yaroslav Nechaev, Cyprien de Lichy, Renxian Zhang
Abstract
In this work, we investigate Data Augmentation methods to improve the performance of state-of-the-art models for four different downstream tasks. Specifically, we propose Generative Adversarial Network using Language Models (GAN-LM) approach that combines a deep generative model with a pre-trained language model to produce diverse augmentations. We compare the GAN-LM to various conventional methods in non-contextual- and contextual-levels on four public datasets: ZESHEL for zero-shot entity linking, TREC for question classification, STS-B for sentence pairs semantic textual similarity (STS), and mSTS for multilingual sentence pairs STS. Additionally, we subsample these datasets to study the impact of such augmentations in low-resource settings where limited amounts of training data is available. Compared to the state-of-the-art methods in downstream tasks, we mostly achieve the best performance using GAN-LM approach. Finally, we investigate the way of combining the GAN-LM with other augmentation methods to complement our proposed approach. The developed code for reproducibility is included in the supplementary material.- Anthology ID:
- 2023.inlg-main.5
- Volume:
- Proceedings of the 16th International Natural Language Generation Conference
- Month:
- September
- Year:
- 2023
- Address:
- Prague, Czechia
- Editors:
- C. Maria Keet, Hung-Yi Lee, Sina Zarrieß
- Venues:
- INLG | SIGDIAL
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 69–79
- Language:
- URL:
- https://aclanthology.org/2023.inlg-main.5
- DOI:
- 10.18653/v1/2023.inlg-main.5
- Cite (ACL):
- Dae Yon Hwang, Yaroslav Nechaev, Cyprien de Lichy, and Renxian Zhang. 2023. GAN-LM: Generative Adversarial Network using Language Models for Downstream Applications. In Proceedings of the 16th International Natural Language Generation Conference, pages 69–79, Prague, Czechia. Association for Computational Linguistics.
- Cite (Informal):
- GAN-LM: Generative Adversarial Network using Language Models for Downstream Applications (Hwang et al., INLG-SIGDIAL 2023)
- PDF:
- https://preview.aclanthology.org/teach-a-man-to-fish/2023.inlg-main.5.pdf