Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization

Yuxian Gu, Pei Ke, Xiaoyan Zhu, Minlie Huang


Abstract
Training language models to learn from human instructions for zero-shot cross-task generalization has attracted much attention in NLP communities. Recently, instruction tuning (IT), which fine-tunes a pre-trained language model on a massive collection of tasks described via human-craft instructions, has been shown effective in instruction learning for unseen tasks. However, IT relies on a large amount of human-annotated samples, which restricts its generalization. Unlike labeled data, unlabeled data are often massive and cheap to obtain. In this work, we study how IT can be improved with unlabeled data. We first empirically explore the IT performance trends versus the number of labeled data, instructions, and training tasks. We find it critical to enlarge the number of training instructions, and the instructions can be underutilized due to the scarcity of labeled data. Then, we propose Unlabeled Data Augmented Instruction Tuning (UDIT) to take better advantage of the instructions during IT by constructing pseudo-labeled data from unlabeled plain texts. We conduct extensive experiments to show UDIT’s effectiveness in various scenarios of tasks and datasets. We also comprehensively analyze the key factors of UDIT to investigate how to better improve IT with unlabeled data. The code is publicly available at https://github.com/thu-coai/UDIT.
Anthology ID:
2022.emnlp-main.105
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1617–1634
Language:
URL:
https://aclanthology.org/2022.emnlp-main.105
DOI:
Bibkey:
Cite (ACL):
Yuxian Gu, Pei Ke, Xiaoyan Zhu, and Minlie Huang. 2022. Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1617–1634, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization (Gu et al., EMNLP 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/author-url/2022.emnlp-main.105.pdf