Towards Efficient Dialogue Pre-training with Transferable and Interpretable Latent Structure
Xueliang Zhao, Lemao Liu, Tingchen Fu, Shuming Shi, Dongyan Zhao, Rui Yan
Abstract
With the availability of massive general-domain dialogue data, pre-trained dialogue generation appears to be super appealing to transfer knowledge from the general domain to downstream applications. In most existing work, such transferable ability is mainly obtained by fitting a large model with hundreds of millions of parameters on massive data in an exhaustive way, leading to inefficient running and poor interpretability. This paper proposes a novel dialogue generation model with a latent structure that is easily transferable from the general domain to downstream tasks in a lightweight and transparent way. Experiments on two benchmarks validate the effectiveness of the proposed model. Thanks to the transferable latent structure, our model is able to yield better dialogue responses than four strong baselines in terms of both automatic and human evaluations, and our model with about 22% parameters particularly delivers a 5x speedup in running time compared with the strongest baseline. Moreover, the proposed model is explainable by interpreting the discrete latent variables.- Anthology ID:
- 2022.emnlp-main.683
- Volume:
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 10051–10063
- Language:
- URL:
- https://aclanthology.org/2022.emnlp-main.683
- DOI:
- Cite (ACL):
- Xueliang Zhao, Lemao Liu, Tingchen Fu, Shuming Shi, Dongyan Zhao, and Rui Yan. 2022. Towards Efficient Dialogue Pre-training with Transferable and Interpretable Latent Structure. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10051–10063, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Towards Efficient Dialogue Pre-training with Transferable and Interpretable Latent Structure (Zhao et al., EMNLP 2022)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/2022.emnlp-main.683.pdf