Abstract
Transfer learning is effective for improving the performance of tasks that are related, and Multi-task learning (MTL) and Cross-lingual learning (CLL) are important instances. This paper argues that hard-parameter sharing, of hard-coding layers shared across different tasks or languages, cannot generalize well, when sharing with a loosely related task. Such case, which we call sparse transfer, might actually hurt performance, a phenomenon known as negative transfer. Our contribution is using adversarial training across tasks, to “soft-code” shared and private spaces, to avoid the shared space gets too sparse. In CLL, our proposed architecture considers another challenge of dealing with low-quality input.- Anthology ID:
- P19-1151
- Volume:
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2019
- Address:
- Florence, Italy
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1560–1568
- Language:
- URL:
- https://aclanthology.org/P19-1151
- DOI:
- 10.18653/v1/P19-1151
- Cite (ACL):
- Haeju Park, Jinyoung Yeo, Gengyu Wang, and Seung-won Hwang. 2019. Soft Representation Learning for Sparse Transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1560–1568, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- Soft Representation Learning for Sparse Transfer (Park et al., ACL 2019)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/P19-1151.pdf
- Data
- MultiNLI, SNLI