Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions
Tian Jin, Zhun Liu, Shengjia Yan, Alexandre Eichenberger, Louis-Philippe Morency
Abstract
Transfer learning using ImageNet pre-trained models has been the de facto approach in a wide range of computer vision tasks. However, fine-tuning still requires task-specific training data. In this paper, we propose N3 (Neural Networks from Natural Language) - a new paradigm of synthesizing task-specific neural networks from language descriptions and a generic pre-trained model. N3 leverages language descriptions to generate parameter adaptations as well as a new task-specific classification layer for a pre-trained neural network, effectively “fine-tuning” the network for a new task using only language descriptions as input. To the best of our knowledge, N3 is the first method to synthesize entire neural networks from natural language. Experimental results show that N3 can out-perform previous natural-language based zero-shot learning methods across 4 different zero-shot image classification benchmarks. We also demonstrate a simple method to help identify keywords in language descriptions leveraged by N3 when synthesizing model parameters.- Anthology ID:
- 2020.acl-main.625
- Volume:
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6994–7007
- Language:
- URL:
- https://aclanthology.org/2020.acl-main.625
- DOI:
- 10.18653/v1/2020.acl-main.625
- Cite (ACL):
- Tian Jin, Zhun Liu, Shengjia Yan, Alexandre Eichenberger, and Louis-Philippe Morency. 2020. Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6994–7007, Online. Association for Computational Linguistics.
- Cite (Informal):
- Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions (Jin et al., ACL 2020)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/2020.acl-main.625.pdf