Consistency Regularization Training for Compositional Generalization
Yongjing Yin, Jiali Zeng, Yafu Li, Fandong Meng, Jie Zhou, Yue Zhang
Abstract
Existing neural models have difficulty generalizing to unseen combinations of seen components. To achieve compositional generalization, models are required to consistently interpret (sub)expressions across contexts. Without modifying model architectures, we improve the capability of Transformer on compositional generalization through consistency regularization training, which promotes representation consistency across samples and prediction consistency for a single sample. Experimental results on semantic parsing and machine translation benchmarks empirically demonstrate the effectiveness and generality of our method. In addition, we find that the prediction consistency scores on in-distribution validation sets can be an alternative for evaluating models during training, when commonly-used metrics are not informative.- Anthology ID:
- 2023.acl-long.72
- Original:
- 2023.acl-long.72v1
- Version 2:
- 2023.acl-long.72v2
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1294–1308
- Language:
- URL:
- https://aclanthology.org/2023.acl-long.72
- DOI:
- 10.18653/v1/2023.acl-long.72
- Cite (ACL):
- Yongjing Yin, Jiali Zeng, Yafu Li, Fandong Meng, Jie Zhou, and Yue Zhang. 2023. Consistency Regularization Training for Compositional Generalization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1294–1308, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Consistency Regularization Training for Compositional Generalization (Yin et al., ACL 2023)
- PDF:
- https://preview.aclanthology.org/landing_page/2023.acl-long.72.pdf