Abstract
Neural machine translation with millions of parameters is vulnerable to unfamiliar inputs. We propose Token Drop to improve generalization and avoid overfitting for the NMT model. Similar to word dropout, whereas we replace dropped token with a special token instead of setting zero to words. We further introduce two self-supervised objectives: Replaced Token Detection and Dropped Token Prediction. Our method aims to force model generating target translation with less information, in this way the model can learn textual representation better. Experiments on Chinese-English and English-Romanian benchmark demonstrate the effectiveness of our approach and our model achieves significant improvements over a strong Transformer baseline.- Anthology ID:
- 2020.coling-main.379
- Volume:
- Proceedings of the 28th International Conference on Computational Linguistics
- Month:
- December
- Year:
- 2020
- Address:
- Barcelona, Spain (Online)
- Editors:
- Donia Scott, Nuria Bel, Chengqing Zong
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 4298–4303
- Language:
- URL:
- https://aclanthology.org/2020.coling-main.379
- DOI:
- 10.18653/v1/2020.coling-main.379
- Cite (ACL):
- Huaao Zhang, Shigui Qiu, Xiangyu Duan, and Min Zhang. 2020. Token Drop mechanism for Neural Machine Translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4298–4303, Barcelona, Spain (Online). International Committee on Computational Linguistics.
- Cite (Informal):
- Token Drop mechanism for Neural Machine Translation (Zhang et al., COLING 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2020.coling-main.379.pdf
- Code
- zhajiahe/Token_Drop