Abstract
We present a novel approach for continuous space language models in statistical machine translation by using Restricted Boltzmann Machines (RBMs). The probability of an n-gram is calculated by the free energy of the RBM instead of a feedforward neural net. Therefore, the calculation is much faster and can be integrated into the translation process instead of using the language model only in a re-ranking step. Furthermore, it is straightforward to introduce additional word factors into the language model. We observed a faster convergence in training if we include automatically generated word classes as an additional word factor. We evaluated the RBM-based language model on the German to English and English to French translation task of TED lectures. Instead of replacing the conventional n-gram-based language model, we trained the RBM-based language model on the more important but smaller in-domain data and combined them in a log-linear way. With this approach we could show improvements of about half a BLEU point on the translation task.- Anthology ID:
- 2012.iwslt-papers.3
- Volume:
- Proceedings of the 9th International Workshop on Spoken Language Translation: Papers
- Month:
- December 6-7
- Year:
- 2012
- Address:
- Hong Kong, Table of contents
- Venue:
- IWSLT
- SIG:
- SIGSLT
- Publisher:
- Note:
- Pages:
- 164–170
- Language:
- URL:
- https://aclanthology.org/2012.iwslt-papers.3
- DOI:
- Cite (ACL):
- Jan Niehues and Alex Waibel. 2012. Continuous space language models using restricted Boltzmann machines. In Proceedings of the 9th International Workshop on Spoken Language Translation: Papers, pages 164–170, Hong Kong, Table of contents.
- Cite (Informal):
- Continuous space language models using restricted Boltzmann machines (Niehues & Waibel, IWSLT 2012)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2012.iwslt-papers.3.pdf