Abstract
Spectral models for learning weighted non-deterministic automata have nice theoretical and algorithmic properties. Despite this, it has been challenging to obtain competitive results in language modeling tasks, for two main reasons. First, in order to capture long-range dependencies of the data, the method must use statistics from long substrings, which results in very large matrices that are difficult to decompose. The second is that the loss function behind spectral learning, based on moment matching, differs from the probabilistic metrics used to evaluate language models. In this work we employ a technique for scaling up spectral learning, and use interpolated predictions that are optimized to maximize perplexity. Our experiments in character-based language modeling show that our method matches the performance of state-of-the-art ngram models, while being very fast to train.- Anthology ID:
- P19-1594
- Volume:
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2019
- Address:
- Florence, Italy
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 5926–5930
- Language:
- URL:
- https://aclanthology.org/P19-1594
- DOI:
- 10.18653/v1/P19-1594
- Cite (ACL):
- Ariadna Quattoni and Xavier Carreras. 2019. Interpolated Spectral NGram Language Models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5926–5930, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- Interpolated Spectral NGram Language Models (Quattoni & Carreras, ACL 2019)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/P19-1594.pdf