N-gram Counts and Language Models from the Common Crawl

Christian Buck, Kenneth Heafield, Bas van Ooyen


Abstract
We contribute 5-gram counts and language models trained on the Common Crawl corpus, a collection over 9 billion web pages. This release improves upon the Google n-gram counts in two key ways: the inclusion of low-count entries and deduplication to reduce boilerplate. By preserving singletons, we were able to use Kneser-Ney smoothing to build large language models. This paper describes how the corpus was processed with emphasis on the problems that arise in working with data at this scale. Our unpruned Kneser-Ney English $5$-gram language model, built on 975 billion deduplicated tokens, contains over 500 billion unique n-grams. We show gains of 0.5-1.4 BLEU by using large language models to translate into various languages.
Anthology ID:
L14-1074
Volume:
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
Month:
May
Year:
2014
Address:
Reykjavik, Iceland
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
3579–3584
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/1097_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Christian Buck, Kenneth Heafield, and Bas van Ooyen. 2014. N-gram Counts and Language Models from the Common Crawl. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 3579–3584, Reykjavik, Iceland. European Language Resources Association (ELRA).
Cite (Informal):
N-gram Counts and Language Models from the Common Crawl (Buck et al., LREC 2014)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/1097_Paper.pdf