Abstract
Language models are used in applications as diverse as speech recognition, optical character recognition and information retrieval. They are used to predict word appearance, and to weight the importance of words in these applications. One basic element of language models is the list of words in a language. Another is the unigram frequency of each word. But this basic information is not available for most languages in the world. Since the multilingual Wikipedia project encourages the production of encyclopedic-like articles in many world languages, we can find there an ever-growing source of text from which to extract these two language modelling elements: word list and frequency. Here we present a simple technique for converting this Wikipedia text into lexicons of weighted unigrams for the more than 280 languages present currently present in Wikipedia. The lexicons produced, and the source code for producing them in a Linux-based system are here made available for free on the Web.- Anthology ID:
- L16-1217
- Volume:
- Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)
- Month:
- May
- Year:
- 2016
- Address:
- Portorož, Slovenia
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association (ELRA)
- Note:
- Pages:
- 1365–1368
- Language:
- URL:
- https://aclanthology.org/L16-1217
- DOI:
- Cite (ACL):
- Gregory Grefenstette. 2016. Extracting Weighted Language Lexicons from Wikipedia. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1365–1368, Portorož, Slovenia. European Language Resources Association (ELRA).
- Cite (Informal):
- Extracting Weighted Language Lexicons from Wikipedia (Grefenstette, LREC 2016)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/L16-1217.pdf