Extracting Weighted Language Lexicons from Wikipedia

Gregory Grefenstette


Abstract
Language models are used in applications as diverse as speech recognition, optical character recognition and information retrieval. They are used to predict word appearance, and to weight the importance of words in these applications. One basic element of language models is the list of words in a language. Another is the unigram frequency of each word. But this basic information is not available for most languages in the world. Since the multilingual Wikipedia project encourages the production of encyclopedic-like articles in many world languages, we can find there an ever-growing source of text from which to extract these two language modelling elements: word list and frequency. Here we present a simple technique for converting this Wikipedia text into lexicons of weighted unigrams for the more than 280 languages present currently present in Wikipedia. The lexicons produced, and the source code for producing them in a Linux-based system are here made available for free on the Web.
Anthology ID:
L16-1217
Volume:
Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)
Month:
May
Year:
2016
Address:
Portorož, Slovenia
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
1365–1368
Language:
URL:
https://aclanthology.org/L16-1217
DOI:
Bibkey:
Cite (ACL):
Gregory Grefenstette. 2016. Extracting Weighted Language Lexicons from Wikipedia. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1365–1368, Portorož, Slovenia. European Language Resources Association (ELRA).
Cite (Informal):
Extracting Weighted Language Lexicons from Wikipedia (Grefenstette, LREC 2016)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/L16-1217.pdf