Geographically-Balanced Gigaword Corpora for 50 Language Varieties

Jonathan Dunn, Ben Adams


Abstract
While text corpora have been steadily increasing in overall size, even very large corpora are not designed to represent global population demographics. For example, recent work has shown that existing English gigaword corpora over-represent inner-circle varieties from the US and the UK. To correct implicit geographic and demographic biases, this paper uses country-level population demographics to guide the construction of gigaword web corpora. The resulting corpora explicitly match the ground-truth geographic distribution of each language, thus equally representing language users from around the world. This is important because it ensures that speakers of under-resourced language varieties (i.e., Indian English or Algerian French) are represented, both in the corpora themselves but also in derivative resources like word embeddings.
Anthology ID:
2020.lrec-1.308
Volume:
Proceedings of the Twelfth Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
2528–2536
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.308
DOI:
Bibkey:
Cite (ACL):
Jonathan Dunn and Ben Adams. 2020. Geographically-Balanced Gigaword Corpora for 50 Language Varieties. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2528–2536, Marseille, France. European Language Resources Association.
Cite (Informal):
Geographically-Balanced Gigaword Corpora for 50 Language Varieties (Dunn & Adams, LREC 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2020.lrec-1.308.pdf