caWaC – A web corpus of Catalan and its application to language modeling and machine translation

Nikola Ljubešić, Antonio Toral


Abstract
In this paper we present the construction process of a web corpus of Catalan built from the content of the .cat top-level domain. For collecting and processing data we use the Brno pipeline with the spiderling crawler and its accompanying tools. To the best of our knowledge the corpus represents the largest existing corpus of Catalan containing 687 million words, which is a significant increase given that until now the biggest corpus of Catalan, CuCWeb, counts 166 million words. We evaluate the resulting resource on the tasks of language modeling and statistical machine translation (SMT) by calculating LM perplexity and incorporating the LM in the SMT pipeline. We compare language models trained on different subsets of the resource with those trained on the Catalan Wikipedia and the target side of the parallel data used to train the SMT system.
Anthology ID:
L14-1647
Volume:
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
Month:
May
Year:
2014
Address:
Reykjavik, Iceland
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
1728–1732
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/841_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Nikola Ljubešić and Antonio Toral. 2014. caWaC – A web corpus of Catalan and its application to language modeling and machine translation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 1728–1732, Reykjavik, Iceland. European Language Resources Association (ELRA).
Cite (Informal):
caWaC – A web corpus of Catalan and its application to language modeling and machine translation (Ljubešić & Toral, LREC 2014)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/841_Paper.pdf