<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="roziewski-stokowiec-2016-languagecrawl">
    <titleInfo>
        <title>LanguageCrawl: A Generic Tool for Building Language Models Upon Common-Crawl</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Szymon</namePart>
        <namePart type="family">Roziewski</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Wojciech</namePart>
        <namePart type="family">Stokowiec</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2016-may</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)</title>
        </titleInfo>
        <originInfo>
            <publisher>European Language Resources Association (ELRA)</publisher>
            <place>
                <placeTerm type="text">Portorož, Slovenia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The web data contains immense amount of data, hundreds of billion words are waiting to be extracted and used for language research. In this work we introduce our tool LanguageCrawl which allows NLP researchers to easily construct web-scale corpus from Common Crawl Archive: a petabyte scale, open repository of web crawl information. Three use-cases are presented: filtering Polish websites, building an N-gram corpora and training continuous skip-gram language model with hierarchical softmax. Each of them has been implemented within the LanguageCrawl toolkit, with the possibility to adjust specified language and N-gram ranks. Special effort has been put on high computing efficiency, by applying highly concurrent multitasking. We make our tool publicly available to enrich NLP resources. We strongly believe that our work will help to facilitate NLP research, especially in under-resourced languages, where the lack of appropriately sized corpora is a serious hindrance to applying data-intensive methods, such as deep neural networks.</abstract>
    <identifier type="citekey">roziewski-stokowiec-2016-languagecrawl</identifier>
    <location>
        <url>https://aclanthology.org/L16-1443</url>
    </location>
    <part>
        <date>2016-may</date>
        <extent unit="page">
            <start>2789</start>
            <end>2793</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T LanguageCrawl: A Generic Tool for Building Language Models Upon Common-Crawl
%A Roziewski, Szymon
%A Stokowiec, Wojciech
%S Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)
%D 2016
%8 may
%I European Language Resources Association (ELRA)
%C Portorož, Slovenia
%F roziewski-stokowiec-2016-languagecrawl
%X The web data contains immense amount of data, hundreds of billion words are waiting to be extracted and used for language research. In this work we introduce our tool LanguageCrawl which allows NLP researchers to easily construct web-scale corpus from Common Crawl Archive: a petabyte scale, open repository of web crawl information. Three use-cases are presented: filtering Polish websites, building an N-gram corpora and training continuous skip-gram language model with hierarchical softmax. Each of them has been implemented within the LanguageCrawl toolkit, with the possibility to adjust specified language and N-gram ranks. Special effort has been put on high computing efficiency, by applying highly concurrent multitasking. We make our tool publicly available to enrich NLP resources. We strongly believe that our work will help to facilitate NLP research, especially in under-resourced languages, where the lack of appropriately sized corpora is a serious hindrance to applying data-intensive methods, such as deep neural networks.
%U https://aclanthology.org/L16-1443
%P 2789-2793

Markdown (Informal)

[LanguageCrawl: A Generic Tool for Building Language Models Upon Common-Crawl](https://aclanthology.org/L16-1443) (Roziewski & Stokowiec, LREC 2016)

• LanguageCrawl: A Generic Tool for Building Language Models Upon Common-Crawl (Roziewski & Stokowiec, LREC 2016)

ACL

• Szymon Roziewski and Wojciech Stokowiec. 2016. LanguageCrawl: A Generic Tool for Building Language Models Upon Common-Crawl. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 2789–2793, Portorož, Slovenia. European Language Resources Association (ELRA).