A Corpus for Evaluating Semantic Multilingual Web Retrieval Systems: The Sense Folder Corpus

Ernesto William De Luca


Abstract
In this paper, we present the multilingual Sense Folder Corpus. After the analysis of different corpora, we describe the requirements that have to be satisfied for evaluating semantic multilingual retrieval approaches. Justified by the unfulfilled requirements explained, we start creating a small bilingual hand-tagged corpus of 502 documents retrieved from Web searches. The documents contained in this collection have been created using Google queries. A single ambiguous word has been searched and related documents (approx. the first 60 documents for every keyword) have been retrieved. The document collection has been extended at the query word level, using single ambiguous words for English (argument, bank, chair, network and rule) and for Italian (argomento, lingua, regola, rete and stampa). The search and annotation process has been done both in a monolingual way for the English and the Italian language. 252 English and 250 Italian documents have been retrieved from Google and saved in their original rank. The performance of semantic multilingual retrieval systems has been evaluated using such a corpus with three baselines (“Random”, “First Sense” and “Most Frequent Sense”) that are formally presented and discussed. The fine-grained evaluation of the Sense Folder approach is discussed in details.
Anthology ID:
L10-1562
Volume:
Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)
Month:
May
Year:
2010
Address:
Valletta, Malta
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2010/pdf/816_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Ernesto William De Luca. 2010. A Corpus for Evaluating Semantic Multilingual Web Retrieval Systems: The Sense Folder Corpus. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta. European Language Resources Association (ELRA).
Cite (Informal):
A Corpus for Evaluating Semantic Multilingual Web Retrieval Systems: The Sense Folder Corpus (De Luca, LREC 2010)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2010/pdf/816_Paper.pdf