Abstract
In this paper we present a novel bilingual (Czech, English) dataset called ShadowSense developed for the purposes of word sense induction (WSI) evaluation. Unlike existing WSI datasets, ShadowSense is annotated by multiple annotators whose inter-annotator agreement represents key reliability score to be used for evaluation of systems automatically inducing word senses. In this paper we clarify the motivation for such an approach, describe the dataset in detail and provide evaluation of three neural WSI systems showing substantial differences compared to traditional evaluation paradigms.- Anthology ID:
- 2024.lrec-main.1286
- Volume:
- Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
- Month:
- May
- Year:
- 2024
- Address:
- Torino, Italia
- Editors:
- Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
- Venues:
- LREC | COLING
- SIG:
- Publisher:
- ELRA and ICCL
- Note:
- Pages:
- 14763–14769
- Language:
- URL:
- https://aclanthology.org/2024.lrec-main.1286
- DOI:
- Cite (ACL):
- Ondřej Herman and Miloš Jakubíček. 2024. ShadowSense: A Multi-annotated Dataset for Evaluating Word Sense Induction. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14763–14769, Torino, Italia. ELRA and ICCL.
- Cite (Informal):
- ShadowSense: A Multi-annotated Dataset for Evaluating Word Sense Induction (Herman & Jakubíček, LREC-COLING 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2024.lrec-main.1286.pdf