Abstract
This report describes first an industrial use case for identifying closely related languages, e.g.dialects, namely the detection of languages of movie subtitle documents. We then presenta 2-stage architecture that is able to detect macrolanguages in the first stage and languagevariants in the second. Using our architecture, we participated in the DSL-TL Shared Task of the VarDial 2023 workshop. We describe the results of our experiments. In the first experiment we report an accuracy of 97.8% on a set of 460 subtitle files. In our second experimentwe used DSL-TL data and achieve a macroaverage F1 of 76% for the binary task, and 54% for the three-way task in the dev set. In the open track, we augment the data with named entities retrieved from Wikidata and achieve minor increases of about 1% for both tracks.- Anthology ID:
- 2023.vardial-1.21
- Volume:
- Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Yves Scherrer, Tommi Jauhiainen, Nikola Ljubešić, Preslav Nakov, Jörg Tiedemann, Marcos Zampieri
- Venue:
- VarDial
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 213–221
- Language:
- URL:
- https://aclanthology.org/2023.vardial-1.21
- DOI:
- 10.18653/v1/2023.vardial-1.21
- Cite (ACL):
- Fritz Hohl and Soh-eun Shim. 2023. VarDial in the Wild: Industrial Applications of LID Systems for Closely-Related Language Varieties. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 213–221, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- VarDial in the Wild: Industrial Applications of LID Systems for Closely-Related Language Varieties (Hohl & Shim, VarDial 2023)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2023.vardial-1.21.pdf