Unsupervised Preference-Aware Language Identification

Xingzhang Ren, Baosong Yang, Dayiheng Liu, Haibo Zhang, Xiaoyu Lv, Liang Yao, Jun Xie


Abstract
Recognizing the language of ambiguous texts has become a main challenge in language identification (LID). When using multilingual applications, users have their own language preferences, which can be regarded as external knowledge for LID. Nevertheless, current studies do not consider the inter-personal variations due to the lack of user annotated training data. To fill this gap, we introduce preference-aware LID and propose a novel unsupervised learning strategy. Concretely, we construct pseudo training set for each user by extracting training samples from a standard LID corpus according to his/her historical language distribution. Besides, we contribute the first user labeled LID test set called “U-LID”. Experimental results reveal that our model can incarnate user traits and significantly outperforms existing LID systems on handling ambiguous texts. Our code and benchmark have been released.
Anthology ID:
2022.findings-acl.303
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3847–3852
Language:
URL:
https://aclanthology.org/2022.findings-acl.303
DOI:
10.18653/v1/2022.findings-acl.303
Bibkey:
Cite (ACL):
Xingzhang Ren, Baosong Yang, Dayiheng Liu, Haibo Zhang, Xiaoyu Lv, Liang Yao, and Jun Xie. 2022. Unsupervised Preference-Aware Language Identification. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3847–3852, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Unsupervised Preference-Aware Language Identification (Ren et al., Findings 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/2022.findings-acl.303.pdf
Software:
 2022.findings-acl.303.software.zip
Code
 xzhren/preferenceawarelid