Large Language Models As Annotators: A Preliminary Evaluation For Annotating Low-Resource Language Content

Savita Bhat, Vasudeva Varma

Abstract
The process of collecting human-generated annotations is time-consuming and resource-hungry. In the case of low-resource (LR) languages such as Indic languages, these efforts are more expensive due to the dearth of data and human experts. Considering their importance in solving downstream applications, there have been concentrated efforts exploring alternatives for human-generated annotations. To that extent, we seek to evaluate multilingual large language models (LLMs) for their potential to substitute or aid human-generated annotation efforts. We use LLMs to re-label publicly available datasets in LR languages for the tasks of natural language inference, sentiment analysis, and news classification. We compare these annotations with existing ground truth labels to analyze the efficacy of using LLMs for annotation tasks. We observe that the performance of these LLMs varies substantially across different tasks and languages. The results show that off-the-shelf use of multilingual LLMs is not appropriate and results in poor performance in two of the three tasks.
Anthology ID:
2023.eval4nlp-1.8
Volume:
Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems
Month:
November
Year:
2023
Address:
Bali, Indonesia
Editors:
Daniel Deutsch, Rotem Dror, Steffen Eger, Yang Gao, Christoph Leiter, Juri Opitz, Andreas Rücklé
Venues:
Eval4NLP | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
100–107
Language:
URL:
https://aclanthology.org/2023.eval4nlp-1.8
DOI:
10.18653/v1/2023.eval4nlp-1.8
Bibkey:
Cite (ACL):
Savita Bhat and Vasudeva Varma. 2023. Large Language Models As Annotators: A Preliminary Evaluation For Annotating Low-Resource Language Content. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, pages 100–107, Bali, Indonesia. Association for Computational Linguistics.
Cite (Informal):
Large Language Models As Annotators: A Preliminary Evaluation For Annotating Low-Resource Language Content (Bhat & Varma, Eval4NLP-WS 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl-24-ws-corrections/2023.eval4nlp-1.8.pdf
PDF Search