@inproceedings{bhat-varma-2023-large,
    title = "Large Language Models As Annotators: A Preliminary Evaluation For Annotating Low-Resource Language Content",
    author = "Bhat, Savita  and
      Varma, Vasudeva",
    editor = {Deutsch, Daniel  and
      Dror, Rotem  and
      Eger, Steffen  and
      Gao, Yang  and
      Leiter, Christoph  and
      Opitz, Juri  and
      R{\"u}ckl{\'e}, Andreas},
    booktitle = "Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems",
    month = nov,
    year = "2023",
    address = "Bali, Indonesia",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2023.eval4nlp-1.8/",
    doi = "10.18653/v1/2023.eval4nlp-1.8",
    pages = "100--107",
    abstract = "The process of collecting human-generated annotations is time-consuming and resource-hungry. In the case of low-resource (LR) languages such as Indic languages, these efforts are more expensive due to the dearth of data and human experts. Considering their importance in solving downstream applications, there have been concentrated efforts exploring alternatives for human-generated annotations. To that extent, we seek to evaluate multilingual large language models (LLMs) for their potential to substitute or aid human-generated annotation efforts. We use LLMs to re-label publicly available datasets in LR languages for the tasks of natural language inference, sentiment analysis, and news classification. We compare these annotations with existing ground truth labels to analyze the efficacy of using LLMs for annotation tasks. We observe that the performance of these LLMs varies substantially across different tasks and languages. The results show that off-the-shelf use of multilingual LLMs is not appropriate and results in poor performance in two of the three tasks."
}