@inproceedings{pavlovic-poesio-2024-effectiveness,
    title = "The Effectiveness of {LLM}s as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation",
    author = "Pavlovic, Maja  and
      Poesio, Massimo",
    editor = "Abercrombie, Gavin  and
      Basile, Valerio  and
      Bernadi, Davide  and
      Dudy, Shiran  and
      Frenda, Simona  and
      Havens, Lucy  and
      Tonelli, Sara",
    booktitle = "Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2024.nlperspectives-1.11/",
    pages = "100--110",
    abstract = "Recent studies focus on exploring the capability of Large Language Models (LLMs) for data annotation. Our work, firstly, offers a comparative overview of twelve such studies that investigate labelling with LLMs, particularly focusing on classification tasks. Secondly, we present an empirical analysis that examines the degree of alignment between the opinion distributions returned by GPT and those provided by human annotators across four subjective datasets. Our analysis supports a minority of studies that are considering diverse perspectives when evaluating data annotation tasks and highlights the need for further research in this direction."
}