@inproceedings{belcavello-etal-2026-evaluating,
    title = "Evaluating the Impact of {LLM}-Assisted Annotation in a Perspectivized Setting: The Case of {F}rame{N}et Annotation",
    author = "Belcavello, Frederico  and
      Matos, Ely E.  and
      Lorenzi, Arthur  and
      Bonoto, Lisandra  and
      P{\'a}dua Ruiz, Livia  and
      Pereira, Luiz Fernando  and
      Herbst, Victor  and
      Navarro, Yulla Liquer  and
      Abreu, Helen de Andrade  and
      Vicente Dutra, L{\'i}via  and
      Torrent, Tiago Timponi",
    editor = "Bunt, Harry",
    booktitle = "Proceedings of the 22nd Joint {ACL} - {ISO} Workshop on Interoperable Semantic Annotation and Representation ({ISA}-22) @ {LREC} 2026",
    month = may,
    year = "2026",
    address = "Palma, Mallorca (Spain)",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/ingest-lrec/2026.isa-1.9/",
    doi = "10.63317/36ot28es26u8",
    pages = "77--87",
    abstract = "The use of LLM-based applications as a means to accelerate and/or substitute human labor in the creation of language resources and datasets is a reality. Nonetheless, despite the potential of such tools for linguistic research, an evaluation of their performance and impact on the creation of annotated datasets, especially under a perspectivized approach to NLP, is still missing. This paper contributes to the reduction of this gap by reporting on an extensive evaluation of the (semi-)automatization of FrameNet-like semantic annotation by the use of an LLM-based semantic role labeler. The methodology employed compares annotation time, coverage, and diversity in three experimental settings: manual, automatic, and semi-automatic annotation. Results show that the hybrid, semi-automatic annotation setting leads to increased frame diversity and similar annotation coverage, when compared to the human-only setting, while the automatic setting performs considerably worse in all metrics, except for annotation time, which remains similar."
}