@inproceedings{pertsas-etal-2024-annotated,
    title = "An Annotated Dataset for Transformer-based Scholarly Information Extraction and Linguistic Linked Data Generation",
    author = "Pertsas, Vayianos  and
      Kasapaki, Marialena  and
      Constantopoulos, Panos",
    editor = "Chiarcos, Christian  and
      Gkirtzou, Katerina  and
      Ionov, Maxim  and
      Khan, Fahad  and
      McCrae, John P.  and
      Ponsoda, Elena Montiel  and
      Chozas, Patricia Mart{\'i}n",
    booktitle = "Proceedings of the 9th Workshop on Linked Data in Linguistics @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2024.ldl-1.11/",
    pages = "84--93",
    abstract = "We present a manually curated and annotated, multidisciplinary dataset of 15,262 sentences from research articles (abstract and main text) that can be used for transformer-based extraction from scholarly publications of three types of entities: 1) research methods, named entities of variable length, 2) research goals, entities that appear as textual spans of variable length with mostly fixed lexico-syntactic-structure, and 3) research activities, entities that appear as textual spans of variable length with complex lexico-syntactic structure. We explore the capabilities of our dataset by using it for training/fine-tuning various ML and transformer-based models. We compare our finetuned models as well as LLM responses (chatGPT 3.5) based on 10-shot learning, by measuring F1 scores in token-based, entity-based strict and entity-based partial evaluations across interdisciplinary and discipline-specific datasets in order to capture any possible differences in discipline-oriented writing styles. Results show that fine tuning of transformer-based models significantly outperforms the performance of few- shot learning of LLMs such as chatGPT, highlighting the significance of annotation datasets in such tasks. Our dataset can also be used as a source for linguistic linked data by itself. We demonstrate this by presenting indicative queries in SPARQL, executed over such an RDF knowledge graph."
}