@inproceedings{saini-etal-2023-90,
    title = "90{\%} F1 Score in Relation Triple Extraction: Is it Real?",
    author = "Saini, Pratik  and
      Pal, Samiran  and
      Nayak, Tapas  and
      Bhattacharya, Indrajit",
    editor = "Hupkes, Dieuwke  and
      Dankers, Verna  and
      Batsuren, Khuyagbaatar  and
      Sinha, Koustuv  and
      Kazemnejad, Amirhossein  and
      Christodoulopoulos, Christos  and
      Cotterell, Ryan  and
      Bruni, Elia",
    booktitle = "Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.genbench-1.1",
    doi = "10.18653/v1/2023.genbench-1.1",
    pages = "1--11",
    abstract = "Extracting relational triples from text is a crucial task for constructing knowledge bases. Recent advancements in joint entity and relation extraction models have demonstrated remarkable F1 scores ({\mbox{$\geq$}} 90{\%}) in accurately extracting relational triples from free text. However, these models have been evaluated under restrictive experimental settings and unrealistic datasets. They overlook sentences with zero triples (zerocardinality), thereby simplifying the task. In this paper, we present a benchmark study of state-of-the-art joint entity and relation extraction models under a more realistic setting. We include sentences that lack any triples in our experiments, providing a comprehensive evaluation. Our findings reveal a significant decline (approximately 10-15{\%} in one dataset and 6-14{\%} in another dataset) in the models{'} F1 scores within this realistic experimental setup. Furthermore, we propose a two-step modeling approach that utilizes a simple BERT-based classifier. This approach leads to overall performance improvement in these models within the realistic experimental setting.",
}