WRF: Weighted Rouge-F1 Metric for Entity Recognition
Lukas Weber, Krishnan Jothi Ramalingam, Matthias Beyer, Axel Zimmermann
Abstract
The continuous progress in Named Entity Recognition allows the identification of complex entities in multiple domains. The traditionally used metrics like precision, recall, and F1-score can only reflect the classification quality of the underlying NER model to a limited extent. Existing metrics do not distinguish between a non-recognition of an entity and a misclassification of an entity. Additionally, the dealing with redundant entities remains unaddressed. We propose WRF, a Weighted Rouge F1 metric for Entity Recognition, to solve the mentioned gaps in currently available metrics. We successfully employ the WRF metric for automotive entity recognition, followed by a comprehensive qualitative and quantitative analysis of the obtained results.- Anthology ID:
- 2023.eval4nlp-1.1
- Volume:
- Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems
- Month:
- November
- Year:
- 2023
- Address:
- Bali, Indonesia
- Editors:
- Daniel Deutsch, Rotem Dror, Steffen Eger, Yang Gao, Christoph Leiter, Juri Opitz, Andreas Rücklé
- Venues:
- Eval4NLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1–11
- Language:
- URL:
- https://aclanthology.org/2023.eval4nlp-1.1
- DOI:
- 10.18653/v1/2023.eval4nlp-1.1
- Cite (ACL):
- Lukas Weber, Krishnan Jothi Ramalingam, Matthias Beyer, and Axel Zimmermann. 2023. WRF: Weighted Rouge-F1 Metric for Entity Recognition. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, pages 1–11, Bali, Indonesia. Association for Computational Linguistics.
- Cite (Informal):
- WRF: Weighted Rouge-F1 Metric for Entity Recognition (Weber et al., Eval4NLP-WS 2023)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2023.eval4nlp-1.1.pdf