What’s wrong with your model? A Quantitative Analysis of Relation Classification

Elisa Bassignana, Rob van der Goot, Barbara Plank

Abstract
With the aim of improving the state-of-the-art (SOTA) on a target task, a standard strategy in Natural Language Processing (NLP) research is to design a new model, or modify the existing SOTA, and then benchmark its performance on the target task. We argue in favor of enriching this chain of actions by a preliminary error-guided analysis: First, explore weaknesses by analyzing the hard cases where the existing model fails, and then target the improvement based on those. Interpretable evaluation has received little attention for structured prediction tasks. Therefore we propose the first in-depth analysis suite for Relation Classification (RC), and show its effectiveness through a case study. We propose a set of potentially influential attributes to focus on (e.g., entity distance, sentence length). Then, we bucket our datasets based on these attributes, and weight the importance of them through correlations. This allows us to identify highly challenging scenarios for the RC model. By exploiting the findings of our analysis, with a carefully targeted adjustment to our architecture, we effectively improve the performance over the baseline by >3 Micro-F1.
Anthology ID:
2024.starsem-1.20
Volume:
Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Danushka Bollegala, Vered Shwartz
Venue:
*SEM
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
252–263
Language:
URL:
https://aclanthology.org/2024.starsem-1.20
DOI:
Bibkey:
Cite (ACL):
Elisa Bassignana, Rob van der Goot, and Barbara Plank. 2024. What’s wrong with your model? A Quantitative Analysis of Relation Classification. In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024), pages 252–263, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
What’s wrong with your model? A Quantitative Analysis of Relation Classification (Bassignana et al., *SEM 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/jeptaln-2024-ingestion/2024.starsem-1.20.pdf
PDF Search