Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text?

Filip Ilievski, Marten Postma, Piek Vossen

[How to correct problems with metadata yourself]

Abstract
Semantic text processing faces the challenge of defining the relation between lexical expressions and the world to which they make reference within a period of time. It is unclear whether the current test sets used to evaluate disambiguation tasks are representative for the full complexity considering this time-anchored relation, resulting in semantic overfitting to a specific period and the frequent phenomena within. We conceptualize and formalize a set of metrics which evaluate this complexity of datasets. We provide evidence for their applicability on five different disambiguation tasks. To challenge semantic overfitting of disambiguation systems, we propose a time-based, metric-aware method for developing datasets in a systematic and semi-automated manner, as well as an event-based QA task.
Anthology ID:
C16-1112
Volume:
Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
Month:
December
Year:
2016
Address:
Osaka, Japan
Editors:
Yuji Matsumoto, Rashmi Prasad
Venue:
COLING
SIG:
Publisher:
The COLING 2016 Organizing Committee
Note:
Pages:
1180–1191
Language:
URL:
https://aclanthology.org/C16-1112
DOI:
Bibkey:
Cite (ACL):
Filip Ilievski, Marten Postma, and Piek Vossen. 2016. Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text?. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1180–1191, Osaka, Japan. The COLING 2016 Organizing Committee.
Cite (Informal):
Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text? (Ilievski et al., COLING 2016)
Copy Citation:
PDF:
https://preview.aclanthology.org/teach-a-man-to-fish/C16-1112.pdf
Data
AMR Bank, ECB+
PDF Search