Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text?
Abstract
Semantic text processing faces the challenge of defining the relation between lexical expressions and the world to which they make reference within a period of time. It is unclear whether the current test sets used to evaluate disambiguation tasks are representative for the full complexity considering this time-anchored relation, resulting in semantic overfitting to a specific period and the frequent phenomena within. We conceptualize and formalize a set of metrics which evaluate this complexity of datasets. We provide evidence for their applicability on five different disambiguation tasks. To challenge semantic overfitting of disambiguation systems, we propose a time-based, metric-aware method for developing datasets in a systematic and semi-automated manner, as well as an event-based QA task.- Anthology ID:
- C16-1112
- Volume:
- Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
- Month:
- December
- Year:
- 2016
- Address:
- Osaka, Japan
- Editors:
- Yuji Matsumoto, Rashmi Prasad
- Venue:
- COLING
- SIG:
- Publisher:
- The COLING 2016 Organizing Committee
- Note:
- Pages:
- 1180–1191
- Language:
- URL:
- https://aclanthology.org/C16-1112
- DOI:
- Cite (ACL):
- Filip Ilievski, Marten Postma, and Piek Vossen. 2016. Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text?. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1180–1191, Osaka, Japan. The COLING 2016 Organizing Committee.
- Cite (Informal):
- Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text? (Ilievski et al., COLING 2016)
- PDF:
- https://preview.aclanthology.org/teach-a-man-to-fish/C16-1112.pdf
- Data
- AMR Bank, ECB+