Abstract
Semantic text processing faces the challenge of defining the relation between lexical expressions and the world to which they make reference within a period of time. It is unclear whether the current test sets used to evaluate disambiguation tasks are representative for the full complexity considering this time-anchored relation, resulting in semantic overfitting to a specific period and the frequent phenomena within. We conceptualize and formalize a set of metrics which evaluate this complexity of datasets. We provide evidence for their applicability on five different disambiguation tasks. To challenge semantic overfitting of disambiguation systems, we propose a time-based, metric-aware method for developing datasets in a systematic and semi-automated manner, as well as an event-based QA task.- Anthology ID:
- C16-1112
- Volume:
- Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
- Month:
- December
- Year:
- 2016
- Address:
- Osaka, Japan
- Editors:
- Yuji Matsumoto, Rashmi Prasad
- Venue:
- COLING
- SIG:
- Publisher:
- The COLING 2016 Organizing Committee
- Note:
- Pages:
- 1180–1191
- Language:
- URL:
- https://aclanthology.org/C16-1112
- DOI:
- Cite (ACL):
- Filip Ilievski, Marten Postma, and Piek Vossen. 2016. Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text?. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1180–1191, Osaka, Japan. The COLING 2016 Organizing Committee.
- Cite (Informal):
- Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text? (Ilievski et al., COLING 2016)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/C16-1112.pdf
- Data
- AMR Bank, ECB+