Abstract
In this paper, we introduce the SORTS Subject-Object Resolution Test Suite of German minimal sentence pairs for model introspection. The full test suite consists of 18,502 transitive clauses with manual annotations of 8 word order patterns, 5 morphological and syntactic and 11 semantic property classes. The test suite has been constructed such that sentences are minimal pairs with respect to a property class. Each property has been selected with a particular focus on its effect on subject-object resolution, the second-most error-prone task within syntactic parsing of German after prepositional phrase attachment (Fischer et al., 2019). The size and detail of annotations make the test suite a valuable resource for natural language processing applications with syntactic and semantic tasks. We use dependency parsing to demonstrate how the test suite allows insights into the process of subject-object resolution. Based on the test suite annotations, word order and case syncretism can be identified as most important factors that affect subject-object resolution.- Anthology ID:
- 2020.coling-main.269
- Volume:
- Proceedings of the 28th International Conference on Computational Linguistics
- Month:
- December
- Year:
- 2020
- Address:
- Barcelona, Spain (Online)
- Editors:
- Donia Scott, Nuria Bel, Chengqing Zong
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 3019–3035
- Language:
- URL:
- https://aclanthology.org/2020.coling-main.269
- DOI:
- 10.18653/v1/2020.coling-main.269
- Cite (ACL):
- Patricia Fischer, Daniël de Kok, and Erhard Hinrichs. 2020. When Beards Start Shaving Men: A Subject-object Resolution Test Suite for Morpho-syntactic and Semantic Model Introspection. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3019–3035, Barcelona, Spain (Online). International Committee on Computational Linguistics.
- Cite (Informal):
- When Beards Start Shaving Men: A Subject-object Resolution Test Suite for Morpho-syntactic and Semantic Model Introspection (Fischer et al., COLING 2020)
- PDF:
- https://preview.aclanthology.org/emnlp22-frontmatter/2020.coling-main.269.pdf
- Code
- divefish/sorts