Abstract
Knowing the state-of-the-art for a particular task is an essential component of any computational linguistics investigation. But can we be truly confident that the current state-of-the-art is indeed the best performing model? In this paper, we study the case of frame semantic parsing, a well-established task with multiple shared datasets. We show that in spite of all the care taken to provide a standard evaluation resource, small variations in data processing can have dramatic consequences for ranking parser performance. This leads us to propose an open-source standardized processing pipeline, which can be shared and reused for robust model comparison.- Anthology ID:
- C18-1267
- Volume:
- Proceedings of the 27th International Conference on Computational Linguistics
- Month:
- August
- Year:
- 2018
- Address:
- Santa Fe, New Mexico, USA
- Editors:
- Emily M. Bender, Leon Derczynski, Pierre Isabelle
- Venue:
- COLING
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3158–3169
- Language:
- URL:
- https://aclanthology.org/C18-1267
- DOI:
- Cite (ACL):
- Alexandre Kabbach, Corentin Ribeyre, and Aurélie Herbelot. 2018. Butterfly Effects in Frame Semantic Parsing: impact of data processing on model ranking. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3158–3169, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
- Cite (Informal):
- Butterfly Effects in Frame Semantic Parsing: impact of data processing on model ranking (Kabbach et al., COLING 2018)
- PDF:
- https://preview.aclanthology.org/naacl24-info/C18-1267.pdf
- Code
- akb89/pyfn
- Data
- FrameNet