A Tale of Three Parsers: Towards Diagnostic Evaluation for Meaning Representation Parsing

Maja Buljan, Joakim Nivre, Stephan Oepen, Lilja Øvrelid


Abstract
We discuss methodological choices in contrastive and diagnostic evaluation in meaning representation parsing, i.e. mapping from natural language utterances to graph-based encodings of its semantic structure. Drawing inspiration from earlier work in syntactic dependency parsing, we transfer and refine several quantitative diagnosis techniques for use in the context of the 2019 shared task on Meaning Representation Parsing (MRP). As in parsing proper, moving evaluation from simple rooted trees to general graphs brings along its own range of challenges. Specifically, we seek to begin to shed light on relative strenghts and weaknesses in different broad families of parsing techniques. In addition to these theoretical reflections, we conduct a pilot experiment on a selection of top-performing MRP systems and one of the five meaning representation frameworks in the shared task. Empirical results suggest that the proposed methodology can be meaningfully applied to parsing into graph-structured target representations, uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches.
Anthology ID:
2020.lrec-1.234
Volume:
Proceedings of the 12th Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
1902–1909
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.234
DOI:
Bibkey:
Cite (ACL):
Maja Buljan, Joakim Nivre, Stephan Oepen, and Lilja Øvrelid. 2020. A Tale of Three Parsers: Towards Diagnostic Evaluation for Meaning Representation Parsing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1902–1909, Marseille, France. European Language Resources Association.
Cite (Informal):
A Tale of Three Parsers: Towards Diagnostic Evaluation for Meaning Representation Parsing (Buljan et al., LREC 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/2020.lrec-1.234.pdf