All Fragments Count in Parser Evaluation

Jasmijn Bastings, Khalil Sima’an


Abstract
PARSEVAL, the default paradigm for evaluating constituency parsers, calculates parsing success (Precision/Recall) as a function of the number of matching labeled brackets across the test set. Nodes in constituency trees, however, are connected together to reflect important linguistic relations such as predicate-argument and direct-dominance relations between categories. In this paper, we present FREVAL, a generalization of PARSEVAL, where the precision and recall are calculated not only for individual brackets, but also for co-occurring, connected brackets (i.e. fragments). FREVAL fragments precision (FLP) and recall (FLR) interpolate the match across the whole spectrum of fragment sizes ranging from those consisting of individual nodes (labeled brackets) to those consisting of full parse trees. We provide evidence that FREVAL is informative for inspecting relative parser performance by comparing a range of existing parsers.
Anthology ID:
L14-1324
Original:
L14-1324v1
Version 2:
L14-1324v2
Volume:
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
Month:
May
Year:
2014
Address:
Reykjavik, Iceland
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
78–82
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/376_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Jasmijn Bastings and Khalil Sima’an. 2014. All Fragments Count in Parser Evaluation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 78–82, Reykjavik, Iceland. European Language Resources Association (ELRA).
Cite (Informal):
All Fragments Count in Parser Evaluation (Bastings & Sima’an, LREC 2014)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/376_Paper.pdf