Zhixing Xu
2022
Align-smatch: A Novel Evaluation Method for Chinese Abstract Meaning Representation Parsing based on Alignment of Concept and Relation
Liming Xiao
|
Bin Li
|
Zhixing Xu
|
Kairui Huo
|
Minxuan Feng
|
Junsheng Zhou
|
Weiguang Qu
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Abstract Meaning Representation is a sentence-level meaning representation, which abstracts the meaning of sentences into a rooted acyclic directed graph. With the continuous expansion of Chinese AMR corpus, more and more scholars have developed parsing systems to automatically parse sentences into Chinese AMR. However, the current parsers can’t deal with concept alignment and relation alignment, let alone the evaluation methods for AMR parsing. Therefore, to make up for the vacancy of Chinese AMR parsing evaluation methods, based on AMR evaluation metric smatch, we have improved the algorithm of generating triples so that to make it compatible with concept alignment and relation alignment. Finally, we obtain a new integrity metric align-smatch for paring evaluation. A comparative research then was conducted on 20 manually annotated AMR and gold AMR, with the result that align-smatch works well in alignments and more robust in evaluating arcs. We also put forward some fine-grained metric for evaluating concept alignment, relation alignment and implicit concepts, in order to further measure parsers’ performance in subtasks.
Search
Co-authors
- Liming Xiao 1
- Bin Li 1
- Kairui Huo (霍凯蕊) 1
- Minxuan Feng 1
- Junsheng Zhou 1
- show all...
Venues
- lrec1