Abstract
Discourse representation structures (DRSs) are scoped semantic representations for texts of arbitrary length. Evaluating the accuracy of predicted DRSs plays a key role in developing semantic parsers and improving their performance. DRSs are typically visualized as boxes which are not straightforward to process automatically. Counter transforms DRSs to clauses and measures clause overlap by searching for variable mappings between two DRSs. However, this metric is computationally costly (with respect to memory and CPU time) and does not scale with longer texts. We introduce Dscorer, an efficient new metric which converts box-style DRSs to graphs and then measures the overlap of n-grams. Experiments show that Dscorer computes accuracy scores that are correlated with Counter at a fraction of the time.- Anthology ID:
- 2020.acl-main.416
- Volume:
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4547–4554
- Language:
- URL:
- https://aclanthology.org/2020.acl-main.416
- DOI:
- 10.18653/v1/2020.acl-main.416
- Cite (ACL):
- Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2020. Dscorer: A Fast Evaluation Metric for Discourse Representation Structure Parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4547–4554, Online. Association for Computational Linguistics.
- Cite (Informal):
- Dscorer: A Fast Evaluation Metric for Discourse Representation Structure Parsing (Liu et al., ACL 2020)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2020.acl-main.416.pdf