Evaluation of Universal Semantic Representation (USR)
Kirti Garg, Soma Paul, Sukhada Sukhada, Fatema Bawahir, Riya Kumari
Abstract
Universal Semantic Representation (USR) is designed as a language-independent information packaging system that captures information at three levels: (a) Lexico-conceptual, (b) Syntactico-Semantic, and (c) Discourse. Unlike other representations that mainly encode predicates and their argument structures, our proposed representation captures the speaker’s vivakṣā- how the speaker views the activity. The idea of “speaker’s vivakṣā is inspired by Indian Grammatical Tradition. There can be some amount of idiosyncrasy of the speaker in the annotation since it is the speaker’s view- point that has been captured in the annotation. Hence the evaluation metrics of such resources need to be also thought through from scratch. This paper presents an extensive evaluation procedure of this semantic representation from two perspectives (a) Inter- Annotator Agreement and (b) one downstream task, namely multilingual Natural Language Generation. We also qualitatively evaluate the experience of natural language generation by manual parsing of USR, so as to understand the readability of USR. We have achieved above 80% Inter-Annotator Agreement for USR annotations and above 80% semantic closeness in multi-lingual generation tasks suggesting the reliability of USR annotations and utility for multi-lingual generations. The qualitative evaluation also suggests high readability and hence the utility of USR as a semantic representation.- Anthology ID:
- 2023.dmr-1.2
- Volume:
- Proceedings of the Fourth International Workshop on Designing Meaning Representations
- Month:
- June
- Year:
- 2023
- Address:
- Nancy, France
- Editors:
- Julia Bonn, Nianwen Xue
- Venues:
- DMR | WS
- SIG:
- SIGSEM
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 13–22
- Language:
- URL:
- https://aclanthology.org/2023.dmr-1.2
- DOI:
- Cite (ACL):
- Kirti Garg, Soma Paul, Sukhada Sukhada, Fatema Bawahir, and Riya Kumari. 2023. Evaluation of Universal Semantic Representation (USR). In Proceedings of the Fourth International Workshop on Designing Meaning Representations, pages 13–22, Nancy, France. Association for Computational Linguistics.
- Cite (Informal):
- Evaluation of Universal Semantic Representation (USR) (Garg et al., DMR-WS 2023)
- PDF:
- https://preview.aclanthology.org/corrections-2024-07/2023.dmr-1.2.pdf