Abstract
Abstract Meaning Representations (AMRs) encode the semantics of sentences in the form of graphs. Vertices represent instances of concepts, and labeled edges represent semantic relations between those instances. Language models (LMs) operate by computing weights of edges of per layer complete graphs whose vertices are words in a sentence or a whole paragraph. In this work, we investigate the ability of the attention heads of two LMs, RoBERTa and GPT2, to detect the semantic relations encoded in an AMR. This is an attempt to show semantic capabilities of those models without finetuning. To do so, we apply both unsupervised and supervised learning techniques.- Anthology ID:
- 2024.starsem-1.26
- Volume:
- Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024)
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Danushka Bollegala, Vered Shwartz
- Venue:
- *SEM
- SIG:
- SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 326–333
- Language:
- URL:
- https://aclanthology.org/2024.starsem-1.26
- DOI:
- Cite (ACL):
- Frédéric Charpentier, Jairo Cugliari, and Adrien Guille. 2024. Exploring Semantics in Pretrained Language Model Attention. In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024), pages 326–333, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- Exploring Semantics in Pretrained Language Model Attention (Charpentier et al., *SEM 2024)
- PDF:
- https://preview.aclanthology.org/jeptaln-2024-ingestion/2024.starsem-1.26.pdf