Abstract
Transformer-based language models are able to capture several linguistic properties such as hierarchical structures like dependency or constituency trees. Whether similar structures for mathematics are extractable from language models has not yet been explored. This work aims to probe current state-of-the-art models for the extractability of Operator Trees from their contextualized embeddings using the structure probe designed by Hewitt and Manning. We release the code and our data set for future analysis.- Anthology ID:
- 2022.mathnlp-1.6
- Volume:
- Proceedings of the 1st Workshop on Mathematical Natural Language Processing (MathNLP)
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates (Hybrid)
- Venue:
- MathNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 40–50
- Language:
- URL:
- https://aclanthology.org/2022.mathnlp-1.6
- DOI:
- Cite (ACL):
- Anja Reusch and Wolfgang Lehner. 2022. Extracting Operator Trees from Model Embeddings. In Proceedings of the 1st Workshop on Mathematical Natural Language Processing (MathNLP), pages 40–50, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- Extracting Operator Trees from Model Embeddings (Reusch & Lehner, MathNLP 2022)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/2022.mathnlp-1.6.pdf