Abstract
Tracking entities throughout a procedure described in a text is challenging due to the dynamic nature of the world described in the process. Firstly, we propose to formulate this task as a question answering problem. This enables us to use pre-trained transformer-based language models on other QA benchmarks by adapting those to the procedural text understanding. Secondly, since the transformer-based language models cannot encode the flow of events by themselves, we propose a Time-Stamped Language Model (TSLM) to encode event information in LMs architecture by introducing the timestamp encoding. Our model evaluated on the Propara dataset shows improvements on the published state-of-the-art results with a 3.1% increase in F1 score. Moreover, our model yields better results on the location prediction task on the NPN-Cooking dataset. This result indicates that our approach is effective for procedural text understanding in general.- Anthology ID:
- 2021.naacl-main.362
- Volume:
- Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Month:
- June
- Year:
- 2021
- Address:
- Online
- Editors:
- Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4560–4570
- Language:
- URL:
- https://aclanthology.org/2021.naacl-main.362
- DOI:
- 10.18653/v1/2021.naacl-main.362
- Cite (ACL):
- Hossein Rajaby Faghihi and Parisa Kordjamshidi. 2021. Time-Stamped Language Model: Teaching Language Models to Understand The Flow of Events. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4560–4570, Online. Association for Computational Linguistics.
- Cite (Informal):
- Time-Stamped Language Model: Teaching Language Models to Understand The Flow of Events (Rajaby Faghihi & Kordjamshidi, NAACL 2021)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2021.naacl-main.362.pdf
- Code
- HLR/TSLM
- Data
- ProPara