Accurate Supervised and Semi-Supervised Machine Reading for Long Documents
Daniel Hewlett, Llion Jones, Alexandre Lacoste, Izzeddin Gur
Abstract
We introduce a hierarchical architecture for machine reading capable of extracting precise information from long documents. The model divides the document into small, overlapping windows and encodes all windows in parallel with an RNN. It then attends over these window encodings, reducing them to a single encoding, which is decoded into an answer using a sequence decoder. This hierarchical approach allows the model to scale to longer documents without increasing the number of sequential steps. In a supervised setting, our model achieves state of the art accuracy of 76.8 on the WikiReading dataset. We also evaluate the model in a semi-supervised setting by downsampling the WikiReading training set to create increasingly smaller amounts of supervision, while leaving the full unlabeled document corpus to train a sequence autoencoder on document windows. We evaluate models that can reuse autoencoder states and outputs without fine-tuning their weights, allowing for more efficient training and inference.- Anthology ID:
- D17-1214
- Volume:
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Month:
- September
- Year:
- 2017
- Address:
- Copenhagen, Denmark
- Editors:
- Martha Palmer, Rebecca Hwa, Sebastian Riedel
- Venue:
- EMNLP
- SIG:
- SIGDAT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2011–2020
- Language:
- URL:
- https://aclanthology.org/D17-1214
- DOI:
- 10.18653/v1/D17-1214
- Cite (ACL):
- Daniel Hewlett, Llion Jones, Alexandre Lacoste, and Izzeddin Gur. 2017. Accurate Supervised and Semi-Supervised Machine Reading for Long Documents. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2011–2020, Copenhagen, Denmark. Association for Computational Linguistics.
- Cite (Informal):
- Accurate Supervised and Semi-Supervised Machine Reading for Long Documents (Hewlett et al., EMNLP 2017)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/D17-1214.pdf
- Data
- WikiReading