Abstract
We present in this work a method for incorporating global context in long documents when making local decisions in sequence labeling problems like NER. Inspired by work in featurized log-linear models (Chieu and Ng, 2002; Sutton and McCallum, 2004), our model learns to attend to multiple mentions of the same word type in generating a representation for each token in context, extending that work to learning representations that can be incorporated into modern neural models. Attending to broader context at test time provides complementary information to pretraining (Gururangan et al., 2020), yields strong gains over equivalently parameterized models lacking such context, and performs best at recognizing entities with high TF-IDF scores (i.e., those that are important within a document).- Anthology ID:
- 2020.findings-emnlp.330
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2020
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3692–3704
- Language:
- URL:
- https://aclanthology.org/2020.findings-emnlp.330
- DOI:
- 10.18653/v1/2020.findings-emnlp.330
- Cite (ACL):
- Matthew Jörke, Jon Gillick, Matthew Sims, and David Bamman. 2020. Attending to Long-Distance Document Context for Sequence Labeling. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3692–3704, Online. Association for Computational Linguistics.
- Cite (Informal):
- Attending to Long-Distance Document Context for Sequence Labeling (Jörke et al., Findings 2020)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/2020.findings-emnlp.330.pdf
- Code
- mjoerke/doc-arc
- Data
- GENIA, LitBank