Abstract
We explore different approaches to explicit entity modelling in language models (LM). We independently replicate two existing models in a controlled setup, introduce a simplified variant of one of the models and analyze their performance in direct comparison. Our results suggest that today’s models are limited as several stochastic variables make learning difficult. We show that the most challenging point in the systems is the decision if the next token is an entity token. The low precision and recall for this variable will lead to severe cascading errors. Our own simplified approach dispenses with the need for latent variables and improves the performance in the entity yes/no decision. A standard well-tuned baseline RNN-LM with a larger number of hidden units outperforms all entity-enabled LMs in terms of perplexity.- Anthology ID:
- W19-2803
- Volume:
- Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference
- Month:
- June
- Year:
- 2019
- Address:
- Minneapolis, USA
- Venue:
- CRAC
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 15–19
- Language:
- URL:
- https://aclanthology.org/W19-2803
- DOI:
- 10.18653/v1/W19-2803
- Cite (ACL):
- Jenny Kunz and Christian Hardmeier. 2019. Entity Decisions in Neural Language Modelling: Approaches and Problems. In Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference, pages 15–19, Minneapolis, USA. Association for Computational Linguistics.
- Cite (Informal):
- Entity Decisions in Neural Language Modelling: Approaches and Problems (Kunz & Hardmeier, CRAC 2019)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/W19-2803.pdf