Entity Decisions in Neural Language Modelling: Approaches and Problems

Jenny Kunz, Christian Hardmeier


Abstract
We explore different approaches to explicit entity modelling in language models (LM). We independently replicate two existing models in a controlled setup, introduce a simplified variant of one of the models and analyze their performance in direct comparison. Our results suggest that today’s models are limited as several stochastic variables make learning difficult. We show that the most challenging point in the systems is the decision if the next token is an entity token. The low precision and recall for this variable will lead to severe cascading errors. Our own simplified approach dispenses with the need for latent variables and improves the performance in the entity yes/no decision. A standard well-tuned baseline RNN-LM with a larger number of hidden units outperforms all entity-enabled LMs in terms of perplexity.
Anthology ID:
W19-2803
Volume:
Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference
Month:
June
Year:
2019
Address:
Minneapolis, USA
Venue:
CRAC
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
15–19
Language:
URL:
https://aclanthology.org/W19-2803
DOI:
10.18653/v1/W19-2803
Bibkey:
Cite (ACL):
Jenny Kunz and Christian Hardmeier. 2019. Entity Decisions in Neural Language Modelling: Approaches and Problems. In Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference, pages 15–19, Minneapolis, USA. Association for Computational Linguistics.
Cite (Informal):
Entity Decisions in Neural Language Modelling: Approaches and Problems (Kunz & Hardmeier, CRAC 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/W19-2803.pdf