Abstract
What is the information captured by neural network models of language? We address this question in the case of character-level recurrent neural language models. These models do not have explicit word representations; do they acquire implicit ones? We assess the lexical capacity of a network using the lexical decision task common in psycholinguistics: the system is required to decide whether or not a string of characters forms a word. We explore how accuracy on this task is affected by the architecture of the network, focusing on cell type (LSTM vs. SRN), depth and width. We also compare these architectural properties to a simple count of the parameters of the network. The overall number of parameters in the network turns out to be the most important predictor of accuracy; in particular, there is little evidence that deeper networks are beneficial for this task.- Anthology ID:
- E17-2020
- Volume:
- Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers
- Month:
- April
- Year:
- 2017
- Address:
- Valencia, Spain
- Editors:
- Mirella Lapata, Phil Blunsom, Alexander Koller
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 125–130
- Language:
- URL:
- https://aclanthology.org/E17-2020
- DOI:
- Cite (ACL):
- Gaël Le Godais, Tal Linzen, and Emmanuel Dupoux. 2017. Comparing Character-level Neural Language Models Using a Lexical Decision Task. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 125–130, Valencia, Spain. Association for Computational Linguistics.
- Cite (Informal):
- Comparing Character-level Neural Language Models Using a Lexical Decision Task (Le Godais et al., EACL 2017)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/E17-2020.pdf