Comparing Character-level Neural Language Models Using a Lexical Decision Task

Gaël Le Godais, Tal Linzen, Emmanuel Dupoux


Abstract
What is the information captured by neural network models of language? We address this question in the case of character-level recurrent neural language models. These models do not have explicit word representations; do they acquire implicit ones? We assess the lexical capacity of a network using the lexical decision task common in psycholinguistics: the system is required to decide whether or not a string of characters forms a word. We explore how accuracy on this task is affected by the architecture of the network, focusing on cell type (LSTM vs. SRN), depth and width. We also compare these architectural properties to a simple count of the parameters of the network. The overall number of parameters in the network turns out to be the most important predictor of accuracy; in particular, there is little evidence that deeper networks are beneficial for this task.
Anthology ID:
E17-2020
Volume:
Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers
Month:
April
Year:
2017
Address:
Valencia, Spain
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
125–130
Language:
URL:
https://aclanthology.org/E17-2020
DOI:
Bibkey:
Cite (ACL):
Gaël Le Godais, Tal Linzen, and Emmanuel Dupoux. 2017. Comparing Character-level Neural Language Models Using a Lexical Decision Task. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 125–130, Valencia, Spain. Association for Computational Linguistics.
Cite (Informal):
Comparing Character-level Neural Language Models Using a Lexical Decision Task (Le Godais et al., EACL 2017)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/E17-2020.pdf