Investigating the Working of Text Classifiers

Devendra Sachan, Manzil Zaheer, Ruslan Salakhutdinov


Abstract
Text classification is one of the most widely studied tasks in natural language processing. Motivated by the principle of compositionality, large multilayer neural network models have been employed for this task in an attempt to effectively utilize the constituent expressions. Almost all of the reported work train large networks using discriminative approaches, which come with a caveat of no proper capacity control, as they tend to latch on to any signal that may not generalize. Using various recent state-of-the-art approaches for text classification, we explore whether these models actually learn to compose the meaning of the sentences or still just focus on some keywords or lexicons for classifying the document. To test our hypothesis, we carefully construct datasets where the training and test splits have no direct overlap of such lexicons, but overall language structure would be similar. We study various text classifiers and observe that there is a big performance drop on these datasets. Finally, we show that even simple models with our proposed regularization techniques, which disincentivize focusing on key lexicons, can substantially improve classification accuracy.
Anthology ID:
C18-1180
Volume:
Proceedings of the 27th International Conference on Computational Linguistics
Month:
August
Year:
2018
Address:
Santa Fe, New Mexico, USA
Editors:
Emily M. Bender, Leon Derczynski, Pierre Isabelle
Venue:
COLING
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2120–2131
Language:
URL:
https://aclanthology.org/C18-1180
DOI:
Bibkey:
Cite (ACL):
Devendra Sachan, Manzil Zaheer, and Ruslan Salakhutdinov. 2018. Investigating the Working of Text Classifiers. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2120–2131, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Cite (Informal):
Investigating the Working of Text Classifiers (Sachan et al., COLING 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl-24-ws-corrections/C18-1180.pdf
Data
IMDb Movie Reviews