Categorical Metadata Representation for Customized Text Classification

Jihyeok Kim, Reinald Kim Amplayo, Kyungjae Lee, Sua Sung, Minji Seo, Seung-won Hwang


Abstract
The performance of text classification has improved tremendously using intelligently engineered neural-based models, especially those injecting categorical metadata as additional information, e.g., using user/product information for sentiment classification. This information has been used to modify parts of the model (e.g., word embeddings, attention mechanisms) such that results can be customized according to the metadata. We observe that current representation methods for categorical metadata, which are devised for human consumption, are not as effective as claimed in popular classification methods, outperformed even by simple concatenation of categorical features in the final layer of the sentence encoder. We conjecture that categorical features are harder to represent for machine use, as available context only indirectly describes the category, and even such context is often scarce (for tail category). To this end, we propose using basis vectors to effectively incorporate categorical metadata on various parts of a neural-based model. This additionally decreases the number of parameters dramatically, especially when the number of categorical features is large. Extensive experiments on various data sets with different properties are performed and show that through our method, we can represent categorical metadata more effectively to customize parts of the model, including unexplored ones, and increase the performance of the model greatly.
Anthology ID:
Q19-1013
Volume:
Transactions of the Association for Computational Linguistics, Volume 7
Month:
Year:
2019
Address:
Cambridge, MA
Venue:
TACL
SIG:
Publisher:
MIT Press
Note:
Pages:
201–215
Language:
URL:
https://aclanthology.org/Q19-1013
DOI:
10.1162/tacl_a_00263
Bibkey:
Cite (ACL):
Jihyeok Kim, Reinald Kim Amplayo, Kyungjae Lee, Sua Sung, Minji Seo, and Seung-won Hwang. 2019. Categorical Metadata Representation for Customized Text Classification. Transactions of the Association for Computational Linguistics, 7:201–215.
Cite (Informal):
Categorical Metadata Representation for Customized Text Classification (Kim et al., TACL 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/Q19-1013.pdf
Video:
 https://vimeo.com/384481958
Code
 zizi1532/BasisCustomize +  additional community code
Data
IMDb Movie Reviews