Abstract
Dropout is a widely used regularization trick to resolve the overfitting issue in large feedforward neural networks trained on a small dataset, which performs poorly on the held-out test subset. Although the effectiveness of this regularization trick has been extensively studied for convolutional neural networks, there is a lack of analysis of it for unsupervised models and in particular, VAE-based neural topic models. In this paper, we have analyzed the consequences of dropout in the encoder as well as in the decoder of the VAE architecture in three widely used neural topic models, namely, contextualized topic model (CTM), ProdLDA, and embedded topic model (ETM) using four publicly available datasets. We characterize the dropout effect on these models in terms of the quality and predictive performance of the generated topics.- Anthology ID:
- 2023.eacl-main.162
- Volume:
- Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2220–2229
- Language:
- URL:
- https://aclanthology.org/2023.eacl-main.162
- DOI:
- Cite (ACL):
- Suman Adhya, Avishek Lahiri, and Debarshi Kumar Sanyal. 2023. Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modeling. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2220–2229, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modeling (Adhya et al., EACL 2023)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/2023.eacl-main.162.pdf