Abstract
Large and complex models have recently been developed that require many parameters and much time to solve various problems in natural language processing. This paper explores an efficient way to avoid models being too complicated and ensure nearly equal performance to models showing the state-of-the-art. We propose a single convolutional neural network (CNN) using the sinusoidal positional encoding (SPE) in text classification. The SPE provides useful position information of a word and can construct a more efficient model architecture than before in a CNN-based approach. Our model can significantly reduce the parameter size (at least 67%) and training time (up to 85%) while maintaining similar performance to the CNN-based approach on multiple benchmark datasets.- Anthology ID:
- 2020.aacl-main.8
- Volume:
- Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing
- Month:
- December
- Year:
- 2020
- Address:
- Suzhou, China
- Venue:
- AACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 63–69
- Language:
- URL:
- https://aclanthology.org/2020.aacl-main.8
- DOI:
- Cite (ACL):
- Byoung-Doo Oh and Yu-Seop Kim. 2020. Lightweight Text Classifier using Sinusoidal Positional Encoding. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 63–69, Suzhou, China. Association for Computational Linguistics.
- Cite (Informal):
- Lightweight Text Classifier using Sinusoidal Positional Encoding (Oh & Kim, AACL 2020)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2020.aacl-main.8.pdf
- Data
- MPQA Opinion Corpus