Discrete Cosine Transform as Universal Sentence Encoder

Nada Almarwani, Mona Diab


Abstract
Modern sentence encoders are used to generate dense vector representations that capture the underlying linguistic characteristics for a sequence of words, including phrases, sentences, or paragraphs. These kinds of representations are ideal for training a classifier for an end task such as sentiment analysis, question answering and text classification. Different models have been proposed to efficiently generate general purpose sentence representations to be used in pretraining protocols. While averaging is the most commonly used efficient sentence encoder, Discrete Cosine Transform (DCT) was recently proposed as an alternative that captures the underlying syntactic characteristics of a given text without compromising practical efficiency compared to averaging. However, as with most other sentence encoders, the DCT sentence encoder was only evaluated in English. To this end, we utilize DCT encoder to generate universal sentence representation for different languages such as German, French, Spanish and Russian. The experimental results clearly show the superior effectiveness of DCT encoding in which consistent performance improvements are achieved over strong baselines on multiple standardized datasets
Anthology ID:
2021.acl-short.53
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
Month:
August
Year:
2021
Address:
Online
Editors:
Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
419–426
Language:
URL:
https://aclanthology.org/2021.acl-short.53
DOI:
10.18653/v1/2021.acl-short.53
Bibkey:
Cite (ACL):
Nada Almarwani and Mona Diab. 2021. Discrete Cosine Transform as Universal Sentence Encoder. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 419–426, Online. Association for Computational Linguistics.
Cite (Informal):
Discrete Cosine Transform as Universal Sentence Encoder (Almarwani & Diab, ACL-IJCNLP 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-2023-videos/2021.acl-short.53.pdf
Video:
 https://preview.aclanthology.org/ingest-acl-2023-videos/2021.acl-short.53.mp4