Gabriele Prato


2020

pdf bib
Fully Quantized Transformer for Machine Translation
Gabriele Prato | Ella Charlaix | Mehdi Rezagholizadeh
Findings of the Association for Computational Linguistics: EMNLP 2020

State-of-the-art neural machine translation methods employ massive amounts of parameters. Drastically reducing computational costs of such methods without affecting performance has been up to this point unsuccessful. To this end, we propose FullyQT: an all-inclusive quantization strategy for the Transformer. To the best of our knowledge, we are the first to show that it is possible to avoid any loss in translation quality with a fully quantized Transformer. Indeed, compared to full-precision, our 8-bit models score greater or equal BLEU on most tasks. Comparing ourselves to all previously proposed methods, we achieve state-of-the-art quantization results.

2019

pdf
Towards Lossless Encoding of Sentences
Gabriele Prato | Mathieu Duchesneau | Sarath Chandar | Alain Tapp
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

A lot of work has been done in the field of image compression via machine learning, but not much attention has been given to the compression of natural language. Compressing text into lossless representations while making features easily retrievable is not a trivial task, yet has huge benefits. Most methods designed to produce feature rich sentence embeddings focus solely on performing well on downstream tasks and are unable to properly reconstruct the original sequence from the learned embedding. In this work, we propose a near lossless method for encoding long sequences of texts as well as all of their sub-sequences into feature rich representations. We test our method on sentiment analysis and show good performance across all sub-sentence and sentence embeddings.