Sanjika Hewavitharana


2022

pdf
Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token
Baohao Liao | David Thulke | Sanjika Hewavitharana | Hermann Ney | Christof Monz
Findings of the Association for Computational Linguistics: EMNLP 2022

The pre-training of masked language models (MLMs) consumes massive computation to achieve good results on downstream NLP tasks, resulting in a large carbon footprint. In the vanilla MLM, the virtual tokens, [MASK]s, act as placeholders and gather the contextualized information from unmasked tokens to restore the corrupted information. It raises the question of whether we can append [MASK]s at a later layer, to reduce the sequence length for earlier layers and make the pre-training more efficient. We show: (1) [MASK]s can indeed be appended at a later layer, being disentangled from the word embedding; (2) The gathering of contextualized information from unmasked tokens can be conducted with a few layers. By further increasing the masking rate from 15% to 50%, we can pre-train RoBERTa-base and RoBERTa-large from scratch with only 78% and 68% of the original computational budget without any degradation on the GLUE benchmark. When pre-training with the original budget, our method outperforms RoBERTa for 6 out of 8 GLUE tasks, on average by 0.4%.

2021

pdf
Back-translation for Large-Scale Multilingual Machine Translation
Baohao Liao | Shahram Khadivi | Sanjika Hewavitharana
Proceedings of the Sixth Conference on Machine Translation

This paper illustrates our approach to the shared task on large-scale multilingual machine translation in the sixth conference on machine translation (WMT-21). In this work, we aim to build a single multilingual translation system with a hypothesis that a universal cross-language representation leads to better multilingual translation performance. We extend the exploration of different back-translation methods from bilingual translation to multilingual translation. Better performance is obtained by the constrained sampling method, which is different from the finding of the bilingual translation. Besides, we also explore the effect of vocabularies and the amount of synthetic data. Surprisingly, the smaller size of vocabularies perform better, and the extensive monolingual English data offers a modest improvement. We submitted to both the small tasks and achieve the second place.

2020

pdf bib
Proceedings of Workshop on Natural Language Processing in E-Commerce
Huasha Zhao | Parikshit Sondhi | Nguyen Bach | Sanjika Hewavitharana | Yifan He | Luo Si | Heng Ji
Proceedings of Workshop on Natural Language Processing in E-Commerce

2018

pdf
Word-based Domain Adaptation for Neural Machine Translation
Shen Yan | Leonard Dahlmann | Pavel Petrushkov | Sanjika Hewavitharana | Shahram Khadivi
Proceedings of the 15th International Conference on Spoken Language Translation

In this paper, we empirically investigate applying word-level weights to adapt neural machine translation to e-commerce domains, where small e-commerce datasets and large out-of-domain datasets are available. In order to mine in-domain like words in the out-of-domain datasets, we compute word weights by using a domain-specific and a non-domain-specific language model followed by smoothing and binary quantization. The baseline model is trained on mixed in-domain and out-of-domain datasets. Experimental results on En → Zh e-commerce domain translation show that compared to continuing training without word weights, it improves MT quality by up to 3.11% BLEU absolute and 1.59% TER. We have also trained models using fine-tuning on the in-domain data. Pre-training a model with word weights improves fine-tuning up to 1.24% BLEU absolute and 1.64% TER, respectively.

2015

pdf
Error-tolerant speech-to-speech translation
Rohit Kumar | Sanjika Hewavitharana | Nina Zinovieva | Matthew E. Roy | Edward Pattison-Gordon
Proceedings of Machine Translation Summit XV: Papers

2014

pdf
Lightly-Supervised Word Sense Translation Error Detection for an Interactive Conversational Spoken Language Translation System
Dennis Mehay | Sankaranarayanan Ananthakrishnan | Sanjika Hewavitharana
Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers

pdf
Anticipatory translation model adaptation for bilingual conversations
Sanjika Hewavitharana | Dennis Mehay | Sankaranarayanan Ananthakrishnan | Rohit Kumar | John Makhoul
Proceedings of the 11th International Workshop on Spoken Language Translation: Papers

Conversational spoken language translation (CSLT) systems facilitate bilingual conversations in which the two participants speak different languages. Bilingual conversations provide additional contextual information that can be used to improve the underlying machine translation system. In this paper, we describe a novel translation model adaptation method that anticipates a participant’s response in the target language, based on his counterpart’s prior turn in the source language. Our proposed strategy uses the source language utterance to perform cross-language retrieval on a large corpus of bilingual conversations in order to obtain a set of potentially relevant target responses. The responses retrieved are used to bias translation choices towards anticipated responses. On an Iraqi-to-English CSLT task, our method achieves a significant improvement over the baseline system in terms of BLEU, TER and METEOR metrics.

2013

pdf
Semi-Supervised Word Sense Disambiguation for Mixed-Initiative Conversational Spoken Language Translation
Sankaranarayanan Ananthakrishnan | Sanjika Hewavitharana | Rohit Kumar | Enoch Kan | Rohit Prasad | Prem Natarajan
Proceedings of Machine Translation Summit XIV: Papers

pdf
Interactive Error Resolution Strategies for Speech-to-Speech Translation Systems
Rohit Kumar | Matthew Roy | Sankaranarayanan Ananthakrishnan | Sanjika Hewavitharana | Frederick Choi
Proceedings of the SIGDIAL 2013 Conference

pdf
Incremental Topic-Based Translation Model Adaptation for Conversational Spoken Language Translation
Sanjika Hewavitharana | Dennis Mehay | Sankaranarayanan Ananthakrishnan | Prem Natarajan
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

2012

pdf bib
Active error detection and resolution for speech-to-speech translation
Rohit Prasad | Rohit Kumar | Sankaranarayanan Ananthakrishnan | Wei Chen | Sanjika Hewavitharana | Matthew Roy | Frederick Choi | Aaron Challenner | Enoch Kan | Arvid Neelakantan | Prem Natarajan
Proceedings of the 9th International Workshop on Spoken Language Translation: Papers

We describe a novel two-way speech-to-speech (S2S) translation system that actively detects a wide variety of common error types and resolves them through user-friendly dialog with the user(s). We present algorithms for detecting out-of-vocabulary (OOV) named entities and terms, sense ambiguities, homophones, idioms, ill-formed input, etc. and discuss novel, interactive strategies for recovering from such errors. We also describe our approach for prioritizing different error types and an extensible architecture for implementing these decisions. We demonstrate the efficacy of our system by presenting analysis on live interactions in the English-to-Iraqi Arabic direction that are designed to invoke different error types for spoken language translation. Our analysis shows that the system can successfully resolve 47% of the errors, resulting in a dramatic improvement in the transfer of problematic concepts.

2011

pdf
Extending a probabilistic phrase alignment approach for SMT
Mridul Gupta | Sanjika Hewavitharana | Stephan Vogel
Proceedings of the 8th International Workshop on Spoken Language Translation: Evaluation Campaign

Phrase alignment is a crucial step in phrase-based statistical machine translation. We explore a way of improving phrase alignment by adding syntactic information in the form of chunks as soft constraints guided by an in-depth and detailed analysis on a hand-aligned data set. We extend a probabilistic phrase alignment model that extracts phrase pairs by optimizing phrase pair boundaries over the sentence pair [1]. The boundaries of the target phrase are chosen such that the overall sentence alignment probability is optimal. Viterbi alignment information is also added in the extended model with a view of improving phrase alignment. We extract phrase pairs using a relatively larger number of features which are discriminatively trained using a large-margin online learning algorithm, i.e., Margin Infused Relaxed Algorithm (MIRA) and integrate it in our approach. Initial experiments show improvements in both phrase alignment and translation quality for Arabic-English on a moderate-size translation task.

pdf
Extracting Parallel Phrases from Comparable Data
Sanjika Hewavitharana | Stephan Vogel
Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web

pdf
Active Learning with Multiple Annotations for Comparable Data Classification Task
Vamshi Ambati | Sanjika Hewavitharana | Stephan Vogel | Jaime Carbonell
Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web

pdf
CMU Haitian Creole-English Translation System for WMT 2011
Sanjika Hewavitharana | Nguyen Bach | Qin Gao | Vamshi Ambati | Stephan Vogel
Proceedings of the Sixth Workshop on Statistical Machine Translation

2008

pdf
Recent Improvements in the CMU Large Scale Chinese-English SMT System
Almut Silja Hildebrand | Kay Rottmann | Mohamed Noamany | Quin Gao | Sanjika Hewavitharana | Nguyen Bach | Stephan Vogel
Proceedings of ACL-08: HLT, Short Papers

2007

pdf
Experiments with a noun-phrase driven statistical machine translation system
Sanjika Hewavitharana | Alon Lavie | Stephan Vogel
Proceedings of Machine Translation Summit XI: Papers

2006

pdf
Thai Grapheme-Based Speech Recognition
Paisarn Charoenpornsawat | Sanjika Hewavitharana | Tanja Schultz
Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers

pdf
The UKA/CMU statistical machine translation system for IWSLT 2006
Matthias Eck | Ian Lane | Nguyen Bach | Sanjika Hewavitharana | Muntsin Kolss | Bing Zhao | Almut Silja Hildebrand | Stephan Vogel | Alex Waibel
Proceedings of the Third International Workshop on Spoken Language Translation: Evaluation Campaign

2005

pdf
Augmenting a statistical translation system with a translation memory
Sanjika Hewavitharana | Stephan Vogel | Alex Waibel
Proceedings of the 10th EAMT Conference: Practical applications of machine translation

pdf
The CMU Statistical Machine Translation System for IWSLT2005
Sanjika Hewavitharana | Bing Zhao | Hildebrand | Almut Silja | Matthias Eck | Chiori Hori | Stephan Vogel | Alex Waibel
Proceedings of the Second International Workshop on Spoken Language Translation

2004

pdf
The ISL statistical translation system for spoken language translation
Stephan Vogel | Sanjika Hewavitharana | Muntsin Kolss | Alex Waibel
Proceedings of the First International Workshop on Spoken Language Translation: Evaluation Campaign