Andrey Boytsov


2023

pdf bib
Comparing Pre-Training Schemes for Luxembourgish BERT Models
Cedric Lothritz | Saad Ezzini | Christoph Purschke | Tegawendé Bissyandé | Jacques Klein | Isabella Olariu | Andrey Boytsov | Clément LeFebvre | Anne Goujon
Proceedings of the 19th Conference on Natural Language Processing (KONVENS 2023)

pdf bib
Evaluating the Impact of Text De-Identification on Downstream NLP Tasks
Cedric Lothritz | Bertrand Lebichot | Kevin Allix | Saad Ezzini | Tegawendé Bissyandé | Jacques Klein | Andrey Boytsov | Clément Lefebvre | Anne Goujon
Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)

Data anonymisation is often required to comply with regulations when transfering information across departments or entities. However, the risk is that this procedure can distort the data and jeopardise the models built on it. Intuitively, the process of training an NLP model on anonymised data may lower the performance of the resulting model when compared to a model trained on non-anonymised data. In this paper, we investigate the impact of de-identification on the performance of nine downstream NLP tasks. We focus on the anonymisation and pseudonymisation of personal names and compare six different anonymisation strategies for two state-of-the-art pre-trained models. Based on these experiments, we formulate recommendations on how the de-identification should be performed to guarantee accurate NLP models. Our results reveal that de-identification does have a negative impact on the performance of NLP models, but this impact is relatively low. We also find that using pseudonymisation techniques involving random names leads to better performance across most tasks.

2022

pdf
LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish
Cedric Lothritz | Bertrand Lebichot | Kevin Allix | Lisa Veiber | Tegawende Bissyande | Jacques Klein | Andrey Boytsov | Clément Lefebvre | Anne Goujon
Proceedings of the Thirteenth Language Resources and Evaluation Conference

Pre-trained Language Models such as BERT have become ubiquitous in NLP where they have achieved state-of-the-art performance in most NLP tasks. While these models are readily available for English and other widely spoken languages, they remain scarce for low-resource languages such as Luxembourgish. In this paper, we present LuxemBERT, a BERT model for the Luxembourgish language that we create using the following approach: we augment the pre-training dataset by considering text data from a closely related language that we partially translate using a simple and straightforward method. We are then able to produce the LuxemBERT model, which we show to be effective for various NLP tasks: it outperforms a simple baseline built with the available Luxembourgish text data as well the multilingual mBERT model, which is currently the only option for transformer-based language models in Luxembourgish. Furthermore, we present datasets for various downstream NLP tasks that we created for this study and will make available to researchers on request.