The Dark Side of the Language: Pre-trained Transformers in the DarkNet

Leonardo Ranaldi, Aria Nourbakhsh, Elena Sofia Ruzzetti, Arianna Patrizi, Dario Onorati, Michele Mastromattei, Francesca Fallucchi, Fabio Massimo Zanzotto

Abstract
Pre-trained Transformers are challenging human performances in many Natural Language Processing tasks. The massive datasets used for pre-training seem to be the key to their success on existing tasks. In this paper, we explore how a range of pre-trained natural language understanding models performs on definitely unseen sentences provided by classification tasks over a DarkNet corpus. Surprisingly, results show that syntactic and lexical neural networks perform on par with pre-trained Transformers even after fine-tuning. Only after what we call extreme domain adaptation, that is, retraining with the masked language model task on all the novel corpus, pre-trained Transformers reach their standard high results. This suggests that huge pre-training corpora may give Transformers unexpected help since they are exposed to many of the possible sentences.
Anthology ID:
2023.ranlp-1.102
Volume:
Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing
Month:
September
Year:
2023
Address:
Varna, Bulgaria
Editors:
Ruslan Mitkov, Galia Angelova
Venue:
RANLP
SIG:
Publisher:
INCOMA Ltd., Shoumen, Bulgaria
Note:
Pages:
949–960
Language:
URL:
https://preview.aclanthology.org/icon-24-ingestion/2023.ranlp-1.102/
DOI:
Bibkey:
Cite (ACL):
Leonardo Ranaldi, Aria Nourbakhsh, Elena Sofia Ruzzetti, Arianna Patrizi, Dario Onorati, Michele Mastromattei, Francesca Fallucchi, and Fabio Massimo Zanzotto. 2023. The Dark Side of the Language: Pre-trained Transformers in the DarkNet. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 949–960, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.
Cite (Informal):
The Dark Side of the Language: Pre-trained Transformers in the DarkNet (Ranaldi et al., RANLP 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/icon-24-ingestion/2023.ranlp-1.102.pdf
PDF Search Fix data