Abstract
State-of-the-art approaches to identifying offensive language online make use of large pre-trained transformer models. However, the inference time, disk, and memory requirements of these transformer models present challenges for their wide usage in the real world. Even the distilled transformer models remain prohibitively large for many usage scenarios. To cope with these challenges, in this paper, we propose transferring knowledge from transformer models to much smaller neural models to make predictions at the token- and at the post-level. We show that this approach leads to lightweight offensive language identification models that perform on par with large transformers but with 100 times fewer parameters and much less memory usage- Anthology ID:
- 2023.findings-acl.241
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2023
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3910–3922
- Language:
- URL:
- https://aclanthology.org/2023.findings-acl.241
- DOI:
- Cite (ACL):
- Tharindu Ranasinghe and Marcos Zampieri. 2023. Teacher and Student Models of Offensive Language in Social Media. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3910–3922, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Teacher and Student Models of Offensive Language in Social Media (Ranasinghe & Zampieri, Findings 2023)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/2023.findings-acl.241.pdf