@inproceedings{shanbhag-etal-2025-non,
    title = "Non-Contextual {BERT} or {F}ast{T}ext? A Comparative Analysis",
    author = "Shanbhag, Abhay  and
      Jadhav, Suramya  and
      Thakurdesai, Amogh  and
      Sinare, Ridhima Bhaskar  and
      Joshi, Raviraj",
    editor = "Das, Sudhansu Bala  and
      Mishra, Pruthwik  and
      Singh, Alok  and
      Muhammad, Shamsuddeen Hassan  and
      Ekbal, Asif  and
      Das, Uday Kumar",
    booktitle = "Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models",
    month = sep,
    year = "2025",
    address = "Varna, Bulgaria",
    publisher = "INCOMA Ltd., Shoumen, BULGARIA",
    url = "https://preview.aclanthology.org/corrections-2026-01/2025.globalnlp-1.4/",
    pages = "27--33",
    abstract = "Natural Language Processing (NLP) for low-resource languages, which lack large annotated datasets, faces significant challenges due to limited high-quality data and linguistic resources. The selection of embeddings plays a critical role in achieving strong performance in NLP tasks. While contextual BERT embeddings require a full forward pass, non-contextual BERT embeddings rely only on table lookup. Existing research has primarily focused on contextual BERT embeddings, leaving non-contextual embeddings largely unexplored. In this study, we analyze the effectiveness of non-contextual embeddings from BERT models (MuRIL and MahaBERT) and FastText models (IndicFT and MahaFT) for tasks such as news classification, sentiment analysis, and hate speech detection in one such low-resource language{---}Marathi. We compare these embeddings with their contextual and compressed variants. Our findings indicate that non-contextual BERT embeddings extracted from the model{'}s first embedding layer outperform FastText embeddings, presenting a promising alternative for low-resource NLP."
}