@inproceedings{alyafeai-ahmad-2021-arabic,
    title = "{A}rabic Compact Language Modelling for Resource Limited Devices",
    author = "Alyafeai, Zaid  and
      Ahmad, Irfan",
    editor = "Habash, Nizar  and
      Bouamor, Houda  and
      Hajj, Hazem  and
      Magdy, Walid  and
      Zaghouani, Wajdi  and
      Bougares, Fethi  and
      Tomeh, Nadi  and
      Abu Farha, Ibrahim  and
      Touileb, Samia",
    booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
    month = apr,
    year = "2021",
    address = "Kyiv, Ukraine (Virtual)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2021.wanlp-1.6/",
    pages = "53--59",
    abstract = "Natural language modelling has gained a lot of interest recently. The current state-of-the-art results are achieved by first training a very large language model and then fine-tuning it on multiple tasks. However, there is little work on smaller more compact language models for resource-limited devices or applications. Not to mention, how to efficiently train such models for a low-resource language like Arabic. In this paper, we investigate how such models can be trained in a compact way for Arabic. We also show how distillation and quantization can be applied to create even smaller models. Our experiments show that our largest model which is 2x smaller than the baseline can achieve better results on multiple tasks with 2x less data for pretraining."
}