@inproceedings{albilali-etal-2021-bert,
    title = "What does {BERT} Learn from {A}rabic Machine Reading Comprehension Datasets?",
    author = "Albilali, Eman  and
      Altwairesh, Nora  and
      Hosny, Manar",
    editor = "Habash, Nizar  and
      Bouamor, Houda  and
      Hajj, Hazem  and
      Magdy, Walid  and
      Zaghouani, Wajdi  and
      Bougares, Fethi  and
      Tomeh, Nadi  and
      Abu Farha, Ibrahim  and
      Touileb, Samia",
    booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
    month = apr,
    year = "2021",
    address = "Kyiv, Ukraine (Virtual)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2021.wanlp-1.4/",
    pages = "32--41",
    abstract = "In machine reading comprehension tasks, a model must extract an answer from the available context given a question and a passage. Recently, transformer-based pre-trained language models have achieved state-of-the-art performance in several natural language processing tasks. However, it is unclear whether such performance reflects true language understanding. In this paper, we propose adversarial examples to probe an Arabic pre-trained language model (AraBERT), leading to a significant performance drop over four Arabic machine reading comprehension datasets. We present a layer-wise analysis for the transformer{'}s hidden states to offer insights into how AraBERT reasons to derive an answer. The experiments indicate that AraBERT relies on superficial cues and keyword matching rather than text understanding. Furthermore, hidden state visualization demonstrates that prediction errors can be recognized from vector representations in earlier layers."
}