@inproceedings{zhang-etal-2022-probing,
    title = "Probing {GPT}-3{'}s Linguistic Knowledge on Semantic Tasks",
    author = "Zhang, Lining  and
      Wang, Mengchen  and
      Chen, Liben  and
      Zhang, Wenxin",
    editor = "Bastings, Jasmijn  and
      Belinkov, Yonatan  and
      Elazar, Yanai  and
      Hupkes, Dieuwke  and
      Saphra, Naomi  and
      Wiegreffe, Sarah",
    booktitle = "Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates (Hybrid)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2022.blackboxnlp-1.24/",
    doi = "10.18653/v1/2022.blackboxnlp-1.24",
    pages = "297--304",
    abstract = "GPT-3 has attracted much attention from both academia and industry. However, it is still unclear what GPT-3 has understood or learned especially in linguistic knowledge. Some studies have shown linguistic phenomena including negation and tense are hard to be recognized by language models such as BERT. In this study, we conduct probing tasks focusing on semantic information. Specifically, we investigate GPT-3{'}s linguistic knowledge on semantic tasks to identify tense, the number of subjects, and the number of objects for a given sentence. We also experiment with different prompt designs and temperatures of the decoding method. Our experiment results suggest that GPT-3 has acquired linguistic knowledge to identify certain semantic information in most cases, but still fails when there are some types of disturbance happening in the sentence. We also perform error analysis to summarize some common types of mistakes that GPT-3 has made when dealing with certain semantic information."
}