@inproceedings{luo-2021-attention,
    title = "Have Attention Heads in {BERT} Learned Constituency Grammar?",
    author = "Luo, Ziyang",
    editor = "Sorodoc, Ionut-Teodor  and
      Sushil, Madhumita  and
      Takmaz, Ece  and
      Agirre, Eneko",
    booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop",
    month = apr,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.eacl-srw.2",
    doi = "10.18653/v1/2021.eacl-srw.2",
    pages = "8--15",
    abstract = "With the success of pre-trained language models in recent years, more and more researchers focus on opening the {``}black box{''} of these models. Following this interest, we carry out a qualitative and quantitative analysis of constituency grammar in attention heads of BERT and RoBERTa. We employ the syntactic distance method to extract implicit constituency grammar from the attention weights of each head. Our results show that there exist heads that can induce some grammar types much better than baselines, suggesting that some heads act as a proxy for constituency grammar. We also analyze how attention heads{'} constituency grammar inducing (CGI) ability changes after fine-tuning with two kinds of tasks, including sentence meaning similarity (SMS) tasks and natural language inference (NLI) tasks. Our results suggest that SMS tasks decrease the average CGI ability of upper layers, while NLI tasks increase it. Lastly, we investigate the connections between CGI ability and natural language understanding ability on QQP and MNLI tasks.",
}