@inproceedings{bao-etal-2019-inspecting,
    title = "Inspecting Unification of Encoding and Matching with Transformer: A Case Study of Machine Reading Comprehension",
    author = "Bao, Hangbo  and
      Dong, Li  and
      Wei, Furu  and
      Wang, Wenhui  and
      Yang, Nan  and
      Cui, Lei  and
      Piao, Songhao  and
      Zhou, Ming",
    editor = "Fisch, Adam  and
      Talmor, Alon  and
      Jia, Robin  and
      Seo, Minjoon  and
      Choi, Eunsol  and
      Chen, Danqi",
    booktitle = "Proceedings of the 2nd Workshop on Machine Reading for Question Answering",
    month = nov,
    year = "2019",
    address = "Hong Kong, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/build-pipeline-with-new-library/D19-5802/",
    doi = "10.18653/v1/D19-5802",
    pages = "14--18",
    abstract = "Most machine reading comprehension (MRC) models separately handle encoding and matching with different network architectures. In contrast, pretrained language models with Transformer layers, such as GPT (Radford et al., 2018) and BERT (Devlin et al., 2018), have achieved competitive performance on MRC. A research question that naturally arises is: apart from the benefits of pre-training, how many performance gain comes from the unified network architecture. In this work, we evaluate and analyze unifying encoding and matching components with Transformer for the MRC task. Experimental results on SQuAD show that the unified model outperforms previous networks that separately treat encoding and matching. We also introduce a metric to inspect whether a Transformer layer tends to perform encoding or matching. The analysis results show that the unified model learns different modeling strategies compared with previous manually-designed models."
}