@inproceedings{koto-fang-2021-handling,
    title = "Handling Variance of Pretrained Language Models in Grading Evidence in the Medical Literature",
    author = "Koto, Fajri  and
      Fang, Biaoyan",
    editor = "Rahimi, Afshin  and
      Lane, William  and
      Zuccon, Guido",
    booktitle = "Proceedings of the 19th Annual Workshop of the Australasian Language Technology Association",
    month = dec,
    year = "2021",
    address = "Online",
    publisher = "Australasian Language Technology Association",
    url = "https://preview.aclanthology.org/fix-sig-urls/2021.alta-1.26/",
    pages = "218--223",
    abstract = "In this paper, we investigate the utility of modern pretrained language models for the evidence grading system in the medical literature based on the ALTA 2021 shared task. We benchmark 1) domain-specific models that are optimized for medical literature and 2) domain-generic models with rich latent discourse representation (i.e. ELECTRA, RoBERTa). Our empirical experiments reveal that these modern pretrained language models suffer from high variance, and the ensemble method can improve the model performance. We found that ELECTRA performs best with an accuracy of 53.6{\%} on the test set, outperforming domain-specific models.1"
}