Vladislav Lialin
2022
Life after BERT: What do Other Muppets Understand about Language?
Vladislav Lialin
|
Kevin Zhao
|
Namrata Shivagunde
|
Anna Rumshisky
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Existing pre-trained transformer analysis works usually focus only on one or two model families at a time, overlooking the variability of the architecture and pre-training objectives. In our work, we utilize the oLMpics bench- mark and psycholinguistic probing datasets for a diverse set of 29 models including T5, BART, and ALBERT. Additionally, we adapt the oLMpics zero-shot setup for autoregres- sive models and evaluate GPT networks of different sizes. Our findings show that none of these models can resolve compositional questions in a zero-shot fashion, suggesting that this skill is not learnable using existing pre-training objectives. Furthermore, we find that global model decisions such as architecture, directionality, size of the dataset, and pre-training objective are not predictive of a model’s linguistic capabilities.
Learning to Ask Like a Physician
Eric Lehman
|
Vladislav Lialin
|
Katelyn Edelwina Legaspi
|
Anne Janelle Sy
|
Patricia Therese Pile
|
Nicole Rose Alberto
|
Richard Raymund Ragasa
|
Corinna Victoria Puyat
|
Marianne Katharina Taliño
|
Isabelle Rose Alberto
|
Pia Gabrielle Alfonso
|
Dana Moukheiber
|
Byron Wallace
|
Anna Rumshisky
|
Jennifer Liang
|
Preethi Raghavan
|
Leo Anthony Celi
|
Peter Szolovits
Proceedings of the 4th Clinical Natural Language Processing Workshop
Existing question answering (QA) datasets derived from electronic health records (EHR) are artificially generated and consequently fail to capture realistic physician information needs. We present Discharge Summary Clinical Questions (DiSCQ), a newly curated question dataset composed of 2,000+ questions paired with the snippets of text (triggers) that prompted each question. The questions are generated by medical experts from 100+ MIMIC-III discharge summaries. We analyze this dataset to characterize the types of information sought by medical experts. We also train baseline models for trigger detection and question generation (QG), paired with unsupervised answer retrieval over EHRs. Our baseline model is able to generate high quality questions in over 62% of cases when prompted with human selected triggers. We release this dataset (and all code to reproduce baseline model results) to facilitate further research into realistic clinical QA and QG: https://github.com/elehman16/discq.
Search
Co-authors
- Anna Rumshisky 2
- Kevin Zhao 1
- Namrata Shivagunde 1
- Eric Lehman 1
- Katelyn Edelwina Legaspi 1
- show all...