Information Extraction from Swedish Medical Prescriptions with Sig-Transformer Encoder

John Pougué Biyong, Bo Wang, Terry Lyons, Alejo Nevado-Holgado


Abstract
Relying on large pretrained language models such as Bidirectional Encoder Representations from Transformers (BERT) for encoding and adding a simple prediction layer has led to impressive performance in many clinical natural language processing (NLP) tasks. In this work, we present a novel extension to the Transformer architecture, by incorporating signature transform with the self-attention model. This architecture is added between embedding and prediction layers. Experiments on a new Swedish prescription data show the proposed architecture to be superior in two of the three information extraction tasks, comparing to baseline models. Finally, we evaluate two different embedding approaches between applying Multilingual BERT and translating the Swedish text to English then encode with a BERT model pretrained on clinical notes.
Anthology ID:
2020.clinicalnlp-1.5
Volume:
Proceedings of the 3rd Clinical Natural Language Processing Workshop
Month:
November
Year:
2020
Address:
Online
Venue:
ClinicalNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
41–54
Language:
URL:
https://aclanthology.org/2020.clinicalnlp-1.5
DOI:
10.18653/v1/2020.clinicalnlp-1.5
Bibkey:
Cite (ACL):
John Pougué Biyong, Bo Wang, Terry Lyons, and Alejo Nevado-Holgado. 2020. Information Extraction from Swedish Medical Prescriptions with Sig-Transformer Encoder. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 41–54, Online. Association for Computational Linguistics.
Cite (Informal):
Information Extraction from Swedish Medical Prescriptions with Sig-Transformer Encoder (Pougué Biyong et al., ClinicalNLP 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/2020.clinicalnlp-1.5.pdf
Optional supplementary material:
 2020.clinicalnlp-1.5.OptionalSupplementaryMaterial.zip
Video:
 https://slideslive.com/38939813