Abstract
Current neural network solutions in scientific document processing employ models pretrained on domain-specific corpi, which are usually limited in model size, as pretraining can be costly and limited by training resources. We introduce a framework that uses data augmentation from such domain-specific pretrained models to transfer domain specific knowledge to larger general pretrained models and improve performance on downstream tasks. Our method improves the performance of Named Entity Recognition in the astrophysical domain by more than 20% compared to domain-specific pretrained models finetuned to the target dataset.- Anthology ID:
- 2022.wiesp-1.10
- Volume:
- Proceedings of the first Workshop on Information Extraction from Scientific Publications
- Month:
- November
- Year:
- 2022
- Address:
- Online
- Venue:
- WIESP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 84–90
- Language:
- URL:
- https://aclanthology.org/2022.wiesp-1.10
- DOI:
- Cite (ACL):
- Po-Wei Huang. 2022. Domain Specific Augmentations as Low Cost Teachers for Large Students. In Proceedings of the first Workshop on Information Extraction from Scientific Publications, pages 84–90, Online. Association for Computational Linguistics.
- Cite (Informal):
- Domain Specific Augmentations as Low Cost Teachers for Large Students (Huang, WIESP 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.wiesp-1.10.pdf