Abstract
Ensembling is a popular method used to improve performance as a last resort. However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members. This paper proposes Multi-Ticket Ensemble, which finetunes different subnetworks of a single pretrained model and ensembles them. We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks and their ensemble outperformed the standard ensemble in some tasks when accurate lottery tickets are found on the tasks.- Anthology ID:
- 2022.bigscience-1.4
- Volume:
- Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models
- Month:
- May
- Year:
- 2022
- Address:
- virtual+Dublin
- Editors:
- Angela Fan, Suzana Ilic, Thomas Wolf, Matthias Gallé
- Venue:
- BigScience
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 42–50
- Language:
- URL:
- https://aclanthology.org/2022.bigscience-1.4
- DOI:
- 10.18653/v1/2022.bigscience-1.4
- Cite (ACL):
- Sosuke Kobayashi, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2022. Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model. In Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models, pages 42–50, virtual+Dublin. Association for Computational Linguistics.
- Cite (Informal):
- Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model (Kobayashi et al., BigScience 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2022.bigscience-1.4.pdf
- Data
- GLUE, MRPC, SQuAD, SST, SST-2, SuperGLUE