MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale

Andreas Rücklé, Jonas Pfeiffer, Iryna Gurevych


Abstract
We study the zero-shot transfer capabilities of text matching models on a massive scale, by self-supervised training on 140 source domains from community question answering forums in English. We investigate the model performances on nine benchmarks of answer selection and question similarity tasks, and show that all 140 models transfer surprisingly well, where the large majority of models substantially outperforms common IR baselines. We also demonstrate that considering a broad selection of source domains is crucial for obtaining the best zero-shot transfer performances, which contrasts the standard procedure that merely relies on the largest and most similar domains. In addition, we extensively study how to best combine multiple source domains. We propose to incorporate self-supervised with supervised multi-task learning on all available source domains. Our best zero-shot transfer model considerably outperforms in-domain BERT and the previous state of the art on six benchmarks. Fine-tuning of our model with in-domain data results in additional large gains and achieves the new state of the art on all nine benchmarks.
Anthology ID:
2020.emnlp-main.194
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2471–2486
Language:
URL:
https://aclanthology.org/2020.emnlp-main.194
DOI:
10.18653/v1/2020.emnlp-main.194
Bibkey:
Cite (ACL):
Andreas Rücklé, Jonas Pfeiffer, and Iryna Gurevych. 2020. MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2471–2486, Online. Association for Computational Linguistics.
Cite (Informal):
MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale (Rücklé et al., EMNLP 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2020.emnlp-main.194.pdf
Video:
 https://slideslive.com/38938833
Code
 ukplab/emnlp2020-multicqa
Data
InsuranceQA