Abstract
Combining a pretrained language model (PLM) with textual patterns has been shown to help in both zero- and few-shot settings. For zero-shot performance, it makes sense to design patterns that closely resemble the text seen during self-supervised pretraining because the model has never seen anything else. Supervised training allows for more flexibility. If we allow for tokens outside the PLM’s vocabulary, patterns can be adapted more flexibly to a PLM’s idiosyncrasies. Contrasting patterns where a “token” can be any continuous vector from those where a discrete choice between vocabulary elements has to be made, we call our method CONtinous pAtterNs (CONAN). We evaluate CONAN on two established benchmarks for lexical inference in context (LIiC) a.k.a. predicate entailment, a challenging natural language understanding task with relatively small training data. In a direct comparison with discrete patterns, CONAN consistently leads to improved performance, setting a new state of the art. Our experiments give valuable insights on the kind of pattern that enhances a PLM’s performance on LIiC and raise important questions regarding our understanding of PLMs using text patterns.- Anthology ID:
- 2021.emnlp-main.556
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Editors:
- Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6952–6959
- Language:
- URL:
- https://aclanthology.org/2021.emnlp-main.556
- DOI:
- 10.18653/v1/2021.emnlp-main.556
- Cite (ACL):
- Martin Schmitt and Hinrich Schütze. 2021. Continuous Entailment Patterns for Lexical Inference in Context. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6952–6959, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Continuous Entailment Patterns for Lexical Inference in Context (Schmitt & Schütze, EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-3/2021.emnlp-main.556.pdf
- Code
- mnschmit/conan
- Data
- SherLIiC