Language Models as Context-sensitive Word Search Engines
Matti Wiegmann, Michael Völske, Benno Stein, Martin Potthast
Abstract
Context-sensitive word search engines are writing assistants that support word choice, phrasing, and idiomatic language use by indexing large-scale n-gram collections and implementing a wildcard search. However, search results become unreliable with increasing context size (e.g., n>=5), when observations become sparse. This paper proposes two strategies for word search with larger n, based on masked and conditional language modeling. We build such search engines using BERT and BART and compare their capabilities in answering English context queries with those of the n-gram-based word search engine Netspeak. Our proposed strategies score within 5 percentage points MRR of n-gram collections while answering up to 5 times as many queries.- Anthology ID:
- 2022.in2writing-1.5
- Volume:
- Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022)
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Ting-Hao 'Kenneth' Huang, Vipul Raheja, Dongyeop Kang, John Joon Young Chung, Daniel Gissin, Mina Lee, Katy Ilonka Gero
- Venue:
- In2Writing
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 39–45
- Language:
- URL:
- https://aclanthology.org/2022.in2writing-1.5
- DOI:
- 10.18653/v1/2022.in2writing-1.5
- Cite (ACL):
- Matti Wiegmann, Michael Völske, Benno Stein, and Martin Potthast. 2022. Language Models as Context-sensitive Word Search Engines. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), pages 39–45, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Language Models as Context-sensitive Word Search Engines (Wiegmann et al., In2Writing 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.in2writing-1.5.pdf
- Code
- webis-de/in2writing22-language-models-as-context-sensitive-word-search-engines
- Data
- CLOTH, WikiText-103, WikiText-2