Abstract
We introduce two simple randomized variants of byte pair encoding (BPE) and explore whether randomizing the selection of merge operations substantially affects a downstream machine translation task. We focus on translation into morphologically rich languages, hypothesizing that this task may show sensitivity to the method of choosing subwords. Analysis using a Bayesian linear model indicates that one variant performs nearly indistinguishably compared to standard BPE while the other degrades performance less than we anticipated. We conclude that although standard BPE is widely used, there exists an interesting universe of potential variations on it worth investigating. Our code is available at: https://github.com/bltlab/random-bpe.- Anthology ID:
- 2023.insights-1.7
- Volume:
- Proceedings of the Fourth Workshop on Insights from Negative Results in NLP
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Shabnam Tafreshi, Arjun Akula, João Sedoc, Aleksandr Drozd, Anna Rogers, Anna Rumshisky
- Venues:
- insights | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 59–66
- Language:
- URL:
- https://aclanthology.org/2023.insights-1.7
- DOI:
- 10.18653/v1/2023.insights-1.7
- Cite (ACL):
- Jonne Saleva and Constantine Lignos. 2023. What changes when you randomly choose BPE merge operations? Not much.. In Proceedings of the Fourth Workshop on Insights from Negative Results in NLP, pages 59–66, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- What changes when you randomly choose BPE merge operations? Not much. (Saleva & Lignos, insights-WS 2023)
- PDF:
- https://preview.aclanthology.org/revert-3132-ingestion-checklist/2023.insights-1.7.pdf