Abstract
Current NLP datasets targeting ambiguity can be solved by a native speaker with relative ease. We present Cryptonite, a large-scale dataset based on cryptic crosswords, which is both linguistically complex and naturally sourced. Each example in Cryptonite is a cryptic clue, a short phrase or sentence with a misleading surface reading, whose solving requires disambiguating semantic, syntactic, and phonetic wordplays, as well as world knowledge. Cryptic clues pose a challenge even for experienced solvers, though top-tier experts can solve them with almost 100% accuracy. Cryptonite is a challenging task for current models; fine-tuning T5-Large on 470k cryptic clues achieves only 7.6% accuracy, on par with the accuracy of a rule-based clue solver (8.6%).- Anthology ID:
- 2021.emnlp-main.344
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Editors:
- Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4186–4192
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2021.emnlp-main.344/
- DOI:
- 10.18653/v1/2021.emnlp-main.344
- Cite (ACL):
- Avia Efrat, Uri Shaham, Dan Kilman, and Omer Levy. 2021. Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4186–4192, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language (Efrat et al., EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2021.emnlp-main.344.pdf
- Code
- aviaefrat/cryptonite