Abstract
Handwritten mathematical expression recognition (HMER) is a multidisciplinary task that generates LaTeX sequences from images. Existing approaches, employing tree decoders within attention-based encoder-decoder architectures, aim to capture the hierarchical tree structure, but are limited by CFGs and pre-generated triplet data, hindering expandability and neglecting visual ambiguity challenges. This article investigates the distinctive language characteristics of LaTeX mathematical expressions, revealing two key observations: 1) the presence of explicit structural symbols, and 2) the treatment of symbols, particularly letters, as minimal units with context-dependent semantics, representing variables or constants. Rooted in these properties, we propose that language models have the potential to synchronously and complementarily provide both structural and semantic information, making them suitable for correction of HMER. To validate our proposition, we propose an architecture called Recognize and Language Fusion Network (RLFN), which integrates recognition and language features to output corrected sequences while jointly optimizing with a string decoder recognition model. Experiments show that RLFN outperforms existing state-of-the-art methods on the CROHME 2014/2016/2019 datasets.- Anthology ID:
- 2023.emnlp-main.247
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4057–4068
- Language:
- URL:
- https://preview.aclanthology.org/remove-affiliations/2023.emnlp-main.247/
- DOI:
- 10.18653/v1/2023.emnlp-main.247
- Cite (ACL):
- Zui Chen, Jiaqi Han, Chaofan Yang, and Yi Zhou. 2023. Language Model is Suitable for Correction of Handwritten Mathematical Expressions Recognition. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4057–4068, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Language Model is Suitable for Correction of Handwritten Mathematical Expressions Recognition (Chen et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/remove-affiliations/2023.emnlp-main.247.pdf