Multilingual Multimodal Text Detection in Indo-Aryan Languages

Nihar Basisth, Eisha Halder, Tushar Sachan, Advaitha Vetagiri, Partha Pakray

Abstract
Multi-language text detection and recognition in complex visual scenes is an essential yet challenging task. Traditional pipelines relying on optical character recognition (OCR) often fail to generalize across different languages, fonts, orientations and imaging conditions. This work proposes a novel approach using the YOLOv5 object detection model architecture for multilanguage text detection in images and videos. We curate and annotate a new dataset of over 4,000 scene text images across 4 Indian languages and use specialized data augmentation techniques to improve model robustness. Transfer learning from a base YOLOv5 model pretrained on COCO is combined with tailored optimization strategies for multi-language text detection. Our approach achieves state-of-theart performance, with over 90% accuracy on multi-language text detection across all four languages in our test set. We demonstrate the effectiveness of fine-tuning YOLOv5 for generalized multi-language text extraction across diverse fonts, scales, orientations, and visual contexts. Our approach’s high accuracy and generalizability could enable numerous applications involving multilingual text processing from imagery and video.
Anthology ID:
2023.icon-1.16
Volume:
Proceedings of the 20th International Conference on Natural Language Processing (ICON)
Month:
December
Year:
2023
Address:
Goa University, Goa, India
Editors:
Jyoti D. Pawar, Sobha Lalitha Devi
Venue:
ICON
SIG:
SIGLEX
Publisher:
NLP Association of India (NLPAI)
Note:
Pages:
162–171
Language:
URL:
https://aclanthology.org/2023.icon-1.16
DOI:
Bibkey:
Cite (ACL):
Nihar Basisth, Eisha Halder, Tushar Sachan, Advaitha Vetagiri, and Partha Pakray. 2023. Multilingual Multimodal Text Detection in Indo-Aryan Languages. In Proceedings of the 20th International Conference on Natural Language Processing (ICON), pages 162–171, Goa University, Goa, India. NLP Association of India (NLPAI).
Cite (Informal):
Multilingual Multimodal Text Detection in Indo-Aryan Languages (Basisth et al., ICON 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2023.icon-1.16.pdf
PDF Search