Block Diagram-to-Text: Understanding Block Diagram Images by Generating Natural Language Descriptors
Abstract
Block diagrams are very popular for representing a workflow or process of a model. Understanding block diagrams by generating summaries can be extremely useful in document summarization. It can also assist people in inferring key insights from block diagrams without requiring a lot of perceptual and cognitive effort. In this paper, we propose a novel task of converting block diagram images into text by presenting a framework called “BloSum”. This framework extracts the contextual meaning from the images in the form of triplets that help the language model in summary generation. We also introduce a new dataset for complex computerized block diagrams, explain the dataset preparation process, and later analyze it. Additionally, to showcase the generalization of the model, we test our method with publicly available handwritten block diagram datasets. Our evaluation with different metrics demonstrates the effectiveness of our approach that outperforms other methods and techniques.- Anthology ID:
- 2022.findings-aacl.15
- Volume:
- Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022
- Month:
- November
- Year:
- 2022
- Address:
- Online only
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 153–168
- Language:
- URL:
- https://aclanthology.org/2022.findings-aacl.15
- DOI:
- Cite (ACL):
- Shreyanshu Bhushan and Minho Lee. 2022. Block Diagram-to-Text: Understanding Block Diagram Images by Generating Natural Language Descriptors. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 153–168, Online only. Association for Computational Linguistics.
- Cite (Informal):
- Block Diagram-to-Text: Understanding Block Diagram Images by Generating Natural Language Descriptors (Bhushan & Lee, Findings 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.findings-aacl.15.pdf