From Syntax to Semantics: Evaluating the Impact of Linguistic Structures on LLM-Based Information Extraction
Anushka Swarup, Avanti Bhandarkar, Ronald Wilson, Tianyu Pan, Damon Woodard
Abstract
Large Language Models (LLMs) have brought significant breakthroughs across all areas of Natural Language Processing (NLP), including Information Extraction (IE). However, knowledge gaps remain regarding their effectiveness in extracting entity-relation triplets, i.e. Joint Relation Extraction (JRE). JRE has been a key operation in creating knowledge bases that can be used to enhance Retrieval Augmented Generation (RAG) systems. Prior work highlights low-quality triplets generated by LLMs. Thus, this work investigates the impact of incorporating linguistic structures, such as constituency and dependency trees and semantic role labeling, to enhance the quality of the extracted triplets. The findings suggest that incorporating specific structural information enhances the uniqueness and topical relevance of the triplets, particularly in scenarios where multiple relationships are present.- Anthology ID:
- 2025.xllm-1.5
- Volume:
- Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025)
- Month:
- August
- Year:
- 2025
- Address:
- Vienna, Austria
- Editors:
- Hao Fei, Kewei Tu, Yuhui Zhang, Xiang Hu, Wenjuan Han, Zixia Jia, Zilong Zheng, Yixin Cao, Meishan Zhang, Wei Lu, N. Siddharth, Lilja Øvrelid, Nianwen Xue, Yue Zhang
- Venues:
- XLLM | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 36–48
- Language:
- URL:
- https://preview.aclanthology.org/landing_page/2025.xllm-1.5/
- DOI:
- Cite (ACL):
- Anushka Swarup, Avanti Bhandarkar, Ronald Wilson, Tianyu Pan, and Damon Woodard. 2025. From Syntax to Semantics: Evaluating the Impact of Linguistic Structures on LLM-Based Information Extraction. In Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025), pages 36–48, Vienna, Austria. Association for Computational Linguistics.
- Cite (Informal):
- From Syntax to Semantics: Evaluating the Impact of Linguistic Structures on LLM-Based Information Extraction (Swarup et al., XLLM 2025)
- PDF:
- https://preview.aclanthology.org/landing_page/2025.xllm-1.5.pdf