Supervised and Unsupervised Methods for Robust Separation of Section Titles and Prose Text in Web Documents

Abhijith Athreya Mysore Gopinath, Shomir Wilson, Norman Sadeh

Abstract
The text in many web documents is organized into a hierarchy of section titles and corresponding prose content, a structure which provides potentially exploitable information on discourse structure and topicality. However, this organization is generally discarded during text collection, and collecting it is not straightforward: the same visual organization can be implemented in a myriad of different ways in the underlying HTML. To remedy this, we present a flexible system for automatically extracting the hierarchical section titles and prose organization of web documents irrespective of differences in HTML representation. This system uses features from syntax, semantics, discourse and markup to build two models which classify HTML text into section titles and prose text. When tested on three different domains of web text, our domain-independent system achieves an overall precision of 0.82 and a recall of 0.98. The domain-dependent variation produces very high precision (0.99) at the expense of recall (0.75). These results exhibit a robust level of accuracy suitable for enhancing question answering, information extraction, and summarization.
Anthology ID:
D18-1099
Volume:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Month:
October-November
Year:
2018
Address:
Brussels, Belgium
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
850–855
Language:
URL:
https://aclanthology.org/D18-1099
DOI:
10.18653/v1/D18-1099
Bibkey:
Cite (ACL):
Abhijith Athreya Mysore Gopinath, Shomir Wilson, and Norman Sadeh. 2018. Supervised and Unsupervised Methods for Robust Separation of Section Titles and Prose Text in Web Documents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 850–855, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
Supervised and Unsupervised Methods for Robust Separation of Section Titles and Prose Text in Web Documents (Mysore Gopinath et al., EMNLP 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/paclic-22-ingestion/D18-1099.pdf
PDF Search