Crawling and Preprocessing Mailing Lists At Scale for Dialog Analysis
Janek Bevendorff, Khalid Al Khatib, Martin Potthast, Benno Stein
Abstract
This paper introduces the Webis Gmane Email Corpus 2019, the largest publicly available and fully preprocessed email corpus to date. We crawled more than 153 million emails from 14,699 mailing lists and segmented them into semantically consistent components using a new neural segmentation model. With 96% accuracy on 15 classes of email segments, our model achieves state-of-the-art performance while being more efficient to train than previous ones. All data, code, and trained models are made freely available alongside the paper.- Anthology ID:
- 2020.acl-main.108
- Volume:
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1151–1158
- Language:
- URL:
- https://aclanthology.org/2020.acl-main.108
- DOI:
- 10.18653/v1/2020.acl-main.108
- Cite (ACL):
- Janek Bevendorff, Khalid Al Khatib, Martin Potthast, and Benno Stein. 2020. Crawling and Preprocessing Mailing Lists At Scale for Dialog Analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1151–1158, Online. Association for Computational Linguistics.
- Cite (Informal):
- Crawling and Preprocessing Mailing Lists At Scale for Dialog Analysis (Bevendorff et al., ACL 2020)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2020.acl-main.108.pdf