Crawling and Preprocessing Mailing Lists At Scale for Dialog Analysis

Janek Bevendorff, Khalid Al Khatib, Martin Potthast, Benno Stein


Abstract
This paper introduces the Webis Gmane Email Corpus 2019, the largest publicly available and fully preprocessed email corpus to date. We crawled more than 153 million emails from 14,699 mailing lists and segmented them into semantically consistent components using a new neural segmentation model. With 96% accuracy on 15 classes of email segments, our model achieves state-of-the-art performance while being more efficient to train than previous ones. All data, code, and trained models are made freely available alongside the paper.
Anthology ID:
2020.acl-main.108
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1151–1158
Language:
URL:
https://aclanthology.org/2020.acl-main.108
DOI:
10.18653/v1/2020.acl-main.108
Bibkey:
Cite (ACL):
Janek Bevendorff, Khalid Al Khatib, Martin Potthast, and Benno Stein. 2020. Crawling and Preprocessing Mailing Lists At Scale for Dialog Analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1151–1158, Online. Association for Computational Linguistics.
Cite (Informal):
Crawling and Preprocessing Mailing Lists At Scale for Dialog Analysis (Bevendorff et al., ACL 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2020.acl-main.108.pdf
Video:
 http://slideslive.com/38928790