Forum 4.0: An Open-Source User Comment Analysis Framework

Marlo Haering, Jakob Smedegaard Andersen, Chris Biemann, Wiebke Loosen, Benjamin Milde, Tim Pietz, Christian Stöcker, Gregor Wiedemann, Olaf Zukunft, Walid Maalej

Abstract
With the increasing number of user comments in diverse domains, including comments on online journalism and e-commerce websites, the manual content analysis of these comments becomes time-consuming and challenging. However, research showed that user comments contain useful information for different domain experts, which is thus worth finding and utilizing. This paper introduces Forum 4.0, an open-source framework to semi-automatically analyze, aggregate, and visualize user comments based on labels defined by domain experts. We demonstrate the applicability of Forum 4.0 with comments analytics scenarios within the domains of online journalism and app stores. We outline the underlying container architecture, including the web-based user interface, the machine learning component, and the task manager for time-consuming tasks. We finally conduct machine learning experiments with simulated annotations and different sampling strategies on existing datasets from both domains to evaluate Forum 4.0’s performance. Forum 4.0 achieves promising classification results (ROC-AUC ≥ 0.9 with 100 annotated samples), utilizing transformer-based embeddings with a lightweight logistic regression model. We explain how Forum 4.0’s architecture is applicable for millions of user comments in real-time, yet at feasible training and classification costs.
Anthology ID:
2021.eacl-demos.8
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations
Month:
April
Year:
2021
Address:
Online
Editors:
Dimitra Gkatzia, Djamé Seddah
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
63–70
Language:
URL:
https://aclanthology.org/2021.eacl-demos.8
DOI:
10.18653/v1/2021.eacl-demos.8
Bibkey:
Cite (ACL):
Marlo Haering, Jakob Smedegaard Andersen, Chris Biemann, Wiebke Loosen, Benjamin Milde, Tim Pietz, Christian Stöcker, Gregor Wiedemann, Olaf Zukunft, and Walid Maalej. 2021. Forum 4.0: An Open-Source User Comment Analysis Framework. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 63–70, Online. Association for Computational Linguistics.
Cite (Informal):
Forum 4.0: An Open-Source User Comment Analysis Framework (Haering et al., EACL 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl24-info/2021.eacl-demos.8.pdf
PDF Search