Abstract
Authorship attribution is the task of assigning an unknown document to an author from a set of candidates. In the past, studies in this field use various evaluation datasets to demonstrate the effectiveness of preprocessing steps, features, and models. However, only a small fraction of works use more than one dataset to prove claims. In this paper, we present a collection of highly diverse authorship attribution datasets, which better generalizes evaluation results from authorship attribution research. Furthermore, we implement a wide variety of previously used machine learning models and show that many approaches show vastly different performances when applied to different datasets. We include pre-trained language models, for the first time testing them in this field in a systematic way. Finally, we propose a set of aggregated scores to evaluate different aspects of the dataset collection.- Anthology ID:
- 2021.eval4nlp-1.18
- Volume:
- Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems
- Month:
- November
- Year:
- 2021
- Address:
- Punta Cana, Dominican Republic
- Venue:
- Eval4NLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 179–188
- Language:
- URL:
- https://aclanthology.org/2021.eval4nlp-1.18
- DOI:
- 10.18653/v1/2021.eval4nlp-1.18
- Cite (ACL):
- Benjamin Murauer and Günther Specht. 2021. Developing a Benchmark for Reducing Data Bias in Authorship Attribution. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 179–188, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Developing a Benchmark for Reducing Data Bias in Authorship Attribution (Murauer & Specht, Eval4NLP 2021)
- PDF:
- https://preview.aclanthology.org/remove-xml-comments/2021.eval4nlp-1.18.pdf