Bias Analysis and Mitigation in the Evaluation of Authorship Verification
Janek Bevendorff, Matthias Hagen, Benno Stein, Martin Potthast
Abstract
The PAN series of shared tasks is well known for its continuous and high quality research in the field of digital text forensics. Among others, PAN contributions include original corpora, tailored benchmarks, and standardized experimentation platforms. In this paper we review, theoretically and practically, the authorship verification task and conclude that the underlying experiment design cannot guarantee pushing forward the state of the art—in fact, it allows for top benchmarking with a surprisingly straightforward approach. In this regard, we present a “Basic and Fairly Flawed” (BAFF) authorship verifier that is on a par with the best approaches submitted so far, and that illustrates sources of bias that should be eliminated. We pinpoint these sources in the evaluation chain and present a refined authorship corpus as effective countermeasure.- Anthology ID:
- P19-1634
- Volume:
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2019
- Address:
- Florence, Italy
- Editors:
- Anna Korhonen, David Traum, Lluís Màrquez
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6301–6306
- Language:
- URL:
- https://aclanthology.org/P19-1634
- DOI:
- 10.18653/v1/P19-1634
- Cite (ACL):
- Janek Bevendorff, Matthias Hagen, Benno Stein, and Martin Potthast. 2019. Bias Analysis and Mitigation in the Evaluation of Authorship Verification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6301–6306, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- Bias Analysis and Mitigation in the Evaluation of Authorship Verification (Bevendorff et al., ACL 2019)
- PDF:
- https://preview.aclanthology.org/emnlp22-frontmatter/P19-1634.pdf
- Code
- webis-de/acl-19