Moral reckoning: How reliable are dictionary-based methods for examining morality in text?

Ines Rehbein, Lilly Brauner, Florian Ertz, Ines Reinig, Simone Ponzetto


Abstract
Due to their availability and ease of use, dictionary-based measures of moral values are a popular tool for text-based analyses of morality that examine human attitudes and behaviour across populations and cultures. In this paper, we revisit the construct validity of different dictionary-based measures of morality in text that have been proposed in the literature. We discuss conceptual challenges for text-based measures of morality and present an annotation experiment where we create a new dataset with human annotations of moral rhetoric in German political manifestos. We compare the results of our human annotations with different measures of moral values, showing that none of them is able to capture the trends observed by trained human coders. Our findings have far-reaching implications for the application of moral dictionaries in the digital humanities.
Anthology ID:
2025.nlp4dh-1.20
Volume:
Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities
Month:
May
Year:
2025
Address:
Albuquerque, USA
Editors:
Mika Hämäläinen, Emily Öhman, Yuri Bizzoni, So Miyagawa, Khalid Alnajjar
Venues:
NLP4DH | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
232–250
Language:
URL:
https://preview.aclanthology.org/fix-sig-urls/2025.nlp4dh-1.20/
DOI:
Bibkey:
Cite (ACL):
Ines Rehbein, Lilly Brauner, Florian Ertz, Ines Reinig, and Simone Ponzetto. 2025. Moral reckoning: How reliable are dictionary-based methods for examining morality in text?. In Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities, pages 232–250, Albuquerque, USA. Association for Computational Linguistics.
Cite (Informal):
Moral reckoning: How reliable are dictionary-based methods for examining morality in text? (Rehbein et al., NLP4DH 2025)
Copy Citation:
PDF:
https://preview.aclanthology.org/fix-sig-urls/2025.nlp4dh-1.20.pdf