Calibration as a Proxy for Fairness and Efficiency in a Perspectivist Ensemble Approach to Irony Detection
Samuel B. Jesus, Guilherme Dal Bianco, Wanderlei Junior, Valerio Basile, Marcos André Gonçalves
Abstract
Identifying subjective phenomena, such as irony in language, poses unique challenges, as these tasks involve subjective interpretation shaped by both cultural and individual perspectives. Unlike conventional models that rely on aggregated annotations, perspectivist approaches aim to capture the diversity of viewpoints by leveraging the knowledge of specific annotator groups, promoting fairness and representativeness. However, such models often incur substantial computational costs, particularly when fine-tuning large-scale pre-trained language models. We also observe that the fine-tuning process can negatively impact fairness, producing certain perspective models that are underrepresented and have limited influence on the outcome. To address these, we explore two complementary strategies: (i) the adoption of traditional machine learning algorithms—such as Support Vector Machines, Random Forests, and XGBoost—as lightweight alternatives; and (ii) the application of calibration techniques to reduce imbalances in inference generation across perspectives. Our results demonstrate up to 12× faster processing with no statistically significant drop in accuracy. Notably, calibration significantly enhances fairness, reducing inter-group bias and leading to more balanced predictions across diverse social perspectives.- Anthology ID:
- 2025.nlperspectives-1.12
- Volume:
- Proceedings of the The 4th Workshop on Perspectivist Approaches to NLP
- Month:
- November
- Year:
- 2025
- Address:
- Suzhou, China
- Editors:
- Gavin Abercrombie, Valerio Basile, Simona Frenda, Sara Tonelli, Shiran Dudy
- Venues:
- NLPerspectives | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 133–141
- Language:
- URL:
- https://preview.aclanthology.org/ingest-emnlp/2025.nlperspectives-1.12/
- DOI:
- Cite (ACL):
- Samuel B. Jesus, Guilherme Dal Bianco, Wanderlei Junior, Valerio Basile, and Marcos André Gonçalves. 2025. Calibration as a Proxy for Fairness and Efficiency in a Perspectivist Ensemble Approach to Irony Detection. In Proceedings of the The 4th Workshop on Perspectivist Approaches to NLP, pages 133–141, Suzhou, China. Association for Computational Linguistics.
- Cite (Informal):
- Calibration as a Proxy for Fairness and Efficiency in a Perspectivist Ensemble Approach to Irony Detection (Jesus et al., NLPerspectives 2025)
- PDF:
- https://preview.aclanthology.org/ingest-emnlp/2025.nlperspectives-1.12.pdf