A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models
Krithika Ramesh, Arnav Chavan, Shrey Pandit, Sunayana Sitaram
Abstract
Compression techniques for deep learning have become increasingly popular, particularly in settings where latency and memory constraints are imposed. Several methods, such as pruning, distillation, and quantization, have been adopted for compressing models, each providing distinct advantages. However, existing literature demonstrates that compressing deep learning models could affect their fairness. Our analysis involves a comprehensive evaluation of pruned, distilled, and quantized language models, which we benchmark across a range of intrinsic and extrinsic metrics for measuring bias in text classification. We also investigate the impact of using multilingual models and evaluation measures. Our findings highlight the significance of considering both the pre-trained model and the chosen compression strategy in developing equitable language technologies. The results also indicate that compression strategies can have an adverse effect on fairness measures.- Anthology ID:
- 2023.acl-long.878
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 15762–15782
- Language:
- URL:
- https://aclanthology.org/2023.acl-long.878
- DOI:
- 10.18653/v1/2023.acl-long.878
- Cite (ACL):
- Krithika Ramesh, Arnav Chavan, Shrey Pandit, and Sunayana Sitaram. 2023. A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15762–15782, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models (Ramesh et al., ACL 2023)
- PDF:
- https://preview.aclanthology.org/emnlp22-frontmatter/2023.acl-long.878.pdf