Abstract
Research on automated program repairs using transformer-based models has recently gained considerable attention. The comprehension of the erroneous behavior of a model enables the identification of its inherent capacity and provides insights for improvement. However, the current landscape of research on program repair models lacks an investigation of their false behavior. Thus, we propose a methodology for diagnosing and treating the false behaviors of transformer-based program repair models. Specifically, we propose 1) a behavior vector that quantifies the behavior of the model when it generates an output, 2) a behavior discriminator (BeDisc) that identifies false behaviors, and 3) two methods for false behavior treatment. Through a large-scale experiment on 55,562 instances employing four datasets and three models, the BeDisc exhibited a balanced accuracy of 86.6% for false behavior classification. The first treatment, namely, early abortion, successfully eliminated 60.4% of false behavior while preserving 97.4% repair accuracy. Furthermore, the second treatment, namely, masked bypassing, resulted in an average improvement of 40.5% in the top-1 repair accuracy. These experimental results demonstrated the importance of investigating false behaviors in program repair models.- Anthology ID:
- 2023.emnlp-main.865
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 14010–14023
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2023.emnlp-main.865/
- DOI:
- 10.18653/v1/2023.emnlp-main.865
- Cite (ACL):
- Youngkyoung Kim, Misoo Kim, and Eunseok Lee. 2023. Improving Transformer-based Program Repair Model through False Behavior Diagnosis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14010–14023, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Improving Transformer-based Program Repair Model through False Behavior Diagnosis (Kim et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2023.emnlp-main.865.pdf