Confounders in Instance Variation for the Analysis of Data Contamination
Behzad Mehrbakhsh, Dario Garigliotti, Fernando Martínez-Plumed, Jose Hernandez-Orallo
Abstract
Test contamination is a serious problem for the evaluation of large language models (LLMs) because it leads to the overestimation of their performance and a quick saturation of benchmarks, even before the actual capability is achieved. One strategy to address this issue is the (adversarial) generation of variations, by including different exemplars and different rephrasings of the questions. However, these two interventions can lead to instances that can be more difficult (accumulating on the expected loss of performance by partly removing the contamination) but also to instances that can be less difficult (cancelling the expected loss of performance), which would make contamination undetectable. Understanding these two phenomena in terms of instance difficulty is critical to determine and measure contamination. In this paper we conduct a comprehensive analysis of these two interventions on an addition task with fine-tuned LLAMA-2 models.- Anthology ID:
- 2024.conda-1.2
- Volume:
- Proceedings of the 1st Workshop on Data Contamination (CONDA)
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Oscar Sainz, Iker García Ferrero, Eneko Agirre, Jon Ander Campos, Alon Jacovi, Yanai Elazar, Yoav Goldberg
- Venues:
- CONDA | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 13–21
- Language:
- URL:
- https://aclanthology.org/2024.conda-1.2
- DOI:
- Cite (ACL):
- Behzad Mehrbakhsh, Dario Garigliotti, Fernando Martínez-Plumed, and Jose Hernandez-Orallo. 2024. Confounders in Instance Variation for the Analysis of Data Contamination. In Proceedings of the 1st Workshop on Data Contamination (CONDA), pages 13–21, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Confounders in Instance Variation for the Analysis of Data Contamination (Mehrbakhsh et al., CONDA-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.conda-1.2.pdf