Abstract
Large language models achieve high performance on many but not all downstream tasks. The interaction between pretraining data and task data is commonly assumed to determine this variance: a task with data that is more similar to a model’s pretraining data is assumed to be easier for that model. We test whether distributional and example-specific similarity measures (embedding-, token- and model-based) correlate with language model performance through a large-scale comparison of the Pile and C4 pretraining datasets with downstream benchmarks. Similarity correlates with performance for multilingual datasets, but in other benchmarks, we surprisingly find that similarity metrics are not correlated with accuracy or even each other. This suggests that the relationship between pretraining data and downstream tasks is more complex than often assumed.- Anthology ID:
- 2023.emnlp-main.695
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 11295–11304
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-main.695
- DOI:
- 10.18653/v1/2023.emnlp-main.695
- Cite (ACL):
- Gregory Yauney, Emily Reif, and David Mimno. 2023. Data Similarity is Not Enough to Explain Language Model Performance. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11295–11304, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Data Similarity is Not Enough to Explain Language Model Performance (Yauney et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2023.emnlp-main.695.pdf