Jocelyn D’Arcy
2026
Too long; didn’t solve
Lucía Cabrera | Jocelyn D’Arcy | Isaac Saxton-Knight
Proceedings of the Workshop on Evaluating Evaluations (EvalEval)
Lucía Cabrera | Jocelyn D’Arcy | Isaac Saxton-Knight
Proceedings of the Workshop on Evaluating Evaluations (EvalEval)
Mathematical benchmarks consisting of a range of mathematics problems are widely used to evaluate the reasoning abilities of large language models, yet little is known about how their structural properties influence model behaviour. In this work, we investigate two structural length variables, prompt length and solution length, and analyse how they relate to model performance on a newly constructed adversarial dataset of expert-authored mathematics problems. Across five evaluated models, we find that both prompt length and solution length are positively associated with model failure. These associations are statistically significant but modest, and we interpret them as descriptive rather than causal. We also include a secondary, exploratory analysis of cross-model disagreement. Because disagreement measures based on variance are mechanically constrained by mean failure, we treat this part of the analysis cautiously. Overall, our main finding is that structural length is linked to empirical difficulty in this benchmark, suggesting that length should be considered as a potential confounder when interpreting mathematical model evaluations.