Accurately predicting human scanpaths duringreading is vital for diverse fields and downstream tasks, from educational technologies toautomatic question answering. To date, however, progress in this direction remains limited by scarce gaze data. We overcome theissue with ScanEZ, a self-supervised framework grounded in cognitive models of reading.ScanEZ jointly models the spatial and temporal dimensions of scanpaths by leveraging synthetic data and a 3-D gaze objective inspired bymasked language modeling. With this framework, we provide evidence that two key factorsin scanpath prediction during reading are: theuse of masked modeling of both spatial andtemporal patterns of eye movements, and cognitive model simulations as an inductive biasto kick-start training. Our approach achievesstate-of-the-art results on established datasets(e.g., up to 31.4% negative log-likelihood improvement on CELER L1), and proves portableacross different experimental conditions.