@inproceedings{xu-etal-2025-tow,
    title = "{T}o{W}: Thoughts of Words Improve Reasoning in Large Language Models",
    author = "Xu, Zhikun  and
      Shen, Ming  and
      Dineen, Jacob  and
      Li, Zhaonan  and
      Ye, Xiao  and
      Lu, Shijie  and
      Rrv, Aswin  and
      Baral, Chitta  and
      Zhou, Ben",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.157/",
    pages = "3057--3075",
    ISBN = "979-8-89176-189-6",
    abstract = "We introduce thoughts of words (ToW), a novel training-time data-augmentation method for next-word prediction. ToW views next-word prediction as a core reasoning task and injects fine-grained thoughts explaining what the next word should be and how it is related to the previous contexts in pre-training texts. Our formulation addresses two fundamental drawbacks of existing next-word prediction learning schemes: they induce factual hallucination and are inefficient for models to learn the implicit reasoning processes in raw texts. While there are many ways to acquire such thoughts of words, we explore the first step of acquiring ToW annotations through distilling from larger models. After continual pre-training with only 70K ToW annotations, we effectively improve models' reasoning performances by 7{\%} to 9{\%} on average and reduce model hallucination by up to 10{\%}. At the same time, ToW is entirely agnostic to tasks and applications, introducing no additional biases on labels or semantics."
}