@inproceedings{chai-etal-2026-information,
    title = "An Information-Theoretic Study of {RLHF}-Induced Uniformity in Large Language Model Outputs",
    author = "Chai, Nolan  and
      Zhang, Tianqi  and
      Warstadt, Alex",
    editor = "Bonial, Claire  and
      Berzak, Yevgeni",
    booktitle = "Proceedings of the 30th Conference on Computational Natural Language Learning",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl-awards/2026.conll-main.43/",
    doi = "10.18653/v1/2026.conll-main.43",
    pages = "707--721",
    ISBN = "979-8-89176-410-1",
    abstract = "Reinforcement Learning with Human Feedback (RLHF) is a common post-training procedure to align the outputs of Large Language Models (LLMs) with human preferences. As a result, one might expect RLHF to induce some elements of human-like audience design into LLMs. However, RLHF and other post-training alignment methods have many complex effects on the outputs of LLMs that have yet to be studied quantitatively. We apply an information-theoretic lens to investigate the changes in the ``naturalness'' of language and the presence of audience design in LLMs before and after post-training. The \textit{Uniform Information Density (UID) Hypothesis} posits that humans optimize language production and comprehension across a noisy channel by transferring information at a more uniform rate. Accordingly, we analyze and compare how information is distributed within model- and human-generated text from different domains. We find that pretrained and post-trained LLMs both show superhuman uniformity across various text domains, and both RLHF and other post-training methods reduce uniformity slightly from their pretrained counterparts. However, RLHF uniquely encourage slower variance in uniformity between documents, potentially demonstrating that training on human preferences encourages consistency in information flow."
}