@inproceedings{chai-etal-2026-information,
    title = "An Information-Theoretic Study of {RLHF}-Induced Uniformity in Large Language Model Outputs",
    author = "Chai, Nolan  and
      Zhang, Tianqi  and
      Warstadt, Alex",
    editor = "Bonial, Claire  and
      Berzak, Yevgeni",
    booktitle = "Proceedings of the 30th Conference on Computational Natural Language Learning",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.conll-main.43/",
    pages = "707--721",
    ISBN = "979-8-89176-410-1",
    abstract = "Reinforcement Learning with Human Feedback(RLHF) is a common post-training procedureto align the outputs of Large Language Mod-els (LLMs) with human preferences. As a re-sult, one might expect RLHF to induce someelements of human-like audience design intoLLMs. However, RLHF and other post-trainingalignment methods have many complex effectson the outputs of LLMs that have yet to be stud-ied quantitatively. We apply an information-theoretic lens to investigate the changes in the{''}naturalness'' of language and the presence ofaudience design in LLMs before and after post-training. The Uniform Information Density(UID) Hypothesis posits that humans optimizelanguage production and comprehension acrossa noisy channel by transferring information ata more uniform rate. Accordingly, we analyzeand compare how information is distributedwithin model- and human-generated text fromdifferent domains. We find that pretrained andpost-trained LLMs both show superhuman uni-formity across various text domains, and bothRLHF and other post-training methods reduceuniformity slightly from their pretrained coun-terparts. However, RLHF uniquely encourageslower variance in uniformity between docu-ments, potentially demonstrating that trainingon human preferences encourages consistencyin information flow."
}