Reinforcement Learning with Human Feedback (RLHF) is a common post-training procedure to align the outputs of Large Language Models (LLMs) with human preferences. As a result, one might expect RLHF to induce some elements of human-like audience design into LLMs. However, RLHF and other post-training alignment methods have many complex effects on the outputs of LLMs that have yet to be studied quantitatively. We apply an information-theoretic lens to investigate the changes in the "naturalness" of language and the presence of audience design in LLMs before and after post-training. The Uniform Information Density (UID) Hypothesis posits that humans optimize language production and comprehension across a noisy channel by transferring information at a more uniform rate. Accordingly, we analyze and compare how information is distributed within model- and human-generated text from different domains. We find that pretrained and post-trained LLMs both show superhuman uniformity across various text domains, and both RLHF and other post-training methods reduce uniformity slightly from their pretrained counterparts. However, RLHF uniquely encourage slower variance in uniformity between documents, potentially demonstrating that training on human preferences encourages consistency in information flow.

Large language models (LLMs) encapsulate vast amounts of knowledge but still remain vulnerable to external misinformation. Existing research mainly studied this susceptibility behavior in a single-turn setting. However, belief can change during a multi-turn conversation, especially a persuasive one. Therefore, in this study, we delve into LLMs’ susceptibility to persuasive conversations, particularly on factual questions that they can answer correctly. We first curate the Farm (i.e., Fact to Misinform) dataset, which contains factual questions paired with systematically generated persuasive misinformation. Then, we develop a testing framework to track LLMs’ belief changes in a persuasive dialogue. Through extensive experiments, we find that LLMs’ correct beliefs on factual knowledge can be easily manipulated by various persuasive strategies.

We introduce VL-CheckList, a toolbox for evaluating Vision-Language Pretraining (VLP) models, including the preliminary datasets that deepen the image-texting ability of a VLP model. Most existing VLP works evaluated their systems by comparing the fine-tuned downstream task performance. However, only average downstream task accuracy provides little information about the pros and cons of each VLP method. In this paper, we demonstrate how minor input changes in language and vision will affect the prediction outputs. Then, we describe the detailed user guidelines to utilize and contribute to the community. We show new findings on one of the representative VLP models to provide an example analysis. The data/code is available at https://github.com/om-ai-lab/VL-CheckList