Comparing the Evaluation and Production of Loophole Behavior in Humans and Large Language Models

Sonia Murthy, Kiera Parece, Sophie Bridgers, Peng Qian, Tomer Ullman

Abstract
In law, lore, and everyday life, loopholes are commonplace. When people exploit a loophole, they understand the intended meaning or goal of another person, but choose to go with a different interpretation. Past and current AI research has shown that artificial intelligence engages in what seems superficially like the exploitation of loopholes, but this is likely anthropomorphization. It remains unclear to what extent current models, especially Large Language Models (LLMs), capture the pragmatic understanding required for engaging in loopholes. We examined the performance of LLMs on two metrics developed for studying loophole behavior in humans: evaluation (ratings of trouble, upset, and humor), and generation (coming up with new loopholes in a given context). We conducted a fine-grained comparison of state-of-the-art LLMs to humans, and find that while many of the models rate loophole behaviors as resulting in less trouble and upset than outright non-compliance (in line with adults), they struggle to recognize the humor in the creative exploitation of loopholes in the way that humans do. Furthermore, only two of the models, GPT 3 and 3.5, are capable of generating loopholes of their own, with GPT3.5 performing closest to the human baseline.
Anthology ID:
2023.findings-emnlp.264
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4010–4025
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.264
DOI:
10.18653/v1/2023.findings-emnlp.264
Bibkey:
Cite (ACL):
Sonia Murthy, Kiera Parece, Sophie Bridgers, Peng Qian, and Tomer Ullman. 2023. Comparing the Evaluation and Production of Loophole Behavior in Humans and Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4010–4025, Singapore. Association for Computational Linguistics.
Cite (Informal):
Comparing the Evaluation and Production of Loophole Behavior in Humans and Large Language Models (Murthy et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-5/2023.findings-emnlp.264.pdf
PDF Search