Naghmeh Farzi
2026
Sycophancy Negatively Affects LLM-as-a-Judge in Conflict Evaluation
Naghmeh Farzi | Laura Dietz | Samuel Carton
Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)
Naghmeh Farzi | Laura Dietz | Samuel Carton
Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)
LLM-as-Judge systems are increasingly used to generate labels and evaluate conversational data, yet their susceptibility to narrative framing remains underexplored. We study whether replacing one speaker’s username with the first-person identifier ’Me’ systematically biases model judgments independent of the underlying evidence. Using the Conversations Gone Awry corpus, we evaluate four LLMs across three judgment tasks (attack detection, attacker identification, and blame attribution), three perspective conditions, and two evidence visibility settings. Our results show that narrative perspective induces strong, task-dependent distortions, particularly in more subjective judgment tasks. We find that models systematically favor the narrator when a speaker is presented as ’Me’, reducing blame and responsibility attribution toward that speaker even when the underlying evidence is unchanged. These findings raise concerns about using LLMs to judge or moderate first-person conversational data.