Takumi Kondo
2025
Counterfactual Evaluation for Blind Attack Detection in LLM-based Evaluation Systems
Lijia Liu
|
Takumi Kondo
|
Kyohei Atarashi
|
Koh Takeuchi
|
Jiyi Li
|
Shigeru Saito
|
Hisashi Kashima
Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics
This paper investigates defenses in LLM-based evaluation, where prompt injection attacks can manipulate scores by deceiving the evaluation system. We formalize blind attacks as a class in which candidate answers are crafted independently of the true answer. To counter such attacks, we propose an evaluation framework that combines standard and counterfactual evaluation. Experiments show it significantly improves attack detection with minimal performance trade-offs for recent models.
Search
Fix author
Co-authors
- Kyohei Atarashi 1
- Hisashi Kashima 1
- Jiyi Li 1
- Lijia Liu 1
- Shigeru Saito 1
- show all...