Abstract
Text simplification seeks to improve readability while retaining the original content and meaning. Our study investigates whether pre-trained classifiers also maintain such coherence by comparing their predictions on both original and simplified inputs. We conduct experiments using 11 pre-trained models, including BERT and OpenAI’s GPT 3.5, across six datasets spanning three languages. Additionally, we conduct a detailed analysis of the correlation between prediction change rates and simplification types/strengths. Our findings reveal alarming inconsistencies across all languages and models. If not promptly addressed, simplified inputs can be easily exploited to craft zero-iteration model-agnostic adversarial attacks with success rates of up to 50%.- Anthology ID:
- 2024.determit-1.17
- Volume:
- Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context @ LREC-COLING 2024
- Month:
- May
- Year:
- 2024
- Address:
- Torino, Italia
- Editors:
- Giorgio Maria Di Nunzio, Federica Vezzani, Liana Ermakova, Hosein Azarbonyad, Jaap Kamps
- Venues:
- DeTermIt | WS
- SIG:
- Publisher:
- ELRA and ICCL
- Note:
- Pages:
- 185–195
- Language:
- URL:
- https://aclanthology.org/2024.determit-1.17
- DOI:
- Cite (ACL):
- Miriam Anschütz, Edoardo Mosca, and Georg Groh. 2024. Simpler Becomes Harder: Do LLMs Exhibit a Coherent Behavior on Simplified Corpora?. In Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context @ LREC-COLING 2024, pages 185–195, Torino, Italia. ELRA and ICCL.
- Cite (Informal):
- Simpler Becomes Harder: Do LLMs Exhibit a Coherent Behavior on Simplified Corpora? (Anschütz et al., DeTermIt-WS 2024)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/2024.determit-1.17.pdf