Abstract
Recent advances in prompt engineering enable large language models (LLMs) to solve multi-hop logical reasoning problems with impressive accuracy. However, there is little existing work investigating the robustness of LLMs with few-shot prompting techniques. Therefore, we introduce a systematic approach to test the robustness of LLMs in multi-hop reasoning tasks via domain-agnostic perturbations. We include perturbations at multiple levels of abstractions (e.g. lexical perturbations such as typos, and semantic perturbations such as the inclusion of intermediate reasoning steps in the questions) to conduct behavioral analysis on the LLMs. Throughout our experiments, we find that models are more sensitive to certain perturbations such as replacing words with their synonyms. We also demonstrate that increasing the proportion of perturbed exemplars in the prompts improves the robustness of few-shot prompting methods.- Anthology ID:
- 2023.emnlp-main.277
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4560–4568
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-main.277
- DOI:
- 10.18653/v1/2023.emnlp-main.277
- Cite (ACL):
- Hongyi Zheng and Abulhair Saparov. 2023. Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4560–4568, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis (Zheng & Saparov, EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/corrections-2024-05/2023.emnlp-main.277.pdf