@inproceedings{balepur-etal-2025-reverse,
    title = "Reverse Question Answering: Can an {LLM} Write a Question so Hard (or Bad) that it Can`t Answer?",
    author = "Balepur, Nishant  and
      Gu, Feng  and
      Ravichander, Abhilasha  and
      Feng, Shi  and
      Boyd-Graber, Jordan Lee  and
      Rudinger, Rachel",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Ingest-2025-COMPUTEL/2025.naacl-short.5/",
    pages = "44--64",
    ISBN = "979-8-89176-190-2",
    abstract = "Question answering (QA){---}giving correct answers to questions{---}is a popular task, but we test **reverse question answering (RQA)**: for an input answer, give a question with that answer. Past work tests QA and RQA separately, but we test them jointly, comparing their difficulty, aiding benchmark design, and checking reasoning consistency. We run 16 LLMs on QA and RQA with trivia questions/answers, revealing: 1) Versus RQA, LLMs are much less accurate in RQA for numerical answers, but slightly more accurate in RQA for textual answers; 2) LLMs often answer their own invalid questions from RQA accurately in QA, so RQA errors are not just from knowledge gaps; 3) RQA errors correlate with question difficulty and inversely correlate with answer frequencies in the Dolma corpus; and 4) LLMs struggle to give valid multi-hop questions. By finding question and answer types that lead to RQA errors, we suggest improvements for LLM reasoning."
}