@inproceedings{hudspeth-etal-2026-respondeoqa,
    title = "{R}espondeo{QA}: A Benchmark for Bilingual {L}atin-{E}nglish Question Answering",
    author = "Hudspeth, Marisa  and
      Burns, Patrick J.  and
      O{'}Connor, Brendan",
    editor = "Piperidis, Stelios  and
      Bel, N{\'u}ria  and
      van den Heuvel, Henk  and
      Ide, Nancy  and
      Krek, Simon  and
      Toral, Antonio",
    booktitle = "Proceedings of the Fifteenth Language Resources and Evaluation Conference",
    month = may,
    year = "2026",
    address = "Palma de Mallorca, Spain",
    publisher = "ELRA Language Resource Association",
    url = "https://preview.aclanthology.org/ingest-nlpsi/2026.lrec-1.80/",
    doi = "10.63317/58p5htfv3nad",
    pages = "1031--1043",
    abstract = "We introduce a benchmark dataset for question answering and translation in bilingual Latin and English settings, containing about 7,800 question{--}answer pairs. The questions are drawn from Latin pedagogical sources, including exams, quizbowl-style trivia, and textbooks ranging from the 1800s to the present. After automated extraction, cleaning, and manual review, the dataset covers a diverse range of question types: knowledge- and skill-based, multihop reasoning, constrained translation, and mixed language pairs. To our knowledge, this is the first QA benchmark centered on Latin. As a case study, we evaluate three large language models{--}LLaMa 3, Qwen QwQ, and OpenAI{'}s o3-mini{--}finding that all perform worse on skill-oriented questions. Although the reasoning models perform better on scansion and literary-device tasks, they offer limited improvement overall. QwQ performs slightly better on questions asked in Latin, but LLaMa3 and o3-mini are more task dependent. This dataset provides a new resource for assessing model capabilities in a specialized linguistic and cultural domain, and the creation process can be easily adapted for other languages. The dataset is available at: \url{https://github.com/slanglab/RespondeoQA}"
}