@inproceedings{sarti-etal-2024-non,
    title = "Non Verbis, Sed Rebus: Large Language Models Are Weak Solvers of {I}talian Rebuses",
    author = "Sarti, Gabriele  and
      Caselli, Tommaso  and
      Nissim, Malvina  and
      Bisazza, Arianna",
    editor = "Dell'Orletta, Felice  and
      Lenci, Alessandro  and
      Montemagni, Simonetta  and
      Sprugnoli, Rachele",
    booktitle = "Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024)",
    month = dec,
    year = "2024",
    address = "Pisa, Italy",
    publisher = "CEUR Workshop Proceedings",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.clicit-1.96/",
    pages = "888--897",
    ISBN = "979-12-210-7060-6",
    abstract = "Rebuses are puzzles requiring constrained multi-step reasoning to identify a hidden phrase from a set of images and letters. In this work, we introduce a large collection of verbalized rebuses for the Italian language and use it to assess the rebus-solving capabilities of state-of-the-art large language models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly on this task, ad-hoc fine-tuning seems to improve models' performance. However, we find that performance gains from training are largely motivated by memorization. Our results suggest that rebus solving remains a challenging test bed to evaluate large language models' linguistic proficiency and sequential instruction-following skills."
}