@inproceedings{alazraki-etal-2026-agentcoma,
    title = "{A}gent{C}o{M}a: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios",
    author = "Alazraki, Lisa  and
      Chen, Lihu  and
      Brassard, Ana  and
      Stacey, Joe  and
      Rahmani, Hossein A.  and
      Rei, Marek",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.380/",
    pages = "8383--8410",
    ISBN = "979-8-89176-390-6",
    abstract = "Large Language Models (LLMs) have achieved high accuracy on complex commonsense and mathematical problems that involve the composition of multiple reasoning steps. However, current compositional benchmarks testing these skills tend to focus on *either* commonsense or math reasoning, whereas LLM agents solving real-world tasks would require a combination of *both*. In this work, we introduce an **Agent**ic **Co**mmonsense and **Ma**th benchmark (AgentCoMa), where each compositional task requires a commonsense reasoning step *and* a math reasoning step. We test it on 61 LLMs of different sizes, model families, and training strategies. We find that LLMs can usually solve both steps in isolation, yet their accuracy drops by nearly 30{\%} on average when the two are combined. This is a substantially greater performance gap than the one we observe in prior compositional benchmarks that combine multiple steps of the same reasoning type. In contrast, non-expert human annotators can solve the compositional questions and the individual steps in AgentCoMa with similarly high accuracy. Furthermore, we conduct a series of interpretability studies to better understand the performance gap, examining neuron patterns, attention maps and membership inference. Our work underscores a substantial degree of model brittleness in the context of mixed-type compositional reasoning and offers a test bed for future improvement."
}