@inproceedings{steindl-etal-2025-effectiveness,
    title = "On the Effectiveness of Prompt-Moderated {LLM}s for Math Tutoring at the Tertiary Level",
    author = {Steindl, Sebastian  and
      Brunner, Fabian  and
      Sissouno, Nada  and
      Schwagerl, Dominik  and
      Sch{\"o}ler-Niewiera, Florian  and
      Sch{\"a}fer, Ulrich},
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.605/",
    doi = "10.18653/v1/2025.findings-emnlp.605",
    pages = "11310--11323",
    ISBN = "979-8-89176-335-7",
    abstract = "Large Language Models (LLMs) have been studied intensively in the context of education, yielding heterogeneous results. Nowadays, these models are also deployed in formal education institutes. While specialized models exist, using prompt-moderated LLMs is widespread. In this study, we therefore investigate the effectiveness of prompt-moderated LLMs for math tutoring at a tertiary-level. We conduct a three-phase study with students (N=49) first receiving a review of the topics, then solving exercises, and finally writing an exam. During the exercises, they are presented with different types of assistance. We analyze the effect of LLM usage on the students' performance, their engagement with the LLM, and their conversation strategies. Our results show that the prompt-moderation had a negative influence when compared to an unmoderated LLM. However, when the assistance was removed again, both LLM groups performed better than the control group, contradicting concerns about shallow learning. We publish the annotated conversations as a dataset to foster future research."
}