RvH-40 at SemEval-2026 Task 11: Disentangling Reasoning from Belief through Symbolic Abstraction

Niek Biesterbos, Mark Den Ouden, Janiek De Rijke


Abstract
Large Language Models (LLMs) often struggle with syllogistic reasoning due to "belief bias," where semantic world knowledge overrides formal logical structure. In this paper, we present our submission for the SemEval-2026 Task 11 shared task. We investigate the discrepancy between a model’s latent logical capabilities and its performance on natural language text. By employing symbolic transformations, specifically variable and pseudoword substitution, we demonstrate that models like Qwen2.5-14B possess strong inherent reasoning skills that are suppressed by linguistic content. We propose a "logic alignment" strategy using Low-Rank Adaptation (LoRA) to bridge this gap. Our final model achieved a near-perfect accuracy of 97.92% on the validation set and 96.34% on the official hidden test set, effectively eliminating content bias while maintaining robust generalization across abstract formats.
Anthology ID:
2026.semeval-1.65
Volume:
Proceedings of the 20th International Workshop on Semantic Evaluation (2026)
Month:
July
Year:
2026
Address:
San Diego, California, USA
Editors:
Ekaterina Kochmar, Debanjan Ghosh, Kai North, Mamoru Komachi
Venues:
SemEval | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
451–456
Language:
URL:
https://preview.aclanthology.org/ingest-acl-workshops/2026.semeval-1.65/
DOI:
Bibkey:
Cite (ACL):
Niek Biesterbos, Mark Den Ouden, and Janiek De Rijke. 2026. RvH-40 at SemEval-2026 Task 11: Disentangling Reasoning from Belief through Symbolic Abstraction. In Proceedings of the 20th International Workshop on Semantic Evaluation (2026), pages 451–456, San Diego, California, USA. Association for Computational Linguistics.
Cite (Informal):
RvH-40 at SemEval-2026 Task 11: Disentangling Reasoning from Belief through Symbolic Abstraction (Biesterbos et al., SemEval 2026)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-workshops/2026.semeval-1.65.pdf