Thiyaga6851 at SemEval-2026 Task 11: Disentangling Content and Formal Reasoning in Large Language Models using Neuro-Symbolic Mapping

Thiyagarajaa Pk, Thenmozhi D.


Abstract
This paper presents our system for SemEval-2026 Task 11 Subtask 1, which evaluates the formal validity of English syllogisms independently of semantic plausibility. To reduce content effects, we use a hybrid neuro-symbolic pipeline that separates natural-language abstraction from logical inference. The system maps each syllogism into categorical propositions using template rules and a learned parser, followed by explicit role mapping for the major, minor, and middle terms. If the abstraction is structurally complete, an exact Venn-style satisfiability solver checks validity; otherwise, the instance is routed to a learned fallback classifier. Our official submission achieved 71.73% accuracy, a Total Content Effect of 11.84, a Combined Score of 20.19, and a rank of 41st. Development analysis shows that symbolic inference is reliable on well-formed abstractions, while most remaining errors arise from paraphrase, multiword terms, and unstable term alignment.
Anthology ID:
2026.semeval-1.276
Volume:
Proceedings of the 20th International Workshop on Semantic Evaluation (2026)
Month:
July
Year:
2026
Address:
San Diego, California, USA
Editors:
Ekaterina Kochmar, Debanjan Ghosh, Kai North, Mamoru Komachi
Venues:
SemEval | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2187–2192
Language:
URL:
https://preview.aclanthology.org/ingest-acl-workshops/2026.semeval-1.276/
DOI:
Bibkey:
Cite (ACL):
Thiyagarajaa Pk and Thenmozhi D.. 2026. Thiyaga6851 at SemEval-2026 Task 11: Disentangling Content and Formal Reasoning in Large Language Models using Neuro-Symbolic Mapping. In Proceedings of the 20th International Workshop on Semantic Evaluation (2026), pages 2187–2192, San Diego, California, USA. Association for Computational Linguistics.
Cite (Informal):
Thiyaga6851 at SemEval-2026 Task 11: Disentangling Content and Formal Reasoning in Large Language Models using Neuro-Symbolic Mapping (Pk & D., SemEval 2026)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-workshops/2026.semeval-1.276.pdf
Supplementarymaterial:
 2026.semeval-1.276.SupplementaryMaterial.zip