
The development of Large Language Models (LLMs) has led to significant progress in natural language understanding, yet their ability to perform pure logical reasoning remains a subject of debate. A primary challenge in this domain is the "content effect" bias, where models struggle to separate the formal validity of an argument from the real-world plausibility of its conclusion. 
% Add example 
For example, the syllogism \emph{"No human is a tree; all trees are women; therefore, some women are not human"} is logically valid despite being factually implausible.

The SemEval 2026 Task 11 \citep{valentino-etal-2026-semeval} on separating content and formal reasoning in LLMs, addresses this fundamental issue by requiring systems to assess the logical validity of syllogisms independently of their factual truth. This task is critical for evaluating whether LLMs can function as reliable reasoners in domains where world knowledge might conflict with logical structure. This paper focuses on Subtask 1, which provides a binary classification challenge to determine the formal validity of syllogisms in English.

Our approach combines Supervised Fine-Tuning (SFT) and Activation Steering. To adapt pretrained instruction-tuned models like LLaMA \citep{DBLP:journals/corr/abs-2302-13971}, Gemma \citep{gemma}, and Qwen \citep{qwen}, we use Low-Rank Adaptation (LoRA) \citep{hu2021loralowrankadaptationlarge}. 

Beyond traditional fine-tuning, we explore activation steering to intervene in the model's internal processing at inference time. Our goal is to determine if these targeted internal adjustments are more effective at mitigating reasoning errors than standard fine-tuning alone.

Participating in this shared task revealed that while fine-tuning models show promising performance, they remain highly sensitive to content-effect bias. In our baseline experiments, Gemma 3 27B achieved the highest accuracy of 86.67\% using a small system prompt. However, official evaluation results showed a significant performance gap when tested on the full test set. Our final submission (participant name "koensnelten") with a fine-tuned model of LLaMa 3 8B and a small system prompt, achieved a combined score of 31.28 with an accuracy of 93.72 \% and a content effect bias of 6.36, ranking 34th overall (out of the 45 submissions). Compared to the other submissions, we have a relatively high content effect bias which suggests that our model struggle a lot ignoring highly plausible but logically invalid conclusions. This shows the difficulty of separating a model's world knowledge from formal logic.

