In this section, we describe our experimental methodology. We first give a general overview of the models and baselines used, then explain our fine-tuning and activation steering setups, and conclude with the logit lens approach, which is used to better understand content effect bias.

\subsection{General Overview} % Koen
 
\paragraph{Baseline} % Tian

To establish a comprehensive baseline, we evaluated various model architectures and sizes, alongside different system prompts and preprocessing techniques. Temperature was consistently set to 0 across all experiments in order to maximize instruction adherence. The most effective configurations for each model and size variant are summarized in Table \ref{tab:model_prompt_accuracy}. We used the baselines to find and select the prompts, which later were also used for the other SFT and steering setups (for details on the prompting strategy see: Appendix~\ref{app:prompts}). 


\paragraph{Models}
In this research we use multiple instruction-tuned language models with different parameter sizes, from smaller models with 1-8B parameters to models with a size of 27-30B parameters. We opted to not choose the largest sizes of these models because research by \citet{eisape2023systematic} shows that the larger models do not always outperform the smaller models. Additionally, we selected the 1-8B models for this research due to their substantially lower computational cost, making them more feasible within our available resources. We focus on the models LLaMA, Gemma and Qwen, because these models are open-source, and that helps with the reproducibility of our research.

In addition to open-source architectures, we evaluated Gemini 3 Flash and Gemini 3 Pro only for the baselines. To ensure a direct comparison with our Supervised Fine-Tuning results, these models were tested using a zero-shot approach with a small system prompt and split-premise preprocessing.

\subsection{Supervised Fine-tuning} % Koen


To adapt the pretrained LLMs to syllogistic reasoning, we use a supervised fine-tuning strategy using the English training data. Since fully fine-tuning large models is computationally very expensive, our approach uses LoRa \cite{hu2021loralowrankadaptationlarge}. We use a single prompt configuration consisting of a short instruction, premises presented on separate lines, and a requirement that the model output be formatted as JSON.

In our first experiment, we performed SFT with LoRA on all small models ranging from 1B to 8B parameters, without any hyperparameter optimization. The hyperparameters are shown in the appendix \ref{app:sft} in table \ref{tab:sft_hyperparams}

For our second experiment, we control both LoRA-specific and general training hyperparameters using a grid search. The LoRA parameters include the rank ($r$), which determines the dimensionality of the update matrices, and the scaling factor alpha ($\alpha$), which regulates the contribution of the LoRA modules to the model output. In addition, we tune standard training parameters such as the learning rate, batch size, and number of training epochs. The full set of values explored during the grid search is summarized in \ref{tab:grid_search_params}.

For our last experiment, we focused on the plausible and implausible labels. Specifically, we trained separate versions of each LLM using only plausible syllogisms and only implausible syllogisms. Using this setup, we also investigate content bias, as training on only one plausibility type may cause the model to rely more on the believability of the content rather than on logical validity. This allows us to compare these specialized models with those trained on the full dataset and assess how plausibility influences reasoning behavior and performance.

\subsection{Activation Steering}

In addition to SFT, we will use activation steering to control model behavior at inference time without altering the weights. This is achieved by adding a precomputed steering vector, $\vec{v}$, to the model's activations. The vector is calculated by subtracting the mean activations of the negative prompt set from those of the positive set ($\vec{v} = \bar{A}_{\text{pos}} - \bar{A}_{\text{neg}}$). This vector, scaled by a hyperparameter $\alpha$, is then used to steer the model's forward pass toward or away from the target behavior \citep{turner_steering_2024}. We will apply these steering techniques to all of the models we also test SFT on to create a fair comparison between the two methods.
We will use several steering approaches with different steering vectors to test different aspects of the syllogistic reasoning process in LLMs. We will evaluate the approaches on the same metrics used for SFT.

\subsubsection{Approach 1: Isolating Logical Validity}

Our first approach creates a steering vector targeting logical validity while controlling for plausibility. The mean positive activations ($\bar{A}_{\text{pos}}$) are calculated from valid syllogisms with balanced plausibility (50\% plausible, 50\% implausible). The mean negative activations ($\bar{A}_{\text{neg}}$) are calculated from invalid syllogisms with the same balanced distribution. The resulting steering vector $\vec{v}=\bar{A}_{\text{pos}}-\bar{A}_{\text{neg}}$ is hypothesized to represent validity in the model's activation space, with plausibility effects canceling out.

\subsubsection{Approach 2: Eliminating Content Effect Bias}

Our second approach directly targets content effect bias by contrasting the two conflicting conditions. The mean positive activations ($\bar{A}_{\text{pos}}$) come from implausible but valid syllogisms, where the model must rely purely on logical structure. The mean negative activations ($\bar{A}_{\text{neg}}$) come from plausible but invalid syllogisms, where the model is most susceptible to content bias. The resulting vector $\vec{v}=\bar{A}_{\text{pos}}-\bar{A}_{\text{neg}}$ is hypothesized to represent the distinction between logic-based and bias-driven reasoning. This approach assumes baseline susceptibility to content effect bias; if the model already performs correctly on these cases, the steering vector will be ineffective.

\subsubsection{Hyperparameter Selection}

Hyperparameter selection follows a two-stage process using the development set. First, we conduct a grid search over $n_{\text{icl}} \in \{2, 4, 8\}$ and $n_{\text{prompts}} \in \{16, 32, 64\}$ while holding the steering coefficient fixed at $\alpha = 2.0$. In the second stage, we take the best-performing configuration and sweep $\alpha$ from 1.0 to 10.0 in 0.5 increments to optimize steering strength. The final optimal parameters are then evaluated on the test set. This two-stage search procedure will allow us to separately optimize the vector quality (via $n_{\text{icl}}$ and $n_{\text{prompts}}$) and the application strength (via $\alpha$). For more details refer to appendix \ref{app:steering_hyper}.

\section{Logit Lens Analysis}

The logit lens technique \citep{nostalgebraist_interpreting_2020} provides insight into a model's intermediate computations by projecting hidden states at each layer. 
For a given layer, we compute a probability distribution over the vocabulary by applying the final layer normalization and the language model head.
We then apply softmax to determine the probabilities assigned to the tokens "valid" and "invalid." By tracking their probability trajectories, we can observe how the model's prediction evolves during the forward pass.

This technique is particularly suited for studying content effect bias. In aligned cases (VP, II) logical structure and plausibility point toward the same answer, while in conflicting cases (VI, IP) we hypothesize an preference for plausibility before logical reasoning emerges in later layers. We apply logit lens analysis to the baseline models and our best fine-tuned model to reveal whether fine-tuning alters internal reasoning dynamics, and use the results to identify intervention layers for activation steering.
