
For the multilingual challenges (\textbf{Subtasks 3 and 4}), we compared two strategies: (I) direct processing of multilingual inputs and (II) automated translation into English prior to inference. 

In \textbf{Subtask 3}, approach (I) proved slightly superior with the 
\textit{QwenEng2FOLmultilingualNewShort} configuration (Acc: 88.02\%, TCE: 6.38, Score: 29.36). However approach (II)  
using the same \textit{Qwen3-4B-Instruct} model (\textit{QwenEng2FOL6}) performed nearly as well, achieving a higher raw accuracy of 90.63\%, albeit with a higher TCE (7.33). Looking at the results for each language individually, there is a large difference in accuracy (for example, 100\% for Dutch compared to 52\% for Swahili in the Qwen3-4B model).Substantial differences were observed between the models; see Appendix \ref{appendix:multilingual} for further details


-------------------
Qwen4B multilingual:
QwenEng2FOLmultilingualNewShort
error: 4
total acc: 88.02

sw accuracy 52.94
pt accuracy 94.12
zh-CN accuracy 94.44
fr accuracy 94.44
nl accuracy 100.0
bn accuracy 76.47
it accuracy 83.33
es accuracy 88.89
te accuracy 88.24
de accuracy 100.0
ru accuracy 82.35

best monolingual (first translated):
QwenEng2FOL6 google translate
experiment fol_otter_output_QwenEng2FOL6
error = 0
total acc:90.63

sw accuracy 88.24
pt accuracy 94.12
zh-CN accuracy 77.78
fr accuracy 100.0
nl accuracy 100.0
bn accuracy 76.47
it accuracy 94.44
es accuracy 88.89
te accuracy 94.12
de accuracy 94.44
ru accuracy 88.24

bestes SmoLLm Modell multilingual: Eng2FOL5multilingual
error: 9
total acc: 79.69

sw accuracy 23.53
pt accuracy 88.24
zh-CN accuracy 94.44
fr accuracy 100.0
nl accuracy 94.12
bn accuracy 64.71
it accuracy 94.44
es accuracy 100.0
te accuracy 47.06
de accuracy 72.22
ru accuracy 64.71





In \textbf{Subtask 4}, the English-translation strategy (II) combined with the winning configuration from Subtask 1 (\textit{Eng2FOL5}) yielded the best results (Acc: 77.08\%, $F_1$-Premises: 64.15\%, TCE: 6.07, Score: 23.89). Direct multilingual processing suffered significantly in this subtask; the corresponding \textit{Qwen} configuration showed a markedly higher content effect (TCE: 14.71). Notably, in 20 out of 192 cases, the LLM failed to produce an output that could be processed by the theorem prover.



mit
masch. Übersetzung monolingual mit SmolLM3-3B	
fol_otter_output_Eng2FOL5
total acc 76.5625 tce 5.0089 score 27.4100	
error 8

beste Multilingual
QwenEng2FOL5multilingualNewShort	
mit Qwen4B
total acc 75.5208
tce 13.6692
score 20.4899
error 20	


























\section{Conclusion}
\subsection{Summary of Approach}
In this paper, we described our contribution to SemEval 2026 Task 11, focusing on the challenge of reasoning disentanglement \cite{valentino2026semeval}. To distinguish between logical validity and semantic content, we evaluated three distinct architectures: a direct neural classification baseline, a Natural-to-FOL translation pipeline, and a structured mapping to syllogistic triples. Our methodology centered on iterative prompt engineering, specifically contrasting short instructions against detailed 1,721-character syntactic guardrails to guide symbolic translation.

\subsection{Key Results}
Our experiments across the four subtasks demonstrated that structured symbolic pivots are highly effective at mitigating the content effect. In Subtask 1, the combination of \texttt{SmolLM-145-1B} and the \texttt{Eng2FOL} prompt achieved a peak accuracy of 91.09\%, significantly outperforming larger general-purpose models like \texttt{Llama-3.2-1B} (75.91\%). Furthermore, Subtask 2-4
As for the Subtasks 2 and 4 

In Subtask 3, TCE might have been affected by prior translation with the deep translator utilizing Google translate correcting or mistranslating connectors in the process and though messing with the accuracy.

\subsection{Analysis and Limitations}
Our error analysis revealed: although we 
- belief bias
-
-

Some limitations: - prompts and examples don't align -> model confused
 .



\subsection{Direct Neural Inference (DNI)}
The DNI approach relied on the LLM's internal representations to predict validity directly from natural language. While accuracy was initially poor in zero-prompt settings, the introduction of a two-step method—simplifying the structure before evaluation—led to measurable improvements. Despite these gains, overall accuracy remained weak, as the model continued to struggle with the content effect and misinterpreting formal logical rules. [accuracy noch einsetzen für beides]
\subsection{Translation-Based Inference(TBI-FOL)}
\subsection{Translation-Based Inference (TBI-SYL)}



#######################################





We distinguish between the following prompt categories:\footnote{All prompt templates used in this study are available in our public repository: \url{https://github.com/your-repo-link/prompts}}

\begin{itemize} 
    \item \textbf{Logic Translation (TBI-FOL):} To address the aforementioned challenges, we compared three prompt variants: \texttt{Eng2FOL} (focusing on explicit syntactic rules, 1721 chars), \texttt{Eng2FOLimproved} (emphasizing semantic decoupling, abstract treatment of symbols, and consistent PascalCase naming, 1634 chars), and \texttt{Eng2FOLultrashort} (a minimalistic, example-driven approach, 320 chars).

    \textbf{Multilingual TBI-FOL:} Two dedicated prompts, \texttt{MultiLingFOL} ( 3033 chars) and \texttt{MultiLingEng2FOLultrashort} ( 502 chars), handled the multilingual scenarios (Subtasks 3 and 4). \texttt{MultiLingFOL} includes a step-by-step reasoning instruction (in English) and a multilingual trigger word mapping for 11 languages to ensure logical fidelity during the \textit{Native-test} condition.

    
    \item \textbf{Syllogistic Mapping (TBI-SYL):} A single specialized prompt, \texttt{SyllTrip} ( 1254 chars), was used to map natural language to Aristotelian triples $(q, S, P)$. It enforces naming consistency and maps quantifiers to the four categorical proposition types (\textit{a, e, i, o}) required for the Prolog-based inference.

    \item \textbf{Linguistic Simplification:} Two prompts, \texttt{LogSimp} ( 1572 chars) and \texttt{LogSimpShortPrompt} ( 1016 chars), were employed to evaluate the impact of text normalization. These prompts instruct the LLM to extract a standardized "Simplified Logic English" (e.g., \texttt{all [Subject] is [Predicate]}, see Figure \ref{fig:example}), enforcing singular forms  capitalization, underscores for multi-word terms, and the elimination of negative atoms (e.g., converting "is not a fish" to a "no" quantifier and not to "not_fish").


    \item \textbf{Direct Neural Inference (DNI):} A single baseline prompt was used to collect binary validity judgments directly from the LLM, providing a point of comparison for the neuro-symbolic pipelines.
\end{itemize}

The prompts described above are combined with several curated sets of few-shot examples to guide the models through the logical transformations. Table~\ref{tab:few_shot_configs} provides details regarding the quantity, average length, and the distribution of validity and plausibility across the example sets.

\begin{table}[h]
\centering
\footnotesize
\begin{tabular}{@{}l|c|cccc|c@{}}
\hline
\textbf{Example Set ID} & \textbf{\#} & \textbf{IV} & \textbf{PV} & \textbf{II} & \textbf{PI} & \textbf{$\varnothing$} \\ \hline
LogSimpEx               & 5           & 1           & 4           & 0           & 0           & 3.4 \\ \hline
SyllTripEx              & 4           & 1           & 3           & 0           & 0           & 3.8 \\
SimpSyllTripEx          & 4           & 1           & 3           & 0           & 0           & 3.8 \\ \hline
Eng2FOLex               & 3           & 0           & 0           & 1           & 2           & 3.0 \\
Eng2FOLexImproved       & 5           & 0           & 0           & 2           & 3           & 3.2 \\
Eng2FOLexImprovedX      & 8           & 0           & 3           & 2           & 3           & 3.1 \\
Eng2FOLexImproved2      & 8           & 2           & 2           & 2           & 2           & 3.6 \\
Eng2FOLexVaried         & 8           & 0           & 5           & 1           & 2           & 3.0 \\
Eng2FOLexLong           & 4           & 0           & 2           & 2           & 0           & 7.5 \\ \hline
MultiLingexImproved     & 22          & 2           & 9           & 1           & 10          & 3.0 \\
MultiLingexShort        & 6           & 1           & 2           & 0           & 3           & 3.0 \\ \hline
\end{tabular}
\caption{Composition of few-shot example sets. \textbf{#} denotes the total number of examples; \textbf{$\varnothing$} represents the average number of logical statements (premises and conclusion) per syllogism. Categories reflect belief-bias conditions: Implausible/Valid (IV), Plausible/Valid (PV), Implausible/Invalid (II), and Plausible/Invalid (PI).}\label{tab:few_shot_configs}
\end{table}


\subsection{Models and Parameters}

We evaluated our pipeline across a diverse selection of smaller Large Language Models (LLMs) ranging from 135M to 4B parameters to investigate scaling effects on formal translation tasks. The following models were utilized:


\begin{description}
    \item[SmolLM-Series:] \texttt{SmolLM2-135M-Instruct}, \texttt{SmolLM2-1.7B-Instruct}, and \texttt{SmolLM3-3B}. These models were chosen for their fully open nature (including training data mixtures) and optimization for hybrid reasoning tasks at a small scale.
    \item \textbf{Qwen-Series:} \texttt{Qwen3-0.6B}, \texttt{Qwen3-1.7B}, and \texttt{Qwen3-4B-Instruct-2507}. Chosen for their competitive performance and established strength in formal logic and instruction-following, particularly in the 4B parameter range.
    \item[Gemma-Series:] \texttt{google/gemma-2-2b-it} and \texttt{google/gemma-3-4b-it}. Included as state-of-the-art models that provide high reasoning capabilities and serve as a performance benchmark for the latest generation of compact models.
    \item[Llama-Series:] \texttt{meta-llama/Llama-3.2-3B-Instruct}. Selected as a widely recognized industry baseline for 3B-parameter models, allowing for a direct performance comparison against the SmolLM3 and Qwen3 architectures.
\end{description}

To ensure reproducibility, we set the temperature to $0.0$ for the majority of models. For the \textit{Qwen} models, a specific configuration was used to ensure generation stability: \texttt{do\_sample=True}, \texttt{temperature=0.1}, \texttt{top\_p=0.9}, and a \texttt{repetition\_penalty} of $1.1$. Inference was performed with batch sizes between 8 and 16, depending on the respective model size.\footnote{A comprehensive list of all hyperparameters and model-specific settings is provided in the \texttt{experiments.yaml} configuration file in our repository.}



\subsection{Data Processing and Rule-based Translation}
\label{sec:processing_translation}

To bridge the gap between natural language and formal reasoning, we implemented a robust processing pipeline that handles both the extraction of LLM outputs and the deterministic translation of propositions.

\paragraph{Extraction and Normalization}
Since LLMs often embed formal outputs within conversational text, we employ a multi-step extraction layer. This involves recursive JSON and literal parsing to isolate logical lists, combined with artifact stripping (e.g., removing trailing explanations). Extracted terms undergo aggressive normalization: articles and redundant prefixes are stripped, and nouns are singularized to ensure that predicates remain consistent across different premises (e.g., \textit{"the birds"} and \textit{"a bird"} are both mapped to \texttt{Bird}).

\paragraph{Rule-based Translation Engine}
For the non-neural translation path, we developed a deterministic engine that maps propositions to FOL. It first attempts a heuristic split based on copula identification (\textit{"is"}, \textit{"are not"}) and handles complex cases like double negations (e.g., \textit{"no X is not Y"}). If the split fails, it employs a hierarchical regex safety net to identify categorical proposition types. By offloading the formalization to this deterministic logic, we ensure that the system can generate valid FOL syntax even if the LLM is only used for linguistic simplification or, in the case of the purely symbolic baseline, bypassed entirely.

\paragraph{Symbolic Standardization and Prover Interface}
Before inference, formulas undergo formal refinement to ensure \textit{Otter} compatibility, including alpha-conversion for unique variable naming, parentheses balancing, and negation standardization. To align First-Order Logic with the existential import of classical syllogisms, i.e.\ the assumption that the subject terms are non-empty, the pipeline automatically injects existence conditions ($\exists x P(x)$) for all detected predicates. For the inference, the conclusion is identified and provided to the prover in its negated form. For successful refutations, we parse Otter's proof history to map used clauses back to their original premises, enabling the identification of minimal sufficient sets for Subtask~2 and 4.


\paragraph{TBI-SYL Adjustments}
The processing for the syllogistic triple pipeline follows a similar normalization logic but maps propositions to $(q, S, P)$ structures. These triples are validated against a Prolog knowledge base that checks their validity against the 24 valid Aristotelian moods. Unlike the FOL path, this specialized solver directly accounts for existential import through its internal encoding of syllogistic figures.


\paragraph{Impact of Model Selection and Scaling}
We evaluate the impact of model capacity by testing various LLMs using our winning configuration (\textit{Ultrashort Prompt}, 8 examples). As shown in Table \ref{tab:models}, \textit{SmolLM3-3B} achieves the highest combined score, marginally outperforming the \textit{Qwen} models due to its superior bias mitigation (lowest TCE). It also significantly outperforms both \textit{Llama} and \textit{Gemma} in this task. While we observe a general scaling trend where larger models yield better results, the \textit{Qwen3-1.7B} model represents a notable exception, demonstrating remarkably high accuracy for its size.

\paragraph{Impact of Model Selection and Scaling}
We evaluate the impact of model capacity by testing various LLMs using our winning configuration (\textit{Ultrashort Prompt}, 8 examples). As shown in Table \ref{tab:models}, \textit{SmolLM3-3B} (Eng2FOL5) achieves the highest combined score, marginally outperforming the \textit{Qwen} models due to its superior bias mitigation (lowest TCE). It significantly outperforms both \textit{Llama} and \textit{Gemma} in this task. While we observe a general scaling trend where larger models yield better results, the \textit{Qwen3-1.7B} model represents a notable exception, demonstrating remarkably high accuracy for its size.

\begin{table}[ht]
\centering
%\scriptsize
\small
\setlength{\tabcolsep}{2pt}
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Model (Experiment)} & \textbf{Comb} & \textbf{Acc}& \textbf{TCE} \\
\midrule
SmolLM3-3B (Eng2FOL5) & \textbf{44.37} & 95.29 & \textbf{2.15} \\
Qwen3-4B-Inst. (Eng2FOL5Qwen4B) & 36.23 & \textbf{96.34} & 4.26 \\
Qwen3-1.7B (Eng2FOL5Qwen1.7B) & 36.03 & 95.81 & 4.26 \\
Llama-3.2-3B-Inst. (Eng2FOL3) & 24.89 & 85.34 & 10.35 \\
Gemma-2-2B-it (Eng2FOL5Gemma2B) & 20.08 & 73.82 & 13.54 \\
SmolLM2-1.7B-Inst. (Eng2FOL5small) & 17.61 & 65.97 & 14.58 \\
Qwen3-0.6B (Eng2FOL5Qwen0.6B) & 17.53 & 65.45 & 14.38 \\
SmolLM2-135M-Inst. (Eng2FOL5tiny) & 11.70 & 50.26 & 26.04 \\
\bottomrule
\end{tabular}
\caption{Impact of model size and family on TBI-FOL performance using a fixed configuration (\texttt{Eng2FOLUltrashort} prompt,  \texttt{Eng2FOLexImprovedX} examples; details in Appendix \ref{appendix:prompts}).}
\label{tab:models}
\end{table}

While Table \ref{tab:models} compares LLMs under identical conditions, Table \ref{tab:best_results} reports the peak performance for each LLM across all tested prompt configurations. This reveals that although most LLMs benefit from the \textit{Ultrashort} setup, individual tuning can further reduce the \textit{Total Content Effect} (TCE) for specific families like \textit{Qwen}.

\begin{table}[ht]
\centering
\scriptsize
\setlength{\tabcolsep}{5pt}
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Model (Best Experiment)} & \textbf{Comb}  & \textbf{Acc} & \textbf{TCE} \\
\midrule
SmolLM3-3B (Eng2FOL5) & \textbf{44.37} & 95.29 & \textbf{2.15} \\
Qwen3-4B-Instruct-2507 (QwenEng2FOL1) & 38.44 & 93.72 & 3.21 \\
Qwen3-1.7B (Eng2FOL5Qwen1.7B) & 36.03 & \textbf{95.81} & 4.26 \\
Llama-3.2-3B-Instruct (Eng2FOL3) & 24.89 & 85.34 & 10.35 \\
gemma-2-2b-it (Eng2FOL5Gemma2B) & 20.08 & 73.82 & 13.54 \\
SmolLM2-1.7B-Instruct (Eng2FOL5small) & 17.61 & 65.97 & 14.58 \\
Qwen3-0.6B (Eng2FOL5Qwen0.6B) & 17.53 & 65.45 & 14.38 \\
SmolLM2-135M-Instruct (Eng2FOL5tiny) & 11.70 & 50.26 & 26.04 \\
gemma-3-4b-it (Eng2FOL11) & 10.41 & 43.46 & 22.92 \\
\bottomrule
\end{tabular}
\caption{Peak performance per model family across all experimental configurations. Each entry represents the best-performing individual setup for that specific architecture (see Appendix \ref{app:results}.}
\label{tab:best_results}
\end{table}

\paragraph{Impact of Prompting Strategy and Few Shot Example Selection}

The transition between prompt strategies yields no uniform advantage across all architectures. For \textit{SmolLM3-3B}, we observe a slight preference for shorter prompts: comparing \texttt{Eng2FOLUltrashort} (Eng2FOL5) to the longer \texttt{Eng2FOL} (Eng2FOL7), accuracy marginally increases from 95.29\% to 96.34\%, yet this is offset by the \textit{Total Content Effect} (TCE) doubling from 2.15 to 4.26 (see Table \ref{tab:results} for full details). Conversely, \textit{Qwen} models exhibit an opposite tendency, showing improved robustness with longer, more descriptive prompts (e.g., \texttt{QwenEng2FOL4} vs. \texttt{QwenEng2FOL2}).

Variations in the few-shot example sets similarly provide an ambiguous picture. Analysis of experiments \texttt{Eng2FOL5}, \texttt{6}, \texttt{8}, and \texttt{9}—which differ solely in example selection—indicates a subtle trend where a higher quantity of examples improves performance. However, explicitly balancing the plausibility-validity distribution within the examples did not yield the expected gains in bias mitigation.






\section{Details on Evaluation Metrics [Todo Wiebke: kontrollieren]}
\label{app:metrics}

The evaluation framework follows \cite{valentino-etal-2026-semeval} to ensure comparability across systems.

\paragraph{Total Content Effect (TCE)}
To measure the susceptibility to belief bias, the dataset is partitioned into four subgroups based on ground truth validity ($V \in \{valid, invalid\}$) and plausibility ($P \in \{plausible, implausible\}$). Let $\text{Acc}_{V,P}$ be the accuracy on a specific subgroup. The TCE is defined as:
\begin{equation}
    TCE = \frac{\Delta_{intra} + \Delta_{inter}}{2}
\end{equation}
where the intra-label and inter-label differences are calculated as:
\begin{small}
\begin{align}
    \Delta_{intra} &= \frac{|\text{Acc}_{v,p} - \text{Acc}_{v,\neg p}| + |\text{Acc}_{\neg v,p} - \text{Acc}_{\neg v,\neg p}|}{2} \\
    \Delta_{inter} &= \frac{|\text{Acc}_{v,p} - \text{Acc}_{\neg v,p}| + |\text{Acc}_{v,\neg p} - \text{Acc}_{\neg v,\neg p}|}{2}
\end{align}
\end{small}

\paragraph{Premise F1-Score ($F1_{prem}$)}
For Subtask 2, let $\mathcal{S}$ be the set of instances. For each instance $i \in \mathcal{S}$, we calculate Precision ($P_i$) and Recall ($R_i$) based on the predicted premises $\hat{Y}_i$ and ground truth premises $Y_i$. The macro-averaged scores are:
\begin{equation}
    P_{macro} = \frac{1}{|\mathcal{S}|} \sum_{i \in \mathcal{S}} \frac{|\hat{Y}_i \cap Y_i|}{|\hat{Y}_i|}
\end{equation}    
\begin{equation}
    R_{macro} = \frac{1}{|\mathcal{S}|} \sum_{i \in \mathcal{S}} \frac{|\hat{Y}_i \cap Y_i|}{|Y_i|}
\end{equation}
The final $F1_{prem}$ is the harmonic mean: $2 \cdot (P_{macro} \cdot R_{macro}) / (P_{macro} + R_{macro})$.

\paragraph{Final Combined Score}
The primary ranking metric integrates all previous measures. First, the Overall Performance $M_{perf}$ is computed:
\begin{equation}
    M_{perf} = \frac{\text{Acc} + F1_{prem}}{2}
\end{equation}
The final score applies a smooth penalty for belief bias using the natural logarithm:
\begin{equation}
    \text{Combined Score} = \frac{M_{perf}}{1 + \ln(1 + TCE)}
\end{equation}




\paragraph{Premise Identification (Subtask 2)}
For Subtask 2, we evaluate the system's ability to identify the minimal set of premises required to support a conclusion. We calculate the \textbf{Macro-Averaged F1-Score} ($F1_{prem}$) across all instances. For each instance, precision and recall are derived from the overlap between predicted premise IDs and the ground truth set. The final score is the harmonic mean of the average precision and average recall across the dataset:
\begin{equation}
    F1_{prem} = 2 \cdot \frac{\text{Precision}_{macro} \cdot \text{Recall}_{macro}}{\text{Precision}_{macro} + \text{Recall}_{macro}} \times 100
\end{equation}

\paragraph{Combined Score}
The official ranking metric rewards configurations that achieve high task performance while maintaining low bias. We first define an \textbf{Overall Performance Metric} ($M_{perf}$) as the arithmetic mean of validity accuracy and the premise F1-score:
\begin{equation}
    M_{perf} = \frac{\text{Acc} + F1_{prem}}{2}
\end{equation}
A logarithmic penalty is then applied based on the observed Content Effect to derive the final \textbf{Combined Score}:
\begin{equation}
    \text{Score}_{comb} = \frac{M_{perf}}{1 + \ln(1 + CE)}
\end{equation}
This "smooth" penalty ensures that high performance achieved through biased heuristics (e.g., relying on plausibility rather than logic) results in a disproportionately lower final standing.


\subsection{[Backup: Steht hier nur, falls ich daraus noch was brauche] Evaluation Metrics}
\label{sec:metrics}

To evaluate the performance of our architectural configurations, we employ a multi-faceted scoring system that accounts for logical validity, premise identification, and resistance to cognitive biases.

\paragraph{Validity and Bias (Subtask 1)}
The primary metric for argument validation is \textbf{Overall Accuracy} (Acc), representing the percentage of correctly predicted validity labels. To quantify the influence of \textit{belief bias}, we calculate the \textbf{Content Effect} (CE). By measuring accuracy across four subgroups—intersecting ground truth validity $\{V, \neg V\}$ with content plausibility $\{P, \neg P\}$—the CE is defined as the average of intra-label and inter-label differences:
\begin{equation}
    CE = \frac{|\text{Acc}_{VP} - \text{Acc}_{V\neg P}| + |\text{Acc}_{\neg VP} - \text{Acc}_{\neg V\neg P}|}{2}
\end{equation}
where, for example, $\text{Acc}_{VP}$ denotes the accuracy on the subgroup of valid arguments with plausible content.
