SubmissionNumber#=%=#31
FinalPaperTitle#=%=#AlienAnnotators at PsyDefDetect: What Lies Between the Lines: Probing Lightweight Open-Source LLMs for Psychological Defense Mechanism Detection
ShortPaperTitle#=%=#
NumberOfPages#=%=#
CopyrightSigned#=%=#
JobTitle#==#
Organization#==#
Abstract#==#Detecting psychological defense mechanisms in therapy dialogue is a clinically valuable but computationally underexplored task. We present our systematic analysis for PsyDefDetect, a shared task at BioNLP@ACL 2026, which frames defense detection as a nine-class utterance-level classification problem based on the Defense Mechanism Rating Scale (DMRS). We systematically evaluate six open-source, instruction-tuned small language models (SLMs, <= 9B parameters) in zero-shot and fine-tuning settings, and compare a clinically-grounded prompt against the organizer-provided baseline. Our official submission achieved 59.96% accuracy and 16.28% Macro F1. Post-submission experiments show that fine-tuning combined with 5-fold cross-validation and logit averaging ensemble substantially improves performance, with the best configuration reaching 34.59% Macro F1 and 65.25% accuracy. We find that clinically-grounded prompts outperform bare label definitions, model scale does not consistently improve zero-shot performance, and fine-tuning dramatically recovers even collapsed zero-shot models. Certain defense tiers remain persistently difficult across all settings, pointing to clinical ambiguity at tier boundaries as a more fundamental bottleneck than data imbalance alone.
Author{1}{Firstname}#=%=#Siam Rahman
Author{1}{Lastname}#=%=#Karip
Author{1}{Username}#=%=#siamkarip
Author{1}{Orcid}#=%=#
Author{1}{Email}#=%=#siamrahmankarip@gmail.com
Author{1}{Affiliation}#=%=#United International University
Author{2}{Firstname}#=%=#Nahid
Author{2}{Lastname}#=%=#Hossain
Author{2}{Orcid}#=%=#
Author{2}{Email}#=%=#nahid@cse.uiu.ac.bd
Author{2}{Affiliation}#=%=#United International University

==========
èéáğö