@inproceedings{karip-hossain-2026-alienannotators,
    title = "{A}lien{A}nnotators at {P}sy{D}ef{D}etect: What Lies Between the Lines: Probing Lightweight Open-Source {LLM}s for Psychological Defense Mechanism Detection",
    author = "Karip, Siam  and
      Hossain, Nahid",
    editor = "Gupta, Deepak  and
      Demner-Fushman, Dina",
    booktitle = "Proceedings of the {B}io{NLP} 2026 (Shared Tasks)",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.bionlp-2.28/",
    pages = "213--223",
    ISBN = "979-8-89176-435-4",
    abstract = "Detecting psychological defense mechanisms in therapy dialogue is a clinically valuable but computationally underexplored task. We present our systematic analysis for PsyDefDetect, a shared task at BioNLP@ACL 2026, which frames defense detection as a nine-class utterance-level classification problem based on the Defense Mechanism Rating Scale (DMRS). We systematically evaluate six open-source, instruction-tuned small language models (SLMs, = 9B parameters) in zero-shot and fine-tuning settings, and compare a clinically-grounded prompt against the organizer-provided baseline. Our official submission achieved 59.96{\%} accuracy and 16.28{\%} Macro F1. Post-submission experiments show that fine-tuning combined with 5-fold cross-validation and logit averaging ensemble substantially improves performance, with the best configuration reaching 34.59{\%} Macro F1 and 65.25{\%} accuracy. We find that clinically-grounded prompts outperform bare label definitions, model scale does not consistently improve zero-shot performance, and fine-tuning dramatically recovers even collapsed zero-shot models. Certain defense tiers remain persistently difficult across all settings, pointing to clinical ambiguity at tier boundaries as a more fundamental bottleneck than data imbalance alone."
}