@inproceedings{vanterpool-aharodnik-2025-handcrafted,
    title = "From Handcrafted Features to {LLM}s: A Comparative Study in Native Language Identification",
    author = "Vanterpool, Aliyah C.  and
      Aharodnik, Katsiaryna",
    editor = "Picazo-Izquierdo, Alicia  and
      Estevanell-Valladares, Ernesto Luis  and
      Mitkov, Ruslan  and
      Guillena, Rafael Mu{\~n}oz  and
      Cerd{\'a}, Ra{\'u}l Garc{\'i}a",
    booktitle = "Proceedings of the First Workshop on Comparative Performance Evaluation: From Rules to Language Models",
    month = sep,
    year = "2025",
    address = "Varna, Bulgaria",
    publisher = "INCOMA Ltd., Shoumen, Bulgaria",
    url = "https://preview.aclanthology.org/corrections-2026-01/2025.r2lm-1.15/",
    pages = "144--153",
    abstract = "This study compares a traditional machine learning feature-engineering approach to a large language models (LLMs) fine-tuning method for Native Language Identification (NLI). We explored the COREFL corpus, which consists of L2 English narratives produced by Spanish and German L1 speakers with lower-advanced English proficiency (C1) (Lozano et al., 2020). For the feature-engineering approach, we extracted language productivity, linguistic diversity, and n-gram features for Support Vector Machine (SVM) classification. We also looked at sentence embeddings with SVM and logistic regression. For the LLM approach, we evaluated BERT-like models and GPT-4. The feature-engineering approach, particularly n-grams, outperformed the LLMs. Sentence-BERT embeddings with SVM achieved the second-highest accuracy (93{\%}), while GPT-4 reached an average accuracy of 90.4{\%} across three runs when prompted with labels. These findings suggest that feature engineering remains a robust method for NLI, especially for smaller datasets with subtle linguistic differences between classes. This study contributes to the comparative analysis of traditional machine learning and transformer-based LLMs, highlighting current LLM limitations in handling domain-specific data and their need for larger training resources."
}